Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Lucida from ByteDance Cool project! It transforms indoor video into editable 3D scenes; reconstructs objects as simulation-ready assets - VLM-based object detection and referring descriptions. - good at pose alignment - high-fidelity real-to-sim digital twins. - multi-view masks, boxes, partial point clouds - Seed3D 2.0 for image-to-3D asset gen

Wildminder

11,281 subscribers

44,774 Aufrufe • vor 2 Tagen •via X (Twitter)

Kunst Wissenschaft & Technologie Bildung

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

🚨 A single image → a fully editable UE5 world. Introducing Lumera: a turnkey pipeline that reconstructs an engine-native 3D scene directly from ONE image — not just pixels, point clouds, or Gaussians. It gives you: → object-level 3D meshes → editable 3D layouts → engine-native lights → HDR environment lighting → a scene ready for Unreal Engine 5 In other words: Image → 3D world → UE5. We believe the future of image-to-3D is not “generate something that looks 3D.” It is generate the actual world project. 🌐 📄

🚨 A single image → a fully editable UE5 world. Introducing Lumera: a turnkey pipeline that reconstructs an engine-native 3D scene directly from ONE image — not just pixels, point clouds, or Gaussians. It gives you: → object-level 3D meshes → editable 3D layouts → engine-native lights → HDR environment lighting → a scene ready for Unreal Engine 5 In other words: Image → 3D world → UE5. We believe the future of image-to-3D is not “generate something that looks 3D.” It is generate the actual world project. 🌐 📄

Hao Zhao

49,886 Aufrufe • vor 5 Tagen

We’re open-sourcing HY-World 2.0, a multimodal world model that generates, reconstructs, and simulates interactive *3D worlds* from text, images, and videos. Outputs can be integrated into game engines and embodied simulation pipelines. Key highlights: 🔹 One-click world generation Turn text or image into interactive 3D worlds automatically. 🔹 Pipeline-ready 3D outputs Editable 3D worlds for Unity and Unreal Engine, with standard 3D exports including mesh, 3DGS, and point clouds. 🔹 Unified world model system One model family for world generation and reconstruction across synthetic and real-world scenes. 🔹 Interactive character mode Explore generated 3D worlds in real time with physics-aware movement and collision support. ✨ Apply for access: 🔗 GitHub: 🤗 Hugging Face: 📄 Technical Report:

We’re open-sourcing HY-World 2.0, a multimodal world model that generates, reconstructs, and simulates interactive 3D worlds from text, images, and videos. Outputs can be integrated into game engines and embodied simulation pipelines. Key highlights: 🔹 One-click world generation Turn text or image into interactive 3D worlds automatically. 🔹 Pipeline-ready 3D outputs Editable 3D worlds for Unity and Unreal Engine, with standard 3D exports including mesh, 3DGS, and point clouds. 🔹 Unified world model system One model family for world generation and reconstruction across synthetic and real-world scenes. 🔹 Interactive character mode Explore generated 3D worlds in real time with physics-aware movement and collision support. ✨ Apply for access: 🔗 GitHub: 🤗 Hugging Face: 📄 Technical Report:

Tencent Hy

372,289 Aufrufe • vor 4 Monaten

Unreal Engine asset packs are about to be worthless. Tencent just shipped HY-World 2.0. You type one sentence. It outputs a real 3D scene with meshes, point clouds, and 3DGS that drags straight into your game engine. Renders in real time on a consumer GPU. Phone video in, digital twin out, one forward pass. The marketplace economy of selling 3D assets just had its Napster moment.

Unreal Engine asset packs are about to be worthless. Tencent just shipped HY-World 2.0. You type one sentence. It outputs a real 3D scene with meshes, point clouds, and 3DGS that drags straight into your game engine. Renders in real time on a consumer GPU. Phone video in, digital twin out, one forward pass. The marketplace economy of selling 3D assets just had its Napster moment.

Ihtesham Ali

45,956 Aufrufe • vor 3 Monaten

📢📢GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction📢📢 Reconstructing high-fidelity 3D scenes from sparse RGB input is hard. It needs a strong 3D prior! We reformulate multi-view scene reconstruction as conditional 3D generation over overlapping spatial chunks, lifting posed image features into a generative shape prior via 3D conditioning. As an example prior, we build on Trellis2, and train it such that its reconstruction is pixel aligned and matches from all views. GenRecon achieves unprecedented reconstruction quality from any sparse RGB input sequence, even from a phone capture. The reconstruction also includes PBR materials which facilitates relighting and virtual object insertion. Amazing work by Katharina Schmid, Nicolas von Lützow, Jozef, Angela Dai

📢📢GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction📢📢 Reconstructing high-fidelity 3D scenes from sparse RGB input is hard. It needs a strong 3D prior! We reformulate multi-view scene reconstruction as conditional 3D generation over overlapping spatial chunks, lifting posed image features into a generative shape prior via 3D conditioning. As an example prior, we build on Trellis2, and train it such that its reconstruction is pixel aligned and matches from all views. GenRecon achieves unprecedented reconstruction quality from any sparse RGB input sequence, even from a phone capture. The reconstruction also includes PBR materials which facilitates relighting and virtual object insertion. Amazing work by Katharina Schmid, Nicolas von Lützow, Jozef, Angela Dai

Matthias Niessner

18,267 Aufrufe • vor 3 Monaten

Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video Contribution quote from the paper: In summary, our main contributions are • a comprehensive pipeline for reconstructing the shape, appearance, and behavior of real-world garments using Gaussian splatting, • an algorithm for registering garment meshes to multi- view videos with an optimization procedure based on Gaussian splatting, and • a Gaussian Garment representation that combines triangle meshes with Gaussian textures to capture photorealistic appearance and can be used as a fully controllable 3D asset.

Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video Contribution quote from the paper: In summary, our main contributions are • a comprehensive pipeline for reconstructing the shape, appearance, and behavior of real-world garments using Gaussian splatting, • an algorithm for registering garment meshes to multi- view videos with an optimization procedure based on Gaussian splatting, and • a Gaussian Garment representation that combines triangle meshes with Gaussian textures to capture photorealistic appearance and can be used as a fully controllable 3D asset.

MrNeRF

27,354 Aufrufe • vor 2 Jahren

ByteDance just announced MagicVideo-V2 Multi-Stage High-Aesthetic Video Generation paper page: The growing demand for high-fidelity video generation from textual descriptions has catalyzed significant research in this field. In this work, we introduce MagicVideo-V2 that integrates the text-to-image model, video motion generator, reference image embedding module and frame interpolation module into an end-to-end video generation pipeline. Benefiting from these architecture designs, MagicVideo-V2 can generate an aesthetically pleasing, high-resolution video with remarkable fidelity and smoothness. It demonstrates superior performance over leading Text-to-Video systems such as Runway, Pika 1.0, Morph, Moon Valley and Stable Video Diffusion model via user evaluation at large scale

ByteDance just announced MagicVideo-V2 Multi-Stage High-Aesthetic Video Generation paper page: The growing demand for high-fidelity video generation from textual descriptions has catalyzed significant research in this field. In this work, we introduce MagicVideo-V2 that integrates the text-to-image model, video motion generator, reference image embedding module and frame interpolation module into an end-to-end video generation pipeline. Benefiting from these architecture designs, MagicVideo-V2 can generate an aesthetically pleasing, high-resolution video with remarkable fidelity and smoothness. It demonstrates superior performance over leading Text-to-Video systems such as Runway, Pika 1.0, Morph, Moon Valley and Stable Video Diffusion model via user evaluation at large scale

AK

181,049 Aufrufe • vor 2 Jahren

How do you go from scanning a real-world space → using it in an app, workflow, or environment? Our product team walks through Scaniverse step-by-step, from capturing spaces to generating production-ready 3D assets and enabling precise, real-world positioning. In the first video, Senior Product Manager, Bec Maxworth, walks through your entry point into Scaniverse from signing up to getting started so you can start building high-fidelity 3D models in minutes. Dive into the full walkthrough here: #NianticSpatial #Scaniverse #Reconstruction #VPS #GeospatialAI #AI #PhysicalAI

How do you go from scanning a real-world space → using it in an app, workflow, or environment? Our product team walks through Scaniverse step-by-step, from capturing spaces to generating production-ready 3D assets and enabling precise, real-world positioning. In the first video, Senior Product Manager, Bec Maxworth, walks through your entry point into Scaniverse from signing up to getting started so you can start building high-fidelity 3D models in minutes. Dive into the full walkthrough here: #NianticSpatial #Scaniverse #Reconstruction #VPS #GeospatialAI #AI #PhysicalAI

Niantic Spatial 🌎

10,739 Aufrufe • vor 4 Monaten

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

AK

62,768 Aufrufe • vor 3 Jahren

One of the coolest 3d real estate demos I've seen in a while. Treedis brings together gaussian splats for area understanding → BIM data to check which units are available → Matterport for indoor views → and image editing AI to restyle decor and see different times of day. The data conflation problem of wrangling all these mismatched 3D assets into one seamless view is hard to do automatically - but this experience shows why it's worth solving. Every digital twin and AR vision video promises this, but here's someone actually building it.

One of the coolest 3d real estate demos I've seen in a while. Treedis brings together gaussian splats for area understanding → BIM data to check which units are available → Matterport for indoor views → and image editing AI to restyle decor and see different times of day. The data conflation problem of wrangling all these mismatched 3D assets into one seamless view is hard to do automatically - but this experience shows why it's worth solving. Every digital twin and AR vision video promises this, but here's someone actually building it.

Bilawal Sidhu

38,590 Aufrufe • vor 9 Monaten

📢📢 𝐏𝐞𝐫𝐜𝐇𝐞𝐚𝐝: 𝐏𝐞𝐫𝐜𝐞𝐩𝐭𝐮𝐚𝐥 𝐇𝐞𝐚𝐝 𝐌𝐨𝐝𝐞𝐥 𝐟𝐨𝐫 𝐒𝐢𝐧𝐠𝐥𝐞-𝐈𝐦𝐚𝐠𝐞 𝟑𝐃 𝐇𝐞𝐚𝐝 𝐑𝐞𝐜𝐨𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 & 𝐄𝐝𝐢𝐭𝐢𝐧𝐠📢📢 PercHead reconstructs realistic 3D heads from a single image and enables disentangled 3D editing via geometric controls and style inputs from images or text. At its core is a generalized 3D head decoder trained with perceptual supervision from DINOv2 and SAM 2.1. We find that our new perceptual loss formulation improves reconstruction fidelity compared to commonly-used methods such as LPIPS. Our trained reconstruction model is able to generate 3D-consistent heads from a single input image. Even with challenging side-view inputs, the model robustly infers missing regions for a coherent, high-fidelity output. In addition, our architecture seamlessly adapts to downstream tasks: by swapping the encoder, we can transform the model into a disentangled 3D editing pipeline. In this scenario, we can control geometry through - potentially hand-drawn - segmentation maps, and condition style via image or text prompt. We also provide an interactive GUI to enable the exploration of our editing pipeline. 🌍 📽️ Great work by Antonio Oroz and Tobias Kirschstein

📢📢 𝐏𝐞𝐫𝐜𝐇𝐞𝐚𝐝: 𝐏𝐞𝐫𝐜𝐞𝐩𝐭𝐮𝐚𝐥 𝐇𝐞𝐚𝐝 𝐌𝐨𝐝𝐞𝐥 𝐟𝐨𝐫 𝐒𝐢𝐧𝐠𝐥𝐞-𝐈𝐦𝐚𝐠𝐞 𝟑𝐃 𝐇𝐞𝐚𝐝 𝐑𝐞𝐜𝐨𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 & 𝐄𝐝𝐢𝐭𝐢𝐧𝐠📢📢 PercHead reconstructs realistic 3D heads from a single image and enables disentangled 3D editing via geometric controls and style inputs from images or text. At its core is a generalized 3D head decoder trained with perceptual supervision from DINOv2 and SAM 2.1. We find that our new perceptual loss formulation improves reconstruction fidelity compared to commonly-used methods such as LPIPS. Our trained reconstruction model is able to generate 3D-consistent heads from a single input image. Even with challenging side-view inputs, the model robustly infers missing regions for a coherent, high-fidelity output. In addition, our architecture seamlessly adapts to downstream tasks: by swapping the encoder, we can transform the model into a disentangled 3D editing pipeline. In this scenario, we can control geometry through - potentially hand-drawn - segmentation maps, and condition style via image or text prompt. We also provide an interactive GUI to enable the exploration of our editing pipeline. 🌍 📽️ Great work by Antonio Oroz and Tobias Kirschstein

Matthias Niessner

18,899 Aufrufe • vor 10 Monaten

Introducing ✨RigidFormer: Learning Rigid Dynamics with Transformers - our attempt to scale learning-based physical dynamics with Transformers. RigidFormer learns rigid dynamics with Transformers. It is a mesh-free, object-centric Transformer for multi-object rigid-body contact dynamics from point clouds. Learning physics with purely neural simulators, without relying on traditional physics engines, is an important and widely studied problem. Prior SOTA methods often use graph neural networks for accuracy and generalization, but still struggle with efficient, high-fidelity simulation at scale. RigidFormer uses only point inputs, matches or outperforms mesh-based baselines on standard benchmarks, runs much faster, generalizes across point resolutions and datasets, and scales to 200+ objects. We also show a preliminary extension to command-conditioned articulated bodies by treating body parts as interacting object-level components. RigidFormer is mesh-free: it does not require mesh connectivity, SDFs, or vertex-level message passing, making it well-suited for point-cloud observations and scalable simulation. This architecture can also be adapted to learn soft-body dynamics by replacing the rigid-body module (differentiable Kabsch alignment). 🎬See our video for more details. Many thanks to my amazing collaborators: Minghao Guo Minghao Guo, Haixu Wu Haixu Wu 吴海旭, Doug Roble, Tuur Stuyck Tuur Stuyck, and Wojciech Matusik Wojciech Matusik. Project page: Paper:

Introducing ✨RigidFormer: Learning Rigid Dynamics with Transformers - our attempt to scale learning-based physical dynamics with Transformers. RigidFormer learns rigid dynamics with Transformers. It is a mesh-free, object-centric Transformer for multi-object rigid-body contact dynamics from point clouds. Learning physics with purely neural simulators, without relying on traditional physics engines, is an important and widely studied problem. Prior SOTA methods often use graph neural networks for accuracy and generalization, but still struggle with efficient, high-fidelity simulation at scale. RigidFormer uses only point inputs, matches or outperforms mesh-based baselines on standard benchmarks, runs much faster, generalizes across point resolutions and datasets, and scales to 200+ objects. We also show a preliminary extension to command-conditioned articulated bodies by treating body parts as interacting object-level components. RigidFormer is mesh-free: it does not require mesh connectivity, SDFs, or vertex-level message passing, making it well-suited for point-cloud observations and scalable simulation. This architecture can also be adapted to learn soft-body dynamics by replacing the rigid-body module (differentiable Kabsch alignment). 🎬See our video for more details. Many thanks to my amazing collaborators: Minghao Guo Minghao Guo, Haixu Wu Haixu Wu 吴海旭, Doug Roble, Tuur Stuyck Tuur Stuyck, and Wojciech Matusik Wojciech Matusik. Project page: Paper:

Zhiyang (Frank) Dou

573,289 Aufrufe • vor 3 Monaten

Another city. Another full 3D Gaussian Splat reconstruction. This entire neighborhood was captured while walking with an Insta360 X5, then processed with our GPU pipeline and transformed into a high-fidelity, machine-readable 3D environment with Over the Reality. What you’re seeing is not an isolated virtual world. It’s a 3D digital twin of a real-world location, built from structured spatial data where every building, street, and detail can become usable by AI, robotics, and XR. And it’s happening at scale, across cities in more than 180 countries every day. Have an Insta360 X5 and want to become a mapper? Apply here: Want to build a 3D Gaussian Splat from a video you already recorded? Do it for free at powered by Over the Reality.

Another city. Another full 3D Gaussian Splat reconstruction. This entire neighborhood was captured while walking with an Insta360 X5, then processed with our GPU pipeline and transformed into a high-fidelity, machine-readable 3D environment with Over the Reality. What you’re seeing is not an isolated virtual world. It’s a 3D digital twin of a real-world location, built from structured spatial data where every building, street, and detail can become usable by AI, robotics, and XR. And it’s happening at scale, across cities in more than 180 countries every day. Have an Insta360 X5 and want to become a mapper? Apply here: Want to build a 3D Gaussian Splat from a video you already recorded? Do it for free at powered by Over the Reality.

Over the Reality 🌐

400,097 Aufrufe • vor 4 Monaten

TEDRA Text-based Editing of Dynamic and Photoreal Actors discuss: Over the past years, significant progress has been made in creating photorealistic and drivable 3D avatars solely from videos of real humans. However, a core remaining challenge is the fine-grained and user-friendly editing of clothing styles by means of textual descriptions. To this end, we present TEDRA, the first method allowing text-based edits of an avatar, which maintains the avatar's high fidelity, space-time coherency, as well as dynamics, and enables skeletal pose and view control. We begin by training a model to create a controllable and high-fidelity digital replica of the real actor. Next, we personalize a pretrained generative diffusion model by fine-tuning it on various frames of the real character captured from different camera angles, ensuring the digital representation faithfully captures the dynamics and movements of the real person. This two-stage process lays the foundation for our approach to dynamic human avatar editing. Utilizing this personalized diffusion model, we modify the dynamic avatar based on a provided text prompt using our Personalized Normal Aligned Score Distillation Sampling (PNA-SDS) within a model-based guidance framework. Additionally, we propose a time step annealing strategy to ensure high-quality edits. Our results demonstrate a clear improvement over prior work in functionality and visual quality.

AK

17,302 Aufrufe • vor 2 Jahren

🚀 Announcing Echo — our new frontier model for 3D world generation. Echo turns a simple text prompt or image into a fully explorable, 3D-consistent world. Instead of disconnected views, the result is a single, coherent spatial representation you can move through freely. This is part of a bigger shift in AI: from generating pixels and tokens to generating spaces. Echo predicts a geometry-grounded 3D scene at metric scale, meaning every novel view, depth map, and interaction comes from the same underlying world — not independent hallucinations. Once generated, the world is interactive in real time. You control the camera, explore from any angle, and render instantly — even on low-end hardware, directly in the browser. High-quality 3D world exploration is no longer gated by expensive equipment. Under the hood, Echo infers a physically grounded 3D representation and converts it into a renderable format. For our web demo, we use 3D Gaussian Splatting (3DGS) for fast, GPU-friendly rendering — but the representation itself is flexible and can be easily adapted. Why this matters: consistent 3D worlds unlock real workflows — digital twins, 3D design, game environments, robotics simulation, and more. From a single photo or a line of text, Echo builds worlds that are reliable, editable, and spatially faithful. Echo also enables scene editing and restyling. Change materials, remove or add objects, explore design variations — all while preserving global 3D consistency. Editing no longer breaks the world. This is only the beginning. Echo is the foundation for future world models with dynamics, physical reasoning, and richer interaction — environments that don’t just look right, but behave right. Explore the generated worlds on our website and sign up for the closed beta. The era of spatial intelligence starts here. 🌍 #Echo #WorldModels #SpatialAI #3DFoundationModels Check it out:

🚀 Announcing Echo — our new frontier model for 3D world generation. Echo turns a simple text prompt or image into a fully explorable, 3D-consistent world. Instead of disconnected views, the result is a single, coherent spatial representation you can move through freely. This is part of a bigger shift in AI: from generating pixels and tokens to generating spaces. Echo predicts a geometry-grounded 3D scene at metric scale, meaning every novel view, depth map, and interaction comes from the same underlying world — not independent hallucinations. Once generated, the world is interactive in real time. You control the camera, explore from any angle, and render instantly — even on low-end hardware, directly in the browser. High-quality 3D world exploration is no longer gated by expensive equipment. Under the hood, Echo infers a physically grounded 3D representation and converts it into a renderable format. For our web demo, we use 3D Gaussian Splatting (3DGS) for fast, GPU-friendly rendering — but the representation itself is flexible and can be easily adapted. Why this matters: consistent 3D worlds unlock real workflows — digital twins, 3D design, game environments, robotics simulation, and more. From a single photo or a line of text, Echo builds worlds that are reliable, editable, and spatially faithful. Echo also enables scene editing and restyling. Change materials, remove or add objects, explore design variations — all while preserving global 3D consistency. Editing no longer breaks the world. This is only the beginning. Echo is the foundation for future world models with dynamics, physical reasoning, and richer interaction — environments that don’t just look right, but behave right. Explore the generated worlds on our website and sign up for the closed beta. The era of spatial intelligence starts here. 🌍 #Echo #WorldModels #SpatialAI #3DFoundationModels Check it out:

SpAItial AI

176,524 Aufrufe • vor 8 Monaten

Want to create an avatar from a single image? FlexAvatar is a transformer model that creates full 360°, high-quality, and expressive 3D head avatar from just a single portrait image in minutes. Real-time Demo: FlexAvatar's lightweight architecture allows both animation and rendering in real-time, enabling interactive user experiences. To create a new 3D head avatar, only one image is required, e.g., from a webcam. The final avatar is ready after 2 minutes. Architecture: Under the hood, FlexAvatar adopts a transformer-based encoder-decoder design. The encoder maps the input image onto a latent avatar space, while the decoder produces 3D Gaussian attribute maps by incorporating the animation signal via cross-attention. The model learns all facial animations directly from the data without relying on pre-built 3D face models. This equips the avatars with realistic facial expressions. The internal avatar latent space can be conveniently used to integrate additional observations of a person via fitting. This enables use-cases where more than one image of a person is available, e.g., from a phone scan of the person. We train jointly on 2D monocular videos and multi-view data. However, in monocular videos, the animation signal leaks the target viewpoint, causing the model to produce incomplete 3D heads. We call this phenomenon entanglement of driving signal and target viewpoint. To prevent entanglement, we introduce bias sinks. These are learnable tokens that indicate whether a training sample stems from a monocular or a multi-view dataset. During training, the model learns to produce incomplete 3D heads only when the monocular token is present. During inference, FlexAvatar then always uses the multi-view token for which the model has learned to produce complete 3D heads. This simple design allows to combine the generalizability from monocular data with the quality of multi-view data. FlexAvatar summary: - Input: Single-image, phone scan, or monocular video - Output: Full 360° head avatar - Expressive animations - Real-time rendering and animation - Generalization to any portrait - Create a new avatar in 2 minutes - Use bias sinks to combine 2D and 3D data 🏠 🌍 🎥 Great work by Tobias Kirschstein and Simon Giebenhain!

Matthias Niessner

96,238 Aufrufe • vor 8 Monaten

Entire cities are being transformed into immersive digital twins faster than ever, creating the digital foundation that can help make them smarter, more connected, and easier to navigate. This complete historic town was captured with a single Insta360 X5 by one of our mappers and reconstructed by OVER into a high-fidelity 3D environment. What once required specialised equipment can now be captured simply by walking. Our mapping network is now covering more than 15,000 new locations every month, and it continues to grow as more mappers join. Every new mapper means more cities, shopping malls, museums, cultural landmarks, public spaces, and businesses being captured and transformed by OVER into detailed digital twins. These environments can support virtual exploration, indoor and outdoor navigation, location-based XR experiences, cultural preservation, Visual AI, VPS, Robotics localisation, and the development of spatial models capable of understanding real-world environments. Explore the complete digital twin and navigate the town:

Entire cities are being transformed into immersive digital twins faster than ever, creating the digital foundation that can help make them smarter, more connected, and easier to navigate. This complete historic town was captured with a single Insta360 X5 by one of our mappers and reconstructed by OVER into a high-fidelity 3D environment. What once required specialised equipment can now be captured simply by walking. Our mapping network is now covering more than 15,000 new locations every month, and it continues to grow as more mappers join. Every new mapper means more cities, shopping malls, museums, cultural landmarks, public spaces, and businesses being captured and transformed by OVER into detailed digital twins. These environments can support virtual exploration, indoor and outdoor navigation, location-based XR experiences, cultural preservation, Visual AI, VPS, Robotics localisation, and the development of spatial models capable of understanding real-world environments. Explore the complete digital twin and navigate the town:

Over the Reality 🌐

21,529 Aufrufe • vor 1 Monat

I'm excited to share the Geo Register Plugin for LichtFeld Studio from the LichtFeld community! This plugin helps bring Gaussian splat scenes into real-world geographic space. It registers a scene to WGS-84 and ECEF coordinates, so you can click any point on the model and get its latitude, longitude and altitude. It supports multiple georeferencing sources, including EXIF GPS data, image position CSVs, RealityScan camera parameters and saved similarity transforms. Once the scene is registered, you can export geo-referenced splat models as LAS, LAZ or 3D Tiles datasets for use in GIS and 3D mapping workflows. Built for anyone working with drone data, photogrammetry, Gaussian splatting, GIS, ArcGIS or CesiumJS. Link in the comment below!

I'm excited to share the Geo Register Plugin for LichtFeld Studio from the LichtFeld community! This plugin helps bring Gaussian splat scenes into real-world geographic space. It registers a scene to WGS-84 and ECEF coordinates, so you can click any point on the model and get its latitude, longitude and altitude. It supports multiple georeferencing sources, including EXIF GPS data, image position CSVs, RealityScan camera parameters and saved similarity transforms. Once the scene is registered, you can export geo-referenced splat models as LAS, LAZ or 3D Tiles datasets for use in GIS and 3D mapping workflows. Built for anyone working with drone data, photogrammetry, Gaussian splatting, GIS, ArcGIS or CesiumJS. Link in the comment below!

MrNeRF

51,983 Aufrufe • vor 3 Monaten