Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Introducing ShapeR, a method for robust conditional 3D shape generation from casually captured sequences. ShapeR leverages a rectified flow transformer conditioned on per-object multimodal data to turn casual image sequences into full metric scene reconstructions. Project Page: Paper: Links to code and huggingface below ⬇️

Yawar Siddiqui

1,740 subscribers

71,078 views • 9 months ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

15 Comments

Yawar Siddiqui9 months ago

Code: Model: Evaluation Dataset: Video:

Jim9 months ago

This is going to be amazing for so many applications especially robotics training data

Zwecharki of Immense AGI fundamentalism9 months ago

Its incredible. Can it be used with unstructured lidar data so we can combine it with iphone lidar + camera mount setups? Or does the lidar have to be specifically aligned with the cameras like on the Aria glasses?

David Ferrera9 months ago

What do I use to capture the data: Meta Quest, iPhone, lidar?

Not Indexed9 months ago

@droneforge

X-Files9 months ago

@intuition_trust You need a drone automated like prometheus

Zwecharki of Immense AGI fundamentalism9 months ago

Can we use this to clear out floaters, low data areas that fuse together or white wall glitches for gaussian splat scans?

Joshua Poddoku9 months ago

Impressive direction, curious how it holds up in cluttered indoor scenes

Amit9 months ago

Rectified flow transformers for 3D generation are a step beyond vanilla diffusion. Conditioning on multimodal signals unlocks compositionality, but metric accuracy from casual sequences still hinges on how you handle scale drift and occlusions.

Reza Sayar9 months ago

wow!! awesome work!! and thanks for releasing everything! 🤗 @pablovelagomez1 lookie 🔥

mrkelly9 months ago

Casual sequences work great. Until your real data doesn't match the training distribution.

Micah Alpern9 months ago

What do you want to use it for? Making a robot that moves?

Nick Harris9 months ago

Is this what Houzz Pro uses to build their mockups with the iphone scans

ObotX - Open Source Robotics9 months ago

nice!

indoor positioning9 months ago

@grok 这个的应用场景

Related Videos

3D Object Tracking without Training Data? In our nature Machine Intelligence paper ( we recast 3D tracking as an inverse neural rendering task where we fit a scene graph to an image that best explains this image. The method generalizes to completely unseen datasets and is explainable. Project and Code: Fun collaboration between Princeton Computer Science and Torc Robotics, with Julian Ost and Tanushree Banerjee leading this project.

3D Object Tracking without Training Data? In our nature Machine Intelligence paper ( we recast 3D tracking as an inverse neural rendering task where we fit a scene graph to an image that best explains this image. The method generalizes to completely unseen datasets and is explainable. Project and Code: Fun collaboration between Princeton Computer Science and Torc Robotics, with Julian Ost and Tanushree Banerjee leading this project.

Felix Heide

27,858 views • 1 year ago

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

AK

62,768 views • 3 years ago

📢📢📢 Excited to share our new work *Autonomous Character-Scene Interaction Synthesis from Text Instruction* (Siggraph Asia 24). It presents a unified model for flexible scene-conditioned motion generation given text, scene, trajectory conditions. The results with smooth interaction look very impressive! 📰Paper: Project: Code and data will be released soon.

📢📢📢 Excited to share our new work Autonomous Character-Scene Interaction Synthesis from Text Instruction (Siggraph Asia 24). It presents a unified model for flexible scene-conditioned motion generation given text, scene, trajectory conditions. The results with smooth interaction look very impressive! 📰Paper: Project: Code and data will be released soon.

Siyuan Huang

11,451 views • 2 years ago

Capturing every combination of human-object interaction is not feasible. We need compositionally. Most interactions are local. A hand holds a cup, a chair supports the pelvis, while much of the body remains free. COSMI, by Daniel Escandar Ilya Petrov builds on this observation to compose existing single-object datasets into 222k multi-object sequences, totaling 275 hours, with up to 5 objects per sequence. LLM reasoning and geometric checks ensure the compositions remain plausible. On this data, we train a single diffusion transformer for interactions with 1 to 5 objects, predicting each object relative to its interacting body part and generalizing to unseen objects and combinations. Data and code will be made available: 🌐

Capturing every combination of human-object interaction is not feasible. We need compositionally. Most interactions are local. A hand holds a cup, a chair supports the pelvis, while much of the body remains free. COSMI, by Daniel Escandar Ilya Petrov builds on this observation to compose existing single-object datasets into 222k multi-object sequences, totaling 275 hours, with up to 5 objects per sequence. LLM reasoning and geometric checks ensure the compositions remain plausible. On this data, we train a single diffusion transformer for interactions with 1 to 5 objects, predicting each object relative to its interacting body part and generalizing to unseen objects and combinations. Data and code will be made available: 🌐

Gerard Pons-Moll

21,576 views • 12 days ago

Introducing Wonder 3D, a new generative AI model in Flow Studio for fast and detailed 3D asset creation. 💬 Text to 3D — Watch your words take shape. Turn a simple prompt into fully textured 3D characters and props. 📸 Image to 3D — Add depth to your visuals. Transform concept art or reference images into dimensional 3D models. 🖼️ Text to Image — Visualize ideas early. Turn descriptions into concept imagery before converting them into 3D. Built for endless storytelling. Give your imagination some dimension. Available in all tiers. Find it under Wonder Tools. 👉 Get started: 💡 Learn more:

Introducing Wonder 3D, a new generative AI model in Flow Studio for fast and detailed 3D asset creation. 💬 Text to 3D — Watch your words take shape. Turn a simple prompt into fully textured 3D characters and props. 📸 Image to 3D — Add depth to your visuals. Transform concept art or reference images into dimensional 3D models. 🖼️ Text to Image — Visualize ideas early. Turn descriptions into concept imagery before converting them into 3D. Built for endless storytelling. Give your imagination some dimension. Available in all tiers. Find it under Wonder Tools. 👉 Get started: 💡 Learn more:

Autodesk Flow Studio

108,426 views • 7 months ago

An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion discuss: We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.

An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion discuss: We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.

AK

66,769 views • 2 years ago

🕹️We are excited to introduce "ChronoEdit: Towards Temporal Reasoning for Image Editing and World Simulation" ChronoEdit reframes image editing as a video generation task to encourage temporal consistency. It leverages a temporal reasoning stage that denoises with “video reasoning tokens” to "reason" on physically plausible edits. See the attached video for results. Project Page: Arxiv: Code and model are coming.

🕹️We are excited to introduce "ChronoEdit: Towards Temporal Reasoning for Image Editing and World Simulation" ChronoEdit reframes image editing as a video generation task to encourage temporal consistency. It leverages a temporal reasoning stage that denoises with “video reasoning tokens” to "reason" on physically plausible edits. See the attached video for results. Project Page: Arxiv: Code and model are coming.

Huan Ling

37,048 views • 1 year ago

WorldFlow3D: Unbounded 3D World Generation 🌍 by Flow Through Hierarchical Distributions, without VAEs ! We reformulate 3D generation as flowing through sequentially finer 3D distributions, cutting training time by more than half ⏱️ compared to existing approaches! Vectorized map layouts provide full scene controllability 🗺️, and a novel flow-field alignment process enables causally coherent, spatially unbounded generation 🌍. This generative method generalizes across both real and synthetic data distributions! Project: Project led by Amogh Joshi and Julian Ost — will be super fun to build on this! 🔥

WorldFlow3D: Unbounded 3D World Generation 🌍 by Flow Through Hierarchical Distributions, without VAEs ! We reformulate 3D generation as flowing through sequentially finer 3D distributions, cutting training time by more than half ⏱️ compared to existing approaches! Vectorized map layouts provide full scene controllability 🗺️, and a novel flow-field alignment process enables causally coherent, spatially unbounded generation 🌍. This generative method generalizes across both real and synthetic data distributions! Project: Project led by Amogh Joshi and Julian Ost — will be super fun to build on this! 🔥

Felix Heide

19,661 views • 6 months ago

Introducing Code Video Model 🎬 Amazing work led by Ziyu Wan and the team! Agentic coding as the engine for controllable video generation. Scene layout, motion, timing, camera trajectories, and physical dynamics, all in editable code. A coding agent turns prompts into executable programs that render proxy videos. Our training-free early-step conditioning lets a pretrained video model follow the proxy’s structure while adopting a reference image’s appearance. From physics-grounded motion and bullet time to 3D/4D reconstruction and robotics simulation, it all starts with code. Watch the demo and explore more visual results 👇 Project page: Paper: Code: Data: Youtube:

Introducing Code Video Model 🎬 Amazing work led by Ziyu Wan and the team! Agentic coding as the engine for controllable video generation. Scene layout, motion, timing, camera trajectories, and physical dynamics, all in editable code. A coding agent turns prompts into executable programs that render proxy videos. Our training-free early-step conditioning lets a pretrained video model follow the proxy’s structure while adopting a reference image’s appearance. From physics-grounded motion and bullet time to 3D/4D reconstruction and robotics simulation, it all starts with code. Watch the demo and explore more visual results 👇 Project page: Paper: Code: Data: Youtube:

Gordon (Guocheng) Qian

39,496 views • 7 days ago

Humanoid robots shouldn't just follow pre-defined movements—they should perform! 🤖✨ Introducing UniAct: A unified model for multimodal motion generation and action streaming. Most humanoids are limited to pre-designed moves. UniAct changes the game by allowing robots to generate live action sequences from: 📝 Text instructions 🎶 Music rhythms 📉 Spatial trajectories 🔄 Cross-modal signals Whether it’s dancing to a beat or following a complex path, UniAct brings humanoid robots to life in real-time. 🚀 🔗 Project: 📄 Paper:

Humanoid robots shouldn't just follow pre-defined movements—they should perform! 🤖✨ Introducing UniAct: A unified model for multimodal motion generation and action streaming. Most humanoids are limited to pre-designed moves. UniAct changes the game by allowing robots to generate live action sequences from: 📝 Text instructions 🎶 Music rhythms 📉 Spatial trajectories 🔄 Cross-modal signals Whether it’s dancing to a beat or following a complex path, UniAct brings humanoid robots to life in real-time. 🚀 🔗 Project: 📄 Paper:

Siyuan Huang

10,672 views • 9 months ago

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

Sherwin Bahmani

66,918 views • 1 year ago

We’re excited to share DiT4DiT, an end-to-end Video-Action Model for robot learning that unifies a video Diffusion Transformer and an action Diffusion Transformer in a single cascaded framework. By leveraging the rich spatiotemporal and physical dynamics learned through video generation, rather than static image-text priors, DiT4DiT achieves state-of-the-art results on LIBERO (98.6%) and RoboCasa GR1 (50.8%) with far less training data, delivering over 10× better sample efficiency and up to 7× faster convergence. Real-world deployment on a humanoid robot further shows robust generalization. We believe this is a step toward making video generation a powerful backbone for robot policy learning. This work builds upon the brilliant foundations laid by Nvidia's GR00T and Cosmos. Project: Paper: Code: Coming soon. In the meantime, you can ask your coding agent to reproduce the method based on GR00T/Cosmos.

We’re excited to share DiT4DiT, an end-to-end Video-Action Model for robot learning that unifies a video Diffusion Transformer and an action Diffusion Transformer in a single cascaded framework. By leveraging the rich spatiotemporal and physical dynamics learned through video generation, rather than static image-text priors, DiT4DiT achieves state-of-the-art results on LIBERO (98.6%) and RoboCasa GR1 (50.8%) with far less training data, delivering over 10× better sample efficiency and up to 7× faster convergence. Real-world deployment on a humanoid robot further shows robust generalization. We believe this is a step toward making video generation a powerful backbone for robot policy learning. This work builds upon the brilliant foundations laid by Nvidia's GR00T and Cosmos. Project: Paper: Code: Coming soon. In the meantime, you can ask your coding agent to reproduce the method based on GR00T/Cosmos.

Shuo Yang

31,872 views • 7 months ago

We are introducing Hunyuan3D-Part: an open-source part-level 3D shape generation model that outperforms all existing open and close-source models. Highlights: 🔹P3-SAM: The industry's first native 3D part segmentation model. 🔹X-Part: A part generation model that achieves state-of-the-art results in controllability and shape quality. Key-features: 1️⃣Eliminates the use of 2D SAM during training, relying solely on a large-scale dataset with 3.7 million shapes and clean part annotations. 2️⃣Introduces a new automated segmentation pipeline in 3D without user intervention. 3️⃣Implements a diffusion-based part decomposition pipeline utilizing both geometry and semantic clues. Code: Weights: Tech reports: 🔸P3-SAM： → Paper: → Project page: 🔸X-Part： → Paper: → Project page: Try it now： → (Light version) Hugging Face demo: → (Full version) Hunyuan3D Studio:

We are introducing Hunyuan3D-Part: an open-source part-level 3D shape generation model that outperforms all existing open and close-source models. Highlights: 🔹P3-SAM: The industry's first native 3D part segmentation model. 🔹X-Part: A part generation model that achieves state-of-the-art results in controllability and shape quality. Key-features: 1️⃣Eliminates the use of 2D SAM during training, relying solely on a large-scale dataset with 3.7 million shapes and clean part annotations. 2️⃣Introduces a new automated segmentation pipeline in 3D without user intervention. 3️⃣Implements a diffusion-based part decomposition pipeline utilizing both geometry and semantic clues. Code: Weights: Tech reports: 🔸P3-SAM： → Paper: → Project page: 🔸X-Part： → Paper: → Project page: Try it now： → (Light version) Hugging Face demo: → (Full version) Hunyuan3D Studio:

Tencent Hy

72,640 views • 1 year ago

Code and data are now online for CameraHMR, our state-of-the-art parametric 3D human pose and shape (HPS) estimation method that will appear at hashtag#3DV2025. There are 4 key contributions that make it so accurate and robust: 1. To get accurate 3D shape and pose as well as good alignment to image features, you need to know the focal length of the camera. To solve this, we train HumanFOV to compute the field of view. 2. We introduce CameraHMR, which integrates HumanFOV into HMR2.0 to exploit the estimated focal length. 3. To get accurate pseudo ground truth (pGT) training data, we compute the focal length for images in 4DHumans dataset and modify SMPLify to take this into account. 4. But SMPLify only uses sparse 2D keypoints, which do not capture body shape. So we train a dense surface keypoint detector, DenseKP, on BEDLAM and run it on 4DHumans, resulting in improved body shape. The resulting method is CamSMPLify. We iterate training CameraHMR and running CamSMPLify on the training set initialized with CameraHMR. This results in much improved pGT for 4DHumans and a SOTA single-image HMR method.

Code and data are now online for CameraHMR, our state-of-the-art parametric 3D human pose and shape (HPS) estimation method that will appear at hashtag#3DV2025. There are 4 key contributions that make it so accurate and robust: 1. To get accurate 3D shape and pose as well as good alignment to image features, you need to know the focal length of the camera. To solve this, we train HumanFOV to compute the field of view. 2. We introduce CameraHMR, which integrates HumanFOV into HMR2.0 to exploit the estimated focal length. 3. To get accurate pseudo ground truth (pGT) training data, we compute the focal length for images in 4DHumans dataset and modify SMPLify to take this into account. 4. But SMPLify only uses sparse 2D keypoints, which do not capture body shape. So we train a dense surface keypoint detector, DenseKP, on BEDLAM and run it on 4DHumans, resulting in improved body shape. The resulting method is CamSMPLify. We iterate training CameraHMR and running CamSMPLify on the training set initialized with CameraHMR. This results in much improved pGT for 4DHumans and a SOTA single-image HMR method.

Michael Black

21,696 views • 1 year ago

Introducing SDXL Turbo: A real-time text-to-image generation model. SDXL Turbo achieves state-of-the-art performance with a new distillation technology, enabling single-step image generation with unprecedented quality, reducing the required step count from 50 to just one. The code, research paper, and weights for non-commercial use are now available on our website. You can test SDXL Turbo on Stability AI’s image editing platform Clipdrop, with a beta demonstration of the real-time text-to-image generation capabilities. Learn more:

Introducing SDXL Turbo: A real-time text-to-image generation model. SDXL Turbo achieves state-of-the-art performance with a new distillation technology, enabling single-step image generation with unprecedented quality, reducing the required step count from 50 to just one. The code, research paper, and weights for non-commercial use are now available on our website. You can test SDXL Turbo on Stability AI’s image editing platform Clipdrop, with a beta demonstration of the real-time text-to-image generation capabilities. Learn more:

Stability AI

977,000 views • 2 years ago

Meet MapAnything – a transformer that directly regresses factored metric 3D scene geometry (from images, calibration, poses, or depth) in an end-to-end way. No pipelines, no extra stages. Just 3D geometry & cameras, straight from any type of input, delivering new state-of-the-art results 🚀 One universal model enables SoTA for: 🔥 Mono Depth Estimation 🔥 Multi-View SfM 🔥 Multi-View Stereo 🔥 Depth Completion 🔥 Registration … and many more possibilities! – plus everything is metric 🎯 We release code for data processing, training, benchmarking & ablations – everything Apache 2.0! Details & Links 👇

Meet MapAnything – a transformer that directly regresses factored metric 3D scene geometry (from images, calibration, poses, or depth) in an end-to-end way. No pipelines, no extra stages. Just 3D geometry & cameras, straight from any type of input, delivering new state-of-the-art results 🚀 One universal model enables SoTA for: 🔥 Mono Depth Estimation 🔥 Multi-View SfM 🔥 Multi-View Stereo 🔥 Depth Completion 🔥 Registration … and many more possibilities! – plus everything is metric 🎯 We release code for data processing, training, benchmarking & ablations – everything Apache 2.0! Details & Links 👇

Nikhil Keetha

123,680 views • 1 year ago

T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation paper page: Despite the stunning ability to generate high-quality images by recent text-to-image models, current approaches often struggle to effectively compose objects with different attributes and relationships into a complex and coherent scene. We propose T2I-CompBench, a comprehensive benchmark for open-world compositional text-to-image generation, consisting of 6,000 compositional text prompts from 3 categories (attribute binding, object relationships, and complex compositions) and 6 sub-categories (color binding, shape binding, texture binding, spatial relationships, non-spatial relationships, and complex compositions). We further propose several evaluation metrics specifically designed to evaluate compositional text-to-image generation. We introduce a new approach, Generative mOdel fine-tuning with Reward-driven Sample selection (GORS), to boost the compositional text-to-image generation abilities of pretrained text-to-image models. Extensive experiments and evaluations are conducted to benchmark previous methods on T2I-CompBench, and to validate the effectiveness of our proposed evaluation metrics and GORS approach.

T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation paper page: Despite the stunning ability to generate high-quality images by recent text-to-image models, current approaches often struggle to effectively compose objects with different attributes and relationships into a complex and coherent scene. We propose T2I-CompBench, a comprehensive benchmark for open-world compositional text-to-image generation, consisting of 6,000 compositional text prompts from 3 categories (attribute binding, object relationships, and complex compositions) and 6 sub-categories (color binding, shape binding, texture binding, spatial relationships, non-spatial relationships, and complex compositions). We further propose several evaluation metrics specifically designed to evaluate compositional text-to-image generation. We introduce a new approach, Generative mOdel fine-tuning with Reward-driven Sample selection (GORS), to boost the compositional text-to-image generation abilities of pretrained text-to-image models. Extensive experiments and evaluations are conducted to benchmark previous methods on T2I-CompBench, and to validate the effectiveness of our proposed evaluation metrics and GORS approach.

AK

17,503 views • 3 years ago