Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

📢📢📢 𝐀𝐂𝟑𝐃: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers TL;DR: for 3D camera control in generative video, it really helps knowing which part of your model you should mess with Internship by Sherwin Bahmani at Snap

Andrea Tagliasacchi 🇨🇦→✈️→🇺🇲 (@CVPR)

16,653 subscribers

23,040 views • 1 year ago •via X (Twitter)

Science & Technology News & Politics Education

Anya Rossi• Live Now

Private livecam show

5 Comments

Andrea Tagliasacchi 🇨🇦1 year ago

TL;DR (expanded): 1) "when" in the diffusion process you condition for camera matters (i.e. noise scheduler) 2) "how" in the diffusion process you condition for camera maters (i.e. architecture) 3) "what data" you give to your diffusion model to condition camera matters

Andrea Tagliasacchi 🇨🇦1 year ago

Why, you ask? 1) camera motion is low-frequency... early denoising iterations deal with low-frequency content 2) early DiT blocks are enough to fine-tune for camera control... more and you lose quality 3) model needs to know what a static view of the dynamic world looks like

Andrea Tagliasacchi 🇨🇦1 year ago

A shout to the collaborators @isskoro @guocheng_qian A. Siarohin @willimenapace @SergeyTulyakov at Snap and @DaveLindell at UofT.

Samarth Sinha1 year ago

@sherwinbahmani Congrats @sherwinbahmani !!

Abdullah Hamdi1 year ago

@sherwinbahmani Congrats to the team ! Amazing work

Related Videos

(1/2) 📢📢𝐃𝐢𝐟𝐟𝐮𝐬𝐢𝐨𝐧𝐀𝐯𝐚𝐭𝐚𝐫𝐬 📢📢 High-fidelity 3D head avatars with precise control over viewpoint, expression, and pose. -> Our parametric 3D model enables control & consistency + 2D diffusion makes it photoreal.

(1/2) 📢📢𝐃𝐢𝐟𝐟𝐮𝐬𝐢𝐨𝐧𝐀𝐯𝐚𝐭𝐚𝐫𝐬 📢📢 High-fidelity 3D head avatars with precise control over viewpoint, expression, and pose. -> Our parametric 3D model enables control & consistency + 2D diffusion makes it photoreal.

Matthias Niessner

66,436 views • 2 years ago

📢Excited to be at #ICLR2025 for our paper: VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control Poster: Thu 3-5:30 PM (#134) Website: Code: Also check out our #CVPR2025 follow-up AC3D:

📢Excited to be at #ICLR2025 for our paper: VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control Poster: Thu 3-5:30 PM (#134) Website: Code: Also check out our #CVPR2025 follow-up AC3D:

Sherwin Bahmani

11,615 views • 1 year ago

Camera control is no longer in the prompt Meet 3D Motion Control → Set the camera path in 3D → Your scene becomes the video reference the model follows → Same move, every take, exactly where you set it Available now in Magnific

Camera control is no longer in the prompt Meet 3D Motion Control → Set the camera path in 3D → Your scene becomes the video reference the model follows → Same move, every take, exactly where you set it Available now in Magnific

Magnific

610,355 views • 1 month ago

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

Sherwin Bahmani

66,842 views • 11 months ago

📢𝐋𝟑𝐃𝐆: 𝐋𝐚𝐭𝐞𝐧𝐭 𝟑𝐃 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧 𝐃𝐢𝐟𝐟𝐮𝐬𝐢𝐨𝐧📢 #SIGGRAPHAsia We propose a generative diffusion model for 3D Gaussians. Key is a learnt latent space which substantially reduces the complexity of the diffusion process, thus facilitating room-scale scene generation! Great work by Barbara Roessle in with Norman Müller, Angela Dai, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder!

📢𝐋𝟑𝐃𝐆: 𝐋𝐚𝐭𝐞𝐧𝐭 𝟑𝐃 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧 𝐃𝐢𝐟𝐟𝐮𝐬𝐢𝐨𝐧📢 #SIGGRAPHAsia We propose a generative diffusion model for 3D Gaussians. Key is a learnt latent space which substantially reduces the complexity of the diffusion process, thus facilitating room-scale scene generation! Great work by Barbara Roessle in with Norman Müller, Angela Dai, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder!

Matthias Niessner

39,496 views • 1 year ago

Introducing Atlas: The world's first multimodal world model that generates image and video frames with pixel-perfect camera control and reconstructs them in 3D. Model the world, move the camera, and simulate space & time.

Introducing Atlas: The world's first multimodal world model that generates image and video frames with pixel-perfect camera control and reconstructs them in 3D. Model the world, move the camera, and simulate space & time.

World Labs

10,679,056 views • 11 days ago

Full creative control is here in 3D-to-Video. 3D-to-Video lets you manage every detail, from objects to camera angles, with total precision. Consistency is guaranteed.

Full creative control is here in 3D-to-Video. 3D-to-Video lets you manage every detail, from objects to camera angles, with total precision. Consistency is guaranteed.

MeshyAI

11,379 views • 11 months ago

Bring your stories to life with a 3D camera. Start with a single frame and turn it into a 3D scene you can move through, shot by shot. Control the camera. Set the pace. Shape the story.

Bring your stories to life with a 3D camera. Start with a single frame and turn it into a 3D scene you can move through, shot by shot. Control the camera. Set the pace. Shape the story.

Moonvalley

18,807 views • 1 year ago

One of the hardest things to achieve with AI is precise character motion. The new model by Kinetix, Kamo-1, is amazing at giving you far more control over your generations. It’s also the first 3D-conditioned model, so it understands the scene in 3D and gives you almost unlimited camera motion. Let me show you how to use it 👇

One of the hardest things to achieve with AI is precise character motion. The new model by Kinetix, Kamo-1, is amazing at giving you far more control over your generations. It’s also the first 3D-conditioned model, so it understands the scene in 3D and gives you almost unlimited camera motion. Let me show you how to use it 👇

Everett World

19,199 views • 9 months ago

"YoNoSplat: You Only Need One Model for Feedforward 3D Gaussian Splatting" TL;DR: a unified 3D Gaussian splatting model that reconstructs high-quality scene geometry and camera poses from unposed/uncalibrated images in a single forward pass.

"YoNoSplat: You Only Need One Model for Feedforward 3D Gaussian Splatting" TL;DR: a unified 3D Gaussian splatting model that reconstructs high-quality scene geometry and camera poses from unposed/uncalibrated images in a single forward pass.

Alexandre Morgand

15,035 views • 6 months ago

Introducing H3 Max Camera Controls, the world's fastest 3D video model. Turn a single viewpoint into a navigable 3D scene in less than 3 seconds. Set the horizontal and vertical camera angle in degrees, and H3 Max Camera Controls moves through the scene while preserving geometry, position, and materials across perspectives. Fast enough to control the camera in real time.

Introducing H3 Max Camera Controls, the world's fastest 3D video model. Turn a single viewpoint into a navigable 3D scene in less than 3 seconds. Set the horizontal and vertical camera angle in degrees, and H3 Max Camera Controls moves through the scene while preserving geometry, position, and materials across perspectives. Fast enough to control the camera in real time.

fal

181,522 views • 1 day ago

🚀Excited to introduce GEN3C #CVPR2025, a generative video model with an explicit 3D cache for precise camera control. 🎥It applies to multiple use cases, including single-view and sparse-view NVS🖼️ and challenging settings like monocular dynamic NVS and driving simulation🚗. Project page:

🚀Excited to introduce GEN3C #CVPR2025, a generative video model with an explicit 3D cache for precise camera control. 🎥It applies to multiple use cases, including single-view and sparse-view NVS🖼️ and challenging settings like monocular dynamic NVS and driving simulation🚗. Project page:

Xuanchi Ren

60,119 views • 1 year ago

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency

AK

31,997 views • 2 years ago

📢 PART 1__[The Making of an UnderClass] with Dr. Claud Anderson_ "Major societies in this country have ways of excluding you intentionally without you knowing it."

📢 PART 1__[The Making of an UnderClass] with Dr. Claud Anderson_ "Major societies in this country have ways of excluding you intentionally without you knowing it."

💪🏾Kelly👊🏿⚖-🏴‍☠"The_War" Counter-War

12,968 views • 1 year ago

📢Excited news from sudoAI (@sudoAI_): The interactive demo of our 3D Generative AI model is online! (alpha test for desktop, mobile / pad coming soon) ✨Transform images & text into stunning 3D models in just 60 seconds!🚀 Try it now! 👉

📢Excited news from sudoAI (@sudoAI_): The interactive demo of our 3D Generative AI model is online! (alpha test for desktop, mobile / pad coming soon) ✨Transform images & text into stunning 3D models in just 60 seconds!🚀 Try it now! 👉

Hao Su

17,515 views • 2 years ago

Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models paper page: Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path. Creating computer-generated videos, however, is a tedious manual process, which can be automated by emerging text-to-video diffusion models. Despite great promise, video diffusion models are difficult to control, hindering a user to apply their own creativity rather than amplifying it. To address this challenge, we present a novel approach that combines the controllability of dynamic 3D meshes with the expressivity and editability of emerging diffusion models. For this purpose, our approach takes an animated, low-fidelity rendered mesh as input and injects the ground truth correspondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generation model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path.

Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models paper page: Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path. Creating computer-generated videos, however, is a tedious manual process, which can be automated by emerging text-to-video diffusion models. Despite great promise, video diffusion models are difficult to control, hindering a user to apply their own creativity rather than amplifying it. To address this challenge, we present a novel approach that combines the controllability of dynamic 3D meshes with the expressivity and editability of emerging diffusion models. For this purpose, our approach takes an animated, low-fidelity rendered mesh as input and injects the ground truth correspondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generation model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path.

AK

117,068 views • 2 years ago

ComfyUI-Mesh2Motion 1.2.0 Custom FBX import is now built into ComfyUI-Mesh2Motion. Load any FBX 3D animation, combine it with professional camera presets, and take precise control over your AI video output.

ComfyUI-Mesh2Motion 1.2.0 Custom FBX import is now built into ComfyUI-Mesh2Motion. Load any FBX 3D animation, combine it with professional camera presets, and take precise control over your AI video output.

jtydhr88

13,972 views • 4 months ago

Using 3D assets with Gen-4 References is a simple way to bring highly detailed and specific models into your generative workflows for even more consistency and control. To do so, simply provide a location plate, a quick comp of your 3D model in that space and a style reference.

Using 3D assets with Gen-4 References is a simple way to bring highly detailed and specific models into your generative workflows for even more consistency and control. To do so, simply provide a location plate, a quick comp of your 3D model in that space and a style reference.

Runway

119,208 views • 1 year ago