Загрузка видео...

Не удалось загрузить видео

На главную

📢📢📢 𝐀𝐂𝟑𝐃: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers TL;DR: for 3D camera control in generative video, it really helps knowing *which* part of your model you should mess with Internship by Sherwin Bahmani at Snap

taiyasaki's profile picture

Andrea Tagliasacchi 🇨🇦→✈️→🇺🇲 (@CVPR)

16,653 subscribers

23,040 просмотров • 1 год назад •via X (Twitter)

Наука и технологииНовости и политикаОбразование
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 5

Фото профиля Andrea Tagliasacchi 🇨🇦
Andrea Tagliasacchi 🇨🇦1 год назад

TL;DR (expanded): 1) "when" in the diffusion process you condition for camera matters (i.e. noise scheduler) 2) "how" in the diffusion process you condition for camera maters (i.e. architecture) 3) "what data" you give to your diffusion model to condition camera matters

Фото профиля Andrea Tagliasacchi 🇨🇦
Andrea Tagliasacchi 🇨🇦1 год назад

Why, you ask? 1) camera motion is low-frequency... early denoising iterations deal with low-frequency content 2) early DiT blocks are enough to fine-tune for camera control... more and you lose quality 3) model needs to know what a static view of the dynamic world looks like

Фото профиля Andrea Tagliasacchi 🇨🇦
Andrea Tagliasacchi 🇨🇦1 год назад

A shout to the collaborators @isskoro @guocheng_qian A. Siarohin @willimenapace @SergeyTulyakov at Snap and @DaveLindell at UofT.

Фото профиля Samarth Sinha
Samarth Sinha1 год назад

@sherwinbahmani Congrats @sherwinbahmani !!

Фото профиля Abdullah Hamdi
Abdullah Hamdi1 год назад

@sherwinbahmani Congrats to the team ! Amazing work

Похожие видео

(1/2) 📢📢𝐃𝐢𝐟𝐟𝐮𝐬𝐢𝐨𝐧𝐀𝐯𝐚𝐭𝐚𝐫𝐬 📢📢 High-fidelity 3D head avatars with precise control over viewpoint, expression, and pose. -> Our parametric 3D model enables control & consistency + 2D diffusion makes it photoreal.
0:32

(1/2) 📢📢𝐃𝐢𝐟𝐟𝐮𝐬𝐢𝐨𝐧𝐀𝐯𝐚𝐭𝐚𝐫𝐬 📢📢 High-fidelity 3D head avatars with precise control over viewpoint, expression, and pose. -> Our parametric 3D model enables control & consistency + 2D diffusion makes it photoreal.

Matthias Niessner

66,414 просмотров • 2 лет назад

📢Excited to be at #ICLR2025 for our paper: VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control Poster: Thu 3-5:30 PM (#134) Website: Code: Also check out our #CVPR2025 follow-up AC3D:
0:29

📢Excited to be at #ICLR2025 for our paper: VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control Poster: Thu 3-5:30 PM (#134) Website: Code: Also check out our #CVPR2025 follow-up AC3D:

Sherwin Bahmani

11,608 просмотров • 1 год назад

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:
0:50

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

Sherwin Bahmani

66,563 просмотров • 9 месяцев назад

Generative 3D worlds are amazing! Here is a generative AI video showing fine grained camera control. Camera movements are all hand places within a generated 3D world.
0:52

Generative 3D worlds are amazing! Here is a generative AI video showing fine grained camera control. Camera movements are all hand places within a generated 3D world.

Brittani Natali

21,043 просмотров • 1 год назад

📢𝐋𝟑𝐃𝐆: 𝐋𝐚𝐭𝐞𝐧𝐭 𝟑𝐃 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧 𝐃𝐢𝐟𝐟𝐮𝐬𝐢𝐨𝐧📢 #SIGGRAPHAsia We propose a generative diffusion model for 3D Gaussians. Key is a learnt latent space which substantially reduces the complexity of the diffusion process, thus facilitating room-scale scene generation! Great work by Barbara Roessle in with Norman Müller, Angela Dai, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder!
0:20

📢𝐋𝟑𝐃𝐆: 𝐋𝐚𝐭𝐞𝐧𝐭 𝟑𝐃 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧 𝐃𝐢𝐟𝐟𝐮𝐬𝐢𝐨𝐧📢 #SIGGRAPHAsia We propose a generative diffusion model for 3D Gaussians. Key is a learnt latent space which substantially reduces the complexity of the diffusion process, thus facilitating room-scale scene generation! Great work by Barbara Roessle in with Norman Müller, Angela Dai, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder!

Matthias Niessner

39,496 просмотров • 1 год назад

Full creative control is here in 3D-to-Video. 3D-to-Video lets you manage every detail, from objects to camera angles, with total precision. Consistency is guaranteed.
0:58

Full creative control is here in 3D-to-Video. 3D-to-Video lets you manage every detail, from objects to camera angles, with total precision. Consistency is guaranteed.

MeshyAI

11,324 просмотров • 9 месяцев назад

Light-X Generative 4D Video Rendering with Camera and Illumination Control
1:00

Light-X Generative 4D Video Rendering with Camera and Illumination Control

AK

31,126 просмотров • 7 месяцев назад

and Wan2.2 again. MoCam - novel view synthesis. Lets you change the camera trajectory of a photo/video. - Video Re-camera - 3D Photo Conversion - Cinematic Control - Precise orbital/pan/zoom control - Corrects geometric artifacts
0:37

and Wan2.2 again. MoCam - novel view synthesis. Lets you change the camera trajectory of a photo/video. - Video Re-camera - 3D Photo Conversion - Cinematic Control - Precise orbital/pan/zoom control - Corrects geometric artifacts

Wildminder

14,069 просмотров • 1 месяц назад

Bring your stories to life with a 3D camera. Start with a single frame and turn it into a 3D scene you can move through, shot by shot. Control the camera. Set the pace. Shape the story.
0:43

Bring your stories to life with a 3D camera. Start with a single frame and turn it into a 3D scene you can move through, shot by shot. Control the camera. Set the pace. Shape the story.

Moonvalley

18,713 просмотров • 11 месяцев назад

📢Animating the Uncaptured 📢 We animate 3D humanoid meshes using video diffusion priors given a text prompt. 🎥 🌍 Realistic motion generation for 3D characters - without motion capture! 🚀 Great work by Marc Benedí Angela Dai
0:24

📢Animating the Uncaptured 📢 We animate 3D humanoid meshes using video diffusion priors given a text prompt. 🎥 🌍 Realistic motion generation for 3D characters - without motion capture! 🚀 Great work by Marc Benedí Angela Dai

Matthias Niessner

11,704 просмотров • 1 год назад

📢📢📢 thrilled to announce "𝟒𝐃-𝐟𝐲: 𝐓𝐞𝐱𝐭-𝐭𝐨-𝟒𝐃 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 𝐔𝐬𝐢𝐧𝐠 𝐇𝐲𝐛𝐫𝐢𝐝 𝐒𝐜𝐨𝐫𝐞 𝐃𝐢𝐬𝐭𝐢𝐥𝐥𝐚𝐭𝐢𝐨𝐧 𝐒𝐚𝐦𝐩𝐥𝐢𝐧𝐠" Way to start Sherwin Bahmani 🎉 PhD co-advised with David Lindell at #UofT.
0:53

📢📢📢 thrilled to announce "𝟒𝐃-𝐟𝐲: 𝐓𝐞𝐱𝐭-𝐭𝐨-𝟒𝐃 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 𝐔𝐬𝐢𝐧𝐠 𝐇𝐲𝐛𝐫𝐢𝐝 𝐒𝐜𝐨𝐫𝐞 𝐃𝐢𝐬𝐭𝐢𝐥𝐥𝐚𝐭𝐢𝐨𝐧 𝐒𝐚𝐦𝐩𝐥𝐢𝐧𝐠" Way to start Sherwin Bahmani 🎉 PhD co-advised with David Lindell at #UofT.

Andrea Tagliasacchi @CVPR

74,639 просмотров • 2 лет назад

Nvidia just dropped GEN3C 3D-Informed World-Consistent Video Generation with Precise Camera Control
1:18

Nvidia just dropped GEN3C 3D-Informed World-Consistent Video Generation with Precise Camera Control

AK

58,157 просмотров • 1 год назад

One of the hardest things to achieve with AI is precise character motion. The new model by Kinetix, Kamo-1, is amazing at giving you far more control over your generations. It’s also the first 3D-conditioned model, so it understands the scene in 3D and gives you almost unlimited camera motion. Let me show you how to use it 👇
0:30

One of the hardest things to achieve with AI is precise character motion. The new model by Kinetix, Kamo-1, is amazing at giving you far more control over your generations. It’s also the first 3D-conditioned model, so it understands the scene in 3D and gives you almost unlimited camera motion. Let me show you how to use it 👇

Everett World

19,199 просмотров • 7 месяцев назад

"YoNoSplat: You Only Need One Model for Feedforward 3D Gaussian Splatting" TL;DR: a unified 3D Gaussian splatting model that reconstructs high-quality scene geometry and camera poses from unposed/uncalibrated images in a single forward pass.
0:38

"YoNoSplat: You Only Need One Model for Feedforward 3D Gaussian Splatting" TL;DR: a unified 3D Gaussian splatting model that reconstructs high-quality scene geometry and camera poses from unposed/uncalibrated images in a single forward pass.

Alexandre Morgand

14,839 просмотров • 4 месяцев назад

📢 GenRecon Code Release 📢 Few images in → complete, high-fidelity 3D scene out! GenRecon builds a generative prior on full scenes, resulting in unprecedented 3D reconstruction quality. 🔗 🌐 📄
0:20

📢 GenRecon Code Release 📢 Few images in → complete, high-fidelity 3D scene out! GenRecon builds a generative prior on full scenes, resulting in unprecedented 3D reconstruction quality. 🔗 🌐 📄

Matthias Niessner

38,772 просмотров • 8 дней назад

📢📢📢 𝐀𝐜𝐜𝐞𝐥𝐞𝐫𝐚𝐭𝐢𝐧𝐠 𝐍𝐞𝐮𝐫𝐚𝐥 𝐅𝐢𝐞𝐥𝐝 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐯𝐢𝐚 𝐒𝐨𝐟𝐭 𝐌𝐢𝐧𝐢𝐧𝐠 by Shakiba Kheradmand et al. TL;DR: importance sampling for accelerating your novel view synthesis workloads (...yes, it should also work for 3DGS)
0:35

📢📢📢 𝐀𝐜𝐜𝐞𝐥𝐞𝐫𝐚𝐭𝐢𝐧𝐠 𝐍𝐞𝐮𝐫𝐚𝐥 𝐅𝐢𝐞𝐥𝐝 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐯𝐢𝐚 𝐒𝐨𝐟𝐭 𝐌𝐢𝐧𝐢𝐧𝐠 by Shakiba Kheradmand et al. TL;DR: importance sampling for accelerating your novel view synthesis workloads (...yes, it should also work for 3DGS)

Andrea Tagliasacchi 🇨🇦

21,810 просмотров • 2 лет назад

🚀Excited to introduce GEN3C #CVPR2025, a generative video model with an explicit 3D cache for precise camera control. 🎥It applies to multiple use cases, including single-view and sparse-view NVS🖼️ and challenging settings like monocular dynamic NVS and driving simulation🚗. Project page:
0:49

🚀Excited to introduce GEN3C #CVPR2025, a generative video model with an explicit 3D cache for precise camera control. 🎥It applies to multiple use cases, including single-view and sparse-view NVS🖼️ and challenging settings like monocular dynamic NVS and driving simulation🚗. Project page:

Xuanchi Ren

59,968 просмотров • 1 год назад

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency
0:21

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency

AK

31,997 просмотров • 2 лет назад

🎥 Take Full Control of the Camera! 🚨 I got early access to the new feature Camera Control with T2V 01 - Director Model by Hailuo AI (MiniMax) the ultimate tool for full camera control in video generation! ✨ What’s exciting? Direct with ease: Use natural language to guide your camera exactly how you want Smooth transitions: Combine multiple moves seamlessly for cinematic perfection Here are 10 sample prompts 🔖Bookmark for later!
1:00

🎥 Take Full Control of the Camera! 🚨 I got early access to the new feature Camera Control with T2V 01 - Director Model by Hailuo AI (MiniMax) the ultimate tool for full camera control in video generation! ✨ What’s exciting? Direct with ease: Use natural language to guide your camera exactly how you want Smooth transitions: Combine multiple moves seamlessly for cinematic perfection Here are 10 sample prompts 🔖Bookmark for later!

MayorkingAI

26,819 просмотров • 1 год назад

📢 A Recipe for Generating 3D Worlds From a Single Image 📢 Our recipe explains how existing generative models can be adapted with minimal training effort to generate 3D worlds from a single input image.
0:28

📢 A Recipe for Generating 3D Worlds From a Single Image 📢 Our recipe explains how existing generative models can be adapted with minimal training effort to generate 3D worlds from a single input image.

Katja Schwarz

13,970 просмотров • 1 год назад