Loading video...

Video Failed to Load

Go Home

(1/3) Can we turn text-to-image models into photorealistic 3D generators? ViewDiff (#CVPR2024) produces realistic, multi-view consistent images of real-world 3D objects in authentic surroundings. Website Video How does it work?

MattNiessner's profile picture

Matthias Niessner

47,801 subscribers

34,753 views • 2 years ago •via X (Twitter)

Science & TechnologyEducation#CVPR2024
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

this is the future of CGI this new OpenMagic AI can add image, video or 3D product into real life video seamlessly with one click and.. it can even turn product image into 3D model automatically 10 wild examples:
0:10

this is the future of CGI this new OpenMagic AI can add image, video or 3D product into real life video seamlessly with one click and.. it can even turn product image into 3D model automatically 10 wild examples:

el.cine

120,965 views • 1 year ago

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.
0:12

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,801 views • 1 year ago

CSM 3D AI: New image-to-3D AI program AI is reshaping the future of 3D design. This amazing tool transforms still images into 3D models, and the best part? You can animate them effortlessly using AI. Link in comment!
0:13

CSM 3D AI: New image-to-3D AI program AI is reshaping the future of 3D design. This amazing tool transforms still images into 3D models, and the best part? You can animate them effortlessly using AI. Link in comment!

el.cine

54,393 views • 1 year ago

🤩Video world models are cool, but it is cooler if they can simulate any 3D physical actions in real time! Introducing RealWonder⚡️: Now you can simulate 3D physical action (robot actions, 3D forces, force fields, etc.) consequences from a single image in real time! 🧵1/6
0:10

🤩Video world models are cool, but it is cooler if they can simulate any 3D physical actions in real time! Introducing RealWonder⚡️: Now you can simulate 3D physical action (robot actions, 3D forces, force fields, etc.) consequences from a single image in real time! 🧵1/6

Hong-Xing (Koven) Yu

29,756 views • 4 months ago

Introducing Objaverse, a massive open dataset of text-paired 3D objects! Nearly 1 million annotated 3D objects to pave the way to build incredible large-scale 3D generative models: 🧵👇 🤗 Hugging Face: 📝ArXiv: #CVPR2023
0:10

Introducing Objaverse, a massive open dataset of text-paired 3D objects! Nearly 1 million annotated 3D objects to pave the way to build incredible large-scale 3D generative models: 🧵👇 🤗 Hugging Face: 📝ArXiv: #CVPR2023

Matt Deitke

1,600,901 views • 3 years ago

.Tencent Hy HY-World 2.0 has landed,fully OPEN SOURCE ! We kept hearing the same thing: world models look great, but you can't use the output. So we made one you can, real 3D assets you can edit, rearrange, and ship into production. Prompt to engine-ready 3D. For real this time： ✨Directly outputs editable, production-ready 3D assets ✨Seamlessly plugs into existing game pipelines ✨Multimodal input: text, image, videos 🪄Character mode: freely explore streets, buildings, and open worlds with realistic physics, collisions, and no time limits 🔗GitHub:
0:11

.Tencent Hy HY-World 2.0 has landed,fully OPEN SOURCE ! We kept hearing the same thing: world models look great, but you can't use the output. So we made one you can, real 3D assets you can edit, rearrange, and ship into production. Prompt to engine-ready 3D. For real this time： ✨Directly outputs editable, production-ready 3D assets ✨Seamlessly plugs into existing game pipelines ✨Multimodal input: text, image, videos 🪄Character mode: freely explore streets, buildings, and open worlds with realistic physics, collisions, and no time limits 🔗GitHub:

Tencent AI

116,648 views • 2 months ago

the fact that i can take an image of a room and turn it into a 3d model in one shot is actually insane this took like 30 seconds from image to 3d model
0:18

the fact that i can take an image of a room and turn it into a 3d model in one shot is actually insane this took like 30 seconds from image to 3d model

Jan

131,884 views • 7 months ago

🚀 Tencent 𝗛𝗬-𝗪𝗼𝗿𝗹𝗱 𝟮.𝟬 just dropped Text or image → 3D world (3DGS + Mesh). Fully automatic. Video world models let you 𝘄𝗮𝘁𝗰𝗵. We let you 𝗲𝘅𝗽𝗹𝗼𝗿𝗲. #worldmodel #HYWorld
0:14

🚀 Tencent 𝗛𝗬-𝗪𝗼𝗿𝗹𝗱 𝟮.𝟬 just dropped Text or image → 3D world (3DGS + Mesh). Fully automatic. Video world models let you 𝘄𝗮𝘁𝗰𝗵. We let you 𝗲𝘅𝗽𝗹𝗼𝗿𝗲. #worldmodel #HYWorld

Tengfei Wang

17,396 views • 2 months ago

We are happy to announce that ComfyUI now natively supports toad Hunyuan3D 2.0 and Hunyuan3D 2.0 MV (Multi-View) model series! 3 workflows to get started: 🔹Hunyuan3D-2 mv: multi-view to 3D 🔹Hunyuan3D-2 mv Turbo: accelerated multi-view 🔹Hunyuan3D-2: single image to 3D
0:16

We are happy to announce that ComfyUI now natively supports toad Hunyuan3D 2.0 and Hunyuan3D 2.0 MV (Multi-View) model series! 3 workflows to get started: 🔹Hunyuan3D-2 mv: multi-view to 3D 🔹Hunyuan3D-2 mv Turbo: accelerated multi-view 🔹Hunyuan3D-2: single image to 3D

ComfyUI

66,892 views • 1 year ago

We introduce TurboEdit -- simple text-based image editing in 1/2 sec! We leverage few-step diffusion models, mapping real images into noise with an encoder. Please see our #ECCV2024 paper: Work w/ Zongze Wu, Nick Kolkin, Jon Brandt, Eli Shechtman 1/
0:15

We introduce TurboEdit -- simple text-based image editing in 1/2 sec! We leverage few-step diffusion models, mapping real images into noise with an encoder. Please see our #ECCV2024 paper: Work w/ Zongze Wu, Nick Kolkin, Jon Brandt, Eli Shechtman 1/

Richard Zhang

39,306 views • 1 year ago

Hunyuan 3D-2.1 turns any flat image into studio-quality 3D models. And you can do it on this Hugging Face space for free.
0:11

Hunyuan 3D-2.1 turns any flat image into studio-quality 3D models. And you can do it on this Hugging Face space for free.

Rohan Paul

222,392 views • 10 months ago

3d world abilities of AI models keep getting better, this is @Kling_AI 1.5 It can move around a space in 3d and you can steer its camera movements All based on a single input image
0:10

3d world abilities of AI models keep getting better, this is @Kling_AI 1.5 It can move around a space in 3d and you can steer its camera movements All based on a single input image

@levelsio

112,201 views • 1 year ago

✨ Made a new mini feature on Photo AI: [ Grab from 3d model ] So the problem is we're at that stage in time (typical for AI) where image-to-3d models are not good enough but are fun to play with, but we know they'll be good enough in 1-2 years With [ Make 3d model ] you already can turn any Photo AI pic into a 3d model but it still looks hyper clunky and deformed, but it works! One cool idea I had to make that more useful and made now: Let people make a 3d model then change the view of the it with the 3d viewer, then press [ o ] and it grabs a frame of the 3d That image you can then [ Remix ] (img2img), and it becomes a real photo again and that in turn you can then turn into a video again with [ Make video ] So that essentially gives you a fully freeform camera position control to take photos with One thing I need to fix is the background/skybox, I kinda need to take the original photo and remove the person and just get the background for the 3d model viewer, in this case it should be white, but it's a start!
0:51

✨ Made a new mini feature on Photo AI: [ Grab from 3d model ] So the problem is we're at that stage in time (typical for AI) where image-to-3d models are not good enough but are fun to play with, but we know they'll be good enough in 1-2 years With [ Make 3d model ] you already can turn any Photo AI pic into a 3d model but it still looks hyper clunky and deformed, but it works! One cool idea I had to make that more useful and made now: Let people make a 3d model then change the view of the it with the 3d viewer, then press [ o ] and it grabs a frame of the 3d That image you can then [ Remix ] (img2img), and it becomes a real photo again and that in turn you can then turn into a video again with [ Make video ] So that essentially gives you a fully freeform camera position control to take photos with One thing I need to fix is the background/skybox, I kinda need to take the original photo and remove the person and just get the background for the 3d model viewer, in this case it should be white, but it's a start!

@levelsio

119,210 views • 1 year ago

(1/2) Check out 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧𝐀𝐯𝐚𝐭𝐚𝐫𝐬: Photorealistic Head Avatars with Rigged 3D Gaussians! We create photorealistic head avatars by animating 3D Gaussians on a parametric face model - edited and rendered in *real-time*!
0:14

(1/2) Check out 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧𝐀𝐯𝐚𝐭𝐚𝐫𝐬: Photorealistic Head Avatars with Rigged 3D Gaussians! We create photorealistic head avatars by animating 3D Gaussians on a parametric face model - edited and rendered in *real-time*!

Matthias Niessner

126,983 views • 2 years ago

Realistic mini 3D models of people
0:15

Realistic mini 3D models of people

Interesting As Fuck

239,269 views • 1 year ago

Introducing 📦𝗔𝗿𝘁𝗶𝗟𝗮𝘁𝗲𝗻𝘁🔧 (SIGGRAPH Asia 2025) — a high-quality 3D diffusion model that explicitly models object articulation, paving the way for richer, more realistic assets in embodied AI and simulation: – Generates fully articulated 3D objects – Physically plausible joints & motion – High-fidelity 3D Gaussian appearance – Supports generation from a single real image arXiv: Project: Code (coming soon):
0:16

Introducing 📦𝗔𝗿𝘁𝗶𝗟𝗮𝘁𝗲𝗻𝘁🔧 (SIGGRAPH Asia 2025) — a high-quality 3D diffusion model that explicitly models object articulation, paving the way for richer, more realistic assets in embodied AI and simulation: – Generates fully articulated 3D objects – Physically plausible joints & motion – High-fidelity 3D Gaussian appearance – Supports generation from a single real image arXiv: Project: Code (coming soon):

Xingang Pan

11,502 views • 7 months ago

📢📢📢 Today I'll be giving a talk at the #CVPR2024 workshop on "Learning 3D with Multi-View Supervision" See you at 16:20 in Summit 331 Get a preview of the upcoming "3D Gaussian Splatting as Markov Chain Monte Carlo"
0:14

📢📢📢 Today I'll be giving a talk at the #CVPR2024 workshop on "Learning 3D with Multi-View Supervision" See you at 16:20 in Summit 331 Get a preview of the upcoming "3D Gaussian Splatting as Markov Chain Monte Carlo"

Andrea Tagliasacchi @CVPR

47,167 views • 2 years ago

this is over.. for developers Gemini 3 can create app that turn 3D model into hologram-like 3D interactive particles and.. use hands to control it use simple text prompts, no need code literally anyone can create $100K apps now
0:16

this is over.. for developers Gemini 3 can create app that turn 3D model into hologram-like 3D interactive particles and.. use hands to control it use simple text prompts, no need code literally anyone can create $100K apps now

el.cine

498,245 views • 7 months ago