Loading video...

Video Failed to Load

Go Home

Early R&D using our SP-6M dataset. Exploring image-to-3D reconstruction from single images, including heavily modified inputs (lighting, hair, etc). Still a work in progress.

Ten_24's profile picture

3D Scanstore

12,241 subscribers

52,940 views • 3 months ago •via X (Twitter)

ArtsEducationScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Finally! #PHORHUM -- our 3D human reconstruction model from a single image -- is available to the research community 🎉 PHORHUM is joint work with Mihai Zanfir & Cristian Sminchisescu. How to get access: 👇
1:09

Finally! #PHORHUM -- our 3D human reconstruction model from a single image -- is available to the research community 🎉 PHORHUM is joint work with Mihai Zanfir & Cristian Sminchisescu. How to get access: 👇

Thiemo Alldieck

12,764 views • 3 years ago

Excited to share our latest work using synthetic image generation for common #DBS targeting sequences! From a single MP2RAGE acquisition, we can create T1, FGATIR, EDGE, GM nulled, etc. & also optimize image contrast. #neurotwitter #NeuroRad
0:22

Excited to share our latest work using synthetic image generation for common #DBS targeting sequences! From a single MP2RAGE acquisition, we can create T1, FGATIR, EDGE, GM nulled, etc. & also optimize image contrast. #neurotwitter #NeuroRad

Erik Middlebrooks, MD

19,642 views • 2 years ago

📢📢 𝐏𝐞𝐫𝐜𝐇𝐞𝐚𝐝: 𝐏𝐞𝐫𝐜𝐞𝐩𝐭𝐮𝐚𝐥 𝐇𝐞𝐚𝐝 𝐌𝐨𝐝𝐞𝐥 𝐟𝐨𝐫 𝐒𝐢𝐧𝐠𝐥𝐞-𝐈𝐦𝐚𝐠𝐞 𝟑𝐃 𝐇𝐞𝐚𝐝 𝐑𝐞𝐜𝐨𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 & 𝐄𝐝𝐢𝐭𝐢𝐧𝐠📢📢 PercHead reconstructs realistic 3D heads from a single image and enables disentangled 3D editing via geometric controls and style inputs from images or text. At its core is a generalized 3D head decoder trained with perceptual supervision from DINOv2 and SAM 2.1. We find that our new perceptual loss formulation improves reconstruction fidelity compared to commonly-used methods such as LPIPS. Our trained reconstruction model is able to generate 3D-consistent heads from a single input image. Even with challenging side-view inputs, the model robustly infers missing regions for a coherent, high-fidelity output. In addition, our architecture seamlessly adapts to downstream tasks: by swapping the encoder, we can transform the model into a disentangled 3D editing pipeline. In this scenario, we can control geometry through - potentially hand-drawn - segmentation maps, and condition style via image or text prompt. We also provide an interactive GUI to enable the exploration of our editing pipeline. 🌍 📽️ Great work by Antonio Oroz and Tobias Kirschstein
0:30

📢📢 𝐏𝐞𝐫𝐜𝐇𝐞𝐚𝐝: 𝐏𝐞𝐫𝐜𝐞𝐩𝐭𝐮𝐚𝐥 𝐇𝐞𝐚𝐝 𝐌𝐨𝐝𝐞𝐥 𝐟𝐨𝐫 𝐒𝐢𝐧𝐠𝐥𝐞-𝐈𝐦𝐚𝐠𝐞 𝟑𝐃 𝐇𝐞𝐚𝐝 𝐑𝐞𝐜𝐨𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 & 𝐄𝐝𝐢𝐭𝐢𝐧𝐠📢📢 PercHead reconstructs realistic 3D heads from a single image and enables disentangled 3D editing via geometric controls and style inputs from images or text. At its core is a generalized 3D head decoder trained with perceptual supervision from DINOv2 and SAM 2.1. We find that our new perceptual loss formulation improves reconstruction fidelity compared to commonly-used methods such as LPIPS. Our trained reconstruction model is able to generate 3D-consistent heads from a single input image. Even with challenging side-view inputs, the model robustly infers missing regions for a coherent, high-fidelity output. In addition, our architecture seamlessly adapts to downstream tasks: by swapping the encoder, we can transform the model into a disentangled 3D editing pipeline. In this scenario, we can control geometry through - potentially hand-drawn - segmentation maps, and condition style via image or text prompt. We also provide an interactive GUI to enable the exploration of our editing pipeline. 🌍 📽️ Great work by Antonio Oroz and Tobias Kirschstein

Matthias Niessner

18,855 views • 8 months ago

UniHair can create 3D hair models from single-view portraits, handling both braided and un-braided styles! It uses a large dataset and advanced techniques to accurately capture complex hairstyles and generalize well to real images. Links ⬇️
1:04

UniHair can create 3D hair models from single-view portraits, handling both braided and un-braided styles! It uses a large dataset and advanced techniques to accurately capture complex hairstyles and generalize well to real images. Links ⬇️

Dreaming Tulpa 🥓👑

25,200 views • 1 year ago

I prompted Omma to build a #threejs app with real dynamic lighting on a 3D mesh - generated from a single image, using Depth Anything v2 + Transformers from Hugging Face ⚡ Depth estimation in the browser 🫧 3D mesh from the depth map 💡 Dynamic lighting reacting to the geometry No manual code. Pure AI prompting. The future of 3D on the web feels wide open.
0:40

I prompted Omma to build a #threejs app with real dynamic lighting on a 3D mesh - generated from a single image, using Depth Anything v2 + Transformers from Hugging Face ⚡ Depth estimation in the browser 🫧 3D mesh from the depth map 💡 Dynamic lighting reacting to the geometry No manual code. Pure AI prompting. The future of 3D on the web feels wide open.

Joseph Azar

14,041 views • 2 months ago

📢Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstruction📢 -> highly accurate face reconstruction by training powerful VITs via surface normals and UV-coordinates estimation. The geometric cues from our 2D foundation model backbone constrain the 3DMM parameters, which allows us to achieve remarkable reconstruction accuracy - works for both single image and videos! In addition, we introduce a new 3D face reconstruction benchmark that evaluates both neutral and posed face geometry. 🌍 📷 Great work by Simon Giebenhain Tobias Kirschstein Martin Rünz Lourdes Agapito
0:32

📢Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstruction📢 -> highly accurate face reconstruction by training powerful VITs via surface normals and UV-coordinates estimation. The geometric cues from our 2D foundation model backbone constrain the 3DMM parameters, which allows us to achieve remarkable reconstruction accuracy - works for both single image and videos! In addition, we introduce a new 3D face reconstruction benchmark that evaluates both neutral and posed face geometry. 🌍 📷 Great work by Simon Giebenhain Tobias Kirschstein Martin Rünz Lourdes Agapito

Matthias Niessner

62,159 views • 1 year ago

📢📢 𝐀𝐯𝐚𝐭𝟑𝐫 📢📢 Avat3r creates high-quality 3D head avatars from just a few input images in a single forward pass with a new dynamic 3DGS reconstruction model. Video: Project: Our core idea is to make Gaussian Reconstruction Models animatable. We find that a simple cross-attention to an expression code sequence is already sufficient to model complex facial expressions. We then incorporate position maps from DUSt3R and feature maps from Sapiens to facilitate the prediction task. While DUSt3R's position maps act as a pixel-aligned initialization for the Gaussians' positions, the Sapiens feature maps help the cross-view transformer to match corresponding image tokens in the 4 input images. One major challenge in creating a 3D head avatar from smartphone images comes from inconsistent facial expressions when the subject could not remain perfectly static during the capture. We eliminate this static requirement by simply showing our model input images with different facial expressions during training. This technique makes our model robust to inconsistent input images later on. Finally, we show that despite the model has been trained with 4 input images, one can even create a 3D head avatar when only a single image is available. To achieve this, we employ a pre-trained 3D GAN to lift the single image to 3D and then render the 4 input images for our model. This allows us to create 3D head avatars from single images and even highly out-of-distribution examples like AI generated faces, paintings or statues. Great work by Tobias Kirschstein from his internship at Meta with Javier Romero, Artem Sevastopolsky, and Shunsuke Saito
0:38

📢📢 𝐀𝐯𝐚𝐭𝟑𝐫 📢📢 Avat3r creates high-quality 3D head avatars from just a few input images in a single forward pass with a new dynamic 3DGS reconstruction model. Video: Project: Our core idea is to make Gaussian Reconstruction Models animatable. We find that a simple cross-attention to an expression code sequence is already sufficient to model complex facial expressions. We then incorporate position maps from DUSt3R and feature maps from Sapiens to facilitate the prediction task. While DUSt3R's position maps act as a pixel-aligned initialization for the Gaussians' positions, the Sapiens feature maps help the cross-view transformer to match corresponding image tokens in the 4 input images. One major challenge in creating a 3D head avatar from smartphone images comes from inconsistent facial expressions when the subject could not remain perfectly static during the capture. We eliminate this static requirement by simply showing our model input images with different facial expressions during training. This technique makes our model robust to inconsistent input images later on. Finally, we show that despite the model has been trained with 4 input images, one can even create a 3D head avatar when only a single image is available. To achieve this, we employ a pre-trained 3D GAN to lift the single image to 3D and then render the 4 input images for our model. This allows us to create 3D head avatars from single images and even highly out-of-distribution examples like AI generated faces, paintings or statues. Great work by Tobias Kirschstein from his internship at Meta with Javier Romero, Artem Sevastopolsky, and Shunsuke Saito

Matthias Niessner

74,698 views • 1 year ago

Our new work PSHuman reconstructs a detailed 3D human mesh from a single-view image in ~1min. Codes have already been released! Welcome to try it! Project page: Codes: Paper:
0:39

Our new work PSHuman reconstructs a detailed 3D human mesh from a single-view image in ~1min. Codes have already been released! Welcome to try it! Project page: Codes: Paper:

Yuan Liu

53,252 views • 1 year ago

We’ve been busy building an AI system to generate 3D worlds from a single image. Check out some early results on our site, where you can interact with our scenes directly in the browser! 1/n
1:12

We’ve been busy building an AI system to generate 3D worlds from a single image. Check out some early results on our site, where you can interact with our scenes directly in the browser! 1/n

World Labs

852,540 views • 1 year ago

Grok Imagine has an excellent image editor. The video below was created from a single image (created by Grok Imagine) that I modified using Grok Imagine's image editor. If you have any trouble using it, let me know in the comments below. I'll try to help you.
0:57

Grok Imagine has an excellent image editor. The video below was created from a single image (created by Grok Imagine) that I modified using Grok Imagine's image editor. If you have any trouble using it, let me know in the comments below. I'll try to help you.

Déborah

1,823,115 views • 5 months ago

Adobe is entering the image-to-3D game! Their new method, LRM, can create high-fidelity 3D meshes from a single image in just 5 seconds 🔥 It's trained on 1 million objects and is able to generate objects from real-world images and generative AI models.
0:28

Adobe is entering the image-to-3D game! Their new method, LRM, can create high-fidelity 3D meshes from a single image in just 5 seconds 🔥 It's trained on 1 million objects and is able to generate objects from real-world images and generative AI models.

Dreaming Tulpa 🥓👑

270,942 views • 2 years ago

DiffSplat Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation DiffSplat is a generative framework to synthesize 3D Gaussian Splats from text prompts & single-view images in ⚡️ 1~2 seconds. It is fine-tuned directly from a pretrained text-to-image diffusion model.
0:24

DiffSplat Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation DiffSplat is a generative framework to synthesize 3D Gaussian Splats from text prompts & single-view images in ⚡️ 1~2 seconds. It is fine-tuned directly from a pretrained text-to-image diffusion model.

AK

38,416 views • 1 year ago

I'm excited to share our new work, VistaDream, which generates a 3D Gaussian field from a single-view image. The codes have already been released. Project page: (with interactive demos) Code: Paper:
0:27

I'm excited to share our new work, VistaDream, which generates a 3D Gaussian field from a single-view image. The codes have already been released. Project page: (with interactive demos) Code: Paper:

Yuan Liu

79,592 views • 1 year ago

From one frame to a whole video! I built a continuous 45-second shot out of a still image. Using a single image I created using Dreamina, I added new scenes with nano-banana. Then, using Kling's First/Last frame and Sora, I made it all work together. Topaz Astra for upscale:
0:45

From one frame to a whole video! I built a continuous 45-second shot out of a still image. Using a single image I created using Dreamina, I added new scenes with nano-banana. Then, using Kling's First/Last frame and Sora, I made it all work together. Topaz Astra for upscale:

Alex Patrascu

11,519 views • 10 months ago

We used machine learning to quickly recreate key moments from the World Cup in 3D using just a single photograph. Read about our process:
0:25

We used machine learning to quickly recreate key moments from the World Cup in 3D using just a single photograph. Read about our process:

NYTimes R&D

100,671 views • 3 years ago

Generate persistent 3D worlds from a single image, bigger and better than ever! We’re excited to share our latest results and invite you to try out our world generation model in a limited beta preview.
0:34

Generate persistent 3D worlds from a single image, bigger and better than ever! We’re excited to share our latest results and invite you to try out our world generation model in a limited beta preview.

World Labs

834,203 views • 10 months ago

3D editing is hard: you need to ground an image + instruction and generate a faithful 3D shape in one forward pass -- no test-time optimization. So, we steer pretrained image-to-3D representations to do text-guided 3D edits; no massive 3D edit-pair dataset needed. Key trap: the “no-edit” solution is a nasty local minimum. We fix it with preference optimization, pushing the model to actually edit. Steer3D is the second work that adapts alignment ideas from LLMs to the 3D modality. SAM 3D also used DPO to improve its 3D generations.
0:22

3D editing is hard: you need to ground an image + instruction and generate a faithful 3D shape in one forward pass -- no test-time optimization. So, we steer pretrained image-to-3D representations to do text-guided 3D edits; no massive 3D edit-pair dataset needed. Key trap: the “no-edit” solution is a nasty local minimum. We fix it with preference optimization, pushing the model to actually edit. Steer3D is the second work that adapts alignment ideas from LLMs to the 3D modality. SAM 3D also used DPO to improve its 3D generations.

Georgia Gkioxari

116,061 views • 7 months ago

new thing: 'WORK IN PROGRESS' (s01e01) recording takeaways from our weekly internal debates -- the goal is to share early ideas (probably wrong!) on themes we're exploring, and invite feedback. this week Alana Levin Daniel Barabander + me talk about non-USD denominated stablecoins
15:28

new thing: 'WORK IN PROGRESS' (s01e01) recording takeaways from our weekly internal debates -- the goal is to share early ideas (probably wrong!) on themes we're exploring, and invite feedback. this week Alana Levin Daniel Barabander + me talk about non-USD denominated stablecoins

Jesse Walden

22,790 views • 9 months ago

We just released HaMeR, our latest work for Hand Mesh Recovery! We reconstruct hands in 3D from a single image with big improvements in accuracy and robustness. Code and models are available. Also colab and Hugging Face demo!
0:52

We just released HaMeR, our latest work for Hand Mesh Recovery! We reconstruct hands in 3D from a single image with big improvements in accuracy and robustness. Code and models are available. Also colab and Hugging Face demo!

Georgios Pavlakos

38,336 views • 2 years ago