Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Personalized 3D Generative Avatars from a Single Portrait Contributions: 1. Generate a personalized 3D avatar from a reference portrait image with controllable facial attributes. 2. Create high-quality synthetic 2D video datasets with diverse attribute editing from a reference portrait image. 3. Use latent space regularization with face morphing supervision... show more

MrNeRF

14,597 subscribers

20,507 views • 1 year ago •via X (Twitter)

Science & Technology Arts

Anya Rossi• Live Now

Private livecam show

3 Comments

MrNeRF1 year ago

Paper: Project:

Erick Ram1 year ago

Finally I'll know what I would look like with silver hair and a decent silver beard.

MrNeRF1 year ago

Haha, true :)

Related Videos

Want to create an avatar from a single image? FlexAvatar is a transformer model that creates full 360°, high-quality, and expressive 3D head avatar from just a single portrait image in minutes. Real-time Demo: FlexAvatar's lightweight architecture allows both animation and rendering in real-time, enabling interactive user experiences. To create a new 3D head avatar, only one image is required, e.g., from a webcam. The final avatar is ready after 2 minutes. Architecture: Under the hood, FlexAvatar adopts a transformer-based encoder-decoder design. The encoder maps the input image onto a latent avatar space, while the decoder produces 3D Gaussian attribute maps by incorporating the animation signal via cross-attention. The model learns all facial animations directly from the data without relying on pre-built 3D face models. This equips the avatars with realistic facial expressions. The internal avatar latent space can be conveniently used to integrate additional observations of a person via fitting. This enables use-cases where more than one image of a person is available, e.g., from a phone scan of the person. We train jointly on 2D monocular videos and multi-view data. However, in monocular videos, the animation signal leaks the target viewpoint, causing the model to produce incomplete 3D heads. We call this phenomenon entanglement of driving signal and target viewpoint. To prevent entanglement, we introduce bias sinks. These are learnable tokens that indicate whether a training sample stems from a monocular or a multi-view dataset. During training, the model learns to produce incomplete 3D heads only when the monocular token is present. During inference, FlexAvatar then always uses the multi-view token for which the model has learned to produce complete 3D heads. This simple design allows to combine the generalizability from monocular data with the quality of multi-view data. FlexAvatar summary: - Input: Single-image, phone scan, or monocular video - Output: Full 360° head avatar - Expressive animations - Real-time rendering and animation - Generalization to any portrait - Create a new avatar in 2 minutes - Use bias sinks to combine 2D and 3D data 🏠 🌍 🎥 Great work by Tobias Kirschstein and Simon Giebenhain!

Matthias Niessner

76,454 views • 5 months ago

SOAP can generate rigged 3D avatars from a single portrait image

SOAP can generate rigged 3D avatars from a single portrait image

Dreaming Tulpa 🥓👑

24,248 views • 1 year ago

Phidias A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion discuss: In 3D modeling, designers often use an existing 3D model as a reference to create new ones. This practice has inspired the development of Phidias, a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generalization ability, and controllability. Our model integrates three key components: 1) meta-ControlNet that dynamically modulates the conditioning strength, 2) dynamic reference routing that mitigates misalignment between the input image and 3D reference, and 3) self-reference augmentations that enable self-supervised training with a progressive curriculum. Collectively, these designs result in a clear improvement over existing methods. Phidias establishes a unified framework for 3D generation using text, image, and 3D conditions with versatile applications.

Phidias A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion discuss: In 3D modeling, designers often use an existing 3D model as a reference to create new ones. This practice has inspired the development of Phidias, a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generalization ability, and controllability. Our model integrates three key components: 1) meta-ControlNet that dynamically modulates the conditioning strength, 2) dynamic reference routing that mitigates misalignment between the input image and 3D reference, and 3) self-reference augmentations that enable self-supervised training with a progressive curriculum. Collectively, these designs result in a clear improvement over existing methods. Phidias establishes a unified framework for 3D generation using text, image, and 3D conditions with versatile applications.

AK

25,120 views • 1 year ago

📢 A Recipe for Generating 3D Worlds From a Single Image 📢 Our recipe explains how existing generative models can be adapted with minimal training effort to generate 3D worlds from a single input image.

📢 A Recipe for Generating 3D Worlds From a Single Image 📢 Our recipe explains how existing generative models can be adapted with minimal training effort to generate 3D worlds from a single input image.

Katja Schwarz

13,970 views • 1 year ago

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors paper page: present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images.

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors paper page: present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images.

AK

305,643 views • 2 years ago

Alibaba released LHM! a new model that can generate high-quality, animatable 3D human avatars from a single image in just a few seconds

Alibaba released LHM! a new model that can generate high-quality, animatable 3D human avatars from a single image in just a few seconds

Dreaming Tulpa 🥓👑

21,853 views • 1 year ago

Download UE5 AVATAR METAVANCE for Free! Create your personalized AVATAR with real-time CG movie-quality rendering, all powered by UE5. Modify 3D models, outfits, and appearances instantly—no need for Maya modeling or complex rigging. Future updates will support 3D model import/export and image-to-3D face generation, making 3D model creation accessible to everyone—no professional skills required! 🚀

CYANPUPPETS

28,742 views • 1 year ago

Introducing 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧 𝐀𝐧𝐲𝐭𝐡𝐢𝐧𝐠, a new 3D generative model with two key properties: - A structured point-cloud latent space enabling flexible editing! - Support multi-modal conditions, e.g., point cloud, text, single/multi-view images arXiv:

Introducing 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧 𝐀𝐧𝐲𝐭𝐡𝐢𝐧𝐠, a new 3D generative model with two key properties: - A structured point-cloud latent space enabling flexible editing! - Support multi-modal conditions, e.g., point cloud, text, single/multi-view images arXiv:

Xingang Pan

55,847 views • 1 year ago

🚀Announcing the release of next-gen avatar platform, It’s now a matter of a few seconds to get your realistic, customizable 3D avatar from a 2d photo using Avaturn’s new generative AI. Developer? Integrate into your game or app, in 15 min. for free.

Avaturn

17,783 views • 3 years ago

📢📢 𝐀𝐯𝐚𝐭𝟑𝐫 📢📢 Avat3r creates high-quality 3D head avatars from just a few input images in a single forward pass with a new dynamic 3DGS reconstruction model. Video: Project: Our core idea is to make Gaussian Reconstruction Models animatable. We find that a simple cross-attention to an expression code sequence is already sufficient to model complex facial expressions. We then incorporate position maps from DUSt3R and feature maps from Sapiens to facilitate the prediction task. While DUSt3R's position maps act as a pixel-aligned initialization for the Gaussians' positions, the Sapiens feature maps help the cross-view transformer to match corresponding image tokens in the 4 input images. One major challenge in creating a 3D head avatar from smartphone images comes from inconsistent facial expressions when the subject could not remain perfectly static during the capture. We eliminate this static requirement by simply showing our model input images with different facial expressions during training. This technique makes our model robust to inconsistent input images later on. Finally, we show that despite the model has been trained with 4 input images, one can even create a 3D head avatar when only a single image is available. To achieve this, we employ a pre-trained 3D GAN to lift the single image to 3D and then render the 4 input images for our model. This allows us to create 3D head avatars from single images and even highly out-of-distribution examples like AI generated faces, paintings or statues. Great work by Tobias Kirschstein from his internship at Meta with Javier Romero, Artem Sevastopolsky, and Shunsuke Saito

Matthias Niessner

74,687 views • 1 year ago

Adobe is entering the image-to-3D game! Their new method, LRM, can create high-fidelity 3D meshes from a single image in just 5 seconds 🔥 It's trained on 1 million objects and is able to generate objects from real-world images and generative AI models.

Adobe is entering the image-to-3D game! Their new method, LRM, can create high-fidelity 3D meshes from a single image in just 5 seconds 🔥 It's trained on 1 million objects and is able to generate objects from real-world images and generative AI models.

Dreaming Tulpa 🥓👑

270,918 views • 2 years ago

Current 3D generative models are slow and low quality. We present GRM, a large-scale model that reconstructs 3D Gaussians in 0.1s and generates high-quality 3D assets from text or single images in a few seconds. Demo: 1/4

Current 3D generative models are slow and low quality. We present GRM, a large-scale model that reconstructs 3D Gaussians in 0.1s and generates high-quality 3D assets from text or single images in a few seconds. Demo: 1/4

Gordon Wetzstein

19,189 views • 2 years ago

Champ Controllable and Consistent Human Image Animation with 3D Parametric Guidance In this study, we introduce a methodology for human image animation by leveraging a 3D human parametric model within a latent diffusion framework to enhance shape alignment and motion

Champ Controllable and Consistent Human Image Animation with 3D Parametric Guidance In this study, we introduce a methodology for human image animation by leveraging a 3D human parametric model within a latent diffusion framework to enhance shape alignment and motion

AK

194,356 views • 2 years ago

Announcing Genmo Video, a generative media platform with a new text-to-video model that can generate immersive live artwork from any prompt or any image. What will you create? 🎨▶️ Free public access: Discord: 👇1/n

Announcing Genmo Video, a generative media platform with a new text-to-video model that can generate immersive live artwork from any prompt or any image. What will you create? 🎨▶️ Free public access: Discord: 👇1/n

Genmo

94,683 views • 3 years ago

DiffSplat Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation DiffSplat is a generative framework to synthesize 3D Gaussian Splats from text prompts & single-view images in ⚡️ 1~2 seconds. It is fine-tuned directly from a pretrained text-to-image diffusion model.

DiffSplat Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation DiffSplat is a generative framework to synthesize 3D Gaussian Splats from text prompts & single-view images in ⚡️ 1~2 seconds. It is fine-tuned directly from a pretrained text-to-image diffusion model.

AK

38,416 views • 1 year ago

High quality AI generated talking heads are coming! GAIA can generate talking avatars from a single portrait image and speech clip. It even supports text prompts like `sad`, `open mouth` or `surprise` to guide video generation. Crazy times ahead 🤯

High quality AI generated talking heads are coming! GAIA can generate talking avatars from a single portrait image and speech clip. It even supports text prompts like `sad`, `open mouth` or `surprise` to guide video generation. Crazy times ahead 🤯

Dreaming Tulpa 🥓👑

659,956 views • 2 years ago

#SIGGRAPH2024 🏆Best Paper Honorable Mentioned Introducing the magic behind #Rodin ( CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets (1/n) CLAY is 3D #stablediffusion + #ControlNet + #LoRA.

#SIGGRAPH2024 🏆Best Paper Honorable Mentioned Introducing the magic behind #Rodin ( CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets (1/n) CLAY is 3D #stablediffusion + #ControlNet + #LoRA.

Deemos

30,855 views • 1 year ago

Using 3D assets with Gen-4 References is a simple way to bring highly detailed and specific models into your generative workflows for even more consistency and control. To do so, simply provide a location plate, a quick comp of your 3D model in that space and a style reference.

Using 3D assets with Gen-4 References is a simple way to bring highly detailed and specific models into your generative workflows for even more consistency and control. To do so, simply provide a location plate, a quick comp of your 3D model in that space and a style reference.

Runway

119,037 views • 1 year ago