Loading video...

Video Failed to Load

Go Home

Meta AI strikes again, with Relightable Gaussian Codec Avatars This is an update to the Meta Codec Avatars 2.0, building on 3D Gaussian Splatting. As a result, we get fully relightable real-time avatars, accurate at the hair strand level 🤯 More details below ⬇️⬇️

alexcarliera's profile picture

Alex Carlier

8,614 subscribers

445,646 views • 2 years ago •via X (Twitter)

EducationGamingScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

8 Comments

Alex Carlier's profile picture
Alex Carlier2 years ago

Another video showing Relightable Gaussian Codec Avatars in more details

Alex Carlier's profile picture
Alex Carlier2 years ago

Follow @alexcarliera for more content about AI & AR! Project page:

Samuel Ekpe's profile picture
Samuel Ekpe2 years ago

Imagine this integrated into Zoom. No need for video calls again!

Alex Carlier's profile picture
Alex Carlier2 years ago

Haha everyone’s dream here it seems 😅

Ghost Pepper's profile picture
Ghost Pepper2 years ago

This is becoming a reality

Alex Carlier's profile picture
Alex Carlier2 years ago

Exactly!

Jerem Dev's profile picture
Jerem Dev2 years ago

I'm fascinated by the advancements in 3D Gaussian Splatting. Mind-blowing!

Alex Carlier's profile picture
Alex Carlier2 years ago

Yes it’s crazy how fast 3DGS have been adopted and improved 🤯

Related Videos

💡Check out our #SIGGRAPHASIA 2024 paper, URAvatar: Universal Relightable Gaussian Codec Avatars. Now high-quality relightable avatars can be created by 𝐞𝐯𝐞𝐫𝐲𝐛𝐨𝐝𝐲! We learn a universal relightble prior and show how to use it for quick adaptation from a phone scan.
0:32

💡Check out our #SIGGRAPHASIA 2024 paper, URAvatar: Universal Relightable Gaussian Codec Avatars. Now high-quality relightable avatars can be created by 𝐞𝐯𝐞𝐫𝐲𝐛𝐨𝐝𝐲! We learn a universal relightble prior and show how to use it for quick adaptation from a phone scan.

Shunsuke Saito

18,811 views • 1 year ago

📢BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading We propose a hybrid neural shading scheme for creating intrinsically decomposed 3DGS head avatars, that allow real-time relighting and animation. 🌍 📷
0:24

📢BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading We propose a hybrid neural shading scheme for creating intrinsically decomposed 3DGS head avatars, that allow real-time relighting and animation. 🌍 📷

Matthias Niessner

23,001 views • 1 year ago

URAvatar: Universal Relightable Gaussian Codec Avatars Contributions (cited): (1) We introduce a universal relightable avatar prior model learned from hundreds of dynamic performance captures with a multi-view and multi-light system. (2) We build a drivable head avatar from a phone scan that can be rendered and relit with global light transport in real-time. (3) A capture system and evaluation protocol to measure the accuracy of relighting under continuous illuminations.
0:32

URAvatar: Universal Relightable Gaussian Codec Avatars Contributions (cited): (1) We introduce a universal relightable avatar prior model learned from hundreds of dynamic performance captures with a multi-view and multi-light system. (2) We build a drivable head avatar from a phone scan that can be rendered and relit with global light transport in real-time. (3) A capture system and evaluation protocol to measure the accuracy of relighting under continuous illuminations.

MrNeRF

50,086 views • 1 year ago

Alibaba just dropped TaoAvatar on Hugging Face Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting
0:41

Alibaba just dropped TaoAvatar on Hugging Face Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting

AK

329,222 views • 1 year ago

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.
6:20

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.

AK

327,040 views • 2 years ago

📢 Check out 𝗥𝗲𝗹𝗶𝗴𝗵𝘁𝗮𝗯𝗹𝗲 𝗚𝗮𝘂𝘀𝘀𝗶𝗮𝗻 𝗖𝗼𝗱𝗲𝗰 𝗔𝘃𝗮𝘁𝗮𝗿𝘀! Our latest codec avatars using 3D Gaussians generalize to novel lighting (OLAT, envmap) with *all-frequency* reflection (see video for hair and eye reflection) in real-time!
0:43

📢 Check out 𝗥𝗲𝗹𝗶𝗴𝗵𝘁𝗮𝗯𝗹𝗲 𝗚𝗮𝘂𝘀𝘀𝗶𝗮𝗻 𝗖𝗼𝗱𝗲𝗰 𝗔𝘃𝗮𝘁𝗮𝗿𝘀! Our latest codec avatars using 3D Gaussians generalize to novel lighting (OLAT, envmap) with *all-frequency* reflection (see video for hair and eye reflection) in real-time!

Shunsuke Saito

45,900 views • 2 years ago

SqueezeMe: Efficient Gaussian Avatars for VR TL;DR: Three of these Gaussian Splatting avatars can be run at 72 frames per second. It runs locally on a Meta Quest 3 VR headset. Abstract (excerpt): While previous methods require a desktop GPU for real-time inference of a single avatar, we aim to squeeze multiple Gaussian avatars onto a portable virtual reality headset with real-time drivable inference. We begin by training a previous work, Animatable Gaussians, on a high-quality dataset captured with 512 cameras. The Gaussians are animated by controlling a base set of Gaussians with linear blend skinning (LBS) motion, and then further adjusting them with a neural network decoder to correct their appearance. When deploying the model on a Meta Quest 3 VR headset, we find two major computational bottlenecks: the decoder and the rendering. To accelerate the decoder, we train the Gaussians in UV-space instead of pixel-space and distill the decoder to a single neural network layer. Further, we discover that neighborhoods of Gaussians can share a single corrective from the decoder, providing an additional speedup. To accelerate the rendering, we develop a custom pipeline in Vulkan that runs on the mobile GPU. Putting it all together, we run 3 Gaussian avatars concurrently at 72 FPS on a VR headset.
0:38

SqueezeMe: Efficient Gaussian Avatars for VR TL;DR: Three of these Gaussian Splatting avatars can be run at 72 frames per second. It runs locally on a Meta Quest 3 VR headset. Abstract (excerpt): While previous methods require a desktop GPU for real-time inference of a single avatar, we aim to squeeze multiple Gaussian avatars onto a portable virtual reality headset with real-time drivable inference. We begin by training a previous work, Animatable Gaussians, on a high-quality dataset captured with 512 cameras. The Gaussians are animated by controlling a base set of Gaussians with linear blend skinning (LBS) motion, and then further adjusting them with a neural network decoder to correct their appearance. When deploying the model on a Meta Quest 3 VR headset, we find two major computational bottlenecks: the decoder and the rendering. To accelerate the decoder, we train the Gaussians in UV-space instead of pixel-space and distill the decoder to a single neural network layer. Further, we discover that neighborhoods of Gaussians can share a single corrective from the decoder, providing an additional speedup. To accelerate the rendering, we develop a custom pipeline in Vulkan that runs on the mobile GPU. Putting it all together, we run 3 Gaussian avatars concurrently at 72 FPS on a VR headset.

MrNeRF

27,104 views • 1 year ago

'3D Gaussian Splatting for Real-Time RF Rendering' rose to the 2nd challenge I presented. The magnifying glass isn't still crystal clear, but the result is compelling. Look at the small details/hairs on the succulent plant. Rendered in real-time. #gaussiansplatting #3D #AI
0:32

'3D Gaussian Splatting for Real-Time RF Rendering' rose to the 2nd challenge I presented. The magnifying glass isn't still crystal clear, but the result is compelling. Look at the small details/hairs on the succulent plant. Rendered in real-time. #gaussiansplatting #3D #AI

Hugues Bruyère

298,206 views • 2 years ago

Codec Avatars can run on Quest 3 - the hardware is there. What’s missing is seamless face scanning and built-in eye & face tracking. Quest 4 will likely have eye tracking, and it may support codec avatars, but I really hope the lightweight headset in 2026 will do it first.
0:38

Codec Avatars can run on Quest 3 - the hardware is there. What’s missing is seamless face scanning and built-in eye & face tracking. Quest 4 will likely have eye tracking, and it may support codec avatars, but I really hope the lightweight headset in 2026 will do it first.

Mixed Reality Guy

10,490 views • 9 months ago

Relightable Full-Body Gaussian Codec Avatars TL;DR: First drivable full-body avatar model that reconstructs perceptually realistic relightable appearance. Contributions: • We propose the first relightable full-body avatar model that jointly models the relightable appearance of the human body, face, and hands for high-fidelity relighting and animation. • To handle full-body articulations with global light transport, we propose learnable zonal harmonics to represent local diffuse radiance transfer in the local coordinate frames of each Gaussian. This results in a reduced number of parameters and improved rendering quality compared to the commonly used spherical harmonics representation. • We reformulate the learnable radiance transfer to explicitly decompose non-local shadowing and propose a dedicated shadow network to predict shadows caused by the articulation of the body. Additionally, we propose a physically based irradiance normalization scheme to ensure that the shadow network can generalize to novel illumination conditions, such as unseen environment maps. • We show that deferred shading can be used for our learned specular radiance transfer function, achieving high-fidelity specular reflections for relightable human avatar modeling without excessively increasing the number of Gaussians.
0:45

Relightable Full-Body Gaussian Codec Avatars TL;DR: First drivable full-body avatar model that reconstructs perceptually realistic relightable appearance. Contributions: • We propose the first relightable full-body avatar model that jointly models the relightable appearance of the human body, face, and hands for high-fidelity relighting and animation. • To handle full-body articulations with global light transport, we propose learnable zonal harmonics to represent local diffuse radiance transfer in the local coordinate frames of each Gaussian. This results in a reduced number of parameters and improved rendering quality compared to the commonly used spherical harmonics representation. • We reformulate the learnable radiance transfer to explicitly decompose non-local shadowing and propose a dedicated shadow network to predict shadows caused by the articulation of the body. Additionally, we propose a physically based irradiance normalization scheme to ensure that the shadow network can generalize to novel illumination conditions, such as unseen environment maps. • We show that deferred shading can be used for our learned specular radiance transfer function, achieving high-fidelity specular reflections for relightable human avatar modeling without excessively increasing the number of Gaussians.

MrNeRF

10,966 views • 1 year ago

3d gaussian splatting is fucking cool. And now you can capture real world spaces just by walking around in your meta quest headset. The real-time feedback ensures you don’t miss a spot. Apple needs to get on this asap:
1:03

3d gaussian splatting is fucking cool. And now you can capture real world spaces just by walking around in your meta quest headset. The real-time feedback ensures you don’t miss a spot. Apple needs to get on this asap:

Bilawal Sidhu

162,888 views • 9 months ago

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.
5:36

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.

AK

140,960 views • 2 years ago

I made a stream overlay for Death Stranding where chat can call into the codec with text-to-speech. I remade their chat avatars in the style of the Death Stranding codec holograms, and we also gave them in-universe names.
0:22

I made a stream overlay for Death Stranding where chat can call into the codec with text-to-speech. I remade their chat avatars in the style of the Death Stranding codec holograms, and we also gave them in-universe names.

shindigs

29,537 views • 9 months ago

🚀New from Meta FAIR: today we’re introducing Seamless Interaction, a research project dedicated to modeling interpersonal dynamics. The project features a family of audiovisual behavioral models, developed in collaboration with Meta’s Codec Avatars lab + Core AI lab, that render speech between two individuals into diverse, expressive full-body gestures and active listening behaviors, allowing the creation of fully embodied avatars in 2D and 3D. These models have potential to create more natural, interactive virtual agents that can engage in human-like social interactions across a variety of settings. Learn more:
0:24

🚀New from Meta FAIR: today we’re introducing Seamless Interaction, a research project dedicated to modeling interpersonal dynamics. The project features a family of audiovisual behavioral models, developed in collaboration with Meta’s Codec Avatars lab + Core AI lab, that render speech between two individuals into diverse, expressive full-body gestures and active listening behaviors, allowing the creation of fully embodied avatars in 2D and 3D. These models have potential to create more natural, interactive virtual agents that can engage in human-like social interactions across a variety of settings. Learn more:

AI at Meta

48,358 views • 1 year ago

⚙️ 404—GEN has released the WORLD'S LARGEST open-source 3D model dataset ⚙️ • 21.5M+ AI-generated 3d models • Open-source and attribution-ready • Larger than all existing 3D datasets combined This is an unmatched scale for Gaussian Splatting research. More details below 🧵 ⬇️
0:37

⚙️ 404—GEN has released the WORLD'S LARGEST open-source 3D model dataset ⚙️ • 21.5M+ AI-generated 3d models • Open-source and attribution-ready • Larger than all existing 3D datasets combined This is an unmatched scale for Gaussian Splatting research. More details below 🧵 ⬇️

404

44,735 views • 1 year ago

Instant Skinned Gaussian Avatars for Web, Mobile and VR Applications Short summary: In our system, we animate a background 3D mesh and have the Gaussian splats follow the mesh’s vertices. During preprocessing, splats are assigned to mesh vertices, and their relative transformations are stored. Once this data is saved, you can instantly use it in your applications without further preprocessing. At runtime, we animate the background 3D mesh, update the Gaussian splats in parallel, and resort all Gaussian splats every frame based on the viewer’s perspective.
1:10

Instant Skinned Gaussian Avatars for Web, Mobile and VR Applications Short summary: In our system, we animate a background 3D mesh and have the Gaussian splats follow the mesh’s vertices. During preprocessing, splats are assigned to mesh vertices, and their relative transformations are stored. Once this data is saved, you can instantly use it in your applications without further preprocessing. At runtime, we animate the background 3D mesh, update the Gaussian splats in parallel, and resort all Gaussian splats every frame based on the viewer’s perspective.

MrNeRF

66,016 views • 8 months ago

We’re so excited to introduce our new AI-Generated Community Avatars! 👧 Users now have access to over 500 unique AI-generated avatars, but here’s the twist—anyone can now create and publish their own avatars for the entire community to enjoy. Try it out below ⬇️
0:44

We’re so excited to introduce our new AI-Generated Community Avatars! 👧 Users now have access to over 500 unique AI-generated avatars, but here’s the twist—anyone can now create and publish their own avatars for the entire community to enjoy. Try it out below ⬇️

HeyGen

13,932 views • 1 year ago

Happy to announce the results of our latest research, which takes 3D Gaussian Splatting to the next level: "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets," which has been accepted at #SIGGRAPH2024!🎉 Find it here:
1:40

Happy to announce the results of our latest research, which takes 3D Gaussian Splatting to the next level: "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets," which has been accepted at #SIGGRAPH2024!🎉 Find it here:

Bernhard Kerbl

356,244 views • 2 years ago

Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video Contribution quote from the paper: In summary, our main contributions are • a comprehensive pipeline for reconstructing the shape, appearance, and behavior of real-world garments using Gaussian splatting, • an algorithm for registering garment meshes to multi- view videos with an optimization procedure based on Gaussian splatting, and • a Gaussian Garment representation that combines triangle meshes with Gaussian textures to capture photorealistic appearance and can be used as a fully controllable 3D asset.
0:20

Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video Contribution quote from the paper: In summary, our main contributions are • a comprehensive pipeline for reconstructing the shape, appearance, and behavior of real-world garments using Gaussian splatting, • an algorithm for registering garment meshes to multi- view videos with an optimization procedure based on Gaussian splatting, and • a Gaussian Garment representation that combines triangle meshes with Gaussian textures to capture photorealistic appearance and can be used as a fully controllable 3D asset.

MrNeRF

27,277 views • 1 year ago