Loading video...

Video Failed to Load

Go Home

Relightable Real-Time Avatars Meta Codec Avatars 2.0 gets an update, building on 3D Gaussian Splatting from Meta. Accuracy is down to the human hair strand level 🔬 🧵 A thread

LinusEkenstam's profile picture

Linus ✦ Ekenstam

239,150 subscribers

351,872 views • 2 years ago •via X (Twitter)

ArtsScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

10 Comments

Linus Ekenstam – eu/acc's profile picture
Linus Ekenstam – eu/acc2 years ago

Lighting It's impressive how this is done in real-time, and shows just how powerful Gaussian splats are.

Linus Ekenstam – eu/acc's profile picture
Linus Ekenstam – eu/acc2 years ago

View

Linus Ekenstam – eu/acc's profile picture
Linus Ekenstam – eu/acc2 years ago

Gaze

Linus Ekenstam – eu/acc's profile picture
Linus Ekenstam – eu/acc2 years ago

Expression

Linus Ekenstam – eu/acc's profile picture
Linus Ekenstam – eu/acc2 years ago

More examples of real-time avatars.

Linus Ekenstam – eu/acc's profile picture
Linus Ekenstam – eu/acc2 years ago

Here is the full video

Linus Ekenstam – eu/acc's profile picture
Linus Ekenstam – eu/acc2 years ago

Read the entire thing, and check out the project.

Linus Ekenstam – eu/acc's profile picture
Linus Ekenstam – eu/acc2 years ago

If you found this interesting and want more AI content directly in your inbox, sign up for my weekly free newsletter below.

Linus Ekenstam – eu/acc's profile picture
Linus Ekenstam – eu/acc2 years ago

If you enjoyed this content, please consider. 1. Follow me at @LinusEkenstam 2. Re-post/like or share this post with a friend

Adam ᯅ's profile picture
Adam ᯅ2 years ago

Amazing gives me Star Wars vibes

Related Videos

💡Check out our #SIGGRAPHASIA 2024 paper, URAvatar: Universal Relightable Gaussian Codec Avatars. Now high-quality relightable avatars can be created by 𝐞𝐯𝐞𝐫𝐲𝐛𝐨𝐝𝐲! We learn a universal relightble prior and show how to use it for quick adaptation from a phone scan.
0:32

💡Check out our #SIGGRAPHASIA 2024 paper, URAvatar: Universal Relightable Gaussian Codec Avatars. Now high-quality relightable avatars can be created by 𝐞𝐯𝐞𝐫𝐲𝐛𝐨𝐝𝐲! We learn a universal relightble prior and show how to use it for quick adaptation from a phone scan.

Shunsuke Saito

18,811 views • 1 year ago

Alibaba just dropped TaoAvatar on Hugging Face Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting
0:41

Alibaba just dropped TaoAvatar on Hugging Face Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting

AK

329,222 views • 1 year ago

📢BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading We propose a hybrid neural shading scheme for creating intrinsically decomposed 3DGS head avatars, that allow real-time relighting and animation. 🌍 📷
0:24

📢BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading We propose a hybrid neural shading scheme for creating intrinsically decomposed 3DGS head avatars, that allow real-time relighting and animation. 🌍 📷

Matthias Niessner

23,001 views • 1 year ago

URAvatar: Universal Relightable Gaussian Codec Avatars Contributions (cited): (1) We introduce a universal relightable avatar prior model learned from hundreds of dynamic performance captures with a multi-view and multi-light system. (2) We build a drivable head avatar from a phone scan that can be rendered and relit with global light transport in real-time. (3) A capture system and evaluation protocol to measure the accuracy of relighting under continuous illuminations.
0:32

URAvatar: Universal Relightable Gaussian Codec Avatars Contributions (cited): (1) We introduce a universal relightable avatar prior model learned from hundreds of dynamic performance captures with a multi-view and multi-light system. (2) We build a drivable head avatar from a phone scan that can be rendered and relit with global light transport in real-time. (3) A capture system and evaluation protocol to measure the accuracy of relighting under continuous illuminations.

MrNeRF

50,052 views • 1 year ago

3d gaussian splatting is fucking cool. And now you can capture real world spaces just by walking around in your meta quest headset. The real-time feedback ensures you don’t miss a spot. Apple needs to get on this asap:
1:03

3d gaussian splatting is fucking cool. And now you can capture real world spaces just by walking around in your meta quest headset. The real-time feedback ensures you don’t miss a spot. Apple needs to get on this asap:

Bilawal Sidhu

162,888 views • 8 months ago

📢 Check out 𝗥𝗲𝗹𝗶𝗴𝗵𝘁𝗮𝗯𝗹𝗲 𝗚𝗮𝘂𝘀𝘀𝗶𝗮𝗻 𝗖𝗼𝗱𝗲𝗰 𝗔𝘃𝗮𝘁𝗮𝗿𝘀! Our latest codec avatars using 3D Gaussians generalize to novel lighting (OLAT, envmap) with *all-frequency* reflection (see video for hair and eye reflection) in real-time!
0:43

📢 Check out 𝗥𝗲𝗹𝗶𝗴𝗵𝘁𝗮𝗯𝗹𝗲 𝗚𝗮𝘂𝘀𝘀𝗶𝗮𝗻 𝗖𝗼𝗱𝗲𝗰 𝗔𝘃𝗮𝘁𝗮𝗿𝘀! Our latest codec avatars using 3D Gaussians generalize to novel lighting (OLAT, envmap) with *all-frequency* reflection (see video for hair and eye reflection) in real-time!

Shunsuke Saito

45,900 views • 2 years ago

📢Happy to present Convex Splatting, a novel way for 3D reconstruction based on 3D smooth convexes. For the first time, a splatting-based method reaches the quality of NeRF sota methods but with real-time rendering and few primitives!! I expect this to replace Gaussian splatting for 3D in the coming months. CODE RELEASED TODAY! joint work with collaborators from Université de Liège Visual Geometry Group (VGG) , KAUST Computer Vision Lab (IVUL) a thread 🧵 1/n
0:38

📢Happy to present Convex Splatting, a novel way for 3D reconstruction based on 3D smooth convexes. For the first time, a splatting-based method reaches the quality of NeRF sota methods but with real-time rendering and few primitives!! I expect this to replace Gaussian splatting for 3D in the coming months. CODE RELEASED TODAY! joint work with collaborators from Université de Liège Visual Geometry Group (VGG) , KAUST Computer Vision Lab (IVUL) a thread 🧵 1/n

Abdullah Hamdi

71,661 views • 1 year ago

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.
6:20

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.

AK

327,040 views • 2 years ago

SqueezeMe: Efficient Gaussian Avatars for VR TL;DR: Three of these Gaussian Splatting avatars can be run at 72 frames per second. It runs locally on a Meta Quest 3 VR headset. Abstract (excerpt): While previous methods require a desktop GPU for real-time inference of a single avatar, we aim to squeeze multiple Gaussian avatars onto a portable virtual reality headset with real-time drivable inference. We begin by training a previous work, Animatable Gaussians, on a high-quality dataset captured with 512 cameras. The Gaussians are animated by controlling a base set of Gaussians with linear blend skinning (LBS) motion, and then further adjusting them with a neural network decoder to correct their appearance. When deploying the model on a Meta Quest 3 VR headset, we find two major computational bottlenecks: the decoder and the rendering. To accelerate the decoder, we train the Gaussians in UV-space instead of pixel-space and distill the decoder to a single neural network layer. Further, we discover that neighborhoods of Gaussians can share a single corrective from the decoder, providing an additional speedup. To accelerate the rendering, we develop a custom pipeline in Vulkan that runs on the mobile GPU. Putting it all together, we run 3 Gaussian avatars concurrently at 72 FPS on a VR headset.
0:38

SqueezeMe: Efficient Gaussian Avatars for VR TL;DR: Three of these Gaussian Splatting avatars can be run at 72 frames per second. It runs locally on a Meta Quest 3 VR headset. Abstract (excerpt): While previous methods require a desktop GPU for real-time inference of a single avatar, we aim to squeeze multiple Gaussian avatars onto a portable virtual reality headset with real-time drivable inference. We begin by training a previous work, Animatable Gaussians, on a high-quality dataset captured with 512 cameras. The Gaussians are animated by controlling a base set of Gaussians with linear blend skinning (LBS) motion, and then further adjusting them with a neural network decoder to correct their appearance. When deploying the model on a Meta Quest 3 VR headset, we find two major computational bottlenecks: the decoder and the rendering. To accelerate the decoder, we train the Gaussians in UV-space instead of pixel-space and distill the decoder to a single neural network layer. Further, we discover that neighborhoods of Gaussians can share a single corrective from the decoder, providing an additional speedup. To accelerate the rendering, we develop a custom pipeline in Vulkan that runs on the mobile GPU. Putting it all together, we run 3 Gaussian avatars concurrently at 72 FPS on a VR headset.

MrNeRF

27,104 views • 1 year ago

1 | Meta Drive is a web3 mobile app that analyzes your real-world rides and rewards you for safe driving🚗💰 The app features Meta Drive NFT cars and Rolling Bones NFT avatars that add an exciting gaming element to the experience🕹️
0:31

1 | Meta Drive is a web3 mobile app that analyzes your real-world rides and rewards you for safe driving🚗💰 The app features Meta Drive NFT cars and Rolling Bones NFT avatars that add an exciting gaming element to the experience🕹️

Meta Drive & Rolling Bones

28,089 views • 3 years ago

Happy to announce the results of our latest research, which takes 3D Gaussian Splatting to the next level: "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets," which has been accepted at #SIGGRAPH2024!🎉 Find it here:
1:40

Happy to announce the results of our latest research, which takes 3D Gaussian Splatting to the next level: "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets," which has been accepted at #SIGGRAPH2024!🎉 Find it here:

Bernhard Kerbl

356,236 views • 2 years ago

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.
5:36

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.

AK

140,960 views • 2 years ago

🚀New from Meta FAIR: today we’re introducing Seamless Interaction, a research project dedicated to modeling interpersonal dynamics. The project features a family of audiovisual behavioral models, developed in collaboration with Meta’s Codec Avatars lab + Core AI lab, that render speech between two individuals into diverse, expressive full-body gestures and active listening behaviors, allowing the creation of fully embodied avatars in 2D and 3D. These models have potential to create more natural, interactive virtual agents that can engage in human-like social interactions across a variety of settings. Learn more:
0:24

🚀New from Meta FAIR: today we’re introducing Seamless Interaction, a research project dedicated to modeling interpersonal dynamics. The project features a family of audiovisual behavioral models, developed in collaboration with Meta’s Codec Avatars lab + Core AI lab, that render speech between two individuals into diverse, expressive full-body gestures and active listening behaviors, allowing the creation of fully embodied avatars in 2D and 3D. These models have potential to create more natural, interactive virtual agents that can engage in human-like social interactions across a variety of settings. Learn more:

AI at Meta

48,358 views • 11 months ago

3D gaussian splatting + time (video) = 4D gaussian splatting is now possible This means in the near future you'll be able to fly around on your phone in any video you recorded A lot like the "braindance" in Cyberpunk 2077
0:29

3D gaussian splatting + time (video) = 4D gaussian splatting is now possible This means in the near future you'll be able to fly around on your phone in any video you recorded A lot like the "braindance" in Cyberpunk 2077

@levelsio

589,960 views • 1 year ago

3D Gaussian Splatting is great, but can it work without the pre-computed camera poses? Introducing: COLMAP-Free 3D Gaussian Splatting Our recent work shows not only it can, but 3D Gaussians make camera pose estimation easy (compared to NeRF) along with reconstruction. 👇🧵
0:50

3D Gaussian Splatting is great, but can it work without the pre-computed camera poses? Introducing: COLMAP-Free 3D Gaussian Splatting Our recent work shows not only it can, but 3D Gaussians make camera pose estimation easy (compared to NeRF) along with reconstruction. 👇🧵

Xiaolong Wang

76,747 views • 2 years ago

'3D Gaussian Splatting for Real-Time RF Rendering' rose to the 2nd challenge I presented. The magnifying glass isn't still crystal clear, but the result is compelling. Look at the small details/hairs on the succulent plant. Rendered in real-time. #gaussiansplatting #3D #AI
0:32

'3D Gaussian Splatting for Real-Time RF Rendering' rose to the 2nd challenge I presented. The magnifying glass isn't still crystal clear, but the result is compelling. Look at the small details/hairs on the succulent plant. Rendered in real-time. #gaussiansplatting #3D #AI

Hugues Bruyère

298,206 views • 2 years ago

Codec Avatars can run on Quest 3 - the hardware is there. What’s missing is seamless face scanning and built-in eye & face tracking. Quest 4 will likely have eye tracking, and it may support codec avatars, but I really hope the lightweight headset in 2026 will do it first.
0:38

Codec Avatars can run on Quest 3 - the hardware is there. What’s missing is seamless face scanning and built-in eye & face tracking. Quest 4 will likely have eye tracking, and it may support codec avatars, but I really hope the lightweight headset in 2026 will do it first.

Mixed Reality Guy

10,490 views • 9 months ago

PERSE Personalized 3D Generative Avatars from A Single Portrait
0:45

PERSE Personalized 3D Generative Avatars from A Single Portrait

AK

45,885 views • 1 year ago

2d-to-3d video Gaussian splatting, fully local on an M4 Max. using Rerun for visualization.
0:21

2d-to-3d video Gaussian splatting, fully local on an M4 Max. using Rerun for visualization.

Jaskirat Singh

103,996 views • 5 months ago