Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Relightable Real-Time Avatars Meta Codec Avatars 2.0 gets an update, building on 3D Gaussian Splatting from Meta. Accuracy is down to the human hair strand level 🔬 🧵 A thread

LinusEkenstam's profile picture

Linus ✦ Ekenstam

239,768 subscribers

351,889 Aufrufe • vor 2 Jahren •via X (Twitter)

KunstWissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

10 Kommentare

Profilbild von Linus Ekenstam – eu/acc
Linus Ekenstam – eu/accvor 2 Jahren

Lighting It's impressive how this is done in real-time, and shows just how powerful Gaussian splats are.

Profilbild von Linus Ekenstam – eu/acc
Linus Ekenstam – eu/accvor 2 Jahren

View

Profilbild von Linus Ekenstam – eu/acc
Linus Ekenstam – eu/accvor 2 Jahren

Gaze

Profilbild von Linus Ekenstam – eu/acc
Linus Ekenstam – eu/accvor 2 Jahren

Expression

Profilbild von Linus Ekenstam – eu/acc
Linus Ekenstam – eu/accvor 2 Jahren

More examples of real-time avatars.

Profilbild von Linus Ekenstam – eu/acc
Linus Ekenstam – eu/accvor 2 Jahren

Here is the full video

Profilbild von Linus Ekenstam – eu/acc
Linus Ekenstam – eu/accvor 2 Jahren

Read the entire thing, and check out the project.

Profilbild von Linus Ekenstam – eu/acc
Linus Ekenstam – eu/accvor 2 Jahren

If you found this interesting and want more AI content directly in your inbox, sign up for my weekly free newsletter below.

Profilbild von Linus Ekenstam – eu/acc
Linus Ekenstam – eu/accvor 2 Jahren

If you enjoyed this content, please consider. 1. Follow me at @LinusEkenstam 2. Re-post/like or share this post with a friend

Profilbild von Adam ᯅ
Adam ᯅvor 2 Jahren

Amazing gives me Star Wars vibes

Ähnliche Videos

💡Check out our #SIGGRAPHASIA 2024 paper, URAvatar: Universal Relightable Gaussian Codec Avatars. Now high-quality relightable avatars can be created by 𝐞𝐯𝐞𝐫𝐲𝐛𝐨𝐝𝐲! We learn a universal relightble prior and show how to use it for quick adaptation from a phone scan.
0:32

💡Check out our #SIGGRAPHASIA 2024 paper, URAvatar: Universal Relightable Gaussian Codec Avatars. Now high-quality relightable avatars can be created by 𝐞𝐯𝐞𝐫𝐲𝐛𝐨𝐝𝐲! We learn a universal relightble prior and show how to use it for quick adaptation from a phone scan.

Shunsuke Saito

18,823 Aufrufe • vor 1 Jahr

Alibaba just dropped TaoAvatar on Hugging Face Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting
0:41

Alibaba just dropped TaoAvatar on Hugging Face Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting

AK

329,222 Aufrufe • vor 1 Jahr

📢BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading We propose a hybrid neural shading scheme for creating intrinsically decomposed 3DGS head avatars, that allow real-time relighting and animation. 🌍 📷
0:24

📢BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading We propose a hybrid neural shading scheme for creating intrinsically decomposed 3DGS head avatars, that allow real-time relighting and animation. 🌍 📷

Matthias Niessner

23,001 Aufrufe • vor 1 Jahr

URAvatar: Universal Relightable Gaussian Codec Avatars Contributions (cited): (1) We introduce a universal relightable avatar prior model learned from hundreds of dynamic performance captures with a multi-view and multi-light system. (2) We build a drivable head avatar from a phone scan that can be rendered and relit with global light transport in real-time. (3) A capture system and evaluation protocol to measure the accuracy of relighting under continuous illuminations.
0:32

URAvatar: Universal Relightable Gaussian Codec Avatars Contributions (cited): (1) We introduce a universal relightable avatar prior model learned from hundreds of dynamic performance captures with a multi-view and multi-light system. (2) We build a drivable head avatar from a phone scan that can be rendered and relit with global light transport in real-time. (3) A capture system and evaluation protocol to measure the accuracy of relighting under continuous illuminations.

MrNeRF

50,088 Aufrufe • vor 1 Jahr

3d gaussian splatting is fucking cool. And now you can capture real world spaces just by walking around in your meta quest headset. The real-time feedback ensures you don’t miss a spot. Apple needs to get on this asap:
1:03

3d gaussian splatting is fucking cool. And now you can capture real world spaces just by walking around in your meta quest headset. The real-time feedback ensures you don’t miss a spot. Apple needs to get on this asap:

Bilawal Sidhu

162,888 Aufrufe • vor 9 Monaten

📢 Check out 𝗥𝗲𝗹𝗶𝗴𝗵𝘁𝗮𝗯𝗹𝗲 𝗚𝗮𝘂𝘀𝘀𝗶𝗮𝗻 𝗖𝗼𝗱𝗲𝗰 𝗔𝘃𝗮𝘁𝗮𝗿𝘀! Our latest codec avatars using 3D Gaussians generalize to novel lighting (OLAT, envmap) with *all-frequency* reflection (see video for hair and eye reflection) in real-time!
0:43

📢 Check out 𝗥𝗲𝗹𝗶𝗴𝗵𝘁𝗮𝗯𝗹𝗲 𝗚𝗮𝘂𝘀𝘀𝗶𝗮𝗻 𝗖𝗼𝗱𝗲𝗰 𝗔𝘃𝗮𝘁𝗮𝗿𝘀! Our latest codec avatars using 3D Gaussians generalize to novel lighting (OLAT, envmap) with *all-frequency* reflection (see video for hair and eye reflection) in real-time!

Shunsuke Saito

45,934 Aufrufe • vor 2 Jahren

📢Happy to present Convex Splatting, a novel way for 3D reconstruction based on 3D smooth convexes. For the first time, a splatting-based method reaches the quality of NeRF sota methods but with real-time rendering and few primitives!! I expect this to replace Gaussian splatting for 3D in the coming months. CODE RELEASED TODAY! joint work with collaborators from Université de Liège Visual Geometry Group (VGG) , KAUST Computer Vision Lab (IVUL) a thread 🧵 1/n
0:38

📢Happy to present Convex Splatting, a novel way for 3D reconstruction based on 3D smooth convexes. For the first time, a splatting-based method reaches the quality of NeRF sota methods but with real-time rendering and few primitives!! I expect this to replace Gaussian splatting for 3D in the coming months. CODE RELEASED TODAY! joint work with collaborators from Université de Liège Visual Geometry Group (VGG) , KAUST Computer Vision Lab (IVUL) a thread 🧵 1/n

Abdullah Hamdi

71,661 Aufrufe • vor 1 Jahr

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.
6:20

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.

AK

327,077 Aufrufe • vor 2 Jahren

I just saw this official Japanese ad for the Quest 3 The extremely jarring transitions from anime to Meta avatars with “anime colored hair” made me die of laughter
0:29

I just saw this official Japanese ad for the Quest 3 The extremely jarring transitions from anime to Meta avatars with “anime colored hair” made me die of laughter

Brad Lynch

126,850 Aufrufe • vor 2 Jahren

SqueezeMe: Efficient Gaussian Avatars for VR TL;DR: Three of these Gaussian Splatting avatars can be run at 72 frames per second. It runs locally on a Meta Quest 3 VR headset. Abstract (excerpt): While previous methods require a desktop GPU for real-time inference of a single avatar, we aim to squeeze multiple Gaussian avatars onto a portable virtual reality headset with real-time drivable inference. We begin by training a previous work, Animatable Gaussians, on a high-quality dataset captured with 512 cameras. The Gaussians are animated by controlling a base set of Gaussians with linear blend skinning (LBS) motion, and then further adjusting them with a neural network decoder to correct their appearance. When deploying the model on a Meta Quest 3 VR headset, we find two major computational bottlenecks: the decoder and the rendering. To accelerate the decoder, we train the Gaussians in UV-space instead of pixel-space and distill the decoder to a single neural network layer. Further, we discover that neighborhoods of Gaussians can share a single corrective from the decoder, providing an additional speedup. To accelerate the rendering, we develop a custom pipeline in Vulkan that runs on the mobile GPU. Putting it all together, we run 3 Gaussian avatars concurrently at 72 FPS on a VR headset.
0:38

SqueezeMe: Efficient Gaussian Avatars for VR TL;DR: Three of these Gaussian Splatting avatars can be run at 72 frames per second. It runs locally on a Meta Quest 3 VR headset. Abstract (excerpt): While previous methods require a desktop GPU for real-time inference of a single avatar, we aim to squeeze multiple Gaussian avatars onto a portable virtual reality headset with real-time drivable inference. We begin by training a previous work, Animatable Gaussians, on a high-quality dataset captured with 512 cameras. The Gaussians are animated by controlling a base set of Gaussians with linear blend skinning (LBS) motion, and then further adjusting them with a neural network decoder to correct their appearance. When deploying the model on a Meta Quest 3 VR headset, we find two major computational bottlenecks: the decoder and the rendering. To accelerate the decoder, we train the Gaussians in UV-space instead of pixel-space and distill the decoder to a single neural network layer. Further, we discover that neighborhoods of Gaussians can share a single corrective from the decoder, providing an additional speedup. To accelerate the rendering, we develop a custom pipeline in Vulkan that runs on the mobile GPU. Putting it all together, we run 3 Gaussian avatars concurrently at 72 FPS on a VR headset.

MrNeRF

27,104 Aufrufe • vor 1 Jahr

1 | Meta Drive is a web3 mobile app that analyzes your real-world rides and rewards you for safe driving🚗💰 The app features Meta Drive NFT cars and Rolling Bones NFT avatars that add an exciting gaming element to the experience🕹️
0:31

1 | Meta Drive is a web3 mobile app that analyzes your real-world rides and rewards you for safe driving🚗💰 The app features Meta Drive NFT cars and Rolling Bones NFT avatars that add an exciting gaming element to the experience🕹️

Meta Drive & Rolling Bones

28,089 Aufrufe • vor 3 Jahren

Happy to announce the results of our latest research, which takes 3D Gaussian Splatting to the next level: "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets," which has been accepted at #SIGGRAPH2024!🎉 Find it here:
1:40

Happy to announce the results of our latest research, which takes 3D Gaussian Splatting to the next level: "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets," which has been accepted at #SIGGRAPH2024!🎉 Find it here:

Bernhard Kerbl

356,272 Aufrufe • vor 2 Jahren

Avatars, meta AI translations, Lip synching and much more. Live Demo
0:58

Avatars, meta AI translations, Lip synching and much more. Live Demo

Linus ✦ Ekenstam

46,523 Aufrufe • vor 1 Jahr

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.
5:36

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.

AK

140,960 Aufrufe • vor 2 Jahren

🚀New from Meta FAIR: today we’re introducing Seamless Interaction, a research project dedicated to modeling interpersonal dynamics. The project features a family of audiovisual behavioral models, developed in collaboration with Meta’s Codec Avatars lab + Core AI lab, that render speech between two individuals into diverse, expressive full-body gestures and active listening behaviors, allowing the creation of fully embodied avatars in 2D and 3D. These models have potential to create more natural, interactive virtual agents that can engage in human-like social interactions across a variety of settings. Learn more:
0:24

🚀New from Meta FAIR: today we’re introducing Seamless Interaction, a research project dedicated to modeling interpersonal dynamics. The project features a family of audiovisual behavioral models, developed in collaboration with Meta’s Codec Avatars lab + Core AI lab, that render speech between two individuals into diverse, expressive full-body gestures and active listening behaviors, allowing the creation of fully embodied avatars in 2D and 3D. These models have potential to create more natural, interactive virtual agents that can engage in human-like social interactions across a variety of settings. Learn more:

AI at Meta

48,410 Aufrufe • vor 1 Jahr

3D gaussian splatting + time (video) = 4D gaussian splatting is now possible This means in the near future you'll be able to fly around on your phone in any video you recorded A lot like the "braindance" in Cyberpunk 2077
0:29

3D gaussian splatting + time (video) = 4D gaussian splatting is now possible This means in the near future you'll be able to fly around on your phone in any video you recorded A lot like the "braindance" in Cyberpunk 2077

@levelsio

590,061 Aufrufe • vor 1 Jahr

3D Gaussian Splatting is great, but can it work without the pre-computed camera poses? Introducing: COLMAP-Free 3D Gaussian Splatting Our recent work shows not only it can, but 3D Gaussians make camera pose estimation easy (compared to NeRF) along with reconstruction. 👇🧵
0:50

3D Gaussian Splatting is great, but can it work without the pre-computed camera poses? Introducing: COLMAP-Free 3D Gaussian Splatting Our recent work shows not only it can, but 3D Gaussians make camera pose estimation easy (compared to NeRF) along with reconstruction. 👇🧵

Xiaolong Wang

76,747 Aufrufe • vor 2 Jahren

'3D Gaussian Splatting for Real-Time RF Rendering' rose to the 2nd challenge I presented. The magnifying glass isn't still crystal clear, but the result is compelling. Look at the small details/hairs on the succulent plant. Rendered in real-time. #gaussiansplatting #3D #AI
0:32

'3D Gaussian Splatting for Real-Time RF Rendering' rose to the 2nd challenge I presented. The magnifying glass isn't still crystal clear, but the result is compelling. Look at the small details/hairs on the succulent plant. Rendered in real-time. #gaussiansplatting #3D #AI

Hugues Bruyère

298,206 Aufrufe • vor 2 Jahren

Codec Avatars can run on Quest 3 - the hardware is there. What’s missing is seamless face scanning and built-in eye & face tracking. Quest 4 will likely have eye tracking, and it may support codec avatars, but I really hope the lightweight headset in 2026 will do it first.
0:38

Codec Avatars can run on Quest 3 - the hardware is there. What’s missing is seamless face scanning and built-in eye & face tracking. Quest 4 will likely have eye tracking, and it may support codec avatars, but I really hope the lightweight headset in 2026 will do it first.

Mixed Reality Guy

10,490 Aufrufe • vor 9 Monaten