Loading video...

Video Failed to Load

Go Home

Instant Skinned Gaussian Avatars for Web, Mobile and VR Applications Short summary: In our system, we animate a background 3D mesh and have the Gaussian splats follow the mesh’s vertices. During preprocessing, splats are assigned to mesh vertices, and their relative transformations are stored. Once this data is saved,...

janusch_patas's profile picture

MrNeRF

16,591 subscribers

66,016 views • 10 months ago •via X (Twitter)

Science & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

🔵 3D Gaussian splats are now available in the Cesium and 3D Tiles ecosystems, including CesiumJS, Cesium for Unreal, and Cesium ion. Learn more and get started:
0:58

🔵 3D Gaussian splats are now available in the Cesium and 3D Tiles ecosystems, including CesiumJS, Cesium for Unreal, and Cesium ion. Learn more and get started:

Cesium

31,428 views • 3 months ago

3D Gaussian splats are a perfect fit for the travel industry. Preview your room before you book (including in VR). @PlayCanvas (and SuperSplat) makes it a breeze to create and publish this content. 🪄 Link 👇
0:40

3D Gaussian splats are a perfect fit for the travel industry. Preview your room before you book (including in VR). @PlayCanvas (and SuperSplat) makes it a breeze to create and publish this content. 🪄 Link 👇

Will Eastcott

10,184 views • 6 months ago

BIG NEWS: Introducing SDF-Splats! 🚨 I just solved one of the biggest problems in 3D Gaussian Splatting: GAPS & HOLES! 🕳️❌ My hybrid approach combines: ✨ Explicit Gaussian Splats (for detail) ✨ Implicit SDFs (for continuity) The result? Seamless 3D reconstruction with NO visible holes, NO transparency artifacts, and REAL-TIME performance at 30-60 FPS! 🔥 Traditional Gaussian Splatting looks amazing but breaks apart in under-sampled regions. SDF-Splats fills every gap using on-demand raymarching through signed distance fields—zero preprocessing, zero neural nets, pure GPU magic! ⚡ Read the full research paper: #GaussianSplatting #SDF Radiance Fields MrNeRF #RealTime3D #VR #3DReconstruction #AIResearch
0:20

BIG NEWS: Introducing SDF-Splats! 🚨 I just solved one of the biggest problems in 3D Gaussian Splatting: GAPS & HOLES! 🕳️❌ My hybrid approach combines: ✨ Explicit Gaussian Splats (for detail) ✨ Implicit SDFs (for continuity) The result? Seamless 3D reconstruction with NO visible holes, NO transparency artifacts, and REAL-TIME performance at 30-60 FPS! 🔥 Traditional Gaussian Splatting looks amazing but breaks apart in under-sampled regions. SDF-Splats fills every gap using on-demand raymarching through signed distance fields—zero preprocessing, zero neural nets, pure GPU magic! ⚡ Read the full research paper: #GaussianSplatting #SDF Radiance Fields MrNeRF #RealTime3D #VR #3DReconstruction #AIResearch

Daniel Skaale

21,382 views • 9 months ago

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.
6:20

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.

AK

327,236 views • 2 years ago

splat-transform is a free and open source library and CLI tool for processing 3D Gaussian splats. It can create lightweight, accurate, voxel-based collision at scale. No human intervention required. This gym scene's voxels are perfectly generated - even large fully transparent windows are correctly filled. The voxel data file is 118KB. Start building 3D Gaussian splat apps today with PlayCanvas and splat-transform. 🧑‍💻
0:43

splat-transform is a free and open source library and CLI tool for processing 3D Gaussian splats. It can create lightweight, accurate, voxel-based collision at scale. No human intervention required. This gym scene's voxels are perfectly generated - even large fully transparent windows are correctly filled. The voxel data file is 118KB. Start building 3D Gaussian splat apps today with PlayCanvas and splat-transform. 🧑‍💻

Will Eastcott

20,508 views • 3 months ago

[NeurIPS '24] DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation Abstract (excerpt) We introduce DreamMesh4D, a novel framework that combines mesh representation with sparse-controlled deformation technique to generate high-quality 4D object from a monocular video. To overcome the limitation of classical texture representation, we bind Gaussian splats to the surface of the triangular mesh for differentiable optimization of both the texture and mesh vertices. In particular, DreamMesh4D begins with a coarse mesh provided by a single image based 3D generation method. Sparse points are then uniformly sampled across the surface of the mesh, and are used to build a deformation graph to drive the motion of the 3D object for the sake of computational efficiency and providing additional constraint. For each step, transformations of sparse control points are predicted using a deformation network, and the mesh vertices as well as the bound surface Gaussians are deformed via a geometric skinning algorithm. The skinning algorithm is a hybrid approach combining LBS (linear blending skinning) and DQS (dual-quaternion skinning), mitigating drawbacks associated with both approaches. The static surface Gaussians and mesh vertices as well as the dynamic deformation network are learned via reference view photometric loss, score distillation loss as well as other regularization losses in a two-stage manner. Extensive experiments demonstrate that our method outperforms prior video-to-4D generation methods in terms of rendering quality and spatial-temporal consistency.
0:20

[NeurIPS '24] DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation Abstract (excerpt) We introduce DreamMesh4D, a novel framework that combines mesh representation with sparse-controlled deformation technique to generate high-quality 4D object from a monocular video. To overcome the limitation of classical texture representation, we bind Gaussian splats to the surface of the triangular mesh for differentiable optimization of both the texture and mesh vertices. In particular, DreamMesh4D begins with a coarse mesh provided by a single image based 3D generation method. Sparse points are then uniformly sampled across the surface of the mesh, and are used to build a deformation graph to drive the motion of the 3D object for the sake of computational efficiency and providing additional constraint. For each step, transformations of sparse control points are predicted using a deformation network, and the mesh vertices as well as the bound surface Gaussians are deformed via a geometric skinning algorithm. The skinning algorithm is a hybrid approach combining LBS (linear blending skinning) and DQS (dual-quaternion skinning), mitigating drawbacks associated with both approaches. The static surface Gaussians and mesh vertices as well as the dynamic deformation network are learned via reference view photometric loss, score distillation loss as well as other regularization losses in a two-stage manner. Extensive experiments demonstrate that our method outperforms prior video-to-4D generation methods in terms of rendering quality and spatial-temporal consistency.

MrNeRF

12,323 views • 1 year ago

Think meshes are the best for e-commerce? Think again. 3D Gaussian Splats are fast and cheap to generate...and with the new SOG format from PlayCanvas, this one is only 5.8MB. 🪄 [1 / 2]
0:32

Think meshes are the best for e-commerce? Think again. 3D Gaussian Splats are fast and cheap to generate...and with the new SOG format from PlayCanvas, this one is only 5.8MB. 🪄 [1 / 2]

Will Eastcott

75,123 views • 11 months ago

Introducing powered by Over the Reality. FreeGaussian is a new way to transform your existing photos, videos, and 360 captures for free into high-quality 3D Gaussian splats, directly in your browser. Whether you already have: ↳ a 360 recording captured with any 360 camera ↳ a set of photos ↳ or even a simple video from a place you visited you can now turn it into a photorealistic 3D Gaussian splat. Just go to upload your content, and let the reconstruction happen. Once it’s ready, you can download your 3D Gaussian splat and point cloud, and use them across your own projects, pipelines, or applications.
0:27

Introducing powered by Over the Reality. FreeGaussian is a new way to transform your existing photos, videos, and 360 captures for free into high-quality 3D Gaussian splats, directly in your browser. Whether you already have: ↳ a 360 recording captured with any 360 camera ↳ a set of photos ↳ or even a simple video from a place you visited you can now turn it into a photorealistic 3D Gaussian splat. Just go to upload your content, and let the reconstruction happen. Once it’s ready, you can download your 3D Gaussian splat and point cloud, and use them across your own projects, pipelines, or applications.

Over the Reality 🌐

86,305 views • 4 months ago

Big news - we shipped Gaussian Splatting 🎉 Capture the impossible at: 🔥 Processing is free, and splats render fast in the browser. You can also export the .ply file. Enjoy 🚀
1:14

Big news - we shipped Gaussian Splatting 🎉 Capture the impossible at: 🔥 Processing is free, and splats render fast in the browser. You can also export the .ply file. Enjoy 🚀

polycam

466,395 views • 2 years ago

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.
5:36

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.

AK

140,992 views • 2 years ago

More and more users are turning their 360 footage into fully explorable 3D Gaussian Splats with View it here: This reconstruction started from a simple capture and is now a fully explorable 3D Gaussian Splat rendered in the FreeGaussian viewer, complete with point cloud reconstruction and high-fidelity spatial detail. More and more creators are discovering that their existing footage can become immersive 3D scenes. And they’re doing it for free with FreeGaussian
0:58

More and more users are turning their 360 footage into fully explorable 3D Gaussian Splats with View it here: This reconstruction started from a simple capture and is now a fully explorable 3D Gaussian Splat rendered in the FreeGaussian viewer, complete with point cloud reconstruction and high-fidelity spatial detail. More and more creators are discovering that their existing footage can become immersive 3D scenes. And they’re doing it for free with FreeGaussian

Over the Reality 🌐

14,759 views • 3 months ago

I like big splats and I cannot lie. PlayCanvas's new LOD streaming system for 3D Gaussian Splatting is a big deal. It's a key step towards splatting the world and making it explorable. Check out a live demo below in your browser.
0:31

I like big splats and I cannot lie. PlayCanvas's new LOD streaming system for 3D Gaussian Splatting is a big deal. It's a key step towards splatting the world and making it explorable. Check out a live demo below in your browser.

Bilawal Sidhu

61,583 views • 10 months ago

This is huge!! Spline now supports Gaussian splats 👀 You can now capture anything in 3D from your phone with Luma, export the .ply file, import it into spline, crop, adjust, and embed on your site. Luma + Spline + Webflow is an insane 3D workflow 🔥
0:37

This is huge!! Spline now supports Gaussian splats 👀 You can now capture anything in 3D from your phone with Luma, export the .ply file, import it into spline, crop, adjust, and embed on your site. Luma + Spline + Webflow is an insane 3D workflow 🔥

Nick St. Pierre

1,139,944 views • 2 years ago

☁️ No more clouds, no more noise! Our new Gaussian Splats are now cleaner and more consistent than ever. OVER’s latest update brings next-level clarity to the world’s largest decentralized 3D mapping network. Your maps have never looked this real. 🌍
0:23

☁️ No more clouds, no more noise! Our new Gaussian Splats are now cleaner and more consistent than ever. OVER’s latest update brings next-level clarity to the world’s largest decentralized 3D mapping network. Your maps have never looked this real. 🌍

Over the Reality 🌐

20,309 views • 10 months ago

Woah, Gaussian splats in mixed reality look so good! Tech stack: lichtfeld studio for splattering training, virtual desktop for PC VR streaming and passthrough MR using a virtual green screen, gracia for VR splat rendering using the green screen background feature
0:26

Woah, Gaussian splats in mixed reality look so good! Tech stack: lichtfeld studio for splattering training, virtual desktop for PC VR streaming and passthrough MR using a virtual green screen, gracia for VR splat rendering using the green screen background feature

Azad Balabanian

38,080 views • 5 months ago

3d visual positioning experiment -- look at the alignment between the 3d mesh and the live camera view. Truly feels magical -- like x-ray vision. Workflow: 1. Scanned a street in 15 mins w/ xgrids 2. Localized against that scan at night, in real-time, while sitting in a car The xgrids scanner (w/ rgb + lidar) is perfect to build maps for humans (3d gaussian splats). Then multiset ai makes it easy to build machine-readable maps for AR and robotics to figure out exactly where they are in 3d space with cm-level accuracy.
0:20

3d visual positioning experiment -- look at the alignment between the 3d mesh and the live camera view. Truly feels magical -- like x-ray vision. Workflow: 1. Scanned a street in 15 mins w/ xgrids 2. Localized against that scan at night, in real-time, while sitting in a car The xgrids scanner (w/ rgb + lidar) is perfect to build maps for humans (3d gaussian splats). Then multiset ai makes it easy to build machine-readable maps for AR and robotics to figure out exactly where they are in 3d space with cm-level accuracy.

Bilawal Sidhu

27,278 views • 9 months ago

People are underestimating Apple in AI. I just ran Apple’s new SHARP model locally and watched my photos turn into 3D Gaussian splats in seconds, then stepped inside them on Vision Pro. This feels like the beginning of something special. You really have to try it.
0:51

People are underestimating Apple in AI. I just ran Apple’s new SHARP model locally and watched my photos turn into 3D Gaussian splats in seconds, then stepped inside them on Vision Pro. This feels like the beginning of something special. You really have to try it.

Justin Ryan

255,483 views • 8 months ago

What happens if you try to push and squash a Gaussian splat? :) Been playing with adding simplified deformation and a lightweight physics simulation layer to Gaussian splats, running directly in real time on mobile. Instead of simulating every splat, this uses a small CPU lattice to drive the deformation, plus a per-splat transform so they actually stretch and compress properly. No mesh conversion, no retraining. Just layered on top of the renderer. You can push, bend, squash a scan and it holds together surprisingly well... Still early, but I can see the potential in making splats feel more real, from static captures to something more tactile, something you can touch, throw, bounce around… or well just fold and squash on the floor :) #GaussianSplatting #3DGS #Unity
0:39

What happens if you try to push and squash a Gaussian splat? :) Been playing with adding simplified deformation and a lightweight physics simulation layer to Gaussian splats, running directly in real time on mobile. Instead of simulating every splat, this uses a small CPU lattice to drive the deformation, plus a per-splat transform so they actually stretch and compress properly. No mesh conversion, no retraining. Just layered on top of the renderer. You can push, bend, squash a scan and it holds together surprisingly well... Still early, but I can see the potential in making splats feel more real, from static captures to something more tactile, something you can touch, throw, bounce around… or well just fold and squash on the floor :) #GaussianSplatting #3DGS #Unity

𝐑𝐔𝐁𝐄𝐍🥽𝐅𝐑𝐎

51,284 views • 4 months ago

2. Breakthrough in Volumetric Video Previous attempts were inconsistent because each frame had to be a new mesh. But Gaussian Splats solve this. It's not only temporally consistent, 60 fps, smaller in filesize but also allows infinite frame retiming. Seeing it in VR at the booth was like an out of body experience. Huge implications for VR. See the 10-min demo: Try it in the browser:
0:26

2. Breakthrough in Volumetric Video Previous attempts were inconsistent because each frame had to be a new mesh. But Gaussian Splats solve this. It's not only temporally consistent, 60 fps, smaller in filesize but also allows infinite frame retiming. Seeing it in VR at the booth was like an out of body experience. Huge implications for VR. See the 10-min demo: Try it in the browser:

Andrew Price

15,627 views • 1 year ago

The greatest Gaussian Splatting VR demo ever? Captured live at NVIDIA GTC — Mariya Sha experiencing 4D Gaussian Splatting in VR for the very first time. You can tell just from her reaction that this was something else! The scene was optimized using QUEEN, NVIDIA AI Developer's advanced research project for fast radiance field rendering. It’s an incredible look at what happens when high fidelity static and dynamic gaussian splats meets VR. Watch the full video on YouTube: Robert Scoble Did you get to do this one? #VR #3D #VFX
0:47

The greatest Gaussian Splatting VR demo ever? Captured live at NVIDIA GTC — Mariya Sha experiencing 4D Gaussian Splatting in VR for the very first time. You can tell just from her reaction that this was something else! The scene was optimized using QUEEN, NVIDIA AI Developer's advanced research project for fast radiance field rendering. It’s an incredible look at what happens when high fidelity static and dynamic gaussian splats meets VR. Watch the full video on YouTube: Robert Scoble Did you get to do this one? #VR #3D #VFX

Jonathan Stephens

28,114 views • 1 year ago