Loading video...

Video Failed to Load

Go Home

Gaussian Shell Maps are a new neural scene representation that connects fields and 3D Gaussians. This representation unlocks the full potential of 3D Gaussian splatting for generative AI applications, such as 3D avatar generation. 1/2

GordonWetzstein's profile picture

Gordon Wetzstein

5,297 subscribers

52,480 views • 2 years ago •via X (Twitter)

Science & TechnologyEducation
LIVE

Anya Rossi• Live Now

Private livecam show

3 Comments

Gordon Wetzstein's profile picture
Gordon Wetzstein2 years ago

With @AbdalRameen, @toomanyyifans, @Vivianszf1, @YinghaoXu1, @Po_lhr, @zfkuang1, @CQFHK, Dit-Yan Yeung 2/2

Hermes ᯅ's profile picture
Hermes ᯅ2 years ago

This looks really interesting. Over the last few weeks I’ve seen papers come out on how to record motion but this looks like it can synthesize the motion which is a whole new level.

Wieland Morgenstern's profile picture
Wieland Morgenstern2 years ago

Lovely looking results, kudos! Bit unclear to me how the shells are actually formed, and how the Gaussians are placed in them. I put my notes from reading the paper here, including the stuff that I don't understand:

Related Videos

VAST AI releases Triplane Meets Gaussian Splatting on Hugging Face Fast and Generalizable Single-View 3D Reconstruction with Transformers demo: TGS enables fast reconstruction from single-view image. It builds the 3D representation upon a hybrid Triplane-Gaussian representation by evaluating a transformer-based framework, from which 3D Gaussians would be decoded
0:23

VAST AI releases Triplane Meets Gaussian Splatting on Hugging Face Fast and Generalizable Single-View 3D Reconstruction with Transformers demo: TGS enables fast reconstruction from single-view image. It builds the 3D representation upon a hybrid Triplane-Gaussian representation by evaluating a transformer-based framework, from which 3D Gaussians would be decoded

AK

113,220 views • 2 years ago

𝗜𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝗶𝗻𝗴 𝗨𝗩𝗚𝗦 We introduce 𝗨𝗩𝗚𝗦, a new 2D representation of 3D Gaussian Splatting (3DGS) that leverages spherical mapping. Website: Paper:
0:23

𝗜𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝗶𝗻𝗴 𝗨𝗩𝗚𝗦 We introduce 𝗨𝗩𝗚𝗦, a new 2D representation of 3D Gaussian Splatting (3DGS) that leverages spherical mapping. Website: Paper:

Nikolaos Sarafianos

10,715 views • 1 year ago

3D Gaussian Splatting is awesome! BUT, the 3D Gaussian primitives often have difficulty reconstructing fine appearance details. 😩 Meet Textured Gaussians! ✋ Our RGBA Textured Gaussians enhance 3D scene appearance modeling. Small change, BIG improvement!
0:34

3D Gaussian Splatting is awesome! BUT, the 3D Gaussian primitives often have difficulty reconstructing fine appearance details. 😩 Meet Textured Gaussians! ✋ Our RGBA Textured Gaussians enhance 3D scene appearance modeling. Small change, BIG improvement!

Jia-Bin Huang

39,666 views • 1 year ago

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.
5:36

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.

AK

140,960 views • 2 years ago

Happy to announce the results of our latest research, which takes 3D Gaussian Splatting to the next level: "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets," which has been accepted at #SIGGRAPH2024!🎉 Find it here:
1:40

Happy to announce the results of our latest research, which takes 3D Gaussian Splatting to the next level: "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets," which has been accepted at #SIGGRAPH2024!🎉 Find it here:

Bernhard Kerbl

356,259 views • 2 years ago

Goodbye SOGS, hello SOG! 👋 PlayCanvas open sources Spatially Ordered Gaussians - a new super-compressed format for 3D Gaussian Splatting.
0:59

Goodbye SOGS, hello SOG! 👋 PlayCanvas open sources Spatially Ordered Gaussians - a new super-compressed format for 3D Gaussian Splatting.

PlayCanvas

48,260 views • 9 months ago

"YoNoSplat: You Only Need One Model for Feedforward 3D Gaussian Splatting" TL;DR: a unified 3D Gaussian splatting model that reconstructs high-quality scene geometry and camera poses from unposed/uncalibrated images in a single forward pass.
0:38

"YoNoSplat: You Only Need One Model for Feedforward 3D Gaussian Splatting" TL;DR: a unified 3D Gaussian splatting model that reconstructs high-quality scene geometry and camera poses from unposed/uncalibrated images in a single forward pass.

Alexandre Morgand

14,839 views • 4 months ago

WaterSplatting: Fast Underwater 3D Scene Reconstruction Using Gaussian Splatting Project: Method ⬇️ 1 I 2
1:07

WaterSplatting: Fast Underwater 3D Scene Reconstruction Using Gaussian Splatting Project: Method ⬇️ 1 I 2

MrNeRF

12,175 views • 1 year ago

a two minute explanation of 3D Gaussian Splatting
2:07

a two minute explanation of 3D Gaussian Splatting

dylan

254,807 views • 2 years ago

100M Gaussians. Streamed on a single RTX 3090. Welcome to NVIDIA AI Developer 's fVDB Reality Capture — bringing 3D Gaussian Splatting, volumetric data, and OpenUSD into one pipeline. 🎥 Full demo → #NVIDIA #GaussianSplatting #3D
0:59

100M Gaussians. Streamed on a single RTX 3090. Welcome to NVIDIA AI Developer 's fVDB Reality Capture — bringing 3D Gaussian Splatting, volumetric data, and OpenUSD into one pipeline. 🎥 Full demo → #NVIDIA #GaussianSplatting #3D

Jonathan Stephens

36,725 views • 7 months ago

3D Gaussian Splatting is great, but can it work without the pre-computed camera poses? Introducing: COLMAP-Free 3D Gaussian Splatting Our recent work shows not only it can, but 3D Gaussians make camera pose estimation easy (compared to NeRF) along with reconstruction. 👇🧵
0:50

3D Gaussian Splatting is great, but can it work without the pre-computed camera poses? Introducing: COLMAP-Free 3D Gaussian Splatting Our recent work shows not only it can, but 3D Gaussians make camera pose estimation easy (compared to NeRF) along with reconstruction. 👇🧵

Xiaolong Wang

76,747 views • 2 years ago

Splatter a Video can turn a video into a 3D Gaussian representation! This allows for enhanced video tracking, depth prediction, motion and appearance editing, and stereoscopic video generation.
0:37

Splatter a Video can turn a video into a 3D Gaussian representation! This allows for enhanced video tracking, depth prediction, motion and appearance editing, and stereoscopic video generation.

Dreaming Tulpa 🥓👑

11,572 views • 2 years ago

3D Gaussian Splatting物質化計画、 by Crysta AI 🧊 3D Printing Thank you!
0:38

3D Gaussian Splatting物質化計画、 by Crysta AI 🧊 3D Printing Thank you!

VoxelKei

43,054 views • 4 months ago

Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video Contribution quote from the paper: In summary, our main contributions are • a comprehensive pipeline for reconstructing the shape, appearance, and behavior of real-world garments using Gaussian splatting, • an algorithm for registering garment meshes to multi- view videos with an optimization procedure based on Gaussian splatting, and • a Gaussian Garment representation that combines triangle meshes with Gaussian textures to capture photorealistic appearance and can be used as a fully controllable 3D asset.
0:20

Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video Contribution quote from the paper: In summary, our main contributions are • a comprehensive pipeline for reconstructing the shape, appearance, and behavior of real-world garments using Gaussian splatting, • an algorithm for registering garment meshes to multi- view videos with an optimization procedure based on Gaussian splatting, and • a Gaussian Garment representation that combines triangle meshes with Gaussian textures to capture photorealistic appearance and can be used as a fully controllable 3D asset.

MrNeRF

27,277 views • 1 year ago

3D Gaussian Splatting in WebGPU is ready
0:26

3D Gaussian Splatting in WebGPU is ready

el.cine

28,817 views • 5 months ago

Photogrammetry × 3D Gaussian Splatting × VRC Light Volumes！！
0:21

Photogrammetry × 3D Gaussian Splatting × VRC Light Volumes！！

とこよし

76,297 views • 11 months ago

This is a next-generation reality capture system using 3D Gaussian Splatting to create photorealistic, walk-through digital models.
0:34

This is a next-generation reality capture system using 3D Gaussian Splatting to create photorealistic, walk-through digital models.

Eyisha Zyer

34,136 views • 9 months ago

Generated the video using Grok and turned it into 3D Gaussian Splatting.
0:43

Generated the video using Grok and turned it into 3D Gaussian Splatting.

Jaskirat Singh

49,096 views • 5 months ago

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:
0:50

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

Sherwin Bahmani

66,490 views • 9 months ago