Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Open Sourcing Forge: 3D Gaussian splat rendering for web developers! 3DGS has become a dominant paradigm for differentiable rendering, combining high visual quality and real-time rendering. However, support for splatting on the web still lags behind its adoption in AI.

spark

5,629 subscribers

120,072 views • 1 year ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

14 Comments

spark1 year ago

At @theworldlabs, we’re building generative 3D foundation models to unlock spatial intelligence. Splats are an important representation for 3D AI, so to enable our vision we built a fast, dynamic web renderer. We believe this tech can be useful to everyone working with 3DGS.

spark1 year ago

Today we’re open sourcing Forge under the MIT license for anyone to use or contribute to, aiming to expand what’s possible with 3DGS with a flexible architecture that enables fully dynamic & programmable Gaussian splatting.

spark1 year ago

Forge integrates seamlessly with @threejs, runs in the browser via WebGL2 (desktop, mobile, and VR), can load most major splat file formats, and requires minimal code to get up and running. It supports multiple splat objects, multiple cameras, and real-time animation/editing.

spark1 year ago

Underlying this is a design that runs user-programmable functions for every splat on the GPU (like shaders in modern 3D graphics) and can be used to create complex animations and effects. We want to help people build new interactive experiences never before seen with 3DGS!

spark1 year ago

You can find full source code, examples, documentation, and more at We can’t wait to see what you’ll build!

MrNeRF1 year ago

@forgegfx @BenMildenhall Awesome!

Georgia Gkioxari1 year ago

@forgegfx Nice!! Looks 🔥🔥

Yanming Di1 year ago

@forgegfx @grok what is splat, when can CS people stop inventing new terms ...

shank ⚡️1 year ago

@forgegfx Cool Will try

Julien Blanchon 🇺🇦1 year ago

@forgegfx Possible to have an integration with threlte ?

Ian Curtis1 year ago

@forgegfx 🤩🔥💯

Saitoshi1 year ago

Great to see Forge open-sourced, @sparkjsdev! Easy integration with Three.js scenes will make it even more accessible for developers. This will be a game changer!

XR# —WebXR with .NET, C# & XAML1 year ago

@forgegfx Awesome! 👏

Patricia1 year ago

@forgegfx Awesome!

Related Videos

Google presents RadSplat Radiance Field-Informed Gaussian Splatting for Robust Real-Time Rendering with 900+ FPS Recent advances in view synthesis and real-time rendering have achieved photorealistic quality at impressive rendering speeds. While Radiance Field-based

Google presents RadSplat Radiance Field-Informed Gaussian Splatting for Robust Real-Time Rendering with 900+ FPS Recent advances in view synthesis and real-time rendering have achieved photorealistic quality at impressive rendering speeds. While Radiance Field-based

AK

139,790 views • 2 years ago

'3D Gaussian Splatting for Real-Time RF Rendering' rose to the 2nd challenge I presented. The magnifying glass isn't still crystal clear, but the result is compelling. Look at the small details/hairs on the succulent plant. Rendered in real-time. #gaussiansplatting #3D #AI

'3D Gaussian Splatting for Real-Time RF Rendering' rose to the 2nd challenge I presented. The magnifying glass isn't still crystal clear, but the result is compelling. Look at the small details/hairs on the succulent plant. Rendered in real-time. #gaussiansplatting #3D #AI

Hugues Bruyère

298,382 views • 3 years ago

Painting with 3D Gaussian Splat Brushes Contributions: • Designing a set of interactive tools and brush parameters for artistic brush creation and control using 3DGS content. • Computing oriented 3D Gaussian splat brushes for stamp-based painting on 3D surfaces represented as meshes or 3DGS scenes. • Deforming the splats in a brush stamp to ensure a smooth appearance of the painted 3D stroke. • Producing seamless brush strokes despite overlapping brush stamps, using diffusion inpainting. • Efficient modeling and rendering of brush strokes to facilitate 3DGS painting in real time.

Painting with 3D Gaussian Splat Brushes Contributions: • Designing a set of interactive tools and brush parameters for artistic brush creation and control using 3DGS content. • Computing oriented 3D Gaussian splat brushes for stamp-based painting on 3D surfaces represented as meshes or 3DGS scenes. • Deforming the splats in a brush stamp to ensure a smooth appearance of the painted 3D stroke. • Producing seamless brush strokes despite overlapping brush stamps, using diffusion inpainting. • Efficient modeling and rendering of brush strokes to facilitate 3DGS painting in real time.

MrNeRF

23,060 views • 1 year ago

Happy to introduce Habitat-GS, a non-intrusive extension of Habitat-Sim that brings dynamic Gaussian Splatting for photorealistic rendering and comes with hundreds of high-quality 3DGS scene assets, aiming to empowering navigation research. Code:

Happy to introduce Habitat-GS, a non-intrusive extension of Habitat-Sim that brings dynamic Gaussian Splatting for photorealistic rendering and comes with hundreds of high-quality 3DGS scene assets, aiming to empowering navigation research. Code:

Sida Peng

12,300 views • 5 months ago

📢Happy to present Convex Splatting, a novel way for 3D reconstruction based on 3D smooth convexes. For the first time, a splatting-based method reaches the quality of NeRF sota methods but with real-time rendering and few primitives!! I expect this to replace Gaussian splatting for 3D in the coming months. CODE RELEASED TODAY! joint work with collaborators from Université de Liège Visual Geometry Group (VGG) , KAUST Computer Vision Lab (IVUL) a thread 🧵 1/n

📢Happy to present Convex Splatting, a novel way for 3D reconstruction based on 3D smooth convexes. For the first time, a splatting-based method reaches the quality of NeRF sota methods but with real-time rendering and few primitives!! I expect this to replace Gaussian splatting for 3D in the coming months. CODE RELEASED TODAY! joint work with collaborators from Université de Liège Visual Geometry Group (VGG) , KAUST Computer Vision Lab (IVUL) a thread 🧵 1/n

Abdullah Hamdi

71,661 views • 1 year ago

Happy to announce the results of our latest research, which takes 3D Gaussian Splatting to the next level: "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets," which has been accepted at #SIGGRAPH2024!🎉 Find it here:

Happy to announce the results of our latest research, which takes 3D Gaussian Splatting to the next level: "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets," which has been accepted at #SIGGRAPH2024!🎉 Find it here:

Bernhard Kerbl

356,584 views • 2 years ago

The PlayCanvas team has solved collision for 3D Gaussian splats. Install splat-transform via NPM to get a CLI tool + library that can output high quality voxel-based collision. Here you can see a splat navigated in first person mode with voxel rendering toggled on/off. 🧵

The PlayCanvas team has solved collision for 3D Gaussian splats. Install splat-transform via NPM to get a CLI tool + library that can output high quality voxel-based collision. Here you can see a splat navigated in first person mode with voxel rendering toggled on/off. 🧵

Will Eastcott

30,303 views • 6 months ago

GPS-Gaussian+: Generalizable Pixel-wise 3D Gaussian Splatting for Real-Time Human-Scene Rendering from Sparse Views TL;DR: Are we witnessing the first steps towards 3DGS live streaming? Contributions: • We introduce a generalizable 3D Gaussian Splatting methodology that employs pixel-wise Gaussian parameter maps defined on 2D source image planes to formulate 3D Gaussians in a feed-forward manner. • We propose a fully differentiable framework composed of an iterative depth estimation module and a Gaussian parameter regression module. The intermediate depth prediction bridges the two components and allows them to benefit from joint training. • We introduce a regularization term and an epipolar attention mechanism to preserve geometry consistency between the two source views when using only rendering loss. Our method generalizes well to unseen characters even in complicated scenes. • We develop a real-time FVV system that achieves high-resolution rendering of characters in the scene without any geometry supervision.

GPS-Gaussian+: Generalizable Pixel-wise 3D Gaussian Splatting for Real-Time Human-Scene Rendering from Sparse Views TL;DR: Are we witnessing the first steps towards 3DGS live streaming? Contributions: • We introduce a generalizable 3D Gaussian Splatting methodology that employs pixel-wise Gaussian parameter maps defined on 2D source image planes to formulate 3D Gaussians in a feed-forward manner. • We propose a fully differentiable framework composed of an iterative depth estimation module and a Gaussian parameter regression module. The intermediate depth prediction bridges the two components and allows them to benefit from joint training. • We introduce a regularization term and an epipolar attention mechanism to preserve geometry consistency between the two source views when using only rendering loss. Our method generalizes well to unseen characters even in complicated scenes. • We develop a real-time FVV system that achieves high-resolution rendering of characters in the scene without any geometry supervision.

MrNeRF

25,893 views • 1 year ago

$Spatial AI is increasingly important, and the newest papers from #NVIDIAResearch, 3DGRT and 3DGUT, represent significant advancements in enabling researchers and developers to explore and innovate with 3D Gaussian Splatting techniques. 💎 3DGRT (Gaussian Ray Tracing) ➡️ - Enables secondary lighting effects such as refractions, shadows, depth of field, and mirrors. - Provides a foundation for further research in ray tracing-based rendering of Gaussian representations. - Achieves real-time rendering performance on modern GPUs. ✨ 3DGUT (Gaussain Unscented Transform) ➡️ - Support for distorted camera models and time-dependent effects like rolling shutter. - Enables phenomena such as reflections and refractions while retaining real-time efficiency. 📥 Code:$

Spatial AI is increasingly important, and the newest papers from #NVIDIAResearch, 3DGRT and 3DGUT, represent significant advancements in enabling researchers and developers to explore and innovate with 3D Gaussian Splatting techniques. 💎 3DGRT (Gaussian Ray Tracing) ➡️ - Enables secondary lighting effects such as refractions, shadows, depth of field, and mirrors. - Provides a foundation for further research in ray tracing-based rendering of Gaussian representations. - Achieves real-time rendering performance on modern GPUs. ✨ 3DGUT (Gaussain Unscented Transform) ➡️ - Support for distorted camera models and time-dependent effects like rolling shutter. - Enables phenomena such as reflections and refractions while retaining real-time efficiency. 📥 Code:

NVIDIA AI Developer

27,782 views • 1 year ago

$FAU Erlangen-Nürnberg presents TRIPS Trilinear Point Splatting for Real-Time Radiance Field Rendering paper page: Point-based radiance field rendering has demonstrated impressive results for novel view synthesis, offering a compelling blend of rendering quality and computational efficiency. However, also latest approaches in this domain are not without their shortcomings. 3D Gaussian Splatting [Kerbl and Kopanas et al. 2023] struggles when tasked with rendering highly detailed scenes, due to blurring and cloudy artifacts. On the other hand, ADOP [R\"uckert et al. 2022] can accommodate crisper images, but the neural reconstruction network decreases performance, it grapples with temporal instability and it is unable to effectively address large gaps in the point cloud. In this paper, we present TRIPS (Trilinear Point Splatting), an approach that combines ideas from both Gaussian Splatting and ADOP. The fundamental concept behind our novel technique involves rasterizing points into a screen-space image pyramid, with the selection of the pyramid layer determined by the projected point size. This approach allows rendering arbitrarily large points using a single trilinear write. A lightweight neural network is then used to reconstruct a hole-free image including detail beyond splat resolution. Importantly, our render pipeline is entirely differentiable, allowing for automatic optimization of both point sizes and positions. Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art methods in terms of rendering quality while maintaining a real-time frame rate of 60 frames per second on readily available hardware. This performance extends to challenging scenarios, such as scenes featuring intricate geometry, expansive landscapes, and auto-exposed footage.$

FAU Erlangen-Nürnberg presents TRIPS Trilinear Point Splatting for Real-Time Radiance Field Rendering paper page: Point-based radiance field rendering has demonstrated impressive results for novel view synthesis, offering a compelling blend of rendering quality and computational efficiency. However, also latest approaches in this domain are not without their shortcomings. 3D Gaussian Splatting [Kerbl and Kopanas et al. 2023] struggles when tasked with rendering highly detailed scenes, due to blurring and cloudy artifacts. On the other hand, ADOP [R\"uckert et al. 2022] can accommodate crisper images, but the neural reconstruction network decreases performance, it grapples with temporal instability and it is unable to effectively address large gaps in the point cloud. In this paper, we present TRIPS (Trilinear Point Splatting), an approach that combines ideas from both Gaussian Splatting and ADOP. The fundamental concept behind our novel technique involves rasterizing points into a screen-space image pyramid, with the selection of the pyramid layer determined by the projected point size. This approach allows rendering arbitrarily large points using a single trilinear write. A lightweight neural network is then used to reconstruct a hole-free image including detail beyond splat resolution. Importantly, our render pipeline is entirely differentiable, allowing for automatic optimization of both point sizes and positions. Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art methods in terms of rendering quality while maintaining a real-time frame rate of 60 frames per second on readily available hardware. This performance extends to challenging scenarios, such as scenes featuring intricate geometry, expansive landscapes, and auto-exposed footage.

AK

45,489 views • 2 years ago

Spark 2.0 Developer Preview is now available: introducing Level-of-Detail (LoD) rendering and streaming for world-scale Gaussian Splats on the web. Learn more ↓

Spark 2.0 Developer Preview is now available: introducing Level-of-Detail (LoD) rendering and streaming for world-scale Gaussian Splats on the web. Learn more ↓

spark

233,091 views • 8 months ago

LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPS Contributions: • LangSplatV2 achieves real-time performance with 476.2 FPS for high-dimensional feature splatting and 384.6 FPS for 3D open-vocabulary text querying. • Delivers a 42× speedup and 47× boost in performance compared to LangSplat. • Improves query accuracy while drastically reducing inference time. • Replaces the heavyweight decoder in LangSplat with a sparse coefficient field, removing the main performance bottleneck. • Introduces a CUDA-optimized sparse coefficient splatting method, enabling fast and high-quality rendering of high-dimensional features. • Enables scalable 3D language interaction in complex scenes, opening up real-time applications previously not possible with LangSplat.

LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPS Contributions: • LangSplatV2 achieves real-time performance with 476.2 FPS for high-dimensional feature splatting and 384.6 FPS for 3D open-vocabulary text querying. • Delivers a 42× speedup and 47× boost in performance compared to LangSplat. • Improves query accuracy while drastically reducing inference time. • Replaces the heavyweight decoder in LangSplat with a sparse coefficient field, removing the main performance bottleneck. • Introduces a CUDA-optimized sparse coefficient splatting method, enabling fast and high-quality rendering of high-dimensional features. • Enables scalable 3D language interaction in complex scenes, opening up real-time applications previously not possible with LangSplat.

MrNeRF

14,365 views • 1 year ago

1000+ FPS 4D Gaussian Splatting for Dynamic Scene Rendering Contributions: • We delve into the temporal redundancy of 4D Gaussian Splatting and explain the main reason for the storage pressure and suboptimal rendering speed. • We introduce 4DGS-1K, a compact and memory-efficient framework to address these issues. It consists of two key components: a spatial-temporal variation score-based pruning strategy and a temporal filter. • Extensive experiments demonstrate that 4DGS-1K not only achieves a substantial storage reduction of approximately 41× but also accelerates rasterization to 1000+ FPS while maintaining high-quality reconstruction.

1000+ FPS 4D Gaussian Splatting for Dynamic Scene Rendering Contributions: • We delve into the temporal redundancy of 4D Gaussian Splatting and explain the main reason for the storage pressure and suboptimal rendering speed. • We introduce 4DGS-1K, a compact and memory-efficient framework to address these issues. It consists of two key components: a spatial-temporal variation score-based pruning strategy and a temporal filter. • Extensive experiments demonstrate that 4DGS-1K not only achieves a substantial storage reduction of approximately 41× but also accelerates rasterization to 1000+ FPS while maintaining high-quality reconstruction.

MrNeRF

12,445 views • 1 year ago

Faster rendering, advanced lighting, multiplayer workflows, expanded AI, and enhanced support for mobile web. See what features our product team is most excited about in Unity 6. Discover More:

Faster rendering, advanced lighting, multiplayer workflows, expanded AI, and enhanced support for mobile web. See what features our product team is most excited about in Unity 6. Discover More:

Unity for Games

40,240 views • 2 years ago

A pivotal moment for real estate! 🏡 Capturing a property in 3D used to mean a $5,000 LiDAR scanner. This house was scanned with a $500 Insta360 X5 - a consumer 360 camera. And best of all? All of the software is free and open source! It's based on a technique called 3D Gaussian splatting: 🪄 Splat creation: Spirula Studio 🖼️ Splat rendering: PlayCanvas So for $500, you can present any property online and let prospective purchasers freely explore as if it was a videogame. 🎮 [1/3]

A pivotal moment for real estate! 🏡 Capturing a property in 3D used to mean a $5,000 LiDAR scanner. This house was scanned with a $500 Insta360 X5 - a consumer 360 camera. And best of all? All of the software is free and open source! It's based on a technique called 3D Gaussian splatting: 🪄 Splat creation: Spirula Studio 🖼️ Splat rendering: PlayCanvas So for $500, you can present any property online and let prospective purchasers freely explore as if it was a videogame. 🎮 [1/3]

Will Eastcott

86,533 views • 1 month ago

The free AVATAR project is expected to launch for testing this week, offering cinematic-quality real-time rendering. You can create personalized characters and even connect with free AI motion capture to produce high-quality real-time animation using a 1080P webcam.

CYANPUPPETS

41,033 views • 1 year ago