Loading video...

Video Failed to Load

Go Home

We will be in ACM SIGGRAPH 2023 with "3D Gaussian Splatting for Real-Time Radiance Field Rendering", have you ever seen radiance fields with 100+ FPS and MipNeRF360 quality? Check out our website here:

gkopanas's profile picture

George Kopanas

2,873 subscribers

135,782 views • 3 years ago •via X (Twitter)

Science & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

3D Gaussian Splatting for Real-Time Radiance Field Rendering paper page: Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (>= 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.
0:18

3D Gaussian Splatting for Real-Time Radiance Field Rendering paper page: Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (>= 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.

AK

633,669 views • 3 years ago

Triangle Splatting for Real-Time Radiance Field Rendering Contributions: (i) We propose Triangle Splatting, a novel approach that directly optimizes unstructured triangles, bridging traditional computer graphics and radiance fields. (ii) We introduce a differentiable window function for soft triangle boundaries, enabling effective gradient flow. (iii) We demonstrate qualitatively and quantitatively that Triangle Splatting outperforms concurrent methods in terms of visual quality and rendering speed, and achieves superior perceptual quality compared to the state-of-the-art Zip-NeRF on indoor scenes. (iv) The optimized triangles are directly compatible with standard mesh-based renderers, enabling seamless integration into traditional graphics pipelines.
0:16

Triangle Splatting for Real-Time Radiance Field Rendering Contributions: (i) We propose Triangle Splatting, a novel approach that directly optimizes unstructured triangles, bridging traditional computer graphics and radiance fields. (ii) We introduce a differentiable window function for soft triangle boundaries, enabling effective gradient flow. (iii) We demonstrate qualitatively and quantitatively that Triangle Splatting outperforms concurrent methods in terms of visual quality and rendering speed, and achieves superior perceptual quality compared to the state-of-the-art Zip-NeRF on indoor scenes. (iv) The optimized triangles are directly compatible with standard mesh-based renderers, enabling seamless integration into traditional graphics pipelines.

MrNeRF

51,407 views • 1 year ago

In the summer of 2023, I cold emailed Jensen Huang and asked to capture a NeRF of him at SIGGRAPH. He responded in about an hour and said yes. A radiance field is, in the simplest terms, akin to a 3D photograph. A moment in time, so completely reconstructed that you can move through it and see it from angles the original cameras never occupied. NeRFs were the original method. Gaussian splatting, which debuted at that same SIGGRAPH, has since become the dominant form of radiance field. I called my late friend James, who told me we needed to begin practicing immediately. We ran capture after capture for weeks until we consistently got the capture time down to ~30 seconds with one camera. Later, in a hallway at the LA Convention Center during SIGGRAPH, I captured the portrait you're seeing now, a full 360° gaussian splat of Jensen, rendered here as a 2D flythrough. Afterward, I continued the conversation with him and members of his team to make the case for radiance fields as a foundational representation for imaging. To my surprise, they listened. Three years later, NVIDIA has several works, including NuRec, fVDB, 3DGRUT, and gsplat all utilizing radiance fields. The landscape has evolved enough that the reasoning is obvious. Gaussian splatting has begun to ship across some of the world’s largest industries, including autonomous vehicles, AEC, geospatial, media and entertainment, robotics, e-commerce, hospitality. It’s become clear that lifelike 3D is here to stay. And yet I think we will look back and be disappointed by how late we started taking 3D portraits of the people around us, just like how we have sparse 2D photos of our grandparents and great grandparents. We have billions of photographs of the people we know and love, but almost no radiance fields of them. I'll be returning to SIGGRAPH in LA where this was initially captured three years ago, with the landscape looking significantly different. Radiance fields are more under deployed than ever relative to what they can do. I'm excited for the future of imaging, and for 2D to transition into 3D. I have a few things up my sleeve that I think will make that case plainly.
0:18

In the summer of 2023, I cold emailed Jensen Huang and asked to capture a NeRF of him at SIGGRAPH. He responded in about an hour and said yes. A radiance field is, in the simplest terms, akin to a 3D photograph. A moment in time, so completely reconstructed that you can move through it and see it from angles the original cameras never occupied. NeRFs were the original method. Gaussian splatting, which debuted at that same SIGGRAPH, has since become the dominant form of radiance field. I called my late friend James, who told me we needed to begin practicing immediately. We ran capture after capture for weeks until we consistently got the capture time down to ~30 seconds with one camera. Later, in a hallway at the LA Convention Center during SIGGRAPH, I captured the portrait you're seeing now, a full 360° gaussian splat of Jensen, rendered here as a 2D flythrough. Afterward, I continued the conversation with him and members of his team to make the case for radiance fields as a foundational representation for imaging. To my surprise, they listened. Three years later, NVIDIA has several works, including NuRec, fVDB, 3DGRUT, and gsplat all utilizing radiance fields. The landscape has evolved enough that the reasoning is obvious. Gaussian splatting has begun to ship across some of the world’s largest industries, including autonomous vehicles, AEC, geospatial, media and entertainment, robotics, e-commerce, hospitality. It’s become clear that lifelike 3D is here to stay. And yet I think we will look back and be disappointed by how late we started taking 3D portraits of the people around us, just like how we have sparse 2D photos of our grandparents and great grandparents. We have billions of photographs of the people we know and love, but almost no radiance fields of them. I'll be returning to SIGGRAPH in LA where this was initially captured three years ago, with the landscape looking significantly different. Radiance fields are more under deployed than ever relative to what they can do. I'm excited for the future of imaging, and for 2D to transition into 3D. I have a few things up my sleeve that I think will make that case plainly.

Radiance Fields

17,663 views • 1 month ago

DroneSplat: 3D Gaussian Splatting for Robust 3D Reconstruction from In-the-Wild Drone Imagery Abstract: Drones have become essential tools for reconstructing wild scenes due to their outstanding maneuverability. Recent advances in radiance field methods have achieved remarkable rendering quality, providing a new avenue for 3D reconstruction from drone imagery. However, dynamic distractors in wild environments challenge the static scene assumption in radiance fields, while limited view constraints hinder the accurate capture of underlying scene geometry. To address these challenges, we introduce DroneSplat, a novel framework designed for robust 3D reconstruction from in-the-wild drone imagery. Our method adaptively adjusts masking thresholds by integrating local-global segmentation heuristics with statistical approaches, enabling precise identification and elimination of dynamic distractors in static scenes. We enhance 3D Gaussian Splatting with multi-view stereo predictions and a voxel-guided optimization strategy, supporting high-quality rendering under limited view constraints. For comprehensive evaluation, we provide a drone-captured 3D reconstruction dataset encompassing both dynamic and static scenes. Extensive experiments demonstrate that DroneSplat outperforms both 3DGS and NeRF baselines in handling in-the-wild drone imagery.
0:19

DroneSplat: 3D Gaussian Splatting for Robust 3D Reconstruction from In-the-Wild Drone Imagery Abstract: Drones have become essential tools for reconstructing wild scenes due to their outstanding maneuverability. Recent advances in radiance field methods have achieved remarkable rendering quality, providing a new avenue for 3D reconstruction from drone imagery. However, dynamic distractors in wild environments challenge the static scene assumption in radiance fields, while limited view constraints hinder the accurate capture of underlying scene geometry. To address these challenges, we introduce DroneSplat, a novel framework designed for robust 3D reconstruction from in-the-wild drone imagery. Our method adaptively adjusts masking thresholds by integrating local-global segmentation heuristics with statistical approaches, enabling precise identification and elimination of dynamic distractors in static scenes. We enhance 3D Gaussian Splatting with multi-view stereo predictions and a voxel-guided optimization strategy, supporting high-quality rendering under limited view constraints. For comprehensive evaluation, we provide a drone-captured 3D reconstruction dataset encompassing both dynamic and static scenes. Extensive experiments demonstrate that DroneSplat outperforms both 3DGS and NeRF baselines in handling in-the-wild drone imagery.

MrNeRF

21,346 views • 1 year ago

A new real-time Radiance Field paper beating 3DGS was just released! 🔥 Similarly to 3D Gaussian Splatting, TRIPS optimizes a point-cloud with color, position & size that gets splatted to the screen. But it does so using a single trilinear write in an image pyramid More info ⬇️
0:13

A new real-time Radiance Field paper beating 3DGS was just released! 🔥 Similarly to 3D Gaussian Splatting, TRIPS optimizes a point-cloud with color, position & size that gets splatted to the screen. But it does so using a single trilinear write in an image pyramid More info ⬇️

Alex Carlier

78,372 views • 2 years ago

F3D-Gaus: Feed-forward 3D-aware Generation on ImageNet with Cycle-Consistent Gaussian Splatting Contributions: • We pioneer 3D-aware generation using generalizable feed-forward Gaussian Splatting representation, achieving significant efficiency and favorable rendering quality on monocular datasets. • We significantly advance the capability of pixel-aligned Gaussian Splatting representations by designing a self-supervised cycle training strategy specifically tailored for monocular datasets. • We further mitigate the artifacts of 3D-aware representations caused by large viewpoint shifts by introducing geometry-aware video priors.
0:16

F3D-Gaus: Feed-forward 3D-aware Generation on ImageNet with Cycle-Consistent Gaussian Splatting Contributions: • We pioneer 3D-aware generation using generalizable feed-forward Gaussian Splatting representation, achieving significant efficiency and favorable rendering quality on monocular datasets. • We significantly advance the capability of pixel-aligned Gaussian Splatting representations by designing a self-supervised cycle training strategy specifically tailored for monocular datasets. • We further mitigate the artifacts of 3D-aware representations caused by large viewpoint shifts by introducing geometry-aware video priors.

MrNeRF

14,229 views • 1 year ago

Segment Any 3D Gaussians paper page: Interactive 3D segmentation in radiance fields is an appealing task since its importance in 3D scene understanding and manipulation. However, existing methods face challenges in either achieving fine-grained, multi-granularity segmentation or contending with substantial computational overhead, inhibiting real-time interaction. In this paper, we introduce Segment Any 3D GAussians (SAGA), a novel 3D interactive segmentation approach that seamlessly blends a 2D segmentation foundation model with 3D Gaussian Splatting (3DGS), a recent breakthrough of radiance fields. SAGA efficiently embeds multi-granularity 2D segmentation results generated by the segmentation foundation model into 3D Gaussian point features through well-designed contrastive training. Evaluation on existing benchmarks demonstrates that SAGA can achieve competitive performance with state-of-the-art methods. Moreover, SAGA achieves multi-granularity segmentation and accommodates various prompts, including points, scribbles, and 2D masks. Notably, SAGA can finish the 3D segmentation within milliseconds, achieving nearly 1000x acceleration compared to previous SOTA.
0:18

Segment Any 3D Gaussians paper page: Interactive 3D segmentation in radiance fields is an appealing task since its importance in 3D scene understanding and manipulation. However, existing methods face challenges in either achieving fine-grained, multi-granularity segmentation or contending with substantial computational overhead, inhibiting real-time interaction. In this paper, we introduce Segment Any 3D GAussians (SAGA), a novel 3D interactive segmentation approach that seamlessly blends a 2D segmentation foundation model with 3D Gaussian Splatting (3DGS), a recent breakthrough of radiance fields. SAGA efficiently embeds multi-granularity 2D segmentation results generated by the segmentation foundation model into 3D Gaussian point features through well-designed contrastive training. Evaluation on existing benchmarks demonstrates that SAGA can achieve competitive performance with state-of-the-art methods. Moreover, SAGA achieves multi-granularity segmentation and accommodates various prompts, including points, scribbles, and 2D masks. Notably, SAGA can finish the 3D segmentation within milliseconds, achieving nearly 1000x acceleration compared to previous SOTA.

AK

69,542 views • 2 years ago

(1/3) 📢📢𝐆𝐆𝐇𝐞𝐚𝐝 𝐅𝐚𝐬𝐭 𝐚𝐧𝐝 𝐆𝐞𝐧𝐞𝐫𝐚𝐥𝐢𝐳𝐚𝐛𝐥𝐞 𝟑𝐃 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧 𝐇𝐞𝐚𝐝𝐬📢📢 #SiggraphAsia'24 We generate photo-realistic 3D heads and render them with Gaussian Splatting at 1k resolution in real-time.
0:16

(1/3) 📢📢𝐆𝐆𝐇𝐞𝐚𝐝 𝐅𝐚𝐬𝐭 𝐚𝐧𝐝 𝐆𝐞𝐧𝐞𝐫𝐚𝐥𝐢𝐳𝐚𝐛𝐥𝐞 𝟑𝐃 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧 𝐇𝐞𝐚𝐝𝐬📢📢 #SiggraphAsia'24 We generate photo-realistic 3D heads and render them with Gaussian Splatting at 1k resolution in real-time.

Matthias Niessner

36,618 views • 1 year ago

Self-Calibrating Gaussian Splatting for Large Field of View Reconstruction Note: Check below for full video. Abstract (cited): "In this paper, we present a self-calibrating framework that jointly optimizes camera parameters, lens distortion, and 3D Gaussian representations, enabling accurate and efficient scene reconstruction. Our technique is particularly effective for high-quality scene reconstruction from large field-of-view (FOV) imagery taken with wide-angle lenses, allowing the scene to be modeled from a smaller number of images. We introduce a novel method for modeling complex lens distortions using a hybrid network that combines invertible residual networks with explicit grids. This design effectively regularizes the optimization process, achieving greater accuracy than conventional camera models. Additionally, we propose a cubemap-based resampling strategy to support large FOV images without sacrificing resolution or introducing distortion artifacts. Our method is compatible with the fast rasterization of Gaussian Splatting, adaptable to a wide variety of camera lens distortions, and demonstrates state-of-the-art performance on both synthetic and real-world datasets."
0:16

Self-Calibrating Gaussian Splatting for Large Field of View Reconstruction Note: Check below for full video. Abstract (cited): "In this paper, we present a self-calibrating framework that jointly optimizes camera parameters, lens distortion, and 3D Gaussian representations, enabling accurate and efficient scene reconstruction. Our technique is particularly effective for high-quality scene reconstruction from large field-of-view (FOV) imagery taken with wide-angle lenses, allowing the scene to be modeled from a smaller number of images. We introduce a novel method for modeling complex lens distortions using a hybrid network that combines invertible residual networks with explicit grids. This design effectively regularizes the optimization process, achieving greater accuracy than conventional camera models. Additionally, we propose a cubemap-based resampling strategy to support large FOV images without sacrificing resolution or introducing distortion artifacts. Our method is compatible with the fast rasterization of Gaussian Splatting, adaptable to a wide variety of camera lens distortions, and demonstrates state-of-the-art performance on both synthetic and real-world datasets."

MrNeRF

17,206 views • 1 year ago

【📢2.0 Website Update】 In case you missed it, our new website is now live! 💻 We have lots of information about us as a company published and will update it often in the future! 🔗Check it out here:
0:15

【📢2.0 Website Update】 In case you missed it, our new website is now live! 💻 We have lots of information about us as a company published and will update it often in the future! 🔗Check it out here:

PixelLink Official (Graduated)

18,543 views • 2 years ago

That's a wrap for 2023! ✨ We've had an incredible year at OPLegends. We want to thank our entire community for everything we have accomplished this year! To celebrate 2023, check out your own personal #OPLegendsRewind at Run /rewind in-game to get your unique code. We have so much planned for 2024, and can't wait for all of you to be a part of our incredible plans. We have celebrated a lot of amazing milestones in 2023, and we truly have the best community in Minecraft. Reply with your rewind URL and with #OPLegendsRewind for a chance to win one of 2X PERMANENT LEGEND RANKS, which will be drawn on January 2nd @ 3PM EST!
0:10

That's a wrap for 2023! ✨ We've had an incredible year at OPLegends. We want to thank our entire community for everything we have accomplished this year! To celebrate 2023, check out your own personal #OPLegendsRewind at Run /rewind in-game to get your unique code. We have so much planned for 2024, and can't wait for all of you to be a part of our incredible plans. We have celebrated a lot of amazing milestones in 2023, and we truly have the best community in Minecraft. Reply with your rewind URL and with #OPLegendsRewind for a chance to win one of 2X PERMANENT LEGEND RANKS, which will be drawn on January 2nd @ 3PM EST!

OPLegends

11,019 views • 2 years ago

Thanks for the tweet. We improve 3D Gaussain splatting for dash cam videos that contain serious reflections and obstructions. With Dash cam videos, it's possible to leverage massive driving data. Visit our page at Codes will be soon.
0:11

Thanks for the tweet. We improve 3D Gaussain splatting for dash cam videos that contain serious reflections and obstructions. With Dash cam videos, it's possible to leverage massive driving data. Visit our page at Codes will be soon.

Xiao-Xiao Long

44,444 views • 2 years ago

Apple just trained a 3D Gaussian head reconstruction model on 10,000+ subjects. Feed-forward. No test-time optimization. New identity in, reconstructed Gaussian head out. The UV-parameterized Gaussian representation decouples the number of Gaussians from the number and resolution of input images, making it practical to train with many high resolution views. And the heads are not just static either: text-conditioned identity generation, plus blendshape-driven latent animation across identities. We've been building in the 3D Gaussian Splatting space for a while. The gap between "research demo" and "works on real people at scale" is closing fast.
0:12

Apple just trained a 3D Gaussian head reconstruction model on 10,000+ subjects. Feed-forward. No test-time optimization. New identity in, reconstructed Gaussian head out. The UV-parameterized Gaussian representation decouples the number of Gaussians from the number and resolution of input images, making it practical to train with many high resolution views. And the heads are not just static either: text-conditioned identity generation, plus blendshape-driven latent animation across identities. We've been building in the 3D Gaussian Splatting space for a while. The gap between "research demo" and "works on real people at scale" is closing fast.

KIRI Engine - 3D Scanner App

12,181 views • 2 months ago

Look at our children enjoying a peaceful boat ride with their peers from other countries here in Rome. We will continue to make education fun, entertaining, and rewarding for our children. In 10 years, I pray you will be alive to see what this generation of our children will become through our sustained effort. They will be greater than we have ever been.
0:15

Look at our children enjoying a peaceful boat ride with their peers from other countries here in Rome. We will continue to make education fun, entertaining, and rewarding for our children. In 10 years, I pray you will be alive to see what this generation of our children will become through our sustained effort. They will be greater than we have ever been.

Alex Onyia

74,084 views • 27 days ago

Georgia Ellenwood Georgia Ellenwood wanted to check in and let you know she will be competing at the 2023 Bell Canadian Track & Field Championships in the Heptathlon at Langley, BC 📣 We also have a special surprise with Georgia coming in a few days… 👀
0:12

Georgia Ellenwood Georgia Ellenwood wanted to check in and let you know she will be competing at the 2023 Bell Canadian Track & Field Championships in the Heptathlon at Langley, BC 📣 We also have a special surprise with Georgia coming in a few days… 👀

Athletics Canada

19,422 views • 3 years ago

We got wagering real $KURO working You will be able to play Brawl against other players and bet real KURO. Time to roll it out for an internal test for our holders very soon We could potentially do wagers with any token tbh
0:16

We got wagering real $KURO working You will be able to play Brawl against other players and bet real KURO. Time to roll it out for an internal test for our holders very soon We could potentially do wagers with any token tbh

Beast Brawl

28,838 views • 8 months ago

👤: can you spoil a little bit what you prepared for the arena tour? 🐧: this is our first time saying this, we will be performing with dancers and a band. it’s coming up real soon, guys come and have fun with us~
0:15

👤: can you spoil a little bit what you prepared for the arena tour? 🐧: this is our first time saying this, we will be performing with dancers and a band. it’s coming up real soon, guys come and have fun with us~

na

10,446 views • 2 months ago