Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

White Light Reference for Machine Learning. Meet our inhouse tech Doggo "Rolo". "More Doggo than Doggo" Since July we've been redesigning our scanning pipeline to work with the new 3D Gaussian Splatting for Real-Time Radiance Field Rendering method from Inria. IR's AeonX capture system has been designed to capture... show more

Infinite-Realities

8,710 subscribers

236,811 views • 2 years ago •via X (Twitter)

Science & Technology Health & Wellness Arts

Anya Rossi• Live Now

Private livecam show

11 Comments

Pianaland 🇪🇺 🇺🇦 🇹🇼🇮🇱2 years ago

Is the fur just geometry? Traditional 3d softwares go arround the theme with a huge variety of solutions. Or this just a reference fo then make a proper 3d model (im imagine that what the cam can’t see there are no data / no model)

Infinite-Realities2 years ago

This is more useful for reference as it's an approximation albeit a very good looking one but I bet the folks at Epic could work some interesting magic with it if they had a commercial license to integrate the method into Unreal.

Dan Lowe2 years ago

That's crazy! Great test subject: The hair detail and shading really shows this approach off.

Furkan Gözükara2 years ago

I read the Readme but confusing So you provided photos of this dog trained model and then viewed with viewer? How is the workflow?

Infinite-Realities2 years ago

The images are aligned in either Reality Capture, or Metashape, or Colmap. They are trained with the Guassian Splatting process then viewed in a modified version of Sibr.

Zvezdan Nedeljkovic2 years ago

I am currently trying to figure out how to install everything and make my first model. As I figuren you need colmap to feed it to the gaussin splatting? What do you export from Reality Capture? Great results btw :)

Infinite-Realities2 years ago

Thanks we don't have a defined pipeline that we can share for the moment but I think others have started to document and share their pipeline. I think more resources will come online soon. We plan to create an install write up at some point.

Giddy Kong: The Gaming Ape2 years ago

You're showing us that Rolo isn't real... but my brain refuses to listen.

Vlad Erium 🇯🇵2 years ago

Looks cool!

darthgera1232 years ago

Hi sorry I dont understand but are these renderings with multiple light conditions?

Infinite-Realities2 years ago

These are viewed in real-time baked lighting results.

Related Videos

Meet our inhouse tech Doggo "Rolo". Cinematic RGB Lighting "More Doggo than Doggo" Since July we've been redesigning our scanning pipeline to work with the amazing 3D Gaussian Splatting for Real-Time Radiance Field Rendering method from Inria. IR's AeonX capture system has been designed to capture "Spatial" memories for the future. We've built on a custom version of Inria's Sibr viewer to handle the playback of our multiple processed 4D frames. GS has reignited our passion for 3D/4D scanning. Thank you to the team at Inria More content to come. #guassiansplatting #ir #aeonx #4d #motionscanning #inria #ximea #idatronic #spatial #youth83

Meet our inhouse tech Doggo "Rolo". Cinematic RGB Lighting "More Doggo than Doggo" Since July we've been redesigning our scanning pipeline to work with the amazing 3D Gaussian Splatting for Real-Time Radiance Field Rendering method from Inria. IR's AeonX capture system has been designed to capture "Spatial" memories for the future. We've built on a custom version of Inria's Sibr viewer to handle the playback of our multiple processed 4D frames. GS has reignited our passion for 3D/4D scanning. Thank you to the team at Inria More content to come. #guassiansplatting #ir #aeonx #4d #motionscanning #inria #ximea #idatronic #spatial #youth83

Infinite-Realities

98,702 views • 2 years ago

"More Doggo than Doggo" Motion scanned @ 60fps. Playing back @ 20fps. Meet our inhouse tech Doggo "Rolo". Since July we've been redesigning our scanning pipeline to work with the new 3D Gaussian Splatting for Real-Time Radiance Field Rendering method from Inria. GS has reignited our passion for 3D/4D scanning. Thank you to the team at Inria IR's AeonX capture system has been designed to capture "Spatial" memories for the future. We've built on a custom version of Inria's Sibr viewer to handle the playback of our multiple processed 4D frames. Our next goal is to optimize the player to be multi-threaded to improve performance over 60fps. GS has reignited our passion for 3D/4D scanning. Thank you to the team at Inria More content to come. #guassiansplatting #ir #aeonx #4d #motionscanning #inria #ximea #idatronic #spatial #youth83

"More Doggo than Doggo" Motion scanned @ 60fps. Playing back @ 20fps. Meet our inhouse tech Doggo "Rolo". Since July we've been redesigning our scanning pipeline to work with the new 3D Gaussian Splatting for Real-Time Radiance Field Rendering method from Inria. GS has reignited our passion for 3D/4D scanning. Thank you to the team at Inria IR's AeonX capture system has been designed to capture "Spatial" memories for the future. We've built on a custom version of Inria's Sibr viewer to handle the playback of our multiple processed 4D frames. Our next goal is to optimize the player to be multi-threaded to improve performance over 60fps. GS has reignited our passion for 3D/4D scanning. Thank you to the team at Inria More content to come. #guassiansplatting #ir #aeonx #4d #motionscanning #inria #ximea #idatronic #spatial #youth83

Infinite-Realities

48,329 views • 2 years ago

We're proud to showcase some recent real-time work with 3D / 4D Gaussian splatting. In January of 2024 we acquired a full 3DGS commercial license with Inria to allow us to offer 4DGS services to clients around the world. We've spent the last 10 months developing our custom 4DGS software pipeline, utilizing our own Nvidia GPU cluster to batch process 100,000's of 4DGS frames rapidly, combined with our state-of-the-art custom built volumetric capture system(s). We can capture anything from complex character interactions for visual effects shots, or dynamic fast fighting scenes, sports, acting, even capturing and storing memories of family and friends for the future! The possibilities are endless. Our focus is always on high-quality and high-fidelity. We don't cut corners and we're continually striving to learn and improve. The below video showcases some 6-dof spatial captures we took of Henry Pearce's family. These scenes are played back in real-time @ 30fps inside a 3rd party engine running in a 120fps virtual environment. This is a culmination of years of hard work and dedication and we are keen to commercialize this technology with you. Please do reach out if you're interested in our capture systems or our 3D/4DGS processing services. HQ YouTube Link: HQ Vimeo Link: #spatialcapture #vr #virtualmemories #inria #3dgs #4dgs #aeonx #ir #ximea Music "Vision" by megiddo music Special thanks to Aras Pranckevičius 🇺🇦🇱🇹

We're proud to showcase some recent real-time work with 3D / 4D Gaussian splatting. In January of 2024 we acquired a full 3DGS commercial license with Inria to allow us to offer 4DGS services to clients around the world. We've spent the last 10 months developing our custom 4DGS software pipeline, utilizing our own Nvidia GPU cluster to batch process 100,000's of 4DGS frames rapidly, combined with our state-of-the-art custom built volumetric capture system(s). We can capture anything from complex character interactions for visual effects shots, or dynamic fast fighting scenes, sports, acting, even capturing and storing memories of family and friends for the future! The possibilities are endless. Our focus is always on high-quality and high-fidelity. We don't cut corners and we're continually striving to learn and improve. The below video showcases some 6-dof spatial captures we took of Henry Pearce's family. These scenes are played back in real-time @ 30fps inside a 3rd party engine running in a 120fps virtual environment. This is a culmination of years of hard work and dedication and we are keen to commercialize this technology with you. Please do reach out if you're interested in our capture systems or our 3D/4DGS processing services. HQ YouTube Link: HQ Vimeo Link: #spatialcapture #vr #virtualmemories #inria #3dgs #4dgs #aeonx #ir #ximea Music "Vision" by megiddo music Special thanks to Aras Pranckevičius 🇺🇦🇱🇹

Infinite-Realities

202,783 views • 2 years ago

4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models Contributions: • We introduce 4D LangSplat for open-vocabulary 4D spatial-temporal queries. To the best of our knowledge, we are the first to construct 4D language fields with object textual captions generated by MLLMs. • To model smooth transitions across states for objects in 4D scenes, we propose a status deformable network to capture continuous temporal changes. • Experiential results show that our method attains state-of-the-art performance for both time-agnostic and time-sensitive open-vocabulary queries.

4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models Contributions: • We introduce 4D LangSplat for open-vocabulary 4D spatial-temporal queries. To the best of our knowledge, we are the first to construct 4D language fields with object textual captions generated by MLLMs. • To model smooth transitions across states for objects in 4D scenes, we propose a status deformable network to capture continuous temporal changes. • Experiential results show that our method attains state-of-the-art performance for both time-agnostic and time-sensitive open-vocabulary queries.

MrNeRF

10,961 views • 1 year ago

Enter the SPLATRIX! We digitized Henry Pearce and converted him over to 4DGS to be played back in real-time. 3D Gaussians Splats in Motion. 4DGS. Our goal was to improve the quality of our AeonX & 4DGS pipeline, with automated background removal to retain fine hair details as well as to improve overall surface consistency on a full torso capture. We're currently using a post process to temporally filter the 2D frames. We still need to solve 4DGS temporal stability. We're hoping to test with Jonathon Luiten dynamic gaussian code soon... 🙏 for that script! We captured this performance under global illumination. This is an easy task compared to RGB+W lighting, which is going to be much harder! This is our next challenge. We're posting these snippet videos to show our progress with 3D/4DGS. We plan to write a full blog post on our 3D/4DGS journey soon. 3D Gaussian Splatting #GaussianSplatting #inria #radiancefields #ir #aeonx #rgbw #idatronic #ximea #sibr #hdri #60fps More info here - Higher Quality Videos:

Enter the SPLATRIX! We digitized Henry Pearce and converted him over to 4DGS to be played back in real-time. 3D Gaussians Splats in Motion. 4DGS. Our goal was to improve the quality of our AeonX & 4DGS pipeline, with automated background removal to retain fine hair details as well as to improve overall surface consistency on a full torso capture. We're currently using a post process to temporally filter the 2D frames. We still need to solve 4DGS temporal stability. We're hoping to test with Jonathon Luiten dynamic gaussian code soon... 🙏 for that script! We captured this performance under global illumination. This is an easy task compared to RGB+W lighting, which is going to be much harder! This is our next challenge. We're posting these snippet videos to show our progress with 3D/4DGS. We plan to write a full blog post on our 3D/4DGS journey soon. 3D Gaussian Splatting #GaussianSplatting #inria #radiancefields #ir #aeonx #rgbw #idatronic #ximea #sibr #hdri #60fps More info here - Higher Quality Videos:

Infinite-Realities

90,662 views • 2 years ago

Happy to announce the results of our latest research, which takes 3D Gaussian Splatting to the next level: "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets," which has been accepted at #SIGGRAPH2024!🎉 Find it here:

Happy to announce the results of our latest research, which takes 3D Gaussian Splatting to the next level: "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets," which has been accepted at #SIGGRAPH2024!🎉 Find it here:

Bernhard Kerbl

356,390 views • 2 years ago

From Mitch: Founder of Shiba Rescue of New Jersey At Shiba Rescue of New Jersey, we sincerely thank Prince and the #DOGGO crypto community on ETH for your generous donation. Your support strengthens our mission and demonstrates how our community can come together to make a difference for Shibas in need. Together, we’re ensuring that every dog has a loving home. Thank you for being part of this journey! This one is for you Shibetoshi Nakamoto from our #DOGGO Community ❤️

From Mitch: Founder of Shiba Rescue of New Jersey At Shiba Rescue of New Jersey, we sincerely thank Prince and the #DOGGO crypto community on ETH for your generous donation. Your support strengthens our mission and demonstrates how our community can come together to make a difference for Shibas in need. Together, we’re ensuring that every dog has a loving home. Thank you for being part of this journey! This one is for you Shibetoshi Nakamoto from our #DOGGO Community ❤️

Prince

55,143 views • 1 year ago

To our Wicked fans, As 2024 is coming to a close, we want to take the time to thank you for joining us on our journey with No Rest for the Wicked. This has been a huge year for all of us at Moon Studios. Launching Wicked in Early Access in April was quite the beginning. We've been listening to your feedback very closely, rapidly rolling out hotfixes, balancing, optimizing and adding quality-of-life improvements to the game. With The Crucible Update we added a whole new rogue-lite mode, setting the stage for more replayability out of the existing Early Access content. Since then we've been hard at work on “The Breach” and our next big Content and Feature updates. We know we’ve been quiet - a lot has been happening for us at Moon Studios during this year, and we will finally be able to share all about it very soon! We will be kicking off 2025 with a new announcement in January and you can also expect better and more frequent communication on our Official Channels throughout the year. Thank you for sticking with us and helping shape No Rest for the Wicked into something truly special. Your passion, feedback and support continue to further fuel our passion since the very first announcement at the The Game Awards. We couldn’t be more proud of our team at Moon Studios and everything we have in store for 2025. Here’s to an even bigger year for No Rest for the Wicked, Moon Studios and all of you! Happy holidays! 🎉🎊🎁 The Moon Studios Team ❤️

To our Wicked fans, As 2024 is coming to a close, we want to take the time to thank you for joining us on our journey with No Rest for the Wicked. This has been a huge year for all of us at Moon Studios. Launching Wicked in Early Access in April was quite the beginning. We've been listening to your feedback very closely, rapidly rolling out hotfixes, balancing, optimizing and adding quality-of-life improvements to the game. With The Crucible Update we added a whole new rogue-lite mode, setting the stage for more replayability out of the existing Early Access content. Since then we've been hard at work on “The Breach” and our next big Content and Feature updates. We know we’ve been quiet - a lot has been happening for us at Moon Studios during this year, and we will finally be able to share all about it very soon! We will be kicking off 2025 with a new announcement in January and you can also expect better and more frequent communication on our Official Channels throughout the year. Thank you for sticking with us and helping shape No Rest for the Wicked into something truly special. Your passion, feedback and support continue to further fuel our passion since the very first announcement at the The Game Awards. We couldn’t be more proud of our team at Moon Studios and everything we have in store for 2025. Here’s to an even bigger year for No Rest for the Wicked, Moon Studios and all of you! Happy holidays! 🎉🎊🎁 The Moon Studios Team ❤️

No Rest for the Wicked

73,244 views • 1 year ago

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We introduced a novel density control strategy in training, which allows our 4DGT to handle longer space-time input while maintaining efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can significantly outperform prior Gaussian-based networks in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We introduced a novel density control strategy in training, which allows our 4DGT to handle longer space-time input while maintaining efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can significantly outperform prior Gaussian-based networks in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.

MrNeRF

34,782 views • 1 year ago

Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video Contribution quote from the paper: In summary, our main contributions are • a comprehensive pipeline for reconstructing the shape, appearance, and behavior of real-world garments using Gaussian splatting, • an algorithm for registering garment meshes to multi- view videos with an optimization procedure based on Gaussian splatting, and • a Gaussian Garment representation that combines triangle meshes with Gaussian textures to capture photorealistic appearance and can be used as a fully controllable 3D asset.

Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video Contribution quote from the paper: In summary, our main contributions are • a comprehensive pipeline for reconstructing the shape, appearance, and behavior of real-world garments using Gaussian splatting, • an algorithm for registering garment meshes to multi- view videos with an optimization procedure based on Gaussian splatting, and • a Gaussian Garment representation that combines triangle meshes with Gaussian textures to capture photorealistic appearance and can be used as a fully controllable 3D asset.

MrNeRF

27,277 views • 1 year ago

🚀 Rocket League has been our arena for greatness, and today, we bid a heartfelt farewell. 🎮💔 As we reflect on the incredible journey of the last two seasons, we want to extend our deepest gratitude to our dedicated players, Coaches, content creators, and the talented casters who've been part of this unforgettable ride. 🙌 🌟 Your dedication, passion, and support have fueled our success and made Team AXLE a force to be reckoned with in the Rocket League world. 🌐 While this chapter comes to an end, the memories we've created together will live on forever. Thank you for being a part of our story. 🚀💥 #FullThrottle

🚀 Rocket League has been our arena for greatness, and today, we bid a heartfelt farewell. 🎮💔 As we reflect on the incredible journey of the last two seasons, we want to extend our deepest gratitude to our dedicated players, Coaches, content creators, and the talented casters who've been part of this unforgettable ride. 🙌 🌟 Your dedication, passion, and support have fueled our success and made Team AXLE a force to be reckoned with in the Rocket League world. 🌐 While this chapter comes to an end, the memories we've created together will live on forever. Thank you for being a part of our story. 🚀💥 #FullThrottle

TEAM AXLE

39,398 views • 2 years ago

Introducing WorldView 3D, a first-of-its-kind satellite tasking product that delivers up-to-date 3D ground truth anywhere on Earth. From GPS-denied autonomy and intelligence analysis to consumer-grade mapping and disaster response, the world's most important missions increasingly depend on accurate, up-to-date 3D terrain. But for years, keeping 3D maps current at global scale hasn't been possible. WorldView 3D changes that. Built on Vantor’s trusted spatial foundation, our new tasking capability empowers customers to task and receive updated 3D data within 24 hours of image collection. For missions that require greater fidelity, they can also produce high-definition 3D maps anywhere in the world. 3D is the future of spatial intelligence, and Vantor is leading the change. Our trusted spatial foundation already includes 100 million sq km of the world mapped inhighly accurate 3D. Now customers can update the locations that matter most with the speed, scale, and fidelity required for critical missions. Read the full announcement:

Introducing WorldView 3D, a first-of-its-kind satellite tasking product that delivers up-to-date 3D ground truth anywhere on Earth. From GPS-denied autonomy and intelligence analysis to consumer-grade mapping and disaster response, the world's most important missions increasingly depend on accurate, up-to-date 3D terrain. But for years, keeping 3D maps current at global scale hasn't been possible. WorldView 3D changes that. Built on Vantor’s trusted spatial foundation, our new tasking capability empowers customers to task and receive updated 3D data within 24 hours of image collection. For missions that require greater fidelity, they can also produce high-definition 3D maps anywhere in the world. 3D is the future of spatial intelligence, and Vantor is leading the change. Our trusted spatial foundation already includes 100 million sq km of the world mapped inhighly accurate 3D. Now customers can update the locations that matter most with the speed, scale, and fidelity required for critical missions. Read the full announcement:

Vantor

16,030 views • 1 month ago

Introducing Teleport 2.0, a next-generation release for our 3D capture solution that sets a new benchmark in cloud-based Gaussian Splatting. Teleport 2.0 is all about lowering the barrier of entry to create and share lifelike 3D captures. New capabilities include: ▪️ Upgraded 3D reconstruction engine: Preserves real-world detail, even in large-scale scenes that can be captured in minutes and automatically processed without manual steps. ▪️Support for any camera: Capture with phones, drones, professional cameras, or other imaging devices. ▪️ Tools for creating engaging immersive experiences: Build virtual tours, or link multiple scans together to build massive explorable digital twins. Read more: #3D #DigitalTwin #GaussianSplatting

Introducing Teleport 2.0, a next-generation release for our 3D capture solution that sets a new benchmark in cloud-based Gaussian Splatting. Teleport 2.0 is all about lowering the barrier of entry to create and share lifelike 3D captures. New capabilities include: ▪️ Upgraded 3D reconstruction engine: Preserves real-world detail, even in large-scale scenes that can be captured in minutes and automatically processed without manual steps. ▪️Support for any camera: Capture with phones, drones, professional cameras, or other imaging devices. ▪️ Tools for creating engaging immersive experiences: Build virtual tours, or link multiple scans together to build massive explorable digital twins. Read more: #3D #DigitalTwin #GaussianSplatting

Varjo

21,211 views • 1 year ago

Another city. Another full 3D Gaussian Splat reconstruction. This entire neighborhood was captured while walking with an Insta360 X5, then processed with our GPU pipeline and transformed into a high-fidelity, machine-readable 3D environment with Over the Reality. What you’re seeing is not an isolated virtual world. It’s a 3D digital twin of a real-world location, built from structured spatial data where every building, street, and detail can become usable by AI, robotics, and XR. And it’s happening at scale, across cities in more than 180 countries every day. Have an Insta360 X5 and want to become a mapper? Apply here: Want to build a 3D Gaussian Splat from a video you already recorded? Do it for free at powered by Over the Reality.

Another city. Another full 3D Gaussian Splat reconstruction. This entire neighborhood was captured while walking with an Insta360 X5, then processed with our GPU pipeline and transformed into a high-fidelity, machine-readable 3D environment with Over the Reality. What you’re seeing is not an isolated virtual world. It’s a 3D digital twin of a real-world location, built from structured spatial data where every building, street, and detail can become usable by AI, robotics, and XR. And it’s happening at scale, across cities in more than 180 countries every day. Have an Insta360 X5 and want to become a mapper? Apply here: Want to build a 3D Gaussian Splat from a video you already recorded? Do it for free at powered by Over the Reality.

Over the Reality 🌐

400,097 views • 3 months ago

We’ve just open-sourced JOSH, our method for 4D human–scene reconstruction in the wild. Huge credit to the first author, my PhD student Zhizheng, who will present this work at ICLR in Brazil next month 🇧🇷 🌐 Project page: 💻 Code: A trivia is that this work was finished more than one year ago, and has been rejected twice...Be resilient!

We’ve just open-sourced JOSH, our method for 4D human–scene reconstruction in the wild. Huge credit to the first author, my PhD student Zhizheng, who will present this work at ICLR in Brazil next month 🇧🇷 🌐 Project page: 💻 Code: A trivia is that this work was finished more than one year ago, and has been rejected twice...Be resilient!

Bolei Zhou

25,539 views • 4 months ago

2014: David Cameron, "We don't need any more on-shore wind.. What's we've got in the planning system is enough to provide for our needs" 2024: Ed Miliband, "The on shore wind ban in England has been in place for more than 9 years. We've been in government for 72 hours, we've lifted it.. We've got to become a clean energy super power for energy independence.. To cut bills.. To create good jobs.. And to tackle the climate crisis" We've lost a decade of green energy revolution because of the Conservative party. Where could we be now if Cameron didn't appease climate skeptics and the fossil fuel industry? Good work Labour 👏

2014: David Cameron, "We don't need any more on-shore wind.. What's we've got in the planning system is enough to provide for our needs" 2024: Ed Miliband, "The on shore wind ban in England has been in place for more than 9 years. We've been in government for 72 hours, we've lifted it.. We've got to become a clean energy super power for energy independence.. To cut bills.. To create good jobs.. And to tackle the climate crisis" We've lost a decade of green energy revolution because of the Conservative party. Where could we be now if Cameron didn't appease climate skeptics and the fossil fuel industry? Good work Labour 👏

Farrukh

799,216 views • 2 years ago

GS^3: Efficient Relighting with Triple Gaussian Splatting Abstract: We present a spatial and angular Gaussian based representation and a triple splatting process, for real-time, high-quality novel lighting-and-view synthesis from multi-view point-lit input images. To describe complex ap pearance, we employ a Lambertian plus a mixture of angular Gaussians as an effective reflectance function for each spatial Gaussian. To generate self-shadow, we splat all spatial Gaussians towards the light source to obtain shadow values, which are further refined by a small multi-layer perceptron. To compensate for other effects like global illumination, another network is trained to compute and add a per-spatial-Gaussian RGB tuple. The effectiveness of our representation is demonstrated on 30 samples with a wide variation in geometry (from solid to fluffy) and appearance (from translucent to anisotropic), as well as using different forms of input data, including rendered images of synthetic/reconstructed objects, photographs captured with a handheld camera and a flash, or from a professional lightstage. We achieve a training time of 40-70 minutes and a rendering speed of 90 fps on a single commodity GPU. Our results compare favorably with state-of-the-art techniques in terms of quality/performance.

GS^3: Efficient Relighting with Triple Gaussian Splatting Abstract: We present a spatial and angular Gaussian based representation and a triple splatting process, for real-time, high-quality novel lighting-and-view synthesis from multi-view point-lit input images. To describe complex ap pearance, we employ a Lambertian plus a mixture of angular Gaussians as an effective reflectance function for each spatial Gaussian. To generate self-shadow, we splat all spatial Gaussians towards the light source to obtain shadow values, which are further refined by a small multi-layer perceptron. To compensate for other effects like global illumination, another network is trained to compute and add a per-spatial-Gaussian RGB tuple. The effectiveness of our representation is demonstrated on 30 samples with a wide variation in geometry (from solid to fluffy) and appearance (from translucent to anisotropic), as well as using different forms of input data, including rendered images of synthetic/reconstructed objects, photographs captured with a handheld camera and a flash, or from a professional lightstage. We achieve a training time of 40-70 minutes and a rendering speed of 90 fps on a single commodity GPU. Our results compare favorably with state-of-the-art techniques in terms of quality/performance.

MrNeRF

17,786 views • 1 year ago

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.

AK

327,105 views • 2 years ago

Rabbi Ellen has been my rabbi for over 30 years. Officiated our wedding, hung the mezzuzah on our door, named our kids and officiated their B’nei Mitzvah. Rabbi Ellen is a critical member of the Jewish community in our city — from teaching the core values of Judaism, to being a champion of peace for people across the world — New York wouldn’t be the same without her. Thank you for taking us down memory lane for a minute! I look forward to standing alongside side for many more years to come.

Rabbi Ellen has been my rabbi for over 30 years. Officiated our wedding, hung the mezzuzah on our door, named our kids and officiated their B’nei Mitzvah. Rabbi Ellen is a critical member of the Jewish community in our city — from teaching the core values of Judaism, to being a champion of peace for people across the world — New York wouldn’t be the same without her. Thank you for taking us down memory lane for a minute! I look forward to standing alongside side for many more years to come.

Brad Lander

15,201 views • 5 months ago

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.

AK

140,992 views • 2 years ago