Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Whilst we figure out a solution for temporal filtering on 3D Gaussian Splatting input frames. We can use a temporal filtering plugin on the screen captures or rendered images to act as a visualization tool to check what the future results might look like. "Processing and rendering a digital...

8Infinite8's profile picture

Infinite-Realities

8,738 subscribers

58,368 Aufrufe • vor 2 Jahren •via X (Twitter)

Wissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

9 Kommentare

Profilbild von thebensimon
thebensimonvor 2 Jahren

Wow! The face details! Spectacular

Profilbild von 3D Scanstore
3D Scanstorevor 2 Jahren

Clean!

Profilbild von catid (e/acc)
catid (e/acc)vor 2 Jahren

Awesome

Profilbild von Mathias Røyrvik
Mathias Røyrvikvor 2 Jahren

I fucking love what you mad tinkerers do. I can't wait to see what you end up with.

Profilbild von Olli Huttunen
Olli Huttunenvor 2 Jahren

Just wow! This tech is going so fast right now! Every day major leaps forward. So exciting times!

Profilbild von Infinite-Realities
Infinite-Realitiesvor 2 Jahren

Thanks for the positive comment Olli! The Inria team created such an amazing tool to experiment with.

Profilbild von Alex Coulombe
Alex Coulombevor 2 Jahren

Hollly cow

Profilbild von arbitrarity
arbitrarityvor 2 Jahren

I'm gonna take a wild guess and say your capture setup for something like this is rather expensive, or quite large, yes? Never having attempted a capture, I'm mostly curious what the key is to ensuring the capture data is high quality? Is it mostly resolution and quantity based? Any resources are appreciated!

Profilbild von Earl Cameron
Earl Cameronvor 2 Jahren

Cc @SebAaltonen

Ähnliche Videos

1000+ FPS 4D Gaussian Splatting for Dynamic Scene Rendering Contributions: • We delve into the temporal redundancy of 4D Gaussian Splatting and explain the main reason for the storage pressure and suboptimal rendering speed. • We introduce 4DGS-1K, a compact and memory-efficient framework to address these issues. It consists of two key components: a spatial-temporal variation score-based pruning strategy and a temporal filter. • Extensive experiments demonstrate that 4DGS-1K not only achieves a substantial storage reduction of approximately 41× but also accelerates rasterization to 1000+ FPS while maintaining high-quality reconstruction.
0:25

1000+ FPS 4D Gaussian Splatting for Dynamic Scene Rendering Contributions: • We delve into the temporal redundancy of 4D Gaussian Splatting and explain the main reason for the storage pressure and suboptimal rendering speed. • We introduce 4DGS-1K, a compact and memory-efficient framework to address these issues. It consists of two key components: a spatial-temporal variation score-based pruning strategy and a temporal filter. • Extensive experiments demonstrate that 4DGS-1K not only achieves a substantial storage reduction of approximately 41× but also accelerates rasterization to 1000+ FPS while maintaining high-quality reconstruction.

MrNeRF

12,366 Aufrufe • vor 1 Jahr

[SIGGRAPH '26] Anchored Temporal Gaussian Splatting for Long Volumetric Video Representation TL;DR: We present ATGS, a novel framework for volumetric video reconstruction that effectively handles long sequences and complex motions. By utilizing time-conditioned anchors and a temporal windowing strategy, ATGS enhances temporal coherence and scalability. Abstract (excerpt): Key insight is that explicitly tracking long term complex motion with individual Gaussian primitives is inherently unstable. Instead, we organize Gaussians around time conditioned anchors that localize their spatial and temporal support, thereby reducing long range motion complexity. We further introduce a temporal windowing strategy to activate only anchors relevant to the queried time, which improves scalability and temporal coherence. In addition, to ensure spatial and temporal stability, we design a compact set of multi level anchor features that encode global features, local spatial features, and local temporal features, jointly constraining Gaussian generation. Extensive experiments demonstrate that ATGS consistently outperforms prior methods on long sequence volumetric videos with complex motions.
0:40

[SIGGRAPH '26] Anchored Temporal Gaussian Splatting for Long Volumetric Video Representation TL;DR: We present ATGS, a novel framework for volumetric video reconstruction that effectively handles long sequences and complex motions. By utilizing time-conditioned anchors and a temporal windowing strategy, ATGS enhances temporal coherence and scalability. Abstract (excerpt): Key insight is that explicitly tracking long term complex motion with individual Gaussian primitives is inherently unstable. Instead, we organize Gaussians around time conditioned anchors that localize their spatial and temporal support, thereby reducing long range motion complexity. We further introduce a temporal windowing strategy to activate only anchors relevant to the queried time, which improves scalability and temporal coherence. In addition, to ensure spatial and temporal stability, we design a compact set of multi level anchor features that encode global features, local spatial features, and local temporal features, jointly constraining Gaussian generation. Extensive experiments demonstrate that ATGS consistently outperforms prior methods on long sequence volumetric videos with complex motions.

MrNeRF

27,839 Aufrufe • vor 4 Monaten

🕹️We are excited to introduce "ChronoEdit: Towards Temporal Reasoning for Image Editing and World Simulation" ChronoEdit reframes image editing as a video generation task to encourage temporal consistency. It leverages a temporal reasoning stage that denoises with “video reasoning tokens” to "reason" on physically plausible edits. See the attached video for results. Project Page: Arxiv: Code and model are coming.
1:38

🕹️We are excited to introduce "ChronoEdit: Towards Temporal Reasoning for Image Editing and World Simulation" ChronoEdit reframes image editing as a video generation task to encourage temporal consistency. It leverages a temporal reasoning stage that denoises with “video reasoning tokens” to "reason" on physically plausible edits. See the attached video for results. Project Page: Arxiv: Code and model are coming.

Huan Ling

36,895 Aufrufe • vor 10 Monaten

Coding some shaders for 3D Gaussian Splatting 😀 Input is a 40 second video processed with Luma Check the reflections on the bag! Demo:
0:20

Coding some shaders for 3D Gaussian Splatting 😀 Input is a 40 second video processed with Luma Check the reflections on the bag! Demo:

Alexandre Devaux

44,165 Aufrufe • vor 2 Jahren

Enter the SPLATRIX! We digitized Henry Pearce and converted him over to 4DGS to be played back in real-time. 3D Gaussians Splats in Motion. 4DGS. Our goal was to improve the quality of our AeonX & 4DGS pipeline, with automated background removal to retain fine hair details as well as to improve overall surface consistency on a full torso capture. We're currently using a post process to temporally filter the 2D frames. We still need to solve 4DGS temporal stability. We're hoping to test with Jonathon Luiten dynamic gaussian code soon... 🙏 for that script! We captured this performance under global illumination. This is an easy task compared to RGB+W lighting, which is going to be much harder! This is our next challenge. We're posting these snippet videos to show our progress with 3D/4DGS. We plan to write a full blog post on our 3D/4DGS journey soon. 3D Gaussian Splatting #GaussianSplatting #inria #radiancefields #ir #aeonx #rgbw #idatronic #ximea #sibr #hdri #60fps More info here - Higher Quality Videos:
1:14

Enter the SPLATRIX! We digitized Henry Pearce and converted him over to 4DGS to be played back in real-time. 3D Gaussians Splats in Motion. 4DGS. Our goal was to improve the quality of our AeonX & 4DGS pipeline, with automated background removal to retain fine hair details as well as to improve overall surface consistency on a full torso capture. We're currently using a post process to temporally filter the 2D frames. We still need to solve 4DGS temporal stability. We're hoping to test with Jonathon Luiten dynamic gaussian code soon... 🙏 for that script! We captured this performance under global illumination. This is an easy task compared to RGB+W lighting, which is going to be much harder! This is our next challenge. We're posting these snippet videos to show our progress with 3D/4DGS. We plan to write a full blog post on our 3D/4DGS journey soon. 3D Gaussian Splatting #GaussianSplatting #inria #radiancefields #ir #aeonx #rgbw #idatronic #ximea #sibr #hdri #60fps More info here - Higher Quality Videos:

Infinite-Realities

90,662 Aufrufe • vor 2 Jahren

Glad to have the opportunity to collaborate with the World Labs team on their new tool Marble! It’s currently the best tool for generating 3D worlds with Gaussian Splatting. Here are some of the results I’ve been getting and putting together in a video.
1:12

Glad to have the opportunity to collaborate with the World Labs team on their new tool Marble! It’s currently the best tool for generating 3D worlds with Gaussian Splatting. Here are some of the results I’ve been getting and putting together in a video.

Cristian Peñas ░░░░░░░░

42,022 Aufrufe • vor 10 Monaten

📢 I’m testing ViggleAI new AI tool, PINOC, and it’s seriously impressive. You can create 3D Gaussian splatting models from just one image and animate them with a video! It’s the first AI I’ve tried that can animate Gaussian splatting models. I also talked to someone from Viggle, and they might add the option to import your own Gaussian splatting models and animate them. If that happens, I’ll probably make a video tutorial showing how to boost the detail level of a Gaussian splatting model by 4x to create highly realistic animated characters.
1:06

📢 I’m testing ViggleAI new AI tool, PINOC, and it’s seriously impressive. You can create 3D Gaussian splatting models from just one image and animate them with a video! It’s the first AI I’ve tried that can animate Gaussian splatting models. I also talked to someone from Viggle, and they might add the option to import your own Gaussian splatting models and animate them. If that happens, I’ll probably make a video tutorial showing how to boost the detail level of a Gaussian splatting model by 4x to create highly realistic animated characters.

Alex

29,776 Aufrufe • vor 2 Monaten

4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models Contributions: • We introduce 4D LangSplat for open-vocabulary 4D spatial-temporal queries. To the best of our knowledge, we are the first to construct 4D language fields with object textual captions generated by MLLMs. • To model smooth transitions across states for objects in 4D scenes, we propose a status deformable network to capture continuous temporal changes. • Experiential results show that our method attains state-of-the-art performance for both time-agnostic and time-sensitive open-vocabulary queries.
0:29

4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models Contributions: • We introduce 4D LangSplat for open-vocabulary 4D spatial-temporal queries. To the best of our knowledge, we are the first to construct 4D language fields with object textual captions generated by MLLMs. • To model smooth transitions across states for objects in 4D scenes, we propose a status deformable network to capture continuous temporal changes. • Experiential results show that our method attains state-of-the-art performance for both time-agnostic and time-sensitive open-vocabulary queries.

MrNeRF

10,961 Aufrufe • vor 1 Jahr

Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video Contribution quote from the paper: In summary, our main contributions are • a comprehensive pipeline for reconstructing the shape, appearance, and behavior of real-world garments using Gaussian splatting, • an algorithm for registering garment meshes to multi- view videos with an optimization procedure based on Gaussian splatting, and • a Gaussian Garment representation that combines triangle meshes with Gaussian textures to capture photorealistic appearance and can be used as a fully controllable 3D asset.
0:20

Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video Contribution quote from the paper: In summary, our main contributions are • a comprehensive pipeline for reconstructing the shape, appearance, and behavior of real-world garments using Gaussian splatting, • an algorithm for registering garment meshes to multi- view videos with an optimization procedure based on Gaussian splatting, and • a Gaussian Garment representation that combines triangle meshes with Gaussian textures to capture photorealistic appearance and can be used as a fully controllable 3D asset.

MrNeRF

27,277 Aufrufe • vor 1 Jahr

Meet our inhouse tech Doggo "Rolo". Cinematic RGB Lighting "More Doggo than Doggo" Since July we've been redesigning our scanning pipeline to work with the amazing 3D Gaussian Splatting for Real-Time Radiance Field Rendering method from Inria. IR's AeonX capture system has been designed to capture "Spatial" memories for the future. We've built on a custom version of Inria's Sibr viewer to handle the playback of our multiple processed 4D frames. GS has reignited our passion for 3D/4D scanning. Thank you to the team at Inria More content to come. #guassiansplatting #ir #aeonx #4d #motionscanning #inria #ximea #idatronic #spatial #youth83
1:57

Meet our inhouse tech Doggo "Rolo". Cinematic RGB Lighting "More Doggo than Doggo" Since July we've been redesigning our scanning pipeline to work with the amazing 3D Gaussian Splatting for Real-Time Radiance Field Rendering method from Inria. IR's AeonX capture system has been designed to capture "Spatial" memories for the future. We've built on a custom version of Inria's Sibr viewer to handle the playback of our multiple processed 4D frames. GS has reignited our passion for 3D/4D scanning. Thank you to the team at Inria More content to come. #guassiansplatting #ir #aeonx #4d #motionscanning #inria #ximea #idatronic #spatial #youth83

Infinite-Realities

98,702 Aufrufe • vor 3 Jahren

📢Happy to present Convex Splatting, a novel way for 3D reconstruction based on 3D smooth convexes. For the first time, a splatting-based method reaches the quality of NeRF sota methods but with real-time rendering and few primitives!! I expect this to replace Gaussian splatting for 3D in the coming months. CODE RELEASED TODAY! joint work with collaborators from Université de Liège Visual Geometry Group (VGG) , KAUST Computer Vision Lab (IVUL) a thread 🧵 1/n
0:38

📢Happy to present Convex Splatting, a novel way for 3D reconstruction based on 3D smooth convexes. For the first time, a splatting-based method reaches the quality of NeRF sota methods but with real-time rendering and few primitives!! I expect this to replace Gaussian splatting for 3D in the coming months. CODE RELEASED TODAY! joint work with collaborators from Université de Liège Visual Geometry Group (VGG) , KAUST Computer Vision Lab (IVUL) a thread 🧵 1/n

Abdullah Hamdi

71,661 Aufrufe • vor 1 Jahr

[SIGGRAPH Asia '24 (TOG)] Representing Long Volumetric Video with Temporal Gaussian Hierarchy Contributions: • We introduce a novel, efficient, and expressive Temporal Gaussian Hierarchy representation for long volumetric video. To our knowledge, our method is the first approach capable of handling minutes of volumetric video data. • We propose a Compact Appearance Model and a new rasterization implementation to facilitate real-time, high-quality dynamic view synthesis while maintaining a compact size. • We propose a system to efficiently model long volumetric videos for the first time and demonstrate state-of-the-art dynamic view synthesis quality on the Neural3DV [Li et al. 2022], ENeRF-Outdoor [Lin et al. 2022], and MobileStage [Xu et al. 2024b] datasets, while also achieving the best rendering speed with reduced training cost and memory usage.
0:53

[SIGGRAPH Asia '24 (TOG)] Representing Long Volumetric Video with Temporal Gaussian Hierarchy Contributions: • We introduce a novel, efficient, and expressive Temporal Gaussian Hierarchy representation for long volumetric video. To our knowledge, our method is the first approach capable of handling minutes of volumetric video data. • We propose a Compact Appearance Model and a new rasterization implementation to facilitate real-time, high-quality dynamic view synthesis while maintaining a compact size. • We propose a system to efficiently model long volumetric videos for the first time and demonstrate state-of-the-art dynamic view synthesis quality on the Neural3DV [Li et al. 2022], ENeRF-Outdoor [Lin et al. 2022], and MobileStage [Xu et al. 2024b] datasets, while also achieving the best rendering speed with reduced training cost and memory usage.

MrNeRF

79,445 Aufrufe • vor 1 Jahr

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.
6:20

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.

AK

327,105 Aufrufe • vor 2 Jahren

White Light Reference for Machine Learning. Meet our inhouse tech Doggo "Rolo". "More Doggo than Doggo" Since July we've been redesigning our scanning pipeline to work with the new 3D Gaussian Splatting for Real-Time Radiance Field Rendering method from Inria. IR's AeonX capture system has been designed to capture "Spatial" memories for the future. We've built on a custom version of Inria's Sibr viewer to handle the playback of our multiple processed 4D frames. GS has reignited our passion for 3D/4D scanning. Thank you to the team at Inria More content to come.. #guassiansplatting #ir #aeonx #4d #motionscanning #inria #ximea #idatronic #spatial #youth83
1:16

White Light Reference for Machine Learning. Meet our inhouse tech Doggo "Rolo". "More Doggo than Doggo" Since July we've been redesigning our scanning pipeline to work with the new 3D Gaussian Splatting for Real-Time Radiance Field Rendering method from Inria. IR's AeonX capture system has been designed to capture "Spatial" memories for the future. We've built on a custom version of Inria's Sibr viewer to handle the playback of our multiple processed 4D frames. GS has reignited our passion for 3D/4D scanning. Thank you to the team at Inria More content to come.. #guassiansplatting #ir #aeonx #4d #motionscanning #inria #ximea #idatronic #spatial #youth83

Infinite-Realities

236,811 Aufrufe • vor 3 Jahren

GS^3: Efficient Relighting with Triple Gaussian Splatting Abstract: We present a spatial and angular Gaussian based representation and a triple splatting process, for real-time, high-quality novel lighting-and-view synthesis from multi-view point-lit input images. To describe complex ap pearance, we employ a Lambertian plus a mixture of angular Gaussians as an effective reflectance function for each spatial Gaussian. To generate self-shadow, we splat all spatial Gaussians towards the light source to obtain shadow values, which are further refined by a small multi-layer perceptron. To compensate for other effects like global illumination, another network is trained to compute and add a per-spatial-Gaussian RGB tuple. The effectiveness of our representation is demonstrated on 30 samples with a wide variation in geometry (from solid to fluffy) and appearance (from translucent to anisotropic), as well as using different forms of input data, including rendered images of synthetic/reconstructed objects, photographs captured with a handheld camera and a flash, or from a professional lightstage. We achieve a training time of 40-70 minutes and a rendering speed of 90 fps on a single commodity GPU. Our results compare favorably with state-of-the-art techniques in terms of quality/performance.
0:40

GS^3: Efficient Relighting with Triple Gaussian Splatting Abstract: We present a spatial and angular Gaussian based representation and a triple splatting process, for real-time, high-quality novel lighting-and-view synthesis from multi-view point-lit input images. To describe complex ap pearance, we employ a Lambertian plus a mixture of angular Gaussians as an effective reflectance function for each spatial Gaussian. To generate self-shadow, we splat all spatial Gaussians towards the light source to obtain shadow values, which are further refined by a small multi-layer perceptron. To compensate for other effects like global illumination, another network is trained to compute and add a per-spatial-Gaussian RGB tuple. The effectiveness of our representation is demonstrated on 30 samples with a wide variation in geometry (from solid to fluffy) and appearance (from translucent to anisotropic), as well as using different forms of input data, including rendered images of synthetic/reconstructed objects, photographs captured with a handheld camera and a flash, or from a professional lightstage. We achieve a training time of 40-70 minutes and a rendering speed of 90 fps on a single commodity GPU. Our results compare favorably with state-of-the-art techniques in terms of quality/performance.

MrNeRF

17,786 Aufrufe • vor 1 Jahr

Is 3D scene generation much closer to being solved all of a sudden? It has been a few days since the release of OpenAI Sora. We run our COLMAP-Free 3D Gaussian Splatting on the released videos. Our method does not need to pre-process cameras and it seems we can directly just get 3D from the videos. Check out our results here. 🧵👇 (1/n
0:25

Is 3D scene generation much closer to being solved all of a sudden? It has been a few days since the release of OpenAI Sora. We run our COLMAP-Free 3D Gaussian Splatting on the released videos. Our method does not need to pre-process cameras and it seems we can directly just get 3D from the videos. Check out our results here. 🧵👇 (1/n

Xiaolong Wang

157,395 Aufrufe • vor 2 Jahren

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model with Gradio demo local demo: This paper studies the human image animation task, which aims to generate a video of a certain reference identity following a particular motion sequence. Existing animation works typically employ the frame-warping technique to animate the reference image towards the target motion. Despite achieving reasonable results, these approaches face challenges in maintaining temporal consistency throughout the animation due to the lack of temporal modeling and poor preservation of reference identity. In this work, we introduce MagicAnimate, a diffusion-based framework that aims at enhancing temporal consistency, preserving reference image faithfully, and improving animation fidelity. To achieve this, we first develop a video diffusion model to encode temporal information. Second, to maintain the appearance coherence across frames, we introduce a novel appearance encoder to retain the intricate details of the reference image. Leveraging these two innovations, we further employ a simple video fusion technique to encourage smooth transitions for long video animation. Empirical results demonstrate the superiority of our method over baseline approaches on two benchmarks. Notably, our approach outperforms the strongest baseline by over 38% in terms of video fidelity on the challenging TikTok dancing dataset. Code and model will be made available.
1:15

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model with Gradio demo local demo: This paper studies the human image animation task, which aims to generate a video of a certain reference identity following a particular motion sequence. Existing animation works typically employ the frame-warping technique to animate the reference image towards the target motion. Despite achieving reasonable results, these approaches face challenges in maintaining temporal consistency throughout the animation due to the lack of temporal modeling and poor preservation of reference identity. In this work, we introduce MagicAnimate, a diffusion-based framework that aims at enhancing temporal consistency, preserving reference image faithfully, and improving animation fidelity. To achieve this, we first develop a video diffusion model to encode temporal information. Second, to maintain the appearance coherence across frames, we introduce a novel appearance encoder to retain the intricate details of the reference image. Leveraging these two innovations, we further employ a simple video fusion technique to encourage smooth transitions for long video animation. Empirical results demonstrate the superiority of our method over baseline approaches on two benchmarks. Notably, our approach outperforms the strongest baseline by over 38% in terms of video fidelity on the challenging TikTok dancing dataset. Code and model will be made available.

AK

810,578 Aufrufe • vor 2 Jahren

GPS-Gaussian+: Generalizable Pixel-wise 3D Gaussian Splatting for Real-Time Human-Scene Rendering from Sparse Views TL;DR: Are we witnessing the first steps towards 3DGS live streaming? Contributions: • We introduce a generalizable 3D Gaussian Splatting methodology that employs pixel-wise Gaussian parameter maps defined on 2D source image planes to formulate 3D Gaussians in a feed-forward manner. • We propose a fully differentiable framework composed of an iterative depth estimation module and a Gaussian parameter regression module. The intermediate depth prediction bridges the two components and allows them to benefit from joint training. • We introduce a regularization term and an epipolar attention mechanism to preserve geometry consistency between the two source views when using only rendering loss. Our method generalizes well to unseen characters even in complicated scenes. • We develop a real-time FVV system that achieves high-resolution rendering of characters in the scene without any geometry supervision.
0:23

GPS-Gaussian+: Generalizable Pixel-wise 3D Gaussian Splatting for Real-Time Human-Scene Rendering from Sparse Views TL;DR: Are we witnessing the first steps towards 3DGS live streaming? Contributions: • We introduce a generalizable 3D Gaussian Splatting methodology that employs pixel-wise Gaussian parameter maps defined on 2D source image planes to formulate 3D Gaussians in a feed-forward manner. • We propose a fully differentiable framework composed of an iterative depth estimation module and a Gaussian parameter regression module. The intermediate depth prediction bridges the two components and allows them to benefit from joint training. • We introduce a regularization term and an epipolar attention mechanism to preserve geometry consistency between the two source views when using only rendering loss. Our method generalizes well to unseen characters even in complicated scenes. • We develop a real-time FVV system that achieves high-resolution rendering of characters in the scene without any geometry supervision.

MrNeRF

25,862 Aufrufe • vor 1 Jahr

FAU Erlangen-Nürnberg presents TRIPS Trilinear Point Splatting for Real-Time Radiance Field Rendering paper page: Point-based radiance field rendering has demonstrated impressive results for novel view synthesis, offering a compelling blend of rendering quality and computational efficiency. However, also latest approaches in this domain are not without their shortcomings. 3D Gaussian Splatting [Kerbl and Kopanas et al. 2023] struggles when tasked with rendering highly detailed scenes, due to blurring and cloudy artifacts. On the other hand, ADOP [R\"uckert et al. 2022] can accommodate crisper images, but the neural reconstruction network decreases performance, it grapples with temporal instability and it is unable to effectively address large gaps in the point cloud. In this paper, we present TRIPS (Trilinear Point Splatting), an approach that combines ideas from both Gaussian Splatting and ADOP. The fundamental concept behind our novel technique involves rasterizing points into a screen-space image pyramid, with the selection of the pyramid layer determined by the projected point size. This approach allows rendering arbitrarily large points using a single trilinear write. A lightweight neural network is then used to reconstruct a hole-free image including detail beyond splat resolution. Importantly, our render pipeline is entirely differentiable, allowing for automatic optimization of both point sizes and positions. Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art methods in terms of rendering quality while maintaining a real-time frame rate of 60 frames per second on readily available hardware. This performance extends to challenging scenarios, such as scenes featuring intricate geometry, expansive landscapes, and auto-exposed footage.
0:21

FAU Erlangen-Nürnberg presents TRIPS Trilinear Point Splatting for Real-Time Radiance Field Rendering paper page: Point-based radiance field rendering has demonstrated impressive results for novel view synthesis, offering a compelling blend of rendering quality and computational efficiency. However, also latest approaches in this domain are not without their shortcomings. 3D Gaussian Splatting [Kerbl and Kopanas et al. 2023] struggles when tasked with rendering highly detailed scenes, due to blurring and cloudy artifacts. On the other hand, ADOP [R\"uckert et al. 2022] can accommodate crisper images, but the neural reconstruction network decreases performance, it grapples with temporal instability and it is unable to effectively address large gaps in the point cloud. In this paper, we present TRIPS (Trilinear Point Splatting), an approach that combines ideas from both Gaussian Splatting and ADOP. The fundamental concept behind our novel technique involves rasterizing points into a screen-space image pyramid, with the selection of the pyramid layer determined by the projected point size. This approach allows rendering arbitrarily large points using a single trilinear write. A lightweight neural network is then used to reconstruct a hole-free image including detail beyond splat resolution. Importantly, our render pipeline is entirely differentiable, allowing for automatic optimization of both point sizes and positions. Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art methods in terms of rendering quality while maintaining a real-time frame rate of 60 frames per second on readily available hardware. This performance extends to challenging scenarios, such as scenes featuring intricate geometry, expansive landscapes, and auto-exposed footage.

AK

45,475 Aufrufe • vor 2 Jahren