Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

Implicit Neural Light Spheres lets you turn panoramic captures into dynamic wide FOV renders (with real-time rendering!). Instead of generating panoramas with image stitching, we use neural light spheres to jointly estimate the camera path and a high-resolution scene reconstruction to produce novel wide field-of-view projections of the environment.... show more

Felix Heide

1,972 subscribers

10,301 görüntüleme • 1 yıl önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Splines instead of Gaussians 😉 Introducing Neural Spline Fields, which can see through occlusions! We learn to represent a stack of misaligned captures as a multi-layer image sandwich. Then you can extract your favorite layer to remove occlusions, reflections, or even your own shadows from the scene! Paper and Code: Fun work by Ilya Chugunov , David Shustin , Ruyu Yan, and Chenyang Lei.

Splines instead of Gaussians 😉 Introducing Neural Spline Fields, which can see through occlusions! We learn to represent a stack of misaligned captures as a multi-layer image sandwich. Then you can extract your favorite layer to remove occlusions, reflections, or even your own shadows from the scene! Paper and Code: Fun work by Ilya Chugunov , David Shustin , Ruyu Yan, and Chenyang Lei.

Felix Heide

137,306 görüntüleme • 2 yıl önce

Evaluating Neural Networks at the Speed of Light (with Light!). See live optical inference in the video below. Excited to share recent academic work on optical neural networks as a collection of computing elements embedded in the camera lens! These elements perform computation optically even before an image is captured, using the photons in the scene instead of GPU computation after the capture. We were able to achieve ImageNet classification more than two orders of magnitude faster than conventional neural networks on today's GPUs at almost no power consumption! To do this, we developed an array of metalenses that perform this computation on light from the scene. Project: Paper: Amazing collaboration with Kaixuan Wei, Xiao Li, Johannes Froech, Praneeth Chakravarthula, James Whitehead, Ethan Tseng , Arka Majumdar .

Evaluating Neural Networks at the Speed of Light (with Light!). See live optical inference in the video below. Excited to share recent academic work on optical neural networks as a collection of computing elements embedded in the camera lens! These elements perform computation optically even before an image is captured, using the photons in the scene instead of GPU computation after the capture. We were able to achieve ImageNet classification more than two orders of magnitude faster than conventional neural networks on today's GPUs at almost no power consumption! To do this, we developed an array of metalenses that perform this computation on light from the scene. Project: Paper: Amazing collaboration with Kaixuan Wei, Xiao Li, Johannes Froech, Praneeth Chakravarthula, James Whitehead, Ethan Tseng , Arka Majumdar .

Felix Heide

29,895 görüntüleme • 1 yıl önce

Fast View Synthesis of Casual Videos paper page: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.

Fast View Synthesis of Casual Videos paper page: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.

AK

20,668 görüntüleme • 2 yıl önce

EnvGS: Modeling View-Dependent Appearance with Environment Gaussian Contributions: • We propose a novel scene representation for accurately modeling complex near-field and high-frequency reflections in real-world environments. • We developed a real-time ray-tracing renderer for 2DGS, enabling joint optimization of our representation for accurate scene reconstruction while achieving real-time rendering speeds. • Extensive experiments show that EnvGS significantly outperforms previous methods. To the best of our knowledge, EnvGS is the first method to achieve real-time photorealistic specular reflections synthesis in real-world scenes.

EnvGS: Modeling View-Dependent Appearance with Environment Gaussian Contributions: • We propose a novel scene representation for accurately modeling complex near-field and high-frequency reflections in real-world environments. • We developed a real-time ray-tracing renderer for 2DGS, enabling joint optimization of our representation for accurate scene reconstruction while achieving real-time rendering speeds. • Extensive experiments show that EnvGS significantly outperforms previous methods. To the best of our knowledge, EnvGS is the first method to achieve real-time photorealistic specular reflections synthesis in real-world scenes.

MrNeRF

44,650 görüntüleme • 1 yıl önce

VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams paper page: Neural Radiance Fields (NeRFs) excel in photorealistically rendering static scenes. However, rendering dynamic, long-duration radiance fields on ubiquitous devices remains challenging, due to data storage and computational constraints. In this paper, we introduce VideoRF, the first approach to enable real-time streaming and rendering of dynamic radiance fields on mobile platforms. At the core is a serialized 2D feature image stream representing the 4D radiance field all in one. We introduce a tailored training scheme directly applied to this 2D domain to impose the temporal and spatial redundancy of the feature image stream. By leveraging the redundancy, we show that the feature image stream can be efficiently compressed by 2D video codecs, which allows us to exploit video hardware accelerators to achieve real-time decoding. On the other hand, based on the feature image stream, we propose a novel rendering pipeline for VideoRF, which has specialized space mappings to query radiance properties efficiently. Paired with a deferred shading model, VideoRF has the capability of real-time rendering on mobile devices thanks to its efficiency. We have developed a real-time interactive player that enables online streaming and rendering of dynamic scenes, offering a seamless and immersive free-viewpoint experience across a range of devices, from desktops to mobile phones.

VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams paper page: Neural Radiance Fields (NeRFs) excel in photorealistically rendering static scenes. However, rendering dynamic, long-duration radiance fields on ubiquitous devices remains challenging, due to data storage and computational constraints. In this paper, we introduce VideoRF, the first approach to enable real-time streaming and rendering of dynamic radiance fields on mobile platforms. At the core is a serialized 2D feature image stream representing the 4D radiance field all in one. We introduce a tailored training scheme directly applied to this 2D domain to impose the temporal and spatial redundancy of the feature image stream. By leveraging the redundancy, we show that the feature image stream can be efficiently compressed by 2D video codecs, which allows us to exploit video hardware accelerators to achieve real-time decoding. On the other hand, based on the feature image stream, we propose a novel rendering pipeline for VideoRF, which has specialized space mappings to query radiance properties efficiently. Paired with a deferred shading model, VideoRF has the capability of real-time rendering on mobile devices thanks to its efficiency. We have developed a real-time interactive player that enables online streaming and rendering of dynamic scenes, offering a seamless and immersive free-viewpoint experience across a range of devices, from desktops to mobile phones.

AK

38,686 görüntüleme • 2 yıl önce

SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes abs: paper page: Existing methods for the 4D reconstruction of general, non-rigidly deforming objects focus on novel-view synthesis and neglect correspondences. However, time consistency enables advanced downstream tasks like 3D editing, motion analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. It then reconstructs the deformations of an estimated canonical model of the geometry and appearance in an online fashion. Since this canonical model is time-invariant, we obtain correspondences even for long-term, long-range motions. We employ neural scene representations to parametrize the components of our method. Like prior dynamic-NeRF methods, we use a backwards deformation model. We find non-trivial adaptations of this model necessary to handle larger motions: We decompose the deformations into a strongly regularized coarse component and a weakly regularized fine component, where the coarse component also extends the deformation field into the space surrounding the object, which enables tracking over time. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.

SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes abs: paper page: Existing methods for the 4D reconstruction of general, non-rigidly deforming objects focus on novel-view synthesis and neglect correspondences. However, time consistency enables advanced downstream tasks like 3D editing, motion analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. It then reconstructs the deformations of an estimated canonical model of the geometry and appearance in an online fashion. Since this canonical model is time-invariant, we obtain correspondences even for long-term, long-range motions. We employ neural scene representations to parametrize the components of our method. Like prior dynamic-NeRF methods, we use a backwards deformation model. We find non-trivial adaptations of this model necessary to handle larger motions: We decompose the deformations into a strongly regularized coarse component and a weakly regularized fine component, where the coarse component also extends the deformation field into the space surrounding the object, which enables tracking over time. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.

AK

76,380 görüntüleme • 2 yıl önce

You might want to take more panoramas with your iPhone. No matter whether you own a visionOS device today or not, there's a good chance that one day you'll want to turn those panoramas into immersive 3D environments you can actually work in. To be honest, not every panorama in my library was as impressive as this one. To get the most out of the feature, you'll want a really wide, high-resolution panorama with a compelling scene. But when you have the right shot, it genuinely looks and feels like you're there.

You might want to take more panoramas with your iPhone. No matter whether you own a visionOS device today or not, there's a good chance that one day you'll want to turn those panoramas into immersive 3D environments you can actually work in. To be honest, not every panorama in my library was as impressive as this one. To get the most out of the feature, you'll want a really wide, high-resolution panorama with a compelling scene. But when you have the right shot, it genuinely looks and feels like you're there.

Phil Traut ᯅ

29,071 görüntüleme • 1 ay önce

$FAU Erlangen-Nürnberg presents TRIPS Trilinear Point Splatting for Real-Time Radiance Field Rendering paper page: Point-based radiance field rendering has demonstrated impressive results for novel view synthesis, offering a compelling blend of rendering quality and computational efficiency. However, also latest approaches in this domain are not without their shortcomings. 3D Gaussian Splatting [Kerbl and Kopanas et al. 2023] struggles when tasked with rendering highly detailed scenes, due to blurring and cloudy artifacts. On the other hand, ADOP [R\"uckert et al. 2022] can accommodate crisper images, but the neural reconstruction network decreases performance, it grapples with temporal instability and it is unable to effectively address large gaps in the point cloud. In this paper, we present TRIPS (Trilinear Point Splatting), an approach that combines ideas from both Gaussian Splatting and ADOP. The fundamental concept behind our novel technique involves rasterizing points into a screen-space image pyramid, with the selection of the pyramid layer determined by the projected point size. This approach allows rendering arbitrarily large points using a single trilinear write. A lightweight neural network is then used to reconstruct a hole-free image including detail beyond splat resolution. Importantly, our render pipeline is entirely differentiable, allowing for automatic optimization of both point sizes and positions. Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art methods in terms of rendering quality while maintaining a real-time frame rate of 60 frames per second on readily available hardware. This performance extends to challenging scenarios, such as scenes featuring intricate geometry, expansive landscapes, and auto-exposed footage.$

FAU Erlangen-Nürnberg presents TRIPS Trilinear Point Splatting for Real-Time Radiance Field Rendering paper page: Point-based radiance field rendering has demonstrated impressive results for novel view synthesis, offering a compelling blend of rendering quality and computational efficiency. However, also latest approaches in this domain are not without their shortcomings. 3D Gaussian Splatting [Kerbl and Kopanas et al. 2023] struggles when tasked with rendering highly detailed scenes, due to blurring and cloudy artifacts. On the other hand, ADOP [R\"uckert et al. 2022] can accommodate crisper images, but the neural reconstruction network decreases performance, it grapples with temporal instability and it is unable to effectively address large gaps in the point cloud. In this paper, we present TRIPS (Trilinear Point Splatting), an approach that combines ideas from both Gaussian Splatting and ADOP. The fundamental concept behind our novel technique involves rasterizing points into a screen-space image pyramid, with the selection of the pyramid layer determined by the projected point size. This approach allows rendering arbitrarily large points using a single trilinear write. A lightweight neural network is then used to reconstruct a hole-free image including detail beyond splat resolution. Importantly, our render pipeline is entirely differentiable, allowing for automatic optimization of both point sizes and positions. Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art methods in terms of rendering quality while maintaining a real-time frame rate of 60 frames per second on readily available hardware. This performance extends to challenging scenarios, such as scenes featuring intricate geometry, expansive landscapes, and auto-exposed footage.

AK

45,459 görüntüleme • 2 yıl önce

SharedNeRF: Leveraging Photorealistic and View-dependent Rendering for Real-time and Remote Collaboration Abstract (excerpt): When collaborators are remote, coordinating the sharing of views of their physical environment becomes challenging. Video-conferencing tools often do not provide the desired viewpoints for a remote viewer. While RGB-D cameras offer 3D views, they lack the necessary fidelity. We introduce SharedNeRF, designed to enhance synchronous remote collaboration by leveraging the photorealistic and view-dependent nature of Neural Radiance Field (NeRF). The system complements the higher visual quality of the NeRF rendering with the instantaneity of a point cloud and combines them through carefully accommodating the dynamic elements within the shared space, such as hand gestures and moving objects. The system employs a head-mounted camera for data collection, creating a volumetric task space on the fly and updating it as the task space changes.

SharedNeRF: Leveraging Photorealistic and View-dependent Rendering for Real-time and Remote Collaboration Abstract (excerpt): When collaborators are remote, coordinating the sharing of views of their physical environment becomes challenging. Video-conferencing tools often do not provide the desired viewpoints for a remote viewer. While RGB-D cameras offer 3D views, they lack the necessary fidelity. We introduce SharedNeRF, designed to enhance synchronous remote collaboration by leveraging the photorealistic and view-dependent nature of Neural Radiance Field (NeRF). The system complements the higher visual quality of the NeRF rendering with the instantaneity of a point cloud and combines them through carefully accommodating the dynamic elements within the shared space, such as hand gestures and moving objects. The system employs a head-mounted camera for data collection, creating a volumetric task space on the fly and updating it as the task space changes.

MrNeRF

10,615 görüntüleme • 1 yıl önce

URAvatar: Universal Relightable Gaussian Codec Avatars Contributions (cited): (1) We introduce a universal relightable avatar prior model learned from hundreds of dynamic performance captures with a multi-view and multi-light system. (2) We build a drivable head avatar from a phone scan that can be rendered and relit with global light transport in real-time. (3) A capture system and evaluation protocol to measure the accuracy of relighting under continuous illuminations.

MrNeRF

50,088 görüntüleme • 1 yıl önce

🚀 Excited to attend my first #NeurIPS2025 and present our lighting-aware SLAM method, NFL-BA! 🔗 🔦 Why this matters: Traditional Bundle Adjustment (BA) in SLAM assumes static lighting, but many real scenarios—endoscopy, search & rescue, subterranean robotics—use co-located light + camera, creating dynamic, near-field lighting that breaks these assumptions. ✨ What we introduce: Near-Field Lighting Bundle Adjustment Loss (NFL-BA) — a formulation that explicitly models near-field illumination inside the BA objective, allowing SLAM systems to jointly reason about geometry, appearance, and lighting. 📈 Results: Modeling lighting directly leads to ~38% improvement in mapping & tracking across dynamic-lighting sequences in both endoscopy and indoor scenes. 🧩 Plug-and-play: NFL-BA integrates seamlessly into existing neural rendering–based SLAM pipelines, using both implicit (NeRF-style) or explicit (3DGS) scene representations. 🌟 If you’re interested in SLAM, neural rendering, or illumination modeling, come check it out on Friday's poster session, 11-2pm, #4407.

🚀 Excited to attend my first #NeurIPS2025 and present our lighting-aware SLAM method, NFL-BA! 🔗 🔦 Why this matters: Traditional Bundle Adjustment (BA) in SLAM assumes static lighting, but many real scenarios—endoscopy, search & rescue, subterranean robotics—use co-located light + camera, creating dynamic, near-field lighting that breaks these assumptions. ✨ What we introduce: Near-Field Lighting Bundle Adjustment Loss (NFL-BA) — a formulation that explicitly models near-field illumination inside the BA objective, allowing SLAM systems to jointly reason about geometry, appearance, and lighting. 📈 Results: Modeling lighting directly leads to ~38% improvement in mapping & tracking across dynamic-lighting sequences in both endoscopy and indoor scenes. 🧩 Plug-and-play: NFL-BA integrates seamlessly into existing neural rendering–based SLAM pipelines, using both implicit (NeRF-style) or explicit (3DGS) scene representations. 🌟 If you’re interested in SLAM, neural rendering, or illumination modeling, come check it out on Friday's poster session, 11-2pm, #4407.

Roni Sengupta

21,162 görüntüleme • 8 ay önce

🍺 LagerNVS (CVPR 2026) 🍺 LagerNVS is a generalizable, feed-forward, real-time Novel View Synthesis network which - performs rendering in real time, - generalizes to in-the-wild data, - works with and without known source cameras, - sets a new state-of-the-art among deterministic methods, - can be paired with a diffusion decoder for generative extrapolation. LagerNVS shows that 3D biases are useful for Novel View Synthesis but explicit 3D representations are not required to achieve them. We use 3D biases in (1) architecture design and (2) pre-training: (1) In NVS with explicit 3D representations (3DGS, NeRF) reconstruction is typically difficult and slow, but rendering is much faster and simpler. We mimic this process in the network design: we use a large (1B params) encoder and a small, lightweight decoder (ViT-B). This allows increasing the network capacity while still achieving real-time rendering. (2) The encoder, initialized from VGGT, was pre-trained with 3D reconstruction objectives, making the initial features 3D aware. Both substantially improve performance. Project page: Code: Paper: Models: Work done with Jianyuan Minghao Chen Christian Rupprecht and Andrea Vedaldi

🍺 LagerNVS (CVPR 2026) 🍺 LagerNVS is a generalizable, feed-forward, real-time Novel View Synthesis network which - performs rendering in real time, - generalizes to in-the-wild data, - works with and without known source cameras, - sets a new state-of-the-art among deterministic methods, - can be paired with a diffusion decoder for generative extrapolation. LagerNVS shows that 3D biases are useful for Novel View Synthesis but explicit 3D representations are not required to achieve them. We use 3D biases in (1) architecture design and (2) pre-training: (1) In NVS with explicit 3D representations (3DGS, NeRF) reconstruction is typically difficult and slow, but rendering is much faster and simpler. We mimic this process in the network design: we use a large (1B params) encoder and a small, lightweight decoder (ViT-B). This allows increasing the network capacity while still achieving real-time rendering. (2) The encoder, initialized from VGGT, was pre-trained with 3D reconstruction objectives, making the initial features 3D aware. Both substantially improve performance. Project page: Code: Paper: Models: Work done with Jianyuan Minghao Chen Christian Rupprecht and Andrea Vedaldi

Stan Szymanowicz

31,651 görüntüleme • 4 ay önce

It's wild what you can one-shot with Gemini 3 and Nano Banana Pro in AIS. Here's an app to relight an image: > Use a 3D environment with a bust of a person and a light source, I should be able to move the light source around to cast shadows in different ways, add multiple light sources, etc. Then I should be able to capture the scene, pass it alongside an input image to Nano Banana Pro to change the lighting on an input

It's wild what you can one-shot with Gemini 3 and Nano Banana Pro in AIS. Here's an app to relight an image: > Use a 3D environment with a bust of a person and a light source, I should be able to move the light source around to cast shadows in different ways, add multiple light sources, etc. Then I should be able to capture the scene, pass it alongside an input image to Nano Banana Pro to change the lighting on an input

fofr

25,907 görüntüleme • 7 ay önce

SpecNeRF: Gaussian Directional Encoding for Specular Reflections paper page: Neural radiance fields have achieved remarkable performance in modeling the appearance of 3D scenes. However, existing approaches still struggle with the view-dependent appearance of glossy surfaces, especially under complex lighting of indoor environments. Unlike existing methods, which typically assume distant lighting like an environment map, we propose a learnable Gaussian directional encoding to better model the view-dependent effects under near-field lighting conditions. Importantly, our new directional encoding captures the spatially-varying nature of near-field lighting and emulates the behavior of prefiltered environment maps. As a result, it enables the efficient evaluation of preconvolved specular color at any 3D location with varying roughness coefficients. We further introduce a data-driven geometry prior that helps alleviate the shape radiance ambiguity in reflection modeling. We show that our Gaussian directional encoding and geometry prior significantly improve the modeling of challenging specular reflections in neural radiance fields, which helps decompose appearance into more physically meaningful components.

SpecNeRF: Gaussian Directional Encoding for Specular Reflections paper page: Neural radiance fields have achieved remarkable performance in modeling the appearance of 3D scenes. However, existing approaches still struggle with the view-dependent appearance of glossy surfaces, especially under complex lighting of indoor environments. Unlike existing methods, which typically assume distant lighting like an environment map, we propose a learnable Gaussian directional encoding to better model the view-dependent effects under near-field lighting conditions. Importantly, our new directional encoding captures the spatially-varying nature of near-field lighting and emulates the behavior of prefiltered environment maps. As a result, it enables the efficient evaluation of preconvolved specular color at any 3D location with varying roughness coefficients. We further introduce a data-driven geometry prior that helps alleviate the shape radiance ambiguity in reflection modeling. We show that our Gaussian directional encoding and geometry prior significantly improve the modeling of challenging specular reflections in neural radiance fields, which helps decompose appearance into more physically meaningful components.

AK

33,933 görüntüleme • 2 yıl önce

The internet is flush with creatives debating the merits of text-to-video models like Sora, but this is the type of AI everyone can get behind. Teleporting your talent into virtual environments is typically a tall order. Green screen is easy, but re-lighting your subject to match a dynamic 3D environment is painful. Skin, hair, clothing all interact with lighting differently. Professionals often rely on a fancy light stage or LED capture volume (ala. Mandalorian), combined with a ton of manual compositing. Meanwhile, ML-based approaches use simplified physical models and are limited by training data. Beeble + NYU is pushing the boundaries of virtual production – making these advanced techniques accessible to all creators, while giving them fine-grain control. It's not just a PBR shader; they utilize neural rendering to emulate light transport effects like sub-surface scattering – so when light interacts with your skin, you don’t look like a waxed up cadaver in Madame Tussauds :) More technical deets below

The internet is flush with creatives debating the merits of text-to-video models like Sora, but this is the type of AI everyone can get behind. Teleporting your talent into virtual environments is typically a tall order. Green screen is easy, but re-lighting your subject to match a dynamic 3D environment is painful. Skin, hair, clothing all interact with lighting differently. Professionals often rely on a fancy light stage or LED capture volume (ala. Mandalorian), combined with a ton of manual compositing. Meanwhile, ML-based approaches use simplified physical models and are limited by training data. Beeble + NYU is pushing the boundaries of virtual production – making these advanced techniques accessible to all creators, while giving them fine-grain control. It's not just a PBR shader; they utilize neural rendering to emulate light transport effects like sub-surface scattering – so when light interacts with your skin, you don’t look like a waxed up cadaver in Madame Tussauds :) More technical deets below

Bilawal Sidhu

67,952 görüntüleme • 2 yıl önce

Unlike ray tracing or path tracing, which require additional runtime computation, this lighting is based on implied patterns, so it incurs the same cost as generating a scene without lighting. As these pattern interpreters scale to recognize more complex patterns, we'll achieve more and better implied interactions essentially for 'free' with each generation. What’s even better is that once we start working with scenes composed of tokenized assets or dynamic neural assets, we’ll have greater control than with traditional pipelines. Moreover, instead of starting with isolated objects and having to apply interactions manually, we'll have assets that naturally interact with their environment unless we explicitly specify otherwise.

Unlike ray tracing or path tracing, which require additional runtime computation, this lighting is based on implied patterns, so it incurs the same cost as generating a scene without lighting. As these pattern interpreters scale to recognize more complex patterns, we'll achieve more and better implied interactions essentially for 'free' with each generation. What’s even better is that once we start working with scenes composed of tokenized assets or dynamic neural assets, we’ll have greater control than with traditional pipelines. Moreover, instead of starting with isolated objects and having to apply interactions manually, we'll have assets that naturally interact with their environment unless we explicitly specify otherwise.

Jon Finger

95,451 görüntüleme • 1 yıl önce

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors paper page: present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images.

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors paper page: present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images.

AK

305,663 görüntüleme • 3 yıl önce

Users state the goal. xBubble handles the building path. With Bubble Code, xBubble helps users create different kinds of Code AI Agents & AI roducts. The key is not just generating a prototype. Bubble Code is built to turn user goals into ready-to-use products with real interfaces, logic, integrations, and deployment paths. AI products should not stop at a mockup. With xBubble, they can be built, deployed, and used directly.

Users state the goal. xBubble handles the building path. With Bubble Code, xBubble helps users create different kinds of Code AI Agents & AI roducts. The key is not just generating a prototype. Bubble Code is built to turn user goals into ready-to-use products with real interfaces, logic, integrations, and deployment paths. AI products should not stop at a mockup. With xBubble, they can be built, deployed, and used directly.

DAPPOS

96,529 görüntüleme • 23 gün önce

Large-scale 3D Scene Generation (all scenes are real-time rendered)!! Physically-grounded generative data without hallucinations is the missing link for robot learning and testing at scale. We introduce a method that directly generates large-scale 3D driving scenes with accurate geometry, allowing for causal view synthesis and generation with object permanence and explicit 3D geometry. This also allows for extreme trajectory extrapolation without failure! We also show that we can build fully data-driven simulators for end-to-end learning with this approach. Project: with the amazing team of Julian Ost, Amogh Joshi , Andrea Ramazzina, Maximilian Bömer, Mario Bijelic.

Large-scale 3D Scene Generation (all scenes are real-time rendered)!! Physically-grounded generative data without hallucinations is the missing link for robot learning and testing at scale. We introduce a method that directly generates large-scale 3D driving scenes with accurate geometry, allowing for causal view synthesis and generation with object permanence and explicit 3D geometry. This also allows for extreme trajectory extrapolation without failure! We also show that we can build fully data-driven simulators for end-to-end learning with this approach. Project: with the amazing team of Julian Ost, Amogh Joshi , Andrea Ramazzina, Maximilian Bömer, Mario Bijelic.

Felix Heide

27,779 görüntüleme • 10 ay önce

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

AK

62,768 görüntüleme • 3 yıl önce