Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Check out our fun work on robust view synthesis! 🤩 Using a casual video as input, our method jointly 1⃣estimates accurate camera poses and 2⃣reconstructs multiple local radiance fields for a large scene. Project🕸️:

Jia-Bin Huang

79,005 subscribers

35,526 Aufrufe • vor 3 Jahren •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

8 Kommentare

Profilbild von Jia-Bin Huang

Jia-Bin Huangvor 3 Jahren

HOW-TO: Estimating accurate poses? 👉 Progressively optimize both radiance fields and poses. HOW-TO: Modeling large scenes? 👉 Dynamically allocate local radiance fields

Profilbild von Jia-Bin Huang

Jia-Bin Huangvor 3 Jahren

How could we handle large scenes? 🤔 a) Normalized Device Coordinate (NDC): 👉 Only work for forward-facing scenes b) Space contraction: 👉 Work best for inward-facing scenes c) Generic scenes: 👉 Allocate multiple local radiance fields along the camera trajectory

Profilbild von Jia-Bin Huang

Jia-Bin Huangvor 3 Jahren

This is @AndreasMeuleman's intern project with Meta last summer, in collaboration with many talented people: Yu-Lun Liu, @gaochen315, @_ChangilKim, Min H. Kim, @JPKopf We will release both the code and data soon! 😍 Stay tuned!

Profilbild von Jia-Bin Huang

Jia-Bin Huangvor 3 Jahren

We will be presenting this paper at #CVPR2023! Looking forward to seeing you all there. Come chat with us for any questions!

Profilbild von Jia-Bin Huang

Jia-Bin Huangvor 3 Jahren

Oh my, the results look amazing! 😍 But can video stabilization methods work just as well? 🤔 Unfortunately, *2D* video stabilization methods cannot handle these large viewpoint changes. Look how shaky the results (from Fusta are on these casual videos!

Profilbild von Jia-Bin Huang

Jia-Bin Huangvor 3 Jahren

Also check out another excellent work (also at #CVPR2023) with a similar capability of handling free camera trajectories!

Profilbild von Anand Bhattad

Anand Bhattadvor 3 Jahren

Looks really good and stable!! Congrats🎉

Profilbild von binz

binzvor 3 Jahren

Results look great, congrats to the team! Will be excited to play around with the repo once released.

Ähnliche Videos

Progressively Optimized Local Radiance Fields for Robust View Synthesis paper page: present an algorithm for reconstructing the radiance field of a large-scale scene from a single casually captured video. The task poses two core challenges. First, most existing radiance field reconstruction approaches rely on accurate pre-estimated camera poses from Structure-from-Motion algorithms, which frequently fail on in-the-wild videos. Second, using a single, global radiance field with finite representational capacity does not scale to longer trajectories in an unbounded scene. For handling unknown poses, we jointly estimate the camera poses with radiance field in a progressive manner. We show that progressive optimization significantly improves the robustness of the reconstruction. For handling large unbounded scenes, we dynamically allocate new local radiance fields trained with frames within a temporal window. This further improves robustness (e.g., performs well even under moderate pose drifts) and allows us to scale to large scenes. Our extensive evaluation on the Tanks and Temples dataset and our collected outdoor dataset, Static Hikes, show that our approach compares favorably with the state-of-the-art.

Progressively Optimized Local Radiance Fields for Robust View Synthesis paper page: present an algorithm for reconstructing the radiance field of a large-scale scene from a single casually captured video. The task poses two core challenges. First, most existing radiance field reconstruction approaches rely on accurate pre-estimated camera poses from Structure-from-Motion algorithms, which frequently fail on in-the-wild videos. Second, using a single, global radiance field with finite representational capacity does not scale to longer trajectories in an unbounded scene. For handling unknown poses, we jointly estimate the camera poses with radiance field in a progressive manner. We show that progressive optimization significantly improves the robustness of the reconstruction. For handling large unbounded scenes, we dynamically allocate new local radiance fields trained with frames within a temporal window. This further improves robustness (e.g., performs well even under moderate pose drifts) and allows us to scale to large scenes. Our extensive evaluation on the Tanks and Temples dataset and our collected outdoor dataset, Static Hikes, show that our approach compares favorably with the state-of-the-art.

AK

140,616 Aufrufe • vor 3 Jahren

New Video! Excited to share our work on fast view synthesis of casual videos. 🤩 high-quality dynamic view synthesis 🤩 applicable for a wide range of casual videos 🤩 fast training and rendering

New Video! Excited to share our work on fast view synthesis of casual videos. 🤩 high-quality dynamic view synthesis 🤩 applicable for a wide range of casual videos 🤩 fast training and rendering

Jia-Bin Huang

27,360 Aufrufe • vor 2 Jahren

📢Introducing 360Anything, our method for lifting any perspective image or video to gravity-aligned 360° panoramas without using any camera or 3D information. This enables consistent novel view synthesis and 3D scene reconstruction. Project page: 🧵

📢Introducing 360Anything, our method for lifting any perspective image or video to gravity-aligned 360° panoramas without using any camera or 3D information. This enables consistent novel view synthesis and 3D scene reconstruction. Project page: 🧵

Ziyi Wu

46,859 Aufrufe • vor 4 Monaten

[SIGGRAPH Asia '25] Editable Physically-based Reflections in Raytraced Gaussian Radiance Fields Contributions: • A reconstruction method for radiance fields, with distinct optimization for diffuse and specular components, using path tracing for the latter. • An efficient and accurate training method that reconstructs the diffuse and specular components of the scene in a single representation. • An efficient ray tracer for Gaussian primitives, fast enough to enable treatment of multiple bounces with minimal computational overhead.

[SIGGRAPH Asia '25] Editable Physically-based Reflections in Raytraced Gaussian Radiance Fields Contributions: • A reconstruction method for radiance fields, with distinct optimization for diffuse and specular components, using path tracing for the latter. • An efficient and accurate training method that reconstructs the diffuse and specular components of the scene in a single representation. • An efficient ray tracer for Gaussian primitives, fast enough to enable treatment of multiple bounces with minimal computational overhead.

MrNeRF

15,810 Aufrufe • vor 8 Monaten

Excited to share our work InfiniDepth (CVPR 2026) — casting monocular depth estimation as a neural implicit field, which enables: 🔍 Arbitrary-Resolution 📐 Accurate Metric Depth 📷 Large-View Novel View Synthesis Feel free to try our code:

Excited to share our work InfiniDepth (CVPR 2026) — casting monocular depth estimation as a neural implicit field, which enables: 🔍 Arbitrary-Resolution 📐 Accurate Metric Depth 📷 Large-View Novel View Synthesis Feel free to try our code:

Sida Peng

38,675 Aufrufe • vor 2 Monaten

I'm excited to share our new work Align3R that estimates camera poses and consistent depth maps from a monocular video of a dynamic scene. Project page: Code: Paper:

I'm excited to share our new work Align3R that estimates camera poses and consistent depth maps from a monocular video of a dynamic scene. Project page: Code: Paper:

Yuan Liu

56,547 Aufrufe • vor 1 Jahr

Capturing out-of-focus photos is sooo frustrating! 😩 Check out our work on defocus control with dual-camera 📸 #CVPR2023 Our method can refocus and adjust the depth of field AFTER you capture the photograph! 🤩 Paper: Project:

Capturing out-of-focus photos is sooo frustrating! 😩 Check out our work on defocus control with dual-camera 📸 #CVPR2023 Our method can refocus and adjust the depth of field AFTER you capture the photograph! 🤩 Paper: Project:

Jia-Bin Huang

41,330 Aufrufe • vor 3 Jahren

Stable Virtual Camera: Generative View Synthesis with Diffusion Models Hint: Check the project website. It is awesome! Contributions: 1. A training strategy for jointly modeling large viewpoint changes and temporal smoothness. 2. A two-pass procedural sampling method for smooth video generation along arbitrarily long camera trajectories. 3. A comprehensive benchmark that evaluates NVS methods across different datasets and settings. 4. An open-source release of model weights to support future research.

Stable Virtual Camera: Generative View Synthesis with Diffusion Models Hint: Check the project website. It is awesome! Contributions: 1. A training strategy for jointly modeling large viewpoint changes and temporal smoothness. 2. A two-pass procedural sampling method for smooth video generation along arbitrarily long camera trajectories. 3. A comprehensive benchmark that evaluates NVS methods across different datasets and settings. 4. An open-source release of model weights to support future research.

MrNeRF

38,619 Aufrufe • vor 1 Jahr

Synthesia research releases HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion a 4D dynamic neural scene representation that captures full-body appearance in motion from multi-view video input, and enables playback from novel, unseen viewpoints project page: paper page:

Synthesia research releases HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion a 4D dynamic neural scene representation that captures full-body appearance in motion from multi-view video input, and enables playback from novel, unseen viewpoints project page: paper page:

AK

160,700 Aufrufe • vor 3 Jahren

Excited to share 🎨🖌️ 3D Paintbrush - a method for generating local stylized textures on meshes using text as input! Our method predicts a localization map & a highly detailed texture map which conforms to it (1/3)

Excited to share 🎨🖌️ 3D Paintbrush - a method for generating local stylized textures on meshes using text as input! Our method predicts a localization map & a highly detailed texture map which conforms to it (1/3)

Rana Hanocka

48,987 Aufrufe • vor 2 Jahren

Dense observations with perfect camera poses are hard to collect. What is a more practical setting for large-scale general object reconstruction? Presenting FORGE - jointly recovering object shape and camera poses from few-view observations, and it generalizes to novel categories in zero-shot. Project page & code: Joint work with Zhenyu Jiang, Kristen Grauman, Yuke Zhu

Dense observations with perfect camera poses are hard to collect. What is a more practical setting for large-scale general object reconstruction? Presenting FORGE - jointly recovering object shape and camera poses from few-view observations, and it generalizes to novel categories in zero-shot. Project page & code: Joint work with Zhenyu Jiang, Kristen Grauman, Yuke Zhu

Hanwen Jiang

18,116 Aufrufe • vor 3 Jahren

Design development for Out, a large scale audiovisual instalation. Check out the full project on our website.

Design development for Out, a large scale audiovisual instalation. Check out the full project on our website.

XK Studio

18,370 Aufrufe • vor 2 Jahren

I'm excited to share our new work Diffusion as Shader (DaS), a versatile controllable video generation method for various tasks: object manipulation, camera control, mesh-to-video, and motion transfer. Project page: Github:

I'm excited to share our new work Diffusion as Shader (DaS), a versatile controllable video generation method for various tasks: object manipulation, camera control, mesh-to-video, and motion transfer. Project page: Github:

Yuan Liu

33,363 Aufrufe • vor 1 Jahr

WorldExplorer: Towards Generating Fully Navigable 3D Scenes Contributions: • We introduce the first method for generating 3D scenes from text that supports high-quality view synthesis while enabling exploration across a wide range of camera poses. • We propose an iterative scene expansion strategy using video diffusion models, driven by trajectory sampling and adaptive collision detection. • We design a scene memory mechanism that conditions each video generation step on relevant past frames, improving view consistency and overall scene coherence.

WorldExplorer: Towards Generating Fully Navigable 3D Scenes Contributions: • We introduce the first method for generating 3D scenes from text that supports high-quality view synthesis while enabling exploration across a wide range of camera poses. • We propose an iterative scene expansion strategy using video diffusion models, driven by trajectory sampling and adaptive collision detection. • We design a scene memory mechanism that conditions each video generation step on relevant past frames, improving view consistency and overall scene coherence.

MrNeRF

23,814 Aufrufe • vor 1 Jahr

Google presents VLOGGER Multimodal Diffusion for Embodied Avatar Synthesis We propose VLOGGER, a method for audio-driven human video generation from a single input image of a person, which builds on the success of recent generative diffusion models. Our method consists of

AK

66,375 Aufrufe • vor 2 Jahren

SceneScape: Text-Driven Consistent Scene Generation abs: project page: text-driven perpetual view generation -- synthesizing long videos of arbitrary scenes solely from an input text describing the scene and camera poses

SceneScape: Text-Driven Consistent Scene Generation abs: project page: text-driven perpetual view generation -- synthesizing long videos of arbitrary scenes solely from an input text describing the scene and camera poses

AK

73,258 Aufrufe • vor 3 Jahren

Fast View Synthesis of Casual Videos paper page: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.

Fast View Synthesis of Casual Videos paper page: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.

AK

20,651 Aufrufe • vor 2 Jahren

Introducing MegaSaM! 🎥 Accurate, fast, & robust structure + camera estimation from casual monocular videos of dynamic scenes! MegaSaM outputs camera parameters and consistent video depth, scaling to long videos with unconstrained camera paths and complex scene dynamics!

Introducing MegaSaM! 🎥 Accurate, fast, & robust structure + camera estimation from casual monocular videos of dynamic scenes! MegaSaM outputs camera parameters and consistent video depth, scaling to long videos with unconstrained camera paths and complex scene dynamics!

Zhengqi Li

56,923 Aufrufe • vor 1 Jahr

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

AK

62,768 Aufrufe • vor 3 Jahren

Google presents RadSplat Radiance Field-Informed Gaussian Splatting for Robust Real-Time Rendering with 900+ FPS Recent advances in view synthesis and real-time rendering have achieved photorealistic quality at impressive rendering speeds. While Radiance Field-based

Google presents RadSplat Radiance Field-Informed Gaussian Splatting for Robust Real-Time Rendering with 900+ FPS Recent advances in view synthesis and real-time rendering have achieved photorealistic quality at impressive rendering speeds. While Radiance Field-based

AK

139,521 Aufrufe • vor 2 Jahren