🚀 Introducing DiffusionRenderer, a neural rendering engine powered by video diffusion models. 🎥 Estimates high-quality geometry and materials from videos, synthesizes photorealistic light transport, enables relighting and material editing with realistic shadows and reflections

Please open-source it 😀

Working on it!

@ZhongpaiGao Anything news ?

@ZhongpaiGao We open-sourced two versions of DiffusionRenderer, check them out! Academic version that reproduce paper results: 🔗 Cosmos version that brings enhanced quality: 🔗

Classic rendering and inverse rendering are often treated separately, with PBR relying on precise scene attributes that inverse models cannot achieve. DiffusionRenderer unifies the tasks in a data-driven framework with two video diffusion models for forward and inverse rendering.

🌐 The inverse renderer is a video diffusion model conditioned on input video and domain embeddings, producing accurate, temporally consistent scene attributes such as normals, albedo, roughness, and metallicity.

Conditioned on input geometry, material buffers, and environment lighting, the forward renderer acts as a neural approximation of path tracing, generating accurate shadows and reflections that are consistent across viewpoints.

Unlike classic PBR which relies on precise geometry and path tracing, DiffusionRenderer synthesizes realistic lighting effects without explicit simulation, outperforming PBR for real-world applications especially when explicit geometry is inaccessible or inaccurate.

DiffusionRenderer enables the relighting of static and dynamic scenes. It produces realistic shadows, reflections, and lighting effects under diverse lighting conditions, from a single video input without requiring explicit scene geometry.

DiffusionRenderer supports photorealistic material editing, enabling adjustments to roughness and metallicity while preserving consistent lighting and shadows.

This work is a great collaboration at @NVIDIAAI by @RfLiang* @ZGojcic @HuanLing6 @yoslber Jon Hasselgren @ZhiHaoLin16 @JunGao33210520 Alexander Keller @nanditav17 @FidlerSanja @zianwang97* 🌐 Project page:

Looks cool! Do you have any 30 or 60 FPS results that you can share? It's really hard to get a sense for these results from these very low-framerate videos.

Thanks! The current results are based on the Stable Video Diffusion model. We’re working towards higher FPS and even better quality, please stay tuned!

@Scobleizer Neural video diffusion for rendering? Now that's fascinating. Would love to test this against our Mars terrain mapping systems. The applications for habitat visualization could be revolutionary.

great work🥰

Impressive technology! Excited to see the possibilities this opens up.