Can we enable free-viewpoint renderings of scenes under various lighting conditions? Yes, we can! And just from SINGLE videos. Here are 🌃 night simulations from input videos with hard shadows.

To reconstruct and re-render the scene under different lighting, we need Inverse Rendering. We need to infer shape, albedo, visibility, and sun and sky illumination from videos. This differs from existing NeRF/3DGS, where the lighting was baked into the models.

How? 🤔 Let's integrate physics and learning! 🤝 Physically based rendering helps explain away shadow volume estimation. Learning-based priors: Help avoid degenerated inverse rendering solutions.

Inverse rendering enables many exciting new applications beyond view synthesis. In the paper, we showcase 🌉 nighttime simulation 📽️ object insertion 💡 changing sun directions

💻 Code is available! Have fun! We will be presenting this work at @3DVconf!

Check out the project page for more results and comparisons. A fun collaboration between 🌽 and 🐢!

Very interesting. What paper is this from???