🚀 Introducing our #SIGGRAPHAsia work “Adaptive Shells”, a novel #NeRF formulation that yields high visual fidelity and greatly accelerates rendering. TLDR: Auto-derived bounding shells result in up to 10x faster inference than InstantNGP! [1/n]

🌐 🌟Key insight: scene complexity varies spatially. Fuzzy content like foliage and hair needs multiple samples in the volume, while hard surfaces can be rendered with one sample. ❓Can we efficiently transition between volume and surface rendering? [2/n]

🔍 To model the scene content complexity, we generalize the SDF formulation of NeuS and introduce a spatially-varying kernel size that encodes the spread of the density, fitting a wide kernel to volume-like regions and a tight kernel to surface-like regions. [3/n]

📏 We use this varying kernel size to extract spatial bounds. Specifically, we apply level set evolution to the implicit isosurface, dilating to produce the outer envelope and eroding it to produce the inner envelope. The evolved implicit envelopes are extracted as meshes. [4/n]

🐚 The two mesh shells form a narrow band with width determined by kernel size. During inference, we cast rays against the shells and evaluate the field only within the enclosed region. Where the envelope is tight, crisp surfaces can be rendered with a single sample! [5/n]

By automatically adapting the number of queries per ray based on the scene content, our method smoothly interpolates between surface and volume rendering, achieving high visual fidelity on both hard surfaces and “fuzzy” contents like hair, fur, and foliage. [6/n]

Our approach enables efficient rendering at very high fidelity. Here we show a demo of real-time rendering within an interactive GUI. [7/n]

When directly applied to challenging unbounded 360 scenes, our method maintains real-time performance at quality competitive with other interactive methods. [8/n]

With the explicit mesh-based shell representation, we can easily adapt our method to a range of downstream tasks, including animation and simulation. [9/n]

This is a great collaboration at @NVIDIAAI by @zianwang97* @TianchangS* @merlin_ND* @nmwsharp @JunGao33210520 Alexander Keller @FidlerSanja Thomas Müller @ZGojcic Project page: #SIGGRAPHAsia #SIGGRAPH #NeRF #RealTimeRendering [10/n]

We are grateful to the original artists of 3D assets appearing in this work: Texturing XYZ, Jhon Maycon, Pierre-Louis Baril, abdoubouam, ckat609, and the BlenderKit team. Check out more details: Video: [11/11]

Very nice work! This offers an elegant continuous version of what we do in TECA ( where we explicitly combine 3D surface models with NeRFs. Your visuals are crazy good.

Thank you Prof. Black for the warm words and pointing the cool work!!

super cool work, congrats! A bit similar to what we did in our paper Meso-GAN by using shell maps with NeRFs for hair and fur rendering.

Thanks Jonathan for pointing the great work!

Well done!

Amazing work!!

🚀🚀🚀

Rigging!?!? Holly molly. Don't tell this to Voxel, SDF, Splat people 😉😉😉

@NVIDIAAIDev Will you release an app like InstantNGP? With VR support?

So we need an app now 🎦🪄😇

Why not compare against SotA?

Pretty amazing

Amazing! How does the storage/memory cost scales with the complexity of the scene/model?

😲 Thank you for sharing! This research sounds fascinating. I appreciate you highlighting such innovative work. 🤓👍

Amazing work ! Would you like to help us extracting the outer/inner shell of this old (4th century) boy:

@jonLorraine9 Fantastic work Zian. Will you release the source code?