📢A new learning-based approach to SfM: #ACEZero No img-to-img matching, optimises image-to-scene correspondences directly. Needs no pose priors. Works on unordered image sets. Efficiently handles thousands of images. Paper: Page:

ACEZero builds an implicit scene representation by incremental training. It iterates between learning the scene representation, and registering new views to the representation. The process starts with a single image and the identity pose.

Because ACEZero optimizes image-to-scene correspondences directly, it naturally avoids the explosion of complexity of image-to-image matching for many images. ACEZero is able to estimate the poses of multiple thousand images in 1 hour on a single GPU.

We demonstrate the robustness and accuracy of ACEZero poses using novel view synthesis. In particular, we fit Nerfacto models on top of ACE Zero poses and render new views.

Existing learning-based SfM approaches do not scale well in the number of images. NoPe-NeRF needs two days to pose 200 images. DUSt3R runs OOM quickly. We were able to run DUSt3R with 50 images on a A100 (40GB). In contrast, ACE Zero can process thousands of images on a V100.

We found the estimated poses of ACEZero to be very similar to those estimated by COLMAP, also achieving similar view synthesis quality. Orange: COLMAP, green: ACEZero

ACEZero builds on the ACE relocaliser and trains it in a self-supervised fashion. The relocaliser is the reason why we do not need pose priors nor sequential images. Either an image is registered close to its optimal pose, or rejected and revisited later.

Watch this 3 minute teaser video to get a rough overview of the method.

Work by @NianticLabs Research, with Jamie Wynn, @ShuaiC8 , Tommaso Cavallari, @AMonszpart , @dantkz and @viprad.

That’s pretty impressive. Awesome work, congrats! Also really happy you compared to the DUSr3R baseline :)

Thanks Jerome! The DUSt3R code is excellent. We were able to get the baseline results in no-time.