A new milestone for real-time accurate 3D spatial computing! Introducing ⚡️Fast-FoundationStereo⚡️, a real-time zero-shot stereo depth estimation model that accelerates the original FoundationStereo by >10x with comparable quality. Details in threads 🧵 (1/N)

This real-time demo shows the model instantly outputting a dense and accurate metric-depth map (with its unprojected 3D point cloud), displayed on the computer screen. (4/N)

Fast-FoundationStereo for the first time bridges the gap between strong zero-shot generalization and real-time speed, which are crucial for runtime sensitive tasks such as robotics and XR. (2/N)

Our code, pretrained weights, and pseudo-labeled internet stereo data will be released. Welcome to watch and star our repo! (8/N) Code: Webpage: Paper:

Our Fast-FoundationStereo accelerates the original FoundationStereo by >10x with comparable quality. It even outperforms some of the existing computational expensive zero-shot stereo matching models. Stats are measured on GPU 3090. (3/N)

We take a divide-and-conquer strategy to address the computational bottleneck of common components adopted in modern stereo matching models, while inheriting the strengths from the teacher model. (6/N)

Obtaining real stereo images with ground-truth metric depth annotation is notoriously difficult. To address this challenge, we propose an automatic data curation pipeline to generate pseudo-labels on internet-scale stereo images from Stereo4D dataset. (7/N)

Our model outperforms existing real-time stereo models by a large margin. The discrepancy is dramatic when visualizing the 3D point cloud. (5/N)

@chris_j_paxton Incredible job! This is super exciting

Long-time follower of your work, it's been of great value in our lab. Would love to get ETA on code/model release just to know whether to invest time into TensorRT-ing OG foundation stereo or wait. Lovely work as always and congrats!

The current plan is to release in Feb or early March.

Any guesses when you will release the code for this method?

it will be very soon!

Congratulations on the awesome release! Would you or any of your co-authors like to present your papers in the @Cohere_Labs Computer Vision community? Cohere Labs Community Page: Here's the playlist to previous talks, if you are interested:

The focus on zero-shot generalization alongside real-time performance is intriguing. How does the model handle varying lighting conditions in practical scenarios?

we do not see much degradation from original FoundationStereo on this front (which is robust). But talk is cheap. Once we release the code, you are welcome to try it out on your scene 😎

What could be a good setup to obtain stereo + ground-truth metric depth to benchmark fast foundation stereo for unique custom environments?

obtaining accurate, pixel-wise, metric-scale depth on real images can be extremely challenging. You can refer to some stereo datasets on how they obtained GT, e.g. Middlebury dataset.

Fast-FoundationStereo achieves a ten-fold speed increase while maintaining similar depth accuracy.