Forget lidar. One single camera. Runs in real time & is open source: A streaming 3D model that reconstructs scenes live, at ~20 FPS, over long sequences. End-to-end. Optimization tricks, cleanup steps? Nope. And it beats both streaming and even some offline methods. Perception is becoming software-first. Closer to... machines that see and understand the world as it unfolds. Thanks for sharing, Yinghao Xu 📍Models: Project page: Code: Paper: —— Weekly robotics and AI insights. Subscribe free:show more

People should read the paper. It outperforms other streaming vision models. The best in class performance is still IMU-corrected lidar image reconstruction. Which even they hint at by their concluding remarks. So no it's not "forget lidar" if you read the paper but rather improvements on streaming 3d construction

Send this model through backrooms and report back please

Curious how much this can be improved upon (without further focus time on one area) using stereoscopic vision, like mammals use. Is double inputs natures redundancy, or is there something more important to spatial reasoning at play?

what gpu is needed to run this in real time?

That is half a pipeline, human memory would then compress that into recognised things and geometric relationships between those things.

Looks great, perhaps useful to note what "real-time" means, i.e. hardware specs and framerate? :)

the 'no cleanup steps' part is doing a lot of work. scale drift over long sequences is a physics problem, not a model one. saw it wreck a clean-looking 3D camera solve

Thanks for sharing! Stay tuned!

Is it able to know what the real-world dimensions of whatever it scans are?

What hardware to run 20fps, hope it's not 10 H100s

What happens when the lights go out?

We dont care about the nominal case for critical application. Lidar actually know how far things are, and know when it doesn't know. Imagine a car running over a mom pushing a stroller because her t-shirt made it believe it was just paint in a wall or something.

Military offset. This comes from technologies like those of the Gripen-NG fighter jet, for ex, which can navigate even without GPS, radar, or INS, using only a camera that compares the filmed terrain with a terrain database where it is located. The difference is on hardware.

@bilawalsidhu bet he would like this a lot!

Thanks for sharing! Stay tuned for more updates from us next month.🥰

20 FPS from one camera is the headline, but the product test is uglier: motion blur, repeated textures, quick turns, bad lighting. If it keeps geometry stable there, robots get a cheaper way to remember a room without lidar.

Fuck I forgot Lidar. What was it?

Zero days without someone pretending Photogrammetry is a new thing.

Isnt this mapping done hand held? If we can add this into drone I wanna see the reliability

where you look and focus the details get better. great paper

Wait, if I could do this and send this to my @maticrobots I totally would to help with its 3D mapping.

You still need lighting right?

This would not work in the wild 🥀

Thanks let us know when this becomes an app I can download.

@TrentTelenko Shiiii

Use this on footage from Severance.

I need this for my smart home setup. 20 FPS indoor reconstruction at this stability means my devices finally map spaces properly instead of guessing boundaries 🏠📡

Very cool

definitly requiring much much more computing power, than say lidar which can work easily on embeded, like rp 5 or older.

Can you achieve such a fast frame rate?

Sweet

I just love Gaussian Splatting

One of the tedious projects I’ve done was on the depth analysis using data of a #LiDAR, and yes you are right as AI has done exceptionally well in this domain. #tesla as well (if I’m not mistaken), is a living proof of it.

Real time 3D scene reconstruction from one single camera beating offline methods is wild, lidar might actually be optional soon

⏫