Skip navigation
Forget lidar. One single camera. Runs in real time & is open source: A streaming 3D model that…
0:00 / 3:30

Forget lidar. One single camera. Runs in real time & is open source: A streaming 3D model that…

Ilir Aliu

@IlirAliu_58K followers

Visit on X
4,992 likesDownloadOriginal post

597,224 views

Forget lidar. One single camera. Runs in real time & is open source: A streaming 3D model that reconstructs scenes live, at ~20 FPS, over long sequences. End-to-end. Optimization tricks, cleanup steps? Nope. And it beats both streaming and even some offline methods. Perception is becoming software-first. Closer to machines that see and understand the world as it unfolds. Thanks for sharing, Yinghao Xu 📍Models: Project page: Code: Paper: —— Weekly robotics and AI insights. Subscribe free:

35 Comments

Comments from the original post on X (Twitter)

  • Clifford Richardson's profile picture

    @CorvusCrypto

    People should read the paper. It outperforms other streaming vision models. The best in class performance is still IMU-corrected lidar image reconstruction. Which even they hint at by their concluding remarks. So no it's not "forget lidar" if you read the paper but rather improvements on streaming 3d construction

    60 likes

  • Mick Giles's profile picture

    @mickgiles

    Send this model through backrooms and report back please

    22 likes

  • Aaron Silvas's profile picture

    @Aaron_Silvas

    Curious how much this can be improved upon (without further focus time on one area) using stereoscopic vision, like mammals use. Is double inputs natures redundancy, or is there something more important to spatial reasoning at play?

    14 likes

  • Alexandre Forget's profile picture

    @alexandreforget

    what gpu is needed to run this in real time?

    7 likes

  • 𝑫𝒂𝒏𝒊𝒆𝒍 𝑺𝒄𝒐𝒕𝒕 𝑴𝒂𝒕𝒕𝒉𝒆𝒘𝒔 🇦🇺's profile picture

    @DanielSMatthews

    That is half a pipeline, human memory would then compress that into recognised things and geometric relationships between those things.

    4 likes

  • Carlos Pinheiro's profile picture

    @Capsbrr

    Looks great, perhaps useful to note what "real-time" means, i.e. hardware specs and framerate? :)

    4 likes

  • Gregor's profile picture

    @bygregorr

    the 'no cleanup steps' part is doing a lot of work. scale drift over long sequences is a physics problem, not a model one. saw it wreck a clean-looking 3D camera solve

    4 likes

  • Yinghao Xu's profile picture

    @YinghaoXu1

    Thanks for sharing! Stay tuned!

    4 likes

  • Panos's profile picture

    @Panoptimos

    Is it able to know what the real-world dimensions of whatever it scans are?

    3 likes

  • Rob's profile picture

    @rbroblambda

    What hardware to run 20fps, hope it's not 10 H100s

    3 likes

  • Rokas Bendikas's profile picture

    @BendikasRokas

    What happens when the lights go out?

    3 likes

  • Simon Archipoff's profile picture

    @SimonArchipoff

    We dont care about the nominal case for critical application. Lidar actually know how far things are, and know when it doesn't know. Imagine a car running over a mom pushing a stroller because her t-shirt made it believe it was just paint in a wall or something.

    2 likes

  • Cítrico's profile picture

    @SeaskySunair

    Military offset. This comes from technologies like those of the Gripen-NG fighter jet, for ex, which can navigate even without GPS, radar, or INS, using only a camera that compares the filmed terrain with a terrain database where it is located. The difference is on hardware.

    2 likes

  • Anurag Rai's profile picture

    @anuragonarun

    @bilawalsidhu bet he would like this a lot!

    2 likes

  • Robbyant's profile picture

    @robbyant_brain

    Thanks for sharing! Stay tuned for more updates from us next month.🥰

    1 like

  • Timur Yessenov's profile picture

    @Timur_Yessenov

    20 FPS from one camera is the headline, but the product test is uglier: motion blur, repeated textures, quick turns, bad lighting. If it keeps geometry stable there, robots get a cheaper way to remember a room without lidar.

    1 like

  • SSOT's profile picture

    @iamssot

    Fuck I forgot Lidar. What was it?

    1 like

  • Dusto's profile picture

    @DustoDimmidome

    Zero days without someone pretending Photogrammetry is a new thing.

    1 like

  • Tony's profile picture

    @Prasen166

    Isnt this mapping done hand held? If we can add this into drone I wanna see the reliability

    1 like

  • H4sh3's profile picture

    @shidumiro

    where you look and focus the details get better. great paper

    1 like