We are publishing our second deep dive today as a follow-up post on SLAM and VIO in egocentric tracking. We go deep into the sensor tradeoffs b/w global shutter and rolling shutter and their implications on SLAM / VIO - specifically how the way the camera reads each frame... can introduce significant tracking errors before our SLAM pipeline even starts processing. We break down why global shutter is the obvious fix but the wrong default, the physics of why rolling shutter dominates every consumer device, and where the fundamental limits lie.show more

Read the full essay here →

such a great read! the best part is its not so trivial yet one of the most important choice when it comes to choosing hardware for downstream SLAM/VIO pipelines

You can also do a combination of continuous time and discrete time approaches. It's often sufficient to use a first order approximation to "de-warp" the distorted features using velocity estimates from VIO or even just IMU. Later loop closures won't affect it too much.

This is very true. Instead of estimating a pose for every row of the image, you could also unwarp features in the image to compensate for the motion effect. This is what is typically done in consumer mobile phones these days, where the effects of rolling shutter may be compensated through some computational photography, so the image may look rolling-shutter-artifact-free in many cases.

quick question, with the rolling shutter example above, did we introduced a depth sensor to get the depth data or was it purely based off of the video ?

The test in the video is simply a feature tracking comparison in rolling shutter vs global shutter mono video. No state estimation or depth or any other data is involved.

👀👀

Is it Lie algebra ?

Lie algebra is a parameterization choice, particularly for non-linear optimization. The blog post itself just discusses simple SE(3) matrices in the setup, but the optimization of the underlying cost function for example - bundle adjustment - usually involves non-linear optimization in different parameter space - the most common choice is the lie algebra space

💪🔥🔥

Putting out learnings in open🚀