Do AI robots see the world like we do? I dove head first into latent space to uncover the attention maps that show how my robot sees and understands the world.

Initially, the robot struggles to locate the capsule to pick, but gradually becomes more certain of the exact pick location. As the task succeeds for capsule 1, attention rapidly shifts to capsule 2 in the background

To me these attention maps prove that AI robot models like ACT make efficient use of the visual information they have access to. I also believe improvements in model performance should be directly visible as more focused attention maps

The algorithm behind this will be open-sourced and compatible with ACT policy in @LeRobotHF (probably this weekend) so stay tuned!

The global AI sector is evolving rapidly, supported by advancements in technology and infrastructure. AIS offers targeted exposure to key players driving these developments.

So such visual policies are like a neural state estimator plus a burnt-in tracker? 🤔

Man this is super cool!

That's super cool, is this just a reconstruction of the attention scores per pixel?

Yes though it’s not quite as simple as that, the transformer model compresses the image a lot, there’s some computer vision magic to upscale that compressed view into the full image you see here that I don’t fully understand 😅

Also, is your web app for lerobot open source? Looks really cool.

The app itself is not but the Python libraries that power it will be though it will take time for us to organise everything into a proper library

dinov2?