正在加载视频...
视频加载失败
Do AI robots see the world like we do? I dove head first into latent space to uncover the attention maps that show how my robot sees and understands the world.
11 条评论
Initially, the robot struggles to locate the capsule to pick, but gradually becomes more certain of the exact pick location. As the task succeeds for capsule 1, attention rapidly shifts to capsule 2 in the background
To me these attention maps prove that AI robot models like ACT make efficient use of the visual information they have access to. I also believe improvements in model performance should be directly visible as more focused attention maps
The algorithm behind this will be open-sourced and compatible with ACT policy in @LeRobotHF (probably this weekend) so stay tuned!
The global AI sector is evolving rapidly, supported by advancements in technology and infrastructure. AIS offers targeted exposure to key players driving these developments.
So such visual policies are like a neural state estimator plus a burnt-in tracker? 🤔
Man this is super cool!
That's super cool, is this just a reconstruction of the attention scores per pixel?
Yes though it’s not quite as simple as that, the transformer model compresses the image a lot, there’s some computer vision magic to upscale that compressed view into the full image you see here that I don’t fully understand 😅
Also, is your web app for lerobot open source? Looks really cool.
The app itself is not but the Python libraries that power it will be though it will take time for us to organise everything into a proper library
dinov2?
![John: "Over in China, they’re doing robot Olympics. They’re doing marathons and all sorts of stuff." Figure founder Brett Adcock: “We look at the distribution of what humans do that is useful, and we [designed our humanoid] to do as much of that as possible.” “These things where we run marathons, or we do backflips, or we do karate moves—they’re not in the main part of the distribution. They don’t matter. And if you really want to size for those, you’re going to build a really expensive and unsafe robot that’s hard to manufacture.” “People want the $10,000 humanoid that can do general-purpose work. We want it to do laundry and dishes and be a companion. I want to ship robots at scale into the workforce to do logistics and healthcare, and build buildings and data centers.” "I don't need [our humanoid] to do backflips to do any of that work. I want robots building other robots."](https://image.24vids.com/tw-2022786577924325714/amplify_video_thumb/2022786433782812673/img/_TrIWe2vwjWaIIhJ.jpg)
