Excited to share our work on STEERing robot behavior! With structured language annotation of offline data, STEER exposes fundamental manipulation skills that can be modulated and combined to enable zero-shot adaptation to new situations and tasks.

By focusing on learning fundamental, modular skills that are also flexible--like “grasp the <object> from the side”--STEER allows a powerful “System 2” (web-scale VLM like Gemini or human) to handle the cognitive heavy-lifting while the policy focuses on the low-level control.

For example, when picking up a potted plant, we’d want the robot to avoid disturbing the plant itself. Without explicit training, we can only hope that it generalizes as desired. STEER first thinks about the approach, then executes that strategy from its learned repertoire.

This also goes for performing new tasks. We show that Gemini understands that to pour, one should grasp the cup around its body (like a human would) and then rotate it back and forth--and that the learned policy, reannotated to expose these skills can be orchestrated accordingly.

I’m excited about using STEER to learn new skills in the real world without any demos but by exploring in the space of the semantically meaningful skills. We tested this with the new pouring task, where the robot improved by 20% after getting feedback on its 10 initial attempts.

Please check out the website for details/videos. This was a really fun project as an SR at Google DeepMind thanks to my awesome hosts @AlexIrpan and @xiao_ted and wonderful collaborators @montseglz, @SeanKirmani, Dmitry Kalashnikov, and @shahdhruv_