Vision-language models can control robots, but what if the prompt is too complex for the robot to follow directly? We developed a way to get robots to “think through” complex instructions, feedback, and interjections. We call it the Hierarchical Interactive Robot (Hi Robot).

Hi Robot uses a high-level VLM policy to parse instructions, camera observations, and user feedback (e.g., interjections) to decide what to do. Full blog post:

Just like when you are trying to cook a new kind of recipe, and you “talk” through it: first I add the salt, then the onions… the robot “talks” to itself to walk through a complex task, and then our π0 VLA follows these steps.

A person can talk to the robot while it’s doing this, interjecting, providing feedback, and making additional requests, like in this shopping scenario.

Hi Robot allows users to define new tasks through prompts, modifying how the robot should perform a task – for example, in this example the robot changes its behavior from the standard “bussing” policy based on user commands.

To learn more about Hi Robot, check out our blog post: The full-length paper is here:

Want to learn how practical AI skills and automations for your business and work? Check out our 50+ step-by-step video tutorials 100% FREE 20+ hours of Ai and Automation goodness absolutely free 🥳

Ummm.... The aluminum pan was recycling... Lol

Pretty cool. Curious - How is this different than reasoning or chain of thought in LLMs?

Very cool

and ppl still think it's not coming for blue collar

damn absoloutely phenomenal work