Chain-of-thought reasoning is a powerful tool to enable language models to work through complex problems. Can we use this with robots? With embodied chain-of-thought, vision-language-action (VLA) models can think through perception and planning! A 🧵👇

Embodied chain of thought allows the VLA (in this case, a finetuned OpenVLA model) to work through a complex task by reasoning over subtasks, detecting objects, and making step-by-step plans. When generating an action, the VLA works through these steps automatically.

How do we train OpenVLA for embodied chain of thought? We distill a variety of other foundation models, such as Gemini and Grounding DINO, into synthetic examples that can teach the VLA to perform embodied chain of thought.

The resulting model can solve complex tasks that require multi stage inferences. It can generalize more effectively to novel objects, perform longer tasks, and understand sophisticated instructions.

The resulting VLA can even interpret human corrections and interventions, incorporating them as corrections into the embodied chain of thought process!

While our main experiments use the Bridge v2 setup: We also tested on a variety of other embodiments from OXE:

This was a really fun collaboration with @MiZawalski, @verityw_, @KarlPertsch, @oier_mees, @chelseabfinn Website: Paper:

For more, check out these posts by Michal and Will:

Thanks for sharing 👍 How do you create videos like these?

You’re making a lot of progress in robotics