How can robots autonomously handle ambiguous situations that require commonsense reasoning? *VLM-PC* provides adaptive high-level planning, so robots can get unstuck by exploring multiple strategies. Paper:

Robots face a long tail of different possible situations. Handling this breadth of tasks typically requires heavy human supervision.

How to effectively enable VLMs as a high-level policy with locomotion? Issue: Prompting a VLM naively can fail: - VLM can misinterpret the scene or the robot’s capabilities - Easy for the robot to get stuck

We find 2 components important for eliciting on-the-fly, adaptive behavior selection with VLMs: 1) In-context reasoning over the history of interactions and (2) Outputting a multi-step plan

Leveraging VLMs in this way allows a robot to handle (zero-shot!) a wide range of complex real-world situations that wide range of complex scenarios that would otherwise require environment-specific engineering or human guidance

Thanks to wonderful collaborators @AlecLessing, @tangerinecoder, Govind Chada, @smithlaura1028, @svlevine, @chelseabfinn! I’ll be presenting this on Thursday at #ICRA2025 in Atlanta! Let me know if you’re around :)