How can robots autonomously handle ambiguous situations that require commonsense reasoning?
@_anniechen_1.4K followers
24,357 views
5 Comments
Comments from the original post on X (Twitter)

@_anniechen_
Robots face a long tail of different possible situations. Handling this breadth of tasks typically requires heavy human supervision.
6 likes

@_anniechen_
How to effectively enable VLMs as a high-level policy with locomotion? Issue: Prompting a VLM naively can fail: - VLM can misinterpret the scene or the robot’s capabilities - Easy for the robot to get stuck
4 likes

@_anniechen_
We find 2 components important for eliciting on-the-fly, adaptive behavior selection with VLMs: 1) In-context reasoning over the history of interactions and (2) Outputting a multi-step plan
4 likes

@_anniechen_
Leveraging VLMs in this way allows a robot to handle (zero-shot!) a wide range of complex real-world situations that wide range of complex scenarios that would otherwise require environment-specific engineering or human guidance
4 likes

@_anniechen_
Thanks to wonderful collaborators @AlecLessing, @tangerinecoder, Govind Chada, @smithlaura1028, @svlevine, @chelseabfinn! I’ll be presenting this on Thursday at #ICRA2025 in Atlanta! Let me know if you’re around :)
4 likes