How to harness foundation models for *generalization in the wild* in robot manipulation? Introducing VoxPoser: use LLM+VLM to label affordances and constraints directly in 3D perceptual space for zero-shot robot manipulation in the real world! 🌐 🧵👇

Data is key for generalization, but robot data is scarce and expensive. Instead of training policies on labeled data, VoxPoser uses LLM+VLM to compose 3D value maps using generated code. Then 6-DoF actions are synthesized by motion planners, all w/o any training or primitives.

Given free-form instructions + RGB-D obs, LLM orchestrates perception calls to VLM and array operations to assign continuous values to voxel map, showing *where to act* and *how to act*. It also parametrizes rotation, velocity, and gripper actions for a complete SE(3) trajectory!

We verified VoxPoser in everyday manipulation tasks in the wild, including articulated and deformable object manipulation. All the results here are synthesized with zero-shot execution.

Just toss your objects too! VoxPoser is robust to disturbances because it replans actions in *real-time* with visual feedback. The 3D value maps are always updated with latest observations, allowing robot to recover from unexpected errors.

LLMs show emergent abilities at scale – same applies to VoxPoser, but on physical behaviors! It can conduct physics experiments, have behavioral commonsense, listen to your fine-grained correction, come up with multi-step visual program, and more.

For more, check out 👇 Project site: Walkthrough video: Paper: Work done w/ @chenwang_j , @RuohanZhang76 , @YunzhuLiYZ , @jiajunwu_cs, and @drfeifei at @StanfordSVL & @StanfordAILab.

@RuohanZhang76 @YunzhuLiYZ @jiajunwu_cs @drfeifei @StanfordSVL @StanfordAILab Excited to share that we have open-sourced the code based on RLBench: We will also be presenting VoxPoser at CoRL next week as an oral presentation. See you in Atlanta! #CoRL @corl_conf

This is great

Thank you Brett!

interesting work!!