How can we help *any* image-input policy generalize better?
@Jesse_Y_Zhang2.1K followers
13,833 views
5 Comments
Comments from the original post on X (Twitter)

@Jesse_Y_Zhang
By offloading high-level reasoning of "what" and "where" to VLMs, PEEK lets robot policies focus on learning "how" to act. It’s a step toward zero-shot generalization in manipulation — giving robots a “peek” at the world.
3 likes

@Jesse_Y_Zhang
Our policy training data consists of: - simple sim envs for 3DDA (3D and RGB input) - BRIDGE-v2 (for π₀ and ACT) This data is labeled with the PEEK VLM to directly draw “where” to focus and “what” to do onto the image.
3 likes

@Jesse_Y_Zhang
We fine-tune policies on PEEK-VLM labeled data. We test zero-shot. 💥 41.4× sim2real improvement (3DDA) 💥 2–3.5× boosts for π₀ and ACT PEEK gives visuomotor policies the minimal cues to generalize across new tasks, scenes, and semantics.
3 likes

@Jesse_Y_Zhang
📄 Paper: 💻 Demo: 🔗 Website, code, data, models: Work done w/ @memmelma (co-lead), @minjunkevink, @fox_dieter17849 , @_jessethomason_ , Fabio Ramos, @ebiyik_ , @abhishekunique7 , @AnqiLi24
4 likes

@Jesse_Y_Zhang
Finally, see @memmelma's tweet thread to learn how the method works!
3 likes