Skip navigation
How can we help *any* image-input policy generalize better?
0:00 / 0:41

How can we help *any* image-input policy generalize better?

Jesse Zhang

@Jesse_Y_Zhang2.1K followers

Visit on X
121 likesDownloadOriginal post

13,833 views

How can we help *any* image-input policy generalize better? 👉 Meet PEEK 🤖 — a framework that uses VLMs to decide *where* to look and *what* to do, so downstream policies — from ACT, 3D-DA, or even π₀ — generalize more effectively! 🧵

5 Comments

Comments from the original post on X (Twitter)

  • Jesse Zhang's profile picture

    @Jesse_Y_Zhang

    By offloading high-level reasoning of "what" and "where" to VLMs, PEEK lets robot policies focus on learning "how" to act. It’s a step toward zero-shot generalization in manipulation — giving robots a “peek” at the world.

    3 likes

  • Jesse Zhang's profile picture

    @Jesse_Y_Zhang

    Our policy training data consists of: - simple sim envs for 3DDA (3D and RGB input) - BRIDGE-v2 (for π₀ and ACT) This data is labeled with the PEEK VLM to directly draw “where” to focus and “what” to do onto the image.

    3 likes

  • Jesse Zhang's profile picture

    @Jesse_Y_Zhang

    We fine-tune policies on PEEK-VLM labeled data. We test zero-shot. 💥 41.4× sim2real improvement (3DDA) 💥 2–3.5× boosts for π₀ and ACT PEEK gives visuomotor policies the minimal cues to generalize across new tasks, scenes, and semantics.

    3 likes

  • Jesse Zhang's profile picture

    @Jesse_Y_Zhang

    📄 Paper: 💻 Demo: 🔗 Website, code, data, models: Work done w/ @memmelma (co-lead), @minjunkevink, @fox_dieter17849 , @_jessethomason_ , Fabio Ramos, @ebiyik_ , @abhishekunique7 , @AnqiLi24

    4 likes

  • Jesse Zhang's profile picture

    @Jesse_Y_Zhang

    Finally, see @memmelma's tweet thread to learn how the method works!

    3 likes