Loading video...

Video Failed to Load

Go Home

How can we help *any* image-input policy generalize better? 👉 Meet PEEK 🤖 — a framework that uses VLMs to decide *where* to look and *what* to do, so downstream policies — from ACT, 3D-DA, or even π₀ — generalize more effectively! 🧵

Jesse_Y_Zhang's profile picture

Jesse Zhang

2,078 subscribers

13,825 views • 11 months ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

5 Comments

Jesse Zhang's profile picture
Jesse Zhang11 months ago

By offloading high-level reasoning of "what" and "where" to VLMs, PEEK lets robot policies focus on learning "how" to act. It’s a step toward zero-shot generalization in manipulation — giving robots a “peek” at the world.

Jesse Zhang's profile picture
Jesse Zhang11 months ago

Our policy training data consists of: - simple sim envs for 3DDA (3D and RGB input) - BRIDGE-v2 (for π₀ and ACT) This data is labeled with the PEEK VLM to directly draw “where” to focus and “what” to do onto the image.

Jesse Zhang's profile picture
Jesse Zhang11 months ago

We fine-tune policies on PEEK-VLM labeled data. We test zero-shot. 💥 41.4× sim2real improvement (3DDA) 💥 2–3.5× boosts for π₀ and ACT PEEK gives visuomotor policies the minimal cues to generalize across new tasks, scenes, and semantics.

Jesse Zhang's profile picture
Jesse Zhang11 months ago

📄 Paper: 💻 Demo: 🔗 Website, code, data, models: Work done w/ @memmelma (co-lead), @minjunkevink, @fox_dieter17849 , @_jessethomason_ , Fabio Ramos, @ebiyik_ , @abhishekunique7 , @AnqiLi24

Jesse Zhang's profile picture
Jesse Zhang11 months ago

Finally, see @memmelma's tweet thread to learn how the method works!

Related Videos

New research from BlackForestLabsAI - Unofficial 🥳 Meet Self-Flow: our self-supervised framework for image, audio, video & world models 🤖 Do generative models really need DINO to learn strong representations? We propose teaching them directly via a joint framework instead 🧵
2:39

New research from BlackForestLabsAI - Unofficial 🥳 Meet Self-Flow: our self-supervised framework for image, audio, video & world models 🤖 Do generative models really need DINO to learn strong representations? We propose teaching them directly via a joint framework instead 🧵

Hila Chefer

69,101 views • 7 months ago

🚨 Without Any Motion Priors, how to make humanoids do versatile parkour jumping🦘, clapping dance🤸, cliff traversal🧗, and box pick-and-move📦 with a unified RL framework? Introduce WoCoCo: 🧗 Whole-body humanoid Control with sequential Contacts 🎯Unified designs for minimal tuning across tasks 🤖Generalize to various high-DoF robots Website:
0:54

🚨 Without Any Motion Priors, how to make humanoids do versatile parkour jumping🦘, clapping dance🤸, cliff traversal🧗, and box pick-and-move📦 with a unified RL framework? Introduce WoCoCo: 🧗 Whole-body humanoid Control with sequential Contacts 🎯Unified designs for minimal tuning across tasks 🤖Generalize to various high-DoF robots Website:

Wenli Xiao

70,449 views • 2 years ago

New work with Alec Radford and David Duvenaud: Have you ever dreamed of talking to someone from the past? Introducing talkie, a 13B model trained only on pre-1931 text. Vintage models should help us to understand how LMs generalize (e.g., can we teach talkie to code?). Thread:
7:58

New work with Alec Radford and David Duvenaud: Have you ever dreamed of talking to someone from the past? Introducing talkie, a 13B model trained only on pre-1931 text. Vintage models should help us to understand how LMs generalize (e.g., can we teach talkie to code?). Thread:

Nick Levine

1,240,210 views • 5 months ago

🤖Robots that think ahead and act in real time. LingBot-VA 2.0 — the first embodied-native foundation model. Not fine-tuned from a video generator. Built from scratch for the physical world. ✅ 93.6% success on bimanual tasks ⚡ 150 Hz single-GPU inference 🎯 20 demos to generalize. This is what happens when you stop adapting and start building natively. 🧵👇
3:48

🤖Robots that think ahead and act in real time. LingBot-VA 2.0 — the first embodied-native foundation model. Not fine-tuned from a video generator. Built from scratch for the physical world. ✅ 93.6% success on bimanual tasks ⚡ 150 Hz single-GPU inference 🎯 20 demos to generalize. This is what happens when you stop adapting and start building natively. 🧵👇

Robbyant

895,496 views • 2 months ago

Robotics fundamentally involves understanding the dynamics of how things change in the world in response to action and force. This is impossible to learn from static images; instead, it’s far more effective and more data-efficient to learn from video. Elvis Nava joins us to talk about @mimicrobotic. One of the key findings from mimic-video is that pretraining on webscale video allows robots to learn physics priors; as a result, policies train faster, generalize better, and are capable of more impressive dexterity, versus training on static images or image-language pairs as per a VLM. Watch Episode #81 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton to learn more!
1:08:43

Robotics fundamentally involves understanding the dynamics of how things change in the world in response to action and force. This is impossible to learn from static images; instead, it’s far more effective and more data-efficient to learn from video. Elvis Nava joins us to talk about @mimicrobotic. One of the key findings from mimic-video is that pretraining on webscale video allows robots to learn physics priors; as a result, policies train faster, generalize better, and are capable of more impressive dexterity, versus training on static images or image-language pairs as per a VLM. Watch Episode #81 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton to learn more!

RoboPapers

48,645 views • 4 months ago

How can we get VLMs to move their eyes—and reason step-by-step in visually grounded ways? 👀 We introduce ViGoRL, a RL method that anchors reasoning to image regions. 🎯 It outperforms vanilla GRPO and SFT across grounding, spatial tasks, and visual search (86.4% on V*). 👇🧵
0:39

How can we get VLMs to move their eyes—and reason step-by-step in visually grounded ways? 👀 We introduce ViGoRL, a RL method that anchors reasoning to image regions. 🎯 It outperforms vanilla GRPO and SFT across grounding, spatial tasks, and visual search (86.4% on V*). 👇🧵

Gabriel Sarch

76,717 views • 1 year ago

If you let VLMs experiment on their own, they can do surprising things! From an image, we let a VLM code a 3D scene from scratch in Blender, and then render to verify/refine in a loop. Even when each step is imperfect, it gets results like this, with zero training.
0:33

If you let VLMs experiment on their own, they can do surprising things! From an image, we let a VLM code a 3D scene from scratch in Blender, and then render to verify/refine in a loop. Even when each step is imperfect, it gets results like this, with zero training.

Angjoo Kanazawa

52,126 views • 8 months ago

I built software to cut docker image sizes by more than 80%. It uses dynamic analysis and and has a LOT more room for improvement. The demo here shows an ETL workflow docker image, cut down from 1.28GB to 193MB (-85%)! If large docker image sizes are becoming a production problem, I'd be happy to explore how we can help.
1:21

I built software to cut docker image sizes by more than 80%. It uses dynamic analysis and and has a LOT more room for improvement. The demo here shows an ETL workflow docker image, cut down from 1.28GB to 193MB (-85%)! If large docker image sizes are becoming a production problem, I'd be happy to explore how we can help.

omkaar

353,266 views • 1 year ago

Elon Musk: We want to make it possible for anyone interested to move to Mars and help build a new civilization. “We want to ultimately make it so that anyone who wants to move to Mars and help build a new civilization can do so. Anyone out there. How cool would that be? And even if you don't want to do it, maybe you have a son or daughter who wants to do that or a friend who wants to do it. And I think it would be the best adventure that one could possibly do is to go and help build a new civilization on a new planet.” From: SpaceX Update, May 29, 2025
0:30

Elon Musk: We want to make it possible for anyone interested to move to Mars and help build a new civilization. “We want to ultimately make it so that anyone who wants to move to Mars and help build a new civilization can do so. Anyone out there. How cool would that be? And even if you don't want to do it, maybe you have a son or daughter who wants to do that or a friend who wants to do it. And I think it would be the best adventure that one could possibly do is to go and help build a new civilization on a new planet.” From: SpaceX Update, May 29, 2025

ELON CLIPS

72,179 views • 1 year ago

Nikola Jokic postgame "We are not even close to where we're supposed to be. I think how bad we've played, we're in a good spot." "We need to start thinking what I can do for this team to help, not what the team can do to help me...We should point (the thumb not the finger)."
0:42

Nikola Jokic postgame "We are not even close to where we're supposed to be. I think how bad we've played, we're in a good spot." "We need to start thinking what I can do for this team to help, not what the team can do to help me...We should point (the thumb not the finger)."

Harrison Wind

314,329 views • 1 year ago

Better policies start with what works. The Clinton Policy Institute was created to help state and local leaders develop, implement and expand evidence-based policies that improve lives. Grounded in the belief that it takes a village to give every child a fair shot, CPI focuses on the issues that matter most to whether children and families can flourish, from economic security and women's and children's health to children's well-being and democratic resilience. Learn more:
1:05

Better policies start with what works. The Clinton Policy Institute was created to help state and local leaders develop, implement and expand evidence-based policies that improve lives. Grounded in the belief that it takes a village to give every child a fair shot, CPI focuses on the issues that matter most to whether children and families can flourish, from economic security and women's and children's health to children's well-being and democratic resilience. Learn more:

Clinton Foundation

348,198 views • 2 months ago

📢📢 𝐀𝐯𝐚𝐭𝟑𝐫 📢📢 Avat3r creates high-quality 3D head avatars from just a few input images in a single forward pass with a new dynamic 3DGS reconstruction model. Video: Project: Our core idea is to make Gaussian Reconstruction Models animatable. We find that a simple cross-attention to an expression code sequence is already sufficient to model complex facial expressions. We then incorporate position maps from DUSt3R and feature maps from Sapiens to facilitate the prediction task. While DUSt3R's position maps act as a pixel-aligned initialization for the Gaussians' positions, the Sapiens feature maps help the cross-view transformer to match corresponding image tokens in the 4 input images. One major challenge in creating a 3D head avatar from smartphone images comes from inconsistent facial expressions when the subject could not remain perfectly static during the capture. We eliminate this static requirement by simply showing our model input images with different facial expressions during training. This technique makes our model robust to inconsistent input images later on. Finally, we show that despite the model has been trained with 4 input images, one can even create a 3D head avatar when only a single image is available. To achieve this, we employ a pre-trained 3D GAN to lift the single image to 3D and then render the 4 input images for our model. This allows us to create 3D head avatars from single images and even highly out-of-distribution examples like AI generated faces, paintings or statues. Great work by Tobias Kirschstein from his internship at Meta with Javier Romero, Artem Sevastopolsky, and Shunsuke Saito
0:38

📢📢 𝐀𝐯𝐚𝐭𝟑𝐫 📢📢 Avat3r creates high-quality 3D head avatars from just a few input images in a single forward pass with a new dynamic 3DGS reconstruction model. Video: Project: Our core idea is to make Gaussian Reconstruction Models animatable. We find that a simple cross-attention to an expression code sequence is already sufficient to model complex facial expressions. We then incorporate position maps from DUSt3R and feature maps from Sapiens to facilitate the prediction task. While DUSt3R's position maps act as a pixel-aligned initialization for the Gaussians' positions, the Sapiens feature maps help the cross-view transformer to match corresponding image tokens in the 4 input images. One major challenge in creating a 3D head avatar from smartphone images comes from inconsistent facial expressions when the subject could not remain perfectly static during the capture. We eliminate this static requirement by simply showing our model input images with different facial expressions during training. This technique makes our model robust to inconsistent input images later on. Finally, we show that despite the model has been trained with 4 input images, one can even create a 3D head avatar when only a single image is available. To achieve this, we employ a pre-trained 3D GAN to lift the single image to 3D and then render the 4 input images for our model. This allows us to create 3D head avatars from single images and even highly out-of-distribution examples like AI generated faces, paintings or statues. Great work by Tobias Kirschstein from his internship at Meta with Javier Romero, Artem Sevastopolsky, and Shunsuke Saito

Matthias Niessner

74,818 views • 1 year ago

Candace Owens VIOLATED HIPAA, and is advocating for others to do so by sending her info. You can access ANY file from ANY hospital that uses Epic, so long as you are at a hospital that uses EPIC... STOP LISTENING TO CANDACE AND DO YOUR OWN RESEARCH
2:09

Candace Owens VIOLATED HIPAA, and is advocating for others to do so by sending her info. You can access ANY file from ANY hospital that uses Epic, so long as you are at a hospital that uses EPIC... STOP LISTENING TO CANDACE AND DO YOUR OWN RESEARCH

Arty Trolls

12,702 views • 2 months ago

We want autistic people, with or without a diagnosis, to take our new survey. It is designed to help you tell us what support you need, so we can understand what works and how more can be put in place:
0:30

We want autistic people, with or without a diagnosis, to take our new survey. It is designed to help you tell us what support you need, so we can understand what works and how more can be put in place:

Autism Action

134,186 views • 2 years ago

Willie Fritz worked to make sure his University of Houston program is a football laboratory where his coordinators Slade Nagle and Austin Armstrong can concentrate on what matters. "We've got so many coaches at this level," Fritz says. "One of the things I try to do is take a lot of things off the plate of my coordinators, my coaches, so that they can work on football. "And that's what they do. They love football. They're talking football all day, every day. "A lot of that is how can we enhance our scheme. And what does our guys do best. What do we need to stay away from. What do we need to do more of. How can we teach this fundamental better."
0:47

Willie Fritz worked to make sure his University of Houston program is a football laboratory where his coordinators Slade Nagle and Austin Armstrong can concentrate on what matters. "We've got so many coaches at this level," Fritz says. "One of the things I try to do is take a lot of things off the plate of my coordinators, my coaches, so that they can work on football. "And that's what they do. They love football. They're talking football all day, every day. "A lot of that is how can we enhance our scheme. And what does our guys do best. What do we need to stay away from. What do we need to do more of. How can we teach this fundamental better."

Chris Baldwin

29,201 views • 2 months ago

🚀 New Paper: Pixel Reasoner 🧠🖼️ How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself? We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning. Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn. 🔍 So we ask: What if we could make VLMs "show their work" by reasoning directly in the pixel space? Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs. To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning ✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks: 📊 84% on InfographicsVQA 🧠 84% on V* benchmark 🧩 74% on TallyQA-Complex It also achieves strong accuracy of 68% on MVBench (a video benchmark). Website: Paper: Code: Demo: (coming soon)
0:43

🚀 New Paper: Pixel Reasoner 🧠🖼️ How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself? We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning. Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn. 🔍 So we ask: What if we could make VLMs "show their work" by reasoning directly in the pixel space? Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs. To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning ✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks: 📊 84% on InfographicsVQA 🧠 84% on V* benchmark 🧩 74% on TallyQA-Complex It also achieves strong accuracy of 68% on MVBench (a video benchmark). Website: Paper: Code: Demo: (coming soon)

Wenhu Chen

82,829 views • 1 year ago

#ICCV2025 🤩3D world generation is cool, but it is cooler to play with the worlds using 3D actions 👆💨, and see what happens! — Introducing *WonderPlay*: Now you can create dynamic 3D scenes that respond to your 3D actions from a single image! Web: 🧵1/7
0:46

#ICCV2025 🤩3D world generation is cool, but it is cooler to play with the worlds using 3D actions 👆💨, and see what happens! — Introducing *WonderPlay*: Now you can create dynamic 3D scenes that respond to your 3D actions from a single image! Web: 🧵1/7

Hong-Xing (Koven) Yu

58,018 views • 1 year ago