Skip navigation
If you want a vision encoder for dexterous manipulation, what should be the most important part to…
0:00 / 1:30

If you want a vision encoder for dexterous manipulation, what should be the most important part to…

Roei Herzig

@roeiherzig2.1K followers

Visit on X
148 likesDownloadOriginal post

27,598 views

If you want a vision encoder for dexterous manipulation, what should be the most important part to model? 🤔 Current standard models like CLIP, SigLIP, and DINOv2 have an incredible grasp of semantics and spatial details. But they lack the action-centric structure needed for downstream visuomotor control. But collecting annotated robotic trajectories at scale is SUPER expensive and largely unrealistic. So, how do we bridge this gap? We introduce CAIP (Contrastive Action-Image Pre-training) ⬇️ 🔸 Action-centric upstream: we align visual observations with action chunks through a contrastive objective. 🔸 Human video as a proxy: we represent 3D human hand poses analogously to robotic end-effector actions, tapping into a massive source of human demonstrations. 🔸 Massive scale: pre-trained on over 32,000 hours of manipulation video, driving both sample efficiency and robust generalization. 🔸 Hardware proven: achieves a 76% average success rate on a real-world Dexmate Vega bimanual Dexmate manipulator with dual 22-DoF Sharpa Wave hands Sharpa . 🔸 State-of-the-art: significantly outperforms strong baselines like DINOv2, SigLIP, MVP, and Qwen3.5 ViT across complex tasks, even under unexpected lighting changes and visual distractors. 🌐 Project: 📝 Blog: 📄 Paper: 💻 Model:

5 Comments

Comments from the original post on X (Twitter)

  • Sharpa's profile picture

    @SharpaRobotics

    Great achievement! another step towards making pre-trained models generalize from egocentric videos. And a good example of how useful it is to have anthropomorphic hand for inference on the robot!

    4 likes

  • Roei Herzig's profile picture

    @roeiherzig

    How easy is it to use? This easy:

    2 likes

  • Roei Herzig's profile picture

    @roeiherzig

    This work would not have been possible without an incredible roster of co-authors and collaborators. First and foremost, a massive thank you to the leads who spearheaded this research and turned a massive two-year vision into reality: 🔸Yuvan Sharma, 🔸Dantong Niu @Dantong_Niu, and 🔸Anirudh Pai @apai253. Building robust robotics takes a village. A huge shoutout to the dedicated team who spent countless hours collecting data and pushing through rigorous evaluations. Flawless execution from: 🔸Zekai Wang, 🔸Zhuoyang Liu @liu_zhuoyang13, 🔸Baifeng Shi @baifeng_shi, 🔸Stefano Saravalle, 🔸Boning Shao, 🔸Ruijie Zheng, 🔸Jing Wang, 🔸Konstantinos Kallidromitis, 🔸Yusuke Kato, and 🔸Fabio Galasso! Finally, special thanks for the invaluable mentorship, support, and help shaping the narrative: 🔸Yuke Zhu @yukez, 🔸Danfei Xu @danfei_xu, 🔸Linxi Fan @DrJimFan, 🔸Trevor Darrell @trevordarrell, and 🔸Jitendra Malik @JitendraMalikCV. A truly monumental team effort from BAIR @berkeley_ai and NVIDIA @nvidia! 🚀

  • Pier-Luc's profile picture

    @PLNadsLab

    This is awesome. Looking forward to building with it

    1 like

  • Nurvai - The Data Layer for Physical AI's profile picture

    @nurvai_ai

    Action-aware vision feels like a key piece for robotics. Recognizing objects is one thing, but understanding what can be done with them is where embodied models need to go. How will these representations change as more robot data becomes available?