正在加载视频...
视频加载失败
🤖 What if a humanoid robot could make a hamburger from raw ingredients—all the way to your plate? 🔥 Excited to announce ViTacFormer: our new pipeline for next-level dexterous manipulation with active vision + high-resolution touch. 🎯 For the first time ever, we demonstrate ~2.5 minutes of continuous, autonomous... show more
11 条评论

Our hardware setup features an active vision system, a bi-manual robot arm, and two high-DoF dexterous hands, SharpaWave, equipped with high-resolution tactile sensors. To enable rich data collection, we use a teleoperation system with a precision exoskeleton for fine-grained arm and finger control, plus a VR interface to control the camera and guide demonstrations.

ViTacFormer is a unified visuo-tactile framework for dexterous manipulation. At its core is a cross-modal representation that fuses visual and tactile inputs via cross-attention layers. Our key insight: predicting future tactile states is more informative than simply perceiving current ones. We validate this both conceptually and empirically. To achieve this, we introduce a tactile prediction head that encourages the shared latent space to capture touch dynamics. The model then auto-regressively leverages predicted tactile signals to generate precise, long-horizon actions.

We benchmarked our proposed pipeline against strong baselines across four challenging manipulation tasks. ViTacFormer significantly outperforms existing methods—achieving around 50% improvement in success rate.

Our ablation study highlights the contribution of each component, confirming their importance to the overall pipeline’s performance.

Here we also show the failure mode from baseline models, highlighting the importance of our tactile signal input, touch processing module, and cross-modal representation learning.

We then explored the full capabilities of our system—and found it can handle super long-horizon tasks end-to-end. 🍔 Sit back and enjoy the hamburger-making policy in action!

This work wouldn’t have been possible without the incredible support from great collaborators @cosm_para13983 and @KaifengZhang4, and my amazing advisors @JitendraMalikCV and @pabbeel. Thank you all! 🙏 Code is fully released; check out our: Homepage: Paper link: Github:

AI isn’t one trade—it’s a layered ecosystem. Chips led, apps followed… what’s next?” 🧠 We mapped the shift across AI subsectors

Awesome!

It could significantly alter job roles and industry standards in food service.

Achieving full automation in food preparation presents complex challenges but could optimize consistency and hygiene.
