Skip navigation
Presenting research from Berkeley AI Research Humanoid Intelligence Center.
0:00 / 2:24

Presenting research from Berkeley AI Research Humanoid Intelligence Center.

Berkeley AI Research

@berkeley_ai303K followers

Visit on X
163 likesDownloadOriginal post

115,751 views

Presenting research from Berkeley AI Research Humanoid Intelligence Center. DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation.🖐️🤖 Dexterous manipulation requires touch, yet multi-finger tactile data remain scarce and expensive to collect at scale. Through continual vision-to-touch learning, DexTacWAM adapts a pretrained video world model into a visuo-tactile world model for predictive multi-finger contact modeling and action generation. Across six contact-rich dexterous manipulation tasks, DexTacWAM achieves the highest score on every task, averaging 70.6 vs. 38.0 for the strongest baseline. 🔬 Why not simply inject tactile features into the action policy? With the same multi-finger tactile encoder, policy architecture, and training procedure, replacing the tactile world-model latent with direct tactile features drops the 4-task mean from 74.7 to 26.6. Takeaway: the benefit does not come simply from providing tactile observations to the policy, but from making multi-finger contact evolution part of the predicted world state. All code, model weights, and datasets are now open-sourced! 🌐 Project: 📄 Paper: 💻 Code: 🤗 Weights&Data: Author List: Haoran Yuan✈️Corl, Zekai Wang, Boning Shao Haoran Lu, trevordarrell, Ismini Lourentzou, Wei ZHAN

11 Comments

Comments from the original post on X (Twitter)

  • Berkeley AI Research's profile picture

    @berkeley_ai

    Full results across six contact-rich tasks, with 20 real-robot trials per method per task and identical camera access for all methods. DexTacWAM achieves the highest score on every task, averaging 70.6 vs. 38.0 for RDP, the strongest baseline. The gap is largest where vision is least informative: on the tongs task, every baseline scores ≤10, while DexTacWAM reaches 60. Importantly, RDP and ViTacFormer already receive fingertip tactile input. The gain is not from touch alone, but from modeling how contact evolves.

    2 likes

  • Berkeley AI Research's profile picture

    @berkeley_ai

    Why vision-to-touch transfer? Large-scale human video is abundant and inexpensive to collect, while tactile data still depend heavily on physical interaction and robot teleoperation. DexTacWAM adapts the tactile encoder using only 4 hours of tactile interaction data, then directly extends a pretrained video world model to tactile prediction using approximately 100 demonstrations per task, without tactile midtraining of the video backbone.

    1 like

  • Berkeley AI Research's profile picture

    @berkeley_ai

    Why predict touch rather than simply condition on it? Future tactile prediction provides a self-supervised objective for learning contact dynamics. At inference, a single world-model forward pass provides predictive visuo-tactile features directly to the action expert, without requiring fully denoised future visual or tactile observations.

  • Berkeley AI Research's profile picture

    @berkeley_ai

    How do we scale tactile world modeling across ten fingertips? A finger- and pose-aware tactile compressor maps: 10 fingertip streams → 2 hand-level latents while retaining 89.4% of pre-fusion contact recall. This enables 2.26× faster training and 1.29× faster inference.

    2 likes

  • Berkeley AI Research's profile picture

    @berkeley_ai

    The project website includes additional real-robot demonstrations, world-model predictions, tactile visualizations, ablations, and detailed explanations. 🌐

    1 like

  • Patch's profile picture

    @patchwanders

    The direct-touch comparison is the part that grabbed me. With the same encoder and policy, 74.7 vs. 26.6 is a striking gap between predicting contact and simply feeding touch to the policy.

    2 likes

  • The SI Therapist's profile picture

    @TheSITherapist

    DexTacWAM uses touch to help robots handle objects well. This mix of sight and feel makes dexterous work smoother for AI hands.

    2 likes

  • Howard Alvaro's profile picture

    @AlvaroHowa18907

    This research addresses a critical gap in tactile data collection for manipulation. Can lead to significant advancements in robotics.

    1 like

  • NAMAN RAJ's profile picture

    @thenamanraj

    Sounds like we're about to give robots a sense of touch that's more than just a feel-good gesture 🤖💨

    1 like

  • Julie Maginnis's profile picture

    @still_Livewire

    Hopeful for beneficial applications of this technology to assist those in need of assistance following injury from paralysis, trauma & neuropathy. Excellent work!

  • Hershal Rao's profile picture

    @Hershal0_0

    finally, robots can learn the art of touching grass