A touch-aware humanoid manipulation policy that cleans the lab for you🧹🧪 Introducing Humanoid Touch Dream: a real-world system for dexterous, contact-rich humanoid loco-manipulation. Our key idea is simple: the policy predicts future hand forces and tactile latents alongside actions, within a single-stage training framework. 1/7

To make this possible, we first addressed the system integration bottleneck. For dexterous, contact-rich humanoid manipulation, we need: • stable whole-body execution • full use of dexterous-hand capability • touch sensing and touch-aware learning Our system combines an RL-based whole-body controller, upper-body IK, dexterous hand retargeting, VR teleoperation, and distributed tactile sensing, giving us a practical platform for collecting high-quality demonstrations for contact-rich humanoid manipulation. 2/7

At the model level, we propose HTD: Humanoid Transformer with Touch Dreaming. HTD is a modular encoder-decoder Transformer that takes: • multi-view vision • robot / hand proprioception • hand-joint force signals • tactile inputs It uses modality tokenizers, a shared Transformer trunk, modular action experts, and modular dream experts. 3/7

The core idea is touch dreaming. Alongside action prediction, the policy also predicts: • future hand-joint forces • future tactile latents Training remains single-stage: no separate tactile pretraining, and no separate tactile world model. An EMA target encoder provides stable latent supervision in a JEPA-style latent prediction setup. This video shows policy rollouts together with dreamed touch signals for visualization and interpretability. In practice, the dream experts are not used during policy rollout. 4/7

We evaluate on 5 real-world contact-rich tasks: • Insert-T: tight-tolerance insertion • Book Organization: pushing + grasping a thin rigid object • Towel Folding: long-horizon deformable manipulation • Cat Litter Scooping: low-profile tool use • Tea Serving: bimanual loco-manipulation with stable transport Together they cover a broad range of manipulation settings, contact modes, and whole-body coordination demands. 5/7

Results: HTD outperforms ACT baselines across all 5 tasks. Average success rate improves by 90.9% relative over the stronger baseline. Ablations also show: • simply adding touch as input is not enough • predicting future touch helps • predicting latent tactile targets works better than raw tactile prediction Latent tactile dreaming gives a 30% relative gain in success rate over raw tactile dreaming. 6/7

Learning Versatile Humanoid Manipulation with Touch Dreaming • Webpage: • Paper: Team: @yaru_niu, Zhenlong Fang, Binghong Chen, @ShuaiZhou234190, @revvozz, Hao Zhang, Bingqing Chen, Chen Qiu, Eric Tseng, Jonathan Francis, @zhao__ding. @cmusafeai @CMUMeche @BoschGlobal @CMU_Robotics 7/7

Based on those motions, I believe this robot could FINALLY clean my cat boxes! I'm sold! Sign me up for one.

For sure!

@YueminMao Great work! Congrats

@YueminMao Thanks Hongyi!

Fan of this work! When using inspire hands did you use heuristics to get force readings? or just joint forces from the teleop?

Thank you, Alfred! Yes, during teleoperation the human operator still needs to rely on visual heuristics to handle contact. Haptic-feedback gloves could be a promising direction for improving the teleoperation interface in future work.

Nice work Yaru! Congrats!

Thanks Yuanhang!

Omg, the crouched duckwalking is absolutely amazing haha

🦆🦆

My DIY projects have this cable chaos too! The tactile handling of contact-rich tasks looks smooth here 🤖

Looks amazing! What is the brand of this tactile dexterous hand?

Thanks Jiageng! We used Inspire DFTP hands for paper’s experiments and Inspire F1 for the lab clean-up demo.

We’ve had robots that can see and robots that can move. Bringing touch into the same prediction loop might be what finally makes those two play nicely in messy environments.

Exactly!