How to learn dexterous manipulation for any robot hand from a single human demonstration? Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.

We study the problem of "functional retargeting": with one human demonstration, learn a dexterous hand policy to manipulate the object to follow the demonstrated trajectory. In contrast to kinematic retargeting which does not produce feasible actions, we use human hand guidance but prioritize object tracking success.

Our method, DexMachina, is a curriculum-based RL algorithm guided by a task reward and auxiliary rewards. Each human demo defines an RL task: we use the object states and human hand data to define the reward terms and residual wrist actions.

Our key idea is a novel curriculum using "virtual object controllers": using the demonstration trajectory, they can drive the object to follow the targets on its own, such that the RL policy can learn through the entire demo sequence without worrying about dropping the object.

For evaluation, we built a simulation benchmark with 5 articulated objects and 7 demo clips from ARCTIC, and curated 6 open-source dexterous robot hand models, with varying sizes and kinematic designs. We show DexMachina significantly outperforms baseline methods, by 21% on average over previous state-of-the-art.

DexMachina lets us perform a functional comparison between different dexterous hands: we evaluate 6 hands on 4 challenging long-horizon tasks, and found that larger, fully actuated hands learn better and faster, and high DoF is more important than having human-like hand sizes – see our paper for more discussions on our empirical findings.

With the recent surge in new dexterous hand hardwares, we hope this work provides a useful platform for identifying desirable hardware capabilities and lower the contribution barrier for future research. Project website: arXiv: Joint work with my amazing collaborators at @Stanford and @NVIDIAAI: @YifanHou2, Dieter Fox, Yashraj Narang, @SongShuran*, @AjayMandlekar*

Which Machine Learning model can beat the market? Check out this post on my free Substack where I share code and commentary for several strategies that beat passively holding a tech stock.

This wins for "best method name"! These are nice results and I'm happy that ARCTIC was useful. I believe that human demonstration is the path to rapid progress in dexterous, task-driven, manipulation.

Clever paper title!

Congrats Mandi!!

Awesome demo