Introducing Omnigrasp: Grasping Diverse Objects with Simulated Humanoids. With Omnigrasp, we show that we can control a humanoid equipped with dexterous hands to grasp diverse objects (>1200) and follow diverse trajectories, with one policy! 🌐: 📜:

Learning physically realistic human-object interaction is one of the major advantage of using humanoids and physics simulation. With Omnigrasp, we learn grasping policy with no paired full-body motion. Here is the results on the Oakink ( which has no paired MoCap data. During inference, all we need is the object mesh and desired trajectory for full body grasping, no need for reference full body pose or grasping point!

The center piece of Omnigrasp is PULSE-X, an extension to PULSE ( ICLR 2024) that has fingers. Having this universal and dexterous motion representation enables us to learn faster and scale better. Much like human don't randomly "jitter" to learn to manipulation objects, PULSE-X provides a strong motion prior so the policy can explore much more effectively. And with PULSE-X as the action space, the state and reward designs are incredibly simple! Just object shape, object state, and desired trajectory. We use pre-grasp from PGDM ( from @SudeepDasari) in the reward design and finds it very effective in guiding the policy to learn grasps.

Here is the grasps Omnigrasp learned. We do not constrain the policy to certain grasps, and it can learn diverse types of grasps via trial and error / RL. It even learns different grasps for the same object (last row).

Big objects? No problem!

This work is done at Meta Reality Labs, Pittsburgh, with the team @jinkuncao @sammy_j_c @awinkler_ @kkitani @xuweipeng000. Thanks everyone!!! Many more funny object interaction videos incoming!

So reference trajectories must include contact force data?

The state contains contact forces, which inform the humanoid of the current contact information of its fingers. c_t is not used in reward computation, so reference trajectory does not need to include force data.

caught bug

Problem has been fixed m'lord.