In-hand object manipulation is a dexterity litmus test for robot hands. Our new system in Science Robotics Science Robotics can dynamically reorient many different objects in hand in the air. 📚Project website: 🧑‍💻Code: 🧵1/n

2/n Successfully reorienting complex shapes in hand is pivotal for versatile robotic tool use. Our system can dynamically reorient diverse real-world objects in hand while in the air, even when the hand faces downward.

3/n We leveraged the massively parallel simulator (Isaac Gym) to train the vision-based controller in simulation and achieved zero-shot transfer to the real robot. 📜 paper:

4/n Learning a controller for complex contact-rich tasks is time-consuming. We sped up training with a teacher-student framework. However, rendering in simulation still slowed training. We proposed a new two-stage student policy training that gave 7x acceleration (one GPU).

5/n We found merging state & vision information in the input spaces works better than merging them in the hidden space. The policy gets easier to train.

6/n Pre-training with synthetic point clouds (no simulation rendering) gave 7x speedup for student vision policy learning.

7/n Enabling in-air object reorientation is challenging— a falling object crashes the party. Our solution is simple yet effective: Train with a supporting table, but penalize contacts between objects and the table. This incentivizes the controller to keep objects aloft.

8/n Another trick that helps improve the sim-to-real transfer is doing robot dynamics identification. We leverage Isaac Gym's massive simulation capability to identify the robot dynamics parameters that make simulated behavior closest to the real robot.

9/n Our system uses just a depth camera and joint encoders—a low-cost, minimal sensor setup. Yet it can dynamically manipulate diverse objects, unlocking dexterous manipulation research for more people.

10/n (n=10) joint work with Megha, Siyang @NJ_WuSiyang , Vikash @Vikashplus , Ted, Pulkit @pulkitology. The code is open-sourced. 🧑‍💻 code: 📜 paper:

@SciRobotics finally! Congrats :)

@SciRobotics thanks, Beomjoon!