Robotic intelligence requires dexterous tool use, but generalizing across tools is hard. Our CoRL23 paper combines semantics (affordances) with low-level control (sim2real) to show functional grasping that generalizes to hammers, drills and more! 1/n

w/ @shagunuppls @kenny__shaw, @pathak2206 Imagine grasping a hammer. You reach for the handle and not the head because of your understanding of how hammers work. In addition to low-level control (from sim) our robot needs a notion of high-level *functional affordances*. 2/

We use two complementary sources of data. First, we use internet data in the form of DINO features to get functional correspondences across objects. These generalize across object instances and categories in the wild! 3/

Second, we use large scale sim data to learn low-level control via RL. Naively training in the 16D joint space can lead to physically implausible behavior. We collect 2hrs of play data to construct an action space of realistic poses for the policy allowing easy transfer. 4/

Simple reward functions and large scale training lead to emergent behavior where grasps are adaptively adjusted on the fly. If the initial pregrasp is inaccurate, the policy automatically adjusts! 5/

The surprising part is that the low-level is trained only on hammers, yet it generalizes to many others at test time!! We use our very own custom dexterous hand LEAP hand ( More details Website: Paper: 6/6

Great job! Will it be open source?