Teaching robots to learn only from RGB human videos is hard! In Feel The Force (FTF), we teach robots to mimic the tactile feedback humans experience when handling objects. This allows for delicate, touch-sensitive tasks—like picking up a raw egg without breaking it. 🧵👇

There are three super simple ideas that makes FTF work: 1. record human touch responses using a latex glove retrofitted with AnySkin. 2. output tactile responses from Point Policy. 3. using a PD controller to pick with desired tactile response.

For the data collection tool, using the transparent gloves allows hand-pose estimators to work well, while being able to easily attach AnySkin.

The policy architecture is pretty much the usual Point Policy, with an additional output of desired tactile response. Training here is the same as Behavior Cloning with an additional tactile loss.

For the delicate tasks we look it, we find that learning without touch feedback is quite poor. This is intuitive as vision based policies find it difficult to reason about critical force amounts before breakage.

This work was led by @AdemiAdeniji & @JoliaChen, and a wonderful collaboration with @vincentjliu @venkyp2000 @haldar_siddhant @Raunaqmb & @pabbeel For the paper, videos and more details:

AI won't replace you, but a person using AI will. Join 500,000+ readers and learn how to use AI in just 5 minutes a day (for free).

👏👏👏👏

Integrating tactile feedback seems crucial for advancing robotic dexterity to match nuanced human object manipulation.

Humanoid robots can never be suitable for homemade use cases if they dont have this

Adding force feedback should improve manipulation precision over vision-only learning.

Modeling tactile signals seems essential for safer manipulation.