Introducing tactile skin sim-to-real for dexterous in-hand translation! We propose a simulation model for ReSkin, a magnetic tactile sensing skin. It can simulate ternary shear and binary normal forces. More:

Three key design decisions for our tractable tactile skin model: 1) We use object point clouds to calculate intersecting collision volumes, in contrast to the typical handful of points for rigid contact. 2) Each taxel’s sensing range extends beyond its collision geometry to mimic a “soft” interaction, allowing for sensor activation without exerting reaction forces on the object. 3) We capture shear by including the full 3D object velocity to compute our tactile signals.

With this tactile information and proprioception, we can train a policy for generalizable in-hand translation. We conduct comprehensive real-world evaluations on OOD objects and different hand orientations.

An interesting thing we find is that Tactile Policies Explore More Finger Gaits. Greater joint state exploration is potentially a factor in task success and an indicator of gait adaptation.

Lead by amazing Jessica Yin and collaborate with @JitendraMalikCV, @LabPikul, Mark Yim, @t_hellebrekers Thanks the support from @AIatMeta, @GRASPlab, @berkeley_ai, and @UWMadEngr Website:  Paper: