How to rotate a tomato with a potato using a robot hand? 🤖🍅🥔 Our new model, Robot Synesthesia, blends touch and vision to manipulate multiple objects, even non-convex ones like a cross!

Trained in simulation, our model generalizes to novel object shapes and transfers zero-shot to a real-world allegro hand. Kinect Camera for vision and FSR sensor for touch form our setup. (2/7)

To achieve this, we convert tactile sensor data into a point cloud, effectively merging vision and touch into one cohesive input. It's as if our robot has the ability to "see" its tactile interactions!  (3/7)

The architecture of our model is straightforward - PointNet processes augmented points from both vision and touch sensors. This is why we call it "Synesthesia", drawing parallels to humans who perceive color through touch. (4/7)

Our experiments visualized the critical set selected by PointNet. It's a mix - 42.7% touch-based points on average, rest mainly from finger tips, edges, and palm. (5/7)

Our policy didn't just stop at rotating tomato and potato. It also took on the challenging Ferrero and toy dinosaurs task, unseen during training. (6/7)

Work done w/ @Ying_yyyyyyyy, Haichuan Che,@binghao_huang,@zhaohengyin, Kang-Won Lee, Yi Wu, Soo-Chul Lim,@xiaolonw Check out our work here  Project: Paper: (7/7)

Eureka!!

Just realized it’s a great neural networks training tool.