Can we bring human-like Touch to robots🤖? Introducing our CoRL work on 3D-ViTac. Humans rely on both vision 👁️ and touch 🫳 for complex tasks. With combined visual-tactile sensing, robots can now tackle challenging tasks, like precise in-hand reorientation, fragile objects grasping. Website: #Robotics #CoRL2024 #Touch #tactile #AI #ML

An open-source tactile sensing system from @Columbia designed to democratize touch to robot data collection. Please check out our website for tutorials ( The sensor is equipped with dense sensing units, each covering an area of 3 millimeter square. These sensors are low-cost and flexible, providing detailed and extensive coverage of physical contacts, effectively complementing visual information. (2/9)

We install flexible tactile sensors on soft gripper to get touch information and use multi-view RGBD cameras to get visual information. To integrate tactile and visual data, we fuse them into a unified 3D representation space that preserves their 3D structures and spatial relationships.(3/9)

With 3D-ViTac, robots can now handle delicate objects, like eggs, with ease! 🥚 In this Egg Steaming task, the robot first uses its right hand to open the egg tray, then carefully grasps and places an egg into a egg cooker (narrow space). (4/9)

The robot keeps retrying until it successfully grasps the grapes even under huge visual occlusions. (5/9)

We observe three key benefits of integrating touch. (1) Tactile sensors provide critical feedback on the presence of contact and the appropriate amount of force to apply. (2) Our policy leverages detailed contact patterns provided by touch to address visual occlusions effectively. (6/9)

This project is led by @binghao_huang and would have been impossible without the hard work from co-authors: @YXWangBot , Xinyi Yang, @LuoYiyue , @YunzhuLiYZ. (7/9)

Awesome!

Nice work. I’ve seen other tactile sensing work emphasize improvements picking and placing deformable objects. But this is the first one I’ve seen that emphasizes visual occlusion performance. The grape demo is really cool! Are there plans for your lab to continue iterating on the hardware for the tactile sensing system?

Thanks, Jason! Yes, we’re continuing to iterate on our hardware. Over the next few weeks, we'll release an easy-to-follow guide for reproducing our flexible sensor. In future iterations, we're planning to develop flexible PCBs to produce sensors directly from the factory, allowing for easy scaling.

Yes the human-like touch will come!