🚨Is it possible to devise an intuitive approach for crowdsourcing trainable data for robots without requiring a physical robot🤖? Can we democratize robot learning for all?🧑‍🤝‍🧑 Check out our latest #CoRL2023 paper-> AR2-D2: Training a Robot Without a Robot

🧵1/10 Manually curated datasets are often the unsung heroes in machine learning, especially in robotics, where human-generated datasets of robot demonstrations are indispensable, especially with recent success in robot learning via imitation learning of these demonstration data.

🧵2/10 AR2-D2 is a framework in the form of an iOS app that people can use to record a video of themselves manipulating any object while simultaneously capturing essential data modalities for training a real robot.

🧵3/10 No robot, no problem! Our system enables parallel data collection without costly real robots. With AR2-D2, you can capture demos outside the lab. Record anywhere, and manipulate diverse objects easily.

🧵4/10 How to get robot data from AR2-D2? Record yourself manipulating an object. AR2-D2 captures 6D hand pose, hand state, RGB frames, and depth and aligns it in real-time with AR robot trajectory. The data collected can be used to train both 2D and 3D BC agents.

🧵5/10 AR2D2 allows for the creation of realistic RGB robot video data with high-fidelity real-time object interaction between real-world objects and virtual robots.

🧵6/10 AR2-D2 makes robot data collection quick, user-friendly, and intuitive. In a study, we compared it to 4 other methods like 3D mouse and kinesthetic teaching. Results? AR2-D2 matches kinesthetic teaching in success rate and user preference, both outperforming other methods.

🧵7/10 To show that the AR data collected via our system can support learning on a real robot. We trained a BC agent for manipulating highly personalized objects and deployed it in the real world.

🧵8/10 We trained PerAct on this AR demonstration with some rapid domain fine-tuning and observed that our demonstrations yielded useful representation for training a real robot, and it trained policies as accurately as the demonstration collected from a real robot.

🧵9/10 We further evaluated our system on more complex tasks (with more key points), such as this “peg-a-tape” task, and it shows that BC policies can learn from the demonstration collected via our system.

🧵10/10 I would like to see how this work can be integrated into future Vision Pro or other AR glasses for a better experience. But more importantly, we hope to see how this work could help to democratize robot learning for all by truly crowdsourcing robot data from all.

Check out our paper at: Project page: iOS App will be released soon