🚨Is it possible to devise an intuitive approach for crowdsourcing trainable data for robots without…
@DJiafei8.6K followers
38,957 views #CoRL2023
11 Comments
Comments from the original post on X (Twitter)

@DJiafei
🧵1/10 Manually curated datasets are often the unsung heroes in machine learning, especially in robotics, where human-generated datasets of robot demonstrations are indispensable, especially with recent success in robot learning via imitation learning of these demonstration data.
2 likes

@DJiafei
🧵2/10 AR2-D2 is a framework in the form of an iOS app that people can use to record a video of themselves manipulating any object while simultaneously capturing essential data modalities for training a real robot.
2 likes

@DJiafei
🧵3/10 No robot, no problem! Our system enables parallel data collection without costly real robots. With AR2-D2, you can capture demos outside the lab. Record anywhere, and manipulate diverse objects easily.
2 likes

@DJiafei
🧵4/10 How to get robot data from AR2-D2? Record yourself manipulating an object. AR2-D2 captures 6D hand pose, hand state, RGB frames, and depth and aligns it in real-time with AR robot trajectory. The data collected can be used to train both 2D and 3D BC agents.
2 likes

@DJiafei
🧵5/10 AR2D2 allows for the creation of realistic RGB robot video data with high-fidelity real-time object interaction between real-world objects and virtual robots.
1 like

@DJiafei
🧵6/10 AR2-D2 makes robot data collection quick, user-friendly, and intuitive. In a study, we compared it to 4 other methods like 3D mouse and kinesthetic teaching. Results? AR2-D2 matches kinesthetic teaching in success rate and user preference, both outperforming other methods.
2 likes

@DJiafei
🧵7/10 To show that the AR data collected via our system can support learning on a real robot. We trained a BC agent for manipulating highly personalized objects and deployed it in the real world.
2 likes

@DJiafei
🧵8/10 We trained PerAct on this AR demonstration with some rapid domain fine-tuning and observed that our demonstrations yielded useful representation for training a real robot, and it trained policies as accurately as the demonstration collected from a real robot.
3 likes

@DJiafei
🧵9/10 We further evaluated our system on more complex tasks (with more key points), such as this “peg-a-tape” task, and it shows that BC policies can learn from the demonstration collected via our system.
2 likes

@DJiafei
🧵10/10 I would like to see how this work can be integrated into future Vision Pro or other AR glasses for a better experience. But more importantly, we hope to see how this work could help to democratize robot learning for all by truly crowdsourcing robot data from all.
3 likes

@DJiafei
Check out our paper at: Project page: iOS App will be released soon
5 likes