Tired of collecting interventions all day to train with DAgger? Introducing IntervenGen from NVIDIA Robotics + Berkeley AI Research. From just 10 corrective human interventions, IntervenGen generates 1000+ to cover broad robot mistake distributions 👇 🧵 1/

Imitation learning is brittle outside the training distribution. Online interventions help but take significant human data and effort to show how to recover from all the possible mistakes a robot may make. Can we instead automatically generate them? 🧵 2/

We build on NVIDIA MimicGen. By executing the robot policy during both *data collection* and *data generation,* the robot encounters novel mistake states, from which we can apply transformed recovery segments. 🧵 3/

We evaluate IntervenGen in contact-rich tasks and observe that it increases robustness up to 39x over existing baselines. IntervenGen with 10 source interventions outperforms a policy trained with 100 human interventions by 24%, with just 12% of the data collection effort. 🧵 4/

IntervenGen is especially helpful for improving robustness to object pose estimation error, where robots have inaccurate beliefs for where relevant objects are in the scene (e.g., due to sensor noise, occlusion, network delay, etc). 🧵 5/

We deployed a policy trained on IntervenGen data on a Franka robot zero-shot, i.e. without any real-world data collection, and it was able to reliably recover from inaccurate pose estimates. 🧵 6/

The policy adapted to dynamic pose changes in the environment without object tracking and was robust to both physical perturbations of the end effector as well as visual distractors in the scene. 🧵 7/

Great to collaborate with @AjayMandlekar @CaelanGarrett @Ken_Goldberg and Dieter Fox! Paper: Website:

@NVIDIARobotics @berkeley_ai @threadreaderapp unroll

@ryan_hoque @NVIDIARobotics @berkeley_ai @ThrowAw75202130 Saluti, here is your unroll: Talk to you soon. 🤖