Can a single neural network policy generalize over poses, objects, obstacles, backgrounds, scene arrangements, in-hand objects, and start/goal states? Introducing Neural MP: A generalist policy for solving motion planning tasks in the real world 🤖 1/N

Quickly and dynamically moving around and in-between obstacles (motion planning) is a crucial skill for robots to manipulate the world around us. Traditional methods (sampling, optimization or search) can be slow and/or require strong assumptions to deploy in the real world. 2/N

Instead of solving each new motion planning problem from scratch, we distill knowledge across millions of problems into a generalist neural network policy. Our Approach: 1) large-scale procedural scene generation 2) multi-modal sequence modeling 3) test-time optimization 3/N

Data Generation involves: 1) Sampling programmatic assets (shelves, microwaves, cubbys, etc.) 2) Adding realistic objects from Objaverse 3) Generating data at scale using a motion planner expert (AIT*) - 1M demos! We distill all of this data into a single, generalist policy 4/N

Neural policies can hallucinate just like ChatGPT - this might not be safe to deploy! Our solution: Using the robot SDF, optimize for paths that have the least intersection of the robot with the scene. This technique improves deployment time success rate by 30-50%! 5/N

Across 64 real-world motion planning problems, Neural MP drastically outperforms prior work, beating out SOTA sampling-based planners by 23%, trajectory optimizers by 17% and learning-based planners by 79%, achieving an overall success rate of 95.83% 6/N

Neural MP extends directly to unstructured, in-the-wild scenes! From defrosting meat in the freezer and doing the dishes to tidying the cabinet and drying the plates, Neural MP does it all! 7/N

Neural MP generalizes gracefully to OOD scenarios as well. The sword in the first video is double the size of any in-hand object in the training set! Meanwhile the model has not seen anything like the bookcase during training, but it's still able to accurately place the book 8/N

Since, we train a closed-loop policy, Neural MP can perform dynamic obstacle avoidance as well! First, Jim tries to attack the robot with a sword, but it has excellent dodging skills. Then, he adds obstacles while the robot moves and it’s still able to safely reach its goal. 9/N

This work was done @CMU_Robotics with co-lead @Jiahui_Yang6709 as well as @mendonca_rl, Youssef Khaky, @rsalakhu, and @pathak2206 The model and hardware deployment code is open-sourced and on Huggingface! Run Neural MP on your robot today, check out 10/N

Congrats!!!! This is so cool