Introduce our #CoRL2023 (Oral) project: "Robot Parkour Learning" Using vision, our robots can climb over high obstacles, leap over large gaps, crawl beneath low barriers, squeeze through thin slits, and run. All done by one neural network running onboard. And it's open-source!

Trained fully in sim, our parkour policy has emergent re-trying behaviors, allowing the robot to attempt overcoming an obstacle multiple times if it initially fails. The robot learns to push against the obstacle, ensuring adequate run-up space for subsequent attempts.

Our parkour policy can be deployed to low-cost robots (e.g. A1, Go1) using only onboard compute (Nvidia Jetson), one onboard depth camera (Intel Realsense) and onboard power. No motion capture, LiDAR, multiple depth cams, heavy compute are used.

How do we train our parkour policy? Stage 1: RL pre-training with soft dynamics constraints. We allow robots to penetrate obstacles using an auto curriculum that encourages robots to gradually learn to overcome obstacles while minimizing penetrations.

Stage 2: RL fine-tuning with hard dynamics constraints. We enforce all dynamics constraints and fine-tune the behaviors learned in the pre-training stage with realistic dynamics.

Stage 3: Distillation. After each individual parkour skill is learned, we use DAgger to distill them into a single vision-based parkour policy (parametrized by a RNN) that has memory and can be deployed to a legged robot using only onboard perception and compute.

This project is led by @ziwenzhuang_leo and me. Advised by @chelseabfinn and @zhaohang0124.(Please consider follow @ziwenzhuang_leo for more cool robot demos in future!) Project website: Code: w/@wang_jianren, Chris and Sören

@chelseabfinn @zhaohang0124 @wang_jianren @StanfordAILab

@chelseabfinn @zhaohang0124 @wang_jianren @StanfordAILab CoRL 2023 Best Systems Paper Finalist!

The future getting wild

Impressive work