Our CoRL 2024 paper shows Reinforcement Learning can allow robots to learn skills via real-world practice, without any demonstrations or simulation engineering. Rewards are provided using language/vision models, and mobility of robots enables autonomous exploration. 1/N

This timelapse demonstrates the learning process, for moving the chair alternatively between the goals shown. We compare the behavior of the initial policy, to that of a later policy after about 8-10 hours of autonomous practice. 2/N

Exploration for complex robots is hard due to the very large action space, and since actions rarely provide useful signal for the task, as seen on the left. Instead, we encourage the robots to grasp objects before exploring how to perform tasks, seen on the right. 3/N

After making some progress on the task, the robot should not stagnate near goal states. We use goal cycles for mobile systems, where one goal serves to reset the other. This allows the robot to preserve state diversity. 4/N

To use this signal rich data to learn skills, we use RL. For greater sample efficiency, we combine RL with behavior priors that contain basic task knowledge. These priors can be planners with a simplified incomplete model, or procedurally generated motions. 5/N

For rewards, we use detection and segmentation models, which given the name of an object of interest, produce the corresponding mask. Combined with low-level depth observations, we get a state estimate of the object. This is compared to the desired goal state for reward. 6/N

Hence our framework consists of 3 main pieces - 1) Task-relevant autonomy, which ensures data collected is likely to have learning signal, 2) Efficient Control, which seeks to learn skills quickly using this signal, 3) Flexible supervision, for defining the learning signal. 7/N

We find task-relevant autonomy is critical for any meaningful progress. Here we compare our approach of efficient control to using only the behavior prior, or RL without the prior (all use task-relevant autonomy). The former is insufficient and the latter learns very slowly. 8/N

Thanks to the AI Institute for this collaboration, and my co-authors Emmanuel Panov, @b_k_bucher, Jiuguang Wang and @pathak2206 ! Please see the website ( and paper ( for more details. 9/9

Nice work, these look like some pretty challenging tasks! Cool to see more people doing online RL directly in the real world :)

Congrats!