Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

We can learn a model that provides shaped "process rewards" for robotic RL, that evolves automatically as the policy gets better. This improves performance on benchmarks, and works in the real world! Some fun new work with Raymond Tsao & Andrew Wagenmaker

Sergey Levine

140,164 subscribers

38,514 views • 4 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

5 Comments

Sergey Levine4 months ago

In the real world, this leads to drastically faster learning, in just a few thousand time steps.

Sergey Levine4 months ago

To learn more, check out website: paper:

Hari4 months ago

@ajwagenmaker This is cool! Every time I do robotics research I remember dense reward is still overpowered for task mastery, and the more self supervised we can learn the better :)

Manan4 months ago

@ajwagenmaker Cool!

Ferbin4 months ago

@ajwagenmaker Does that hold with actual robots, or just in simulation?

Related Videos

Building upon SimpleVLA-RL, we have implemented real-world RL on long-horizon dexterous tasks and witnessed a non-trivial (~relatively 300%) performance improvement over the SFT model, along with surprising capabilities on auto-recovery. Blog coming soon. The entire process uses very little data and training compute—basically costing no more than a single robotic arm—hinting that real-world generality for machines is actually within sight.

Building upon SimpleVLA-RL, we have implemented real-world RL on long-horizon dexterous tasks and witnessed a non-trivial (~relatively 300%) performance improvement over the SFT model, along with surprising capabilities on auto-recovery. Blog coming soon. The entire process uses very little data and training compute—basically costing no more than a single robotic arm—hinting that real-world generality for machines is actually within sight.

Ning Ding

93,483 views • 9 months ago

RL in the real world presents some big challenges, but also some really big opportunities. In our new work, HIL-SERL, Charles Xu, Jeffrey Wu, Jianlan Luo show that real-world RL can learn a huge range of precise and robust tasks, and perform them much faster than imitation.

RL in the real world presents some big challenges, but also some really big opportunities. In our new work, HIL-SERL, Charles Xu, Jeffrey Wu, Jianlan Luo show that real-world RL can learn a huge range of precise and robust tasks, and perform them much faster than imitation.

Sergey Levine

36,618 views • 2 years ago

If you want a robot to do something well, you need to know how to talk to it. If you don't, you can learn, with Semantic Action RL! In our paper, Jagdeep Bhatia, Andrew Wagenmaker, Will Chen show how RL over VLA prompts enables new tasks and learns blazing fast in the real world!

If you want a robot to do something well, you need to know how to talk to it. If you don't, you can learn, with Semantic Action RL! In our paper, Jagdeep Bhatia, Andrew Wagenmaker, Will Chen show how RL over VLA prompts enables new tasks and learns blazing fast in the real world!

Sergey Levine

32,162 views • 3 months ago

How do we run RL with real-time chunking (RTC)? In this work we figured out how to use a small RL policy with a large robot foundation model, where the RL policy observed more recent images (due to faster inference) and steers the policy toward better behaviors! A fun collaboration with Siemens, led by Brian Zhu, Momen Khalil, Emanuele Poggi from Siemens and E Harrison from Berkeley, with lots of amazing contributors!

How do we run RL with real-time chunking (RTC)? In this work we figured out how to use a small RL policy with a large robot foundation model, where the RL policy observed more recent images (due to faster inference) and steers the policy toward better behaviors! A fun collaboration with Siemens, led by Brian Zhu, Momen Khalil, Emanuele Poggi from Siemens and E Harrison from Berkeley, with lots of amazing contributors!

Sergey Levine

47,700 views • 1 month ago

Our vision is for AI that uses world models to adapt in new and dynamic environments and efficiently learn new skills. We’re sharing V-JEPA 2, a new world model with state-of-the-art performance in visual understanding and prediction. V-JEPA 2 is a 1.2 billion-parameter model, trained on video, that can enable zero-shot planning in robots—allowing them to plan and execute tasks in unfamiliar environments. Learn more about V-JEPA 2 ➡️ As we continue working toward our goal of achieving advanced machine intelligence (AMI), we’re also releasing three new benchmarks for evaluating how well existing models can reason about the physical world from video. Learn more and download the new benchmarks ➡️

Our vision is for AI that uses world models to adapt in new and dynamic environments and efficiently learn new skills. We’re sharing V-JEPA 2, a new world model with state-of-the-art performance in visual understanding and prediction. V-JEPA 2 is a 1.2 billion-parameter model, trained on video, that can enable zero-shot planning in robots—allowing them to plan and execute tasks in unfamiliar environments. Learn more about V-JEPA 2 ➡️ As we continue working toward our goal of achieving advanced machine intelligence (AMI), we’re also releasing three new benchmarks for evaluating how well existing models can reason about the physical world from video. Learn more and download the new benchmarks ➡️

AI at Meta

310,986 views • 1 year ago

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Jingyun Yang

88,312 views • 3 years ago

Our model can now learn from its own experience with RL! Our new π*0.6 model can more than double throughput over a base model trained without RL, and can perform real-world tasks: making espresso drinks, folding diverse laundry, and assembling boxes. More in the thread below.

Our model can now learn from its own experience with RL! Our new π*0.6 model can more than double throughput over a base model trained without RL, and can perform real-world tasks: making espresso drinks, folding diverse laundry, and assembling boxes. More in the thread below.

Physical Intelligence

712,988 views • 11 months ago

Self-supervised representation learning looks a bit like RL. What if we literally use RL as a SSL method for visual representations? Turns out that it works quite well. In new work by Dibya Ghosh, we show how this can be done:

Self-supervised representation learning looks a bit like RL. What if we literally use RL as a SSL method for visual representations? Turns out that it works quite well. In new work by Dibya Ghosh, we show how this can be done:

Sergey Levine

48,840 views • 1 year ago

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning *beyond* just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning beyond just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

Abhishek Gupta

13,659 views • 1 year ago

🦋 Last week, our team took delivery of a new quadruped robotic dog from Unitree — the Unitree Go 2 AIR. With Letta and the Animus framework, developers can build embodied agents that think, learn, and act in the real world. Watch how we plan to bring Animus to life on this new hardware. Build 🤖 on Animus

🦋 Last week, our team took delivery of a new quadruped robotic dog from Unitree — the Unitree Go 2 AIR. With Letta and the Animus framework, developers can build embodied agents that think, learn, and act in the real world. Watch how we plan to bring Animus to life on this new hardware. Build 🤖 on Animus

Animus

33,084 views • 1 year ago

The more AI can do, the more we need to ask what it should and shouldn’t do. OpenAI researcher Jason Wolfe joins host Andrew Mayne to explore the Model Spec, the public framework that defines how models are intended to behave. They break down how it works in practice, from the chain of command that resolves conflicting instructions to the way it evolves over time through real-world use, feedback, and new model capabilities.

The more AI can do, the more we need to ask what it should and shouldn’t do. OpenAI researcher Jason Wolfe joins host Andrew Mayne to explore the Model Spec, the public framework that defines how models are intended to behave. They break down how it works in practice, from the chain of command that resolves conflicting instructions to the way it evolves over time through real-world use, feedback, and new model capabilities.

OpenAI

226,383 views • 7 months ago

Some scene reconstruction R&D at Simulon that produces more detailed meshes quickly and is able to exclude moving objects without explicit segmentation. This improves occlusion of real-world objects and provides better meshes for 3D creators. Coming to the beta soon.

Some scene reconstruction R&D at Simulon that produces more detailed meshes quickly and is able to exclude moving objects without explicit segmentation. This improves occlusion of real-world objects and provides better meshes for 3D creators. Coming to the beta soon.

Divesh Naidoo

50,643 views • 2 years ago

We trained a robotic foundation model that can drive mobile robots in six different countries, and navigate Sproul Plaza in midday on the UC Berkeley campus! Some cool new work w/ noriaki_hirose, Lydia Ignatova, Kyle Stachowicz, Catherine Glossop, Dhruv Shah

We trained a robotic foundation model that can drive mobile robots in six different countries, and navigate Sproul Plaza in midday on the UC Berkeley campus! Some cool new work w/ noriaki_hirose, Lydia Ignatova, Kyle Stachowicz, Catherine Glossop, Dhruv Shah

Sergey Levine

52,979 views • 1 year ago

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐

📢 Announcing one of the most exciting works from us this year on scalable robot policy evaluation through real-to-sim transfer, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is orders of magnitude slower than the development of language agents. We propose a new framework where simulation performance strongly correlates with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing state-of-the-art correlation between simulation and reality for deformable object manipulation. It provides a scalable and reproducible evaluation engine for robot learning. 🌐

Yunzhu Li

39,960 views • 11 months ago

Among the policies we evaluated so far (ACT, DiT, SmolVLA, pi0, pi0.5), fine-tuned pi0.5 achieves the best performance on VLA-REPLICA. The trend is consistent with recent simulation benchmarks such as RoboLab. The policy behaviors in the real world:

Among the policies we evaluated so far (ACT, DiT, SmolVLA, pi0, pi0.5), fine-tuned pi0.5 achieves the best performance on VLA-REPLICA. The trend is consistent with recent simulation benchmarks such as RoboLab. The policy behaviors in the real world:

Yu Xiang

19,066 views • 5 months ago

Years before AI infrastructure became the most competitive market in tech, we were proving that behind-the-meter computing works at scale. Bitcoin gave us that proof. Real assets. Real operations. Real performance through real market cycles. Now that same model — the same power strategy, the same operational discipline — is expanding to serve AI and high-performance compute demand. The workload evolves. The infrastructure holds. $SLNH

Years before AI infrastructure became the most competitive market in tech, we were proving that behind-the-meter computing works at scale. Bitcoin gave us that proof. Real assets. Real operations. Real performance through real market cycles. Now that same model — the same power strategy, the same operational discipline — is expanding to serve AI and high-performance compute demand. The workload evolves. The infrastructure holds. $SLNH

Soluna

11,244 views • 4 months ago

New course 🚨 Learn to build AI apps that can process very long documents with the Jamba model in this course, built in partnership with AI21 Labs and taught by Chen Wang and Chen Almagor. Learn more and join for free:

New course 🚨 Learn to build AI apps that can process very long documents with the Jamba model in this course, built in partnership with AI21 Labs and taught by Chen Wang and Chen Almagor. Learn more and join for free:

DeepLearning.AI

10,882 views • 1 year ago

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

Chris Paxton

34,102 views • 4 months ago