Loading video...

Video Failed to Load

Go Home

(1/n) Since its publication in 2017, PPO has essentially become synonymous with RL. Today, we are excited to provide you with a better alternative - EPO.

wang_jianren's profile picture

Jianren Wang

1,497 subscribers

145,870 views • 1 year ago •via X (Twitter)

Science & TechnologyEducation
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

PPO has long dominated robot locomotion training in simulation. SAC, despite its sample efficiency, couldn't keep up. We analyze why: 🔗 🔥Integrated into RSL-RL, our approach requires only minimal changes, making SAC a drop-in alternative out of the box.
0:19

PPO has long dominated robot locomotion training in simulation. SAC, despite its sample efficiency, couldn't keep up. We analyze why: 🔗 🔥Integrated into RSL-RL, our approach requires only minimal changes, making SAC a drop-in alternative out of the box.

Robotic Systems Lab

45,107 views • 2 months ago

We found a new way to get language models to reason. 🤯 No RL, no training, no verifiers, no prompting. ❌ With better sampling, base models can achieve single-shot reasoning on par with (or better than!) GRPO while avoiding its characteristic loss in generation diversity.
0:11

We found a new way to get language models to reason. 🤯 No RL, no training, no verifiers, no prompting. ❌ With better sampling, base models can achieve single-shot reasoning on par with (or better than!) GRPO while avoiding its characteristic loss in generation diversity.

Aayush Karan

278,309 views • 10 months ago

Today, we are excited to announce a major partnership with GSK to deploy the latest Boltz models across GSK’s research organization!
0:15

Today, we are excited to announce a major partnership with GSK to deploy the latest Boltz models across GSK’s research organization!

Gabriele Corso

40,855 views • 2 months ago

Model-Free Reinforcement Learning (MFRL) has been alluring, especially with supercharged compute with physics on GPU. However, the methods use 0-th order gradients, and are often not the best optimizers. Can we do better than PPO in continuous control for robotics? Turns out yes! 🥳 tl;dr: Faster, better RL than PPO in continuous control 💪 The answer lies in using more information from the simulation. We are juicing the simulation on GPU as it is, why not use it for gradients as well? This has been a driving question in a series of our works. We first studied this problem in ICLR 2022 paper on Short Horizon Actor Critic Naive gradient based methods are stuck in local minima and have exploding/vanishing gradients. SHAC solved this problem truncated rollouts and model based value estimation, where the model is Differentiable Sim. This boosted sample efficiency and wall-clock time immensely especially in high dimensional systems such as humanoids Yet, given enough compute PPO often caught up. Our follow up paper on on Adaptive Horizon Actor Critic at ICML 2024 discovers the cause and provides a fix. However, we find that even when given ground-truth dynamics, not all gradients are useful due to sample error. 1st-Order Model-Based Reinforcement Learning methods employing differentiable simulation provide gradients with reduced variance but are susceptible to bias in scenarios involving stiff dynamics, such as physical contact. We find that back-propagating through contact and long trajectories drastically reduces gradient accuracy. Using this insight, we propose AHAC to dynamically adapt its roll-out horizon to avoid differentiating through stiff contact. AHAC is a first-order model-based RL algorithm that learns high-dimensional tasks in minutes (wall clock) and outperforms PPO by 40%, even in the limit of data provided to PPO. This work is led by Ignat Georgiev alongside Krishnan Srinivasan, Jie Xu, Eric Heiden and ample assistance from warp team at NVIDIA Robotics (Miles Macklin)
0:10

Model-Free Reinforcement Learning (MFRL) has been alluring, especially with supercharged compute with physics on GPU. However, the methods use 0-th order gradients, and are often not the best optimizers. Can we do better than PPO in continuous control for robotics? Turns out yes! 🥳 tl;dr: Faster, better RL than PPO in continuous control 💪 The answer lies in using more information from the simulation. We are juicing the simulation on GPU as it is, why not use it for gradients as well? This has been a driving question in a series of our works. We first studied this problem in ICLR 2022 paper on Short Horizon Actor Critic Naive gradient based methods are stuck in local minima and have exploding/vanishing gradients. SHAC solved this problem truncated rollouts and model based value estimation, where the model is Differentiable Sim. This boosted sample efficiency and wall-clock time immensely especially in high dimensional systems such as humanoids Yet, given enough compute PPO often caught up. Our follow up paper on on Adaptive Horizon Actor Critic at ICML 2024 discovers the cause and provides a fix. However, we find that even when given ground-truth dynamics, not all gradients are useful due to sample error. 1st-Order Model-Based Reinforcement Learning methods employing differentiable simulation provide gradients with reduced variance but are susceptible to bias in scenarios involving stiff dynamics, such as physical contact. We find that back-propagating through contact and long trajectories drastically reduces gradient accuracy. Using this insight, we propose AHAC to dynamically adapt its roll-out horizon to avoid differentiating through stiff contact. AHAC is a first-order model-based RL algorithm that learns high-dimensional tasks in minutes (wall clock) and outperforms PPO by 40%, even in the limit of data provided to PPO. This work is led by Ignat Georgiev alongside Krishnan Srinivasan, Jie Xu, Eric Heiden and ample assistance from warp team at NVIDIA Robotics (Miles Macklin)

Animesh Garg

52,308 views • 2 years ago

Are you excited to be with God forever in paradise? A. Yes B. No
0:10

Are you excited to be with God forever in paradise? A. Yes B. No

Faith Alone Saves ✟

231,151 views • 2 years ago

Trump: When we do well, they shouldn't have to raise interest rates. They should in many cases lower because you know why? We become a better credit. When we're doing well, we become a better credit. When you become a better credit, you're supposed to cut rates.
0:19

Trump: When we do well, they shouldn't have to raise interest rates. They should in many cases lower because you know why? We become a better credit. When we're doing well, we become a better credit. When you become a better credit, you're supposed to cut rates.

Acyn

80,499 views • 3 hours ago

1/ 📣 Native USDC is Going xChain: Wormhole integrates Circle’s CCTP We're incredibly excited to announce that Wormhole has partnered with Circle to provide a seamless cross-chain development experience. Read everything about it in this 🧵
0:16

1/ 📣 Native USDC is Going xChain: Wormhole integrates Circle’s CCTP We're incredibly excited to announce that Wormhole has partnered with Circle to provide a seamless cross-chain development experience. Read everything about it in this 🧵

Wormhole

96,995 views • 3 years ago

Imagine believing that millions of these things erected almost silently overnight in every corner of The World are simply here to provide you with better internet.
0:14

Imagine believing that millions of these things erected almost silently overnight in every corner of The World are simply here to provide you with better internet.

Concerned Citizen

1,595,759 views • 1 year ago

you will be able to become friends with jaoba. STOP BEING RUDE IN MY DMS WE ARE DOING IT
0:16

you will be able to become friends with jaoba. STOP BEING RUDE IN MY DMS WE ARE DOING IT

not yeucc

133,718 views • 9 months ago

Making some business deals with my new partner Hailey Blush today, I'm really excited to partner with her and her associates with "X" and I think she's excited too! In fact we decided that signing with a smooch was better than pen!! 🤗😘🥰💯✨❤️‍🔥💗🤍 #blondegirl #blondeshavemorefun #sealitwithakiss #newpartnerships #cutegirls
0:15

Making some business deals with my new partner Hailey Blush today, I'm really excited to partner with her and her associates with "X" and I think she's excited too! In fact we decided that signing with a smooch was better than pen!! 🤗😘🥰💯✨❤️‍🔥💗🤍 #blondegirl #blondeshavemorefun #sealitwithakiss #newpartnerships #cutegirls

Tisha

45,079 views • 4 months ago

Ever since seeing Daniel getting along so well with a girl on campus, Kevin has been acting differently. "Hyung, are you upset with me about something today?" "Why won't you say anything?"
0:11

Ever since seeing Daniel getting along so well with a girl on campus, Kevin has been acting differently. "Hyung, are you upset with me about something today?" "Why won't you say anything?"

Kevin_boxer

50,859 views • 2 months ago

just a reminder that conspiracy theorists are the easiest people to sell to now you can become one in 2 mins with AI crazy times we live in
0:12

just a reminder that conspiracy theorists are the easiest people to sell to now you can become one in 2 mins with AI crazy times we live in

MAX

76,427 views • 7 months ago

Gun Fu Art with a Capital "A". In Tensor City, survival demands you become its maestro.
0:17

Gun Fu Art with a Capital "A". In Tensor City, survival demands you become its maestro.

SPINE — This is Gun Fu

28,320 views • 8 months ago

For the 8th year in row, and thanks to you, Air France has received the N°1 Customer Relationship Award in the « Transport » category. Year after year, we do our very best to maintain a strong link with you and to keep you satisfied. Thank you!
0:11

For the 8th year in row, and thanks to you, Air France has received the N°1 Customer Relationship Award in the « Transport » category. Year after year, we do our very best to maintain a strong link with you and to keep you satisfied. Thank you!

Air France

19,021 views • 3 years ago

Class of 2027, we are excited to get on the phone with you guys very soon! #GoWildcats
0:17

Class of 2027, we are excited to get on the phone with you guys very soon! #GoWildcats

ACU Women’s Basketball

14,408 views • 1 year ago

We are incredibly excited to announce that MANSA has raised $10M, led by and Polymorphic Capital , to revolutionize cross-border payments with with stablecoin-based liquidity solutions.
0:19

We are incredibly excited to announce that MANSA has raised $10M, led by and Polymorphic Capital , to revolutionize cross-border payments with with stablecoin-based liquidity solutions.

MANSA

124,359 views • 1 year ago