Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Does off-policy value-based RL scale? In LLMs, larger scale predictably improves performance. Value-based RL learns from arbitrary data and is sample-efficient, but folk wisdom says it doesn't scale 🧵⬇️We show predictability for scaling value-based RL!

_oleh's profile picture

Oleg Rybkin

1,333 subscribers

23,968 Aufrufe • vor 1 Jahr •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

🚨Current scalable RL algos train a policy w/o value func, which is limiting with learning in open-ended, non-stationary, dynamic environments. But, how to scale value-based RL with more data/compute is unclear... Not anymore: presenting scaling laws for value-based RL 🧵⬇️
0:10

🚨Current scalable RL algos train a policy w/o value func, which is limiting with learning in open-ended, non-stationary, dynamic environments. But, how to scale value-based RL with more data/compute is unclear... Not anymore: presenting scaling laws for value-based RL 🧵⬇️

Aviral Kumar

37,284 Aufrufe • vor 1 Jahr

Introducing CQN: Coarse-to-fine Q-Network, a value-based RL algorithm for continuous control🦾Initialized with 20~50 demonstrations, it learns to solve real-world robotic tasks within 10 mins of training, without any pre-training and shaped rewards! (1/4)
0:10

Introducing CQN: Coarse-to-fine Q-Network, a value-based RL algorithm for continuous control🦾Initialized with 20~50 demonstrations, it learns to solve real-world robotic tasks within 10 mins of training, without any pre-training and shaped rewards! (1/4)

Younggyo Seo

16,413 Aufrufe • vor 1 Jahr

New research from Databricks: LLMs Can Learn to Reason via Off-Policy RL Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL) shows you don’t need strict on-policy training to improve reasoning. It matches or beats Group Relative Policy Optimization (GRPO), stays stable with large policy lag, and uses ~3× fewer training generations. For Databricks customers, it’s a simpler, practical, and equally powerful approach to RL that Databricks is pioneering internally — and bringing directly to Databricks customers, so enterprises can improve agents using the same methods we use for our in-house agents, without complex infrastructure changes.
0:13

New research from Databricks: LLMs Can Learn to Reason via Off-Policy RL Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL) shows you don’t need strict on-policy training to improve reasoning. It matches or beats Group Relative Policy Optimization (GRPO), stays stable with large policy lag, and uses ~3× fewer training generations. For Databricks customers, it’s a simpler, practical, and equally powerful approach to RL that Databricks is pioneering internally — and bringing directly to Databricks customers, so enterprises can improve agents using the same methods we use for our in-house agents, without complex infrastructure changes.

Databricks AI Research

12,439 Aufrufe • vor 3 Monaten

US-based K-Scale Labs launched pre-orders for its open-source humanoid, priced at $9K The K-Bot stands 4′7″, weighs 77lbs, and integrates with K-Scale's open-source stack - covering RL/VLA models, Rust firmware, sim2real pipeline, and hardware design
0:17

US-based K-Scale Labs launched pre-orders for its open-source humanoid, priced at $9K The K-Bot stands 4′7″, weighs 77lbs, and integrates with K-Scale's open-source stack - covering RL/VLA models, Rust firmware, sim2real pipeline, and hardware design

Brett Adcock

10,994 Aufrufe • vor 11 Monaten

1/ While most RL methods use shallow MLPs (~2–5 layers), we show that scaling up to 1000-layers for contrastive RL (CRL) can significantly boost performance, ranging from doubling performance to 50x on a diverse suite of robotic tasks. Webpage+Paper+Code:
0:15

1/ While most RL methods use shallow MLPs (~2–5 layers), we show that scaling up to 1000-layers for contrastive RL (CRL) can significantly boost performance, ranging from doubling performance to 50x on a diverse suite of robotic tasks. Webpage+Paper+Code:

Kevin Wang

154,516 Aufrufe • vor 1 Jahr

Your value doesn't decrease based on someone's inability to see your worth 💌
0:12

Your value doesn't decrease based on someone's inability to see your worth 💌

Persephanii Aka Thick Yonce

380,674 Aufrufe • vor 2 Jahren

Why am I working on RL for LLMs, when I can go work on RL for Olaf... #humanoidssummit
0:10

Why am I working on RL for LLMs, when I can go work on RL for Olaf... #humanoidssummit

Shane Gu

12,440 Aufrufe • vor 6 Monaten

Frontier research just crossed a new threshold. Mind Lab is now public. And it begins with trillion-scale RL trained at only 10% of the usual compute.
0:13

Frontier research just crossed a new threshold. Mind Lab is now public. And it begins with trillion-scale RL trained at only 10% of the usual compute.

Chidanand Tripathi

89,813 Aufrufe • vor 6 Monaten

Thanks AK! Finally, robot can do continuous, agile, autonomous, adaptive jumping over stair and stepping stone Key idea: combine the pros of model-free RL and model-based control. RL (for CoM refs) + QP (for GRF) + WBC (for torque) Open-sourced:
0:10

Thanks AK! Finally, robot can do continuous, agile, autonomous, adaptive jumping over stair and stepping stone Key idea: combine the pros of model-free RL and model-based control. RL (for CoM refs) + QP (for GRF) + WBC (for torque) Open-sourced:

Guanya Shi

32,142 Aufrufe • vor 1 Jahr

Does LLM RL post-training need to be on-policy?
0:13

Does LLM RL post-training need to be on-policy?

Kianté Brantley

113,263 Aufrufe • vor 3 Monaten

RL is back! But is it always the best choice? In a new paper, we investigate under what circumstances neuroevolution outperforms RL in transfer learning tasks. See more details in the thread below 🧵 While NE performs best in simpler domains, it will be interesting to see if the lessons learned here can also be applied to more complex systems/tasks (LLMs?).
0:13

RL is back! But is it always the best choice? In a new paper, we investigate under what circumstances neuroevolution outperforms RL in transfer learning tasks. See more details in the thread below 🧵 While NE performs best in simpler domains, it will be interesting to see if the lessons learned here can also be applied to more complex systems/tasks (LLMs?).

Sebastian Risi

11,130 Aufrufe • vor 10 Monaten

X_Acc_Flags - for ios The value of ‘Location (accurate)’ is coming from X’s servers, so I have no idea what it was based on Source-Code :
0:17

X_Acc_Flags - for ios The value of ‘Location (accurate)’ is coming from X’s servers, so I have no idea what it was based on Source-Code :

 CrazyMind

19,392 Aufrufe • vor 6 Monaten

"I think just based on vehicle autonomy, we can 10x the value of the company." 一 Elon Musk
0:13

"I think just based on vehicle autonomy, we can 10x the value of the company." 一 Elon Musk

DogeDesigner

37,893 Aufrufe • vor 11 Monaten

Crypto can’t scale without solving value transfer. Not just between blockchains but across fiat, stablecoins, and bank networks. We’re building it in Kima! From CBDCs to DEXs, from TradFi to DeFi this is how money moves in the bridgeless Kima era. Join us. 💥
0:12

Crypto can’t scale without solving value transfer. Not just between blockchains but across fiat, stablecoins, and bank networks. We’re building it in Kima! From CBDCs to DEXs, from TradFi to DeFi this is how money moves in the bridgeless Kima era. Join us. 💥

Kima Network

25,026 Aufrufe • vor 11 Monaten

Polygon 2.0 is a concrete vision to build the Value Layer of the Internet, and expand Ethereum to Internet-scale. It is a series of proposals for unlimited scalability & unified liquidity.
0:14

Polygon 2.0 is a concrete vision to build the Value Layer of the Internet, and expand Ethereum to Internet-scale. It is a series of proposals for unlimited scalability & unified liquidity.

Polygon | POL

72,469 Aufrufe • vor 3 Jahren

This figure from HIL-SERL is one of the clearest visualisations of how RL learns differently from imitation learning. The difference comes down to this: imitation learning treats each (state, action) pair as independent. A correction at timestep 20 teaches nothing about timestep 19 or 21. RL propagates reward backward through time. One successful insertion updates the value estimate of every state along the trajectory. So RL builds a full map of "which states lead to success"; imitation learning just memorizes individual snapshots. Setup: a robot inserting a RAM stick into a motherboard slot. Each dot is an end-effector position (Y = lateral, Z = height). Starting position is randomized. Left to right = training progressing. Top row (RL): the policy builds a funnel. Broad at the top, narrowing into the target. It systematically fills in the state space, learning which paths lead to success from many different starting positions. Bottom row (imitation learning / HG-DAgger, same human data): sparse, diffuse, no funnel. The policy only learns near states the human demonstrated. Both have access to the same data, including human corrections, but a completely different structure emerges.
0:14

This figure from HIL-SERL is one of the clearest visualisations of how RL learns differently from imitation learning. The difference comes down to this: imitation learning treats each (state, action) pair as independent. A correction at timestep 20 teaches nothing about timestep 19 or 21. RL propagates reward backward through time. One successful insertion updates the value estimate of every state along the trajectory. So RL builds a full map of "which states lead to success"; imitation learning just memorizes individual snapshots. Setup: a robot inserting a RAM stick into a motherboard slot. Each dot is an end-effector position (Y = lateral, Z = height). Starting position is randomized. Left to right = training progressing. Top row (RL): the policy builds a funnel. Broad at the top, narrowing into the target. It systematically fills in the state space, learning which paths lead to success from many different starting positions. Bottom row (imitation learning / HG-DAgger, same human data): sparse, diffuse, no funnel. The policy only learns near states the human demonstrated. Both have access to the same data, including human corrections, but a completely different structure emerges.

Dominique Paul

24,399 Aufrufe • vor 3 Monaten

Pay for what you create, not for seats. FLORA's introducing credit-based pricing: make your ROI measurable at scale.
0:13

Pay for what you create, not for seats. FLORA's introducing credit-based pricing: make your ROI measurable at scale.

FLORA ©

22,998 Aufrufe • vor 4 Monaten

Some personal news: I recently joined Cursor. Cursor is a small, ambitious team, and they’ve created my favorite AI systems. We’re now building frontier RL models at scale in real-world coding environments. Excited for how good coding is going to be.
0:18

Some personal news: I recently joined Cursor. Cursor is a small, ambitious team, and they’ve created my favorite AI systems. We’re now building frontier RL models at scale in real-world coding environments. Excited for how good coding is going to be.

Sasha Rush

335,873 Aufrufe • vor 1 Jahr

Video diffusion models are just overqualified depth estimators! Deterministic single-pass depth estimation based on WanV2.1. - SOTA 5.5 AbsRel on ScanNet - data-efficient than baselines; - no temporal flicker + infinite-length estimation w/ zero scale drift.
0:18

Video diffusion models are just overqualified depth estimators! Deterministic single-pass depth estimation based on WanV2.1. - SOTA 5.5 AbsRel on ScanNet - data-efficient than baselines; - no temporal flicker + infinite-length estimation w/ zero scale drift.

Wildminder

49,235 Aufrufe • vor 2 Monaten

How to return a value based on a criteria using the IF function. 🤓 #excel
0:19

How to return a value based on a criteria using the IF function. 🤓 #excel

Excel Dictionary

153,781 Aufrufe • vor 3 Jahren