Does off-policy value-based RL scale? In LLMs, larger scale... predictably improves performance. Value-based RL learns from arbitrary data and is sample-efficient, but folk wisdom says it doesn't scale 🧵⬇️We show predictability for scaling value-based RL!show more

🚨Current scalable RL algos train a policy w/o value... func, which is limiting with learning in open-ended, non-stationary, dynamic environments. But, how to scale value-based RL with more data/compute is unclear... Not anymore: presenting scaling laws for value-based RL 🧵⬇️show more

Introducing CQN: Coarse-to-fine Q-Network, a value-based RL algorithm for... continuous control🦾Initialized with 20~50 demonstrations, it learns to solve real-world robotic tasks within 10 mins of training, without any pre-training and shaped rewards! (1/4)show more

New research from Databricks: LLMs Can Learn to Reason... via Off-Policy RL Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL) shows you don’t need strict on-policy training to improve reasoning. It matches or beats Group Relative Policy Optimization (GRPO), stays stable with large policy lag, and uses ~3× fewer training generations. For Databricks customers, it’s a simpler, practical, and equally powerful approach to RL that Databricks is pioneering internally — and bringing directly to Databricks customers, so enterprises can improve agents using the same methods we use for our in-house agents, without complex infrastructure changes.show more

US-based K-Scale Labs launched pre-orders for its open-source humanoid,... priced at $9K The K-Bot stands 4′7″, weighs 77lbs, and integrates with K-Scale's open-source stack - covering RL/VLA models, Rust firmware, sim2real pipeline, and hardware designshow more

1/ While most RL methods use shallow MLPs (~2–5... layers), we show that scaling up to 1000-layers for contrastive RL (CRL) can significantly boost performance, ranging from doubling performance to 50x on a diverse suite of robotic tasks. Webpage+Paper+Code:show more

Your value doesn't decrease based on someone's inability to... see your worth 💌show more

Why am I working on RL for LLMs, when... I can go work on RL for Olaf... #humanoidssummitshow more

Frontier research just crossed a new threshold. Mind Lab... is now public. And it begins with trillion-scale RL trained at only 10% of the usual compute.show more

Thanks AK! Finally, robot can do continuous, agile, autonomous,... adaptive jumping over stair and stepping stone Key idea: combine the pros of model-free RL and model-based control. RL (for CoM refs) + QP (for GRF) + WBC (for torque) Open-sourced:show more

RL is back! But is it always the best... choice? In a new paper, we investigate under what circumstances neuroevolution outperforms RL in transfer learning tasks. See more details in the thread below 🧵 While NE performs best in simpler domains, it will be interesting to see if the lessons learned here can also be applied to more complex systems/tasks (LLMs?).show more

Does LLM RL post-training need to be on-policy?

X_Acc_Flags - for ios The value of ‘Location (accurate)’... is coming from X’s servers, so I have no idea what it was based on Source-Code :show more

"I think just based on vehicle autonomy, we can... 10x the value of the company." 一 Elon Muskshow more

Crypto can’t scale without solving value transfer. Not just... between blockchains but across fiat, stablecoins, and bank networks. We’re building it in Kima! From CBDCs to DEXs, from TradFi to DeFi this is how money moves in the bridgeless Kima era. Join us. 💥show more

Polygon 2.0 is a concrete vision to build the... Value Layer of the Internet, and expand Ethereum to Internet-scale. It is a series of proposals for unlimited scalability & unified liquidity.show more

Pay for what you create, not for seats. FLORA's... introducing credit-based pricing: make your ROI measurable at scale.show more

This figure from HIL-SERL is one of the clearest... visualisations of how RL learns differently from imitation learning. The difference comes down to this: imitation learning treats each (state, action) pair as independent. A correction at timestep 20 teaches nothing about timestep 19 or 21. RL propagates reward backward through time. One successful insertion updates the value estimate of every state along the trajectory. So RL builds a full map of "which states lead to success"; imitation learning just memorizes individual snapshots. Setup: a robot inserting a RAM stick into a motherboard slot. Each dot is an end-effector position (Y = lateral, Z = height). Starting position is randomized. Left to right = training progressing. Top row (RL): the policy builds a funnel. Broad at the top, narrowing into the target. It systematically fills in the state space, learning which paths lead to success from many different starting positions. Bottom row (imitation learning / HG-DAgger, same human data): sparse, diffuse, no funnel. The policy only learns near states the human demonstrated. Both have access to the same data, including human corrections, but a completely different structure emerges.show more

Some personal news: I recently joined Cursor. Cursor is... a small, ambitious team, and they’ve created my favorite AI systems. We’re now building frontier RL models at scale in real-world coding environments. Excited for how good coding is going to be.show more

Video diffusion models are just overqualified depth estimators! Deterministic... single-pass depth estimation based on WanV2.1. - SOTA 5.5 AbsRel on ScanNet - data-efficient than baselines; - no temporal flicker + infinite-length estimation w/ zero scale drift.show more

How to return a value based on a criteria... using the IF function. 🤓 #excelshow more