training for RL 2027 🥰😋

Peaking with *NEW* RL 2027 Video Settings

Does LLM RL post-training need to be on-policy?

New project! Flow Policy Gradients for Robot Control tldr;... a simple online RL recipe for training and fine-tuning flow policies for robots co-led w/ Hongsuk Benjamin Choi:show more

I've gotten a mujoco sim RL training loop for... a unitree robot at 200k SPS. I'm looking into the physics for friction, contact dynamics. My goal: can I reproduce & beat the mujoco playground RL baselines This is running in my web browser with raylib. Its the baselineshow more

What is Spider girl training for?🌚😋

1/ RL for improving task success has made LLMs... remarkably capable. But what happens when the model fails? Turns out, training for graceful failure can be surprisingly tricky! Very excited to share our new paper: Training Language Models to Fail Gracefully 🧵show more

Why am I working on RL for LLMs, when... I can go work on RL for Olaf... #humanoidssummitshow more

An electric horse you say? Dave's trying to bring... me to his lair (100% RL, sim2real 60m of training)show more

Training hard for what’s next, I gotta look good 😋

Introducing CQN: Coarse-to-fine Q-Network, a value-based RL algorithm for... continuous control🦾Initialized with 20~50 demonstrations, it learns to solve real-world robotic tasks within 10 mins of training, without any pre-training and shaped rewards! (1/4)show more

GPT-6 Astra test 4/n Prompt: "Implement pen spinning with... a dexterous hand. Use Isaac Lab for RL training, use the Sharpa hand, and create the pen mesh yourself. Give me a trained RL policy and a visualization video. You are free to search the web and download papers or anything else you need." After running autonomously for a day and a half (including training the policy), this is what we got: Credit: my student Chengyang Lishow more

Reinforcement learning should be able to improve upon behaviors... seen when training. In practice, RL agents often struggle to generalize to new long-horizon behaviors. Our new paper studies *horizon generalization*, the degree RL algorithms generalize to reaching distant goals. 1/show more

🚨 New: Integrating Harbor (Harbor Framework) for end-to-end Computer-Use... evaluation(for Windows and Linux) at scale with Thinking Machines' Tinker, OSWorld, Daytona, and bare-metal servers. We just added support for Computer Use, Tinker, and OSWorld to Harbor - a framework for evaluating agents and generating RL training data by running large-scale rollouts across parallel sandboxed environments and collecting trajectories for SFT and RL. Repo and blogpost below 👇show more

Training large models with RL typically runs on a... fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive. So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.show more

fell for a g*rl in the holy month….

here’s some dopamine for your wednesday afternoon 😋🥰

RL is painfully slow 😭 — bottlenecked by super-long... CoT rollout. 🔭 Sparse attention should help, but naive sparse rollout hits a brutal efficiency–stability tradeoff: A tedious trial-and-error sparsity sweep for each dense policy is required before an actual RL run. 🐤Sparrow chirps no more pain! Introduce Sparrow: Sparse Rollout for stable and efficient long-context RL. Sparrow finds that: 💡As long as we keep the tail distribution mismatch throughout the sparse rollout above a critical threshold, the RL training will be stable. 💡Even cooler! Through comprehensive control studies of Qwen3-1.7B, 4B, 8B thinking models RL with 40K rollout max length, the critical threshold stays constant across model sizes. 💡Sparrow then finds the optimal dynamic sparse schedule to reach the threshold with minimal cost. 💡Sparrow's findings are empirically validated to generalize in Qwen3-14B, and hold on both Math and Coding RL. 🐤Sparrow empirically helps achieve 2.2× / 2.4× / 2.0× rollout speedup on Qwen3 1.7B / 4B / 8B thinking models, while keeping training stability over extended RL steps. We release the 🐤bird in the following formats. [1/n] Paper: Code: Blog:show more