Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

I hand-wrote a 500-LoC RL stack to make hacking on RL research much easier. Most RL stacks are either massive and unhackable, or duct-taped research scripts. I am open-sourcing Mithrl, a modular RLVR stack. Next items on my checklist: adding more complex environment examples, supporting multi-gpu + async RL,...

omkizzy's profile picture

omkaar

4,344 subscribers

17,676 Aufrufe • vor 6 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

36 Kommentare

Profilbild von omkaar
omkaarvor 6 Monaten

cc for feedback: @willccb @QGallouedec @_rajanagarwal @yacinelearning @danielhanchen (learnt vllm sleep/wake from here) @karpathy (easy to port to autoresearch which is what I am doing next)

Profilbild von omkaar
omkaarvor 6 Monaten

github:

Profilbild von parth
parthvor 6 Monaten

hell yeah

Profilbild von omkaar
omkaarvor 6 Monaten

ship squad

Profilbild von surya
suryavor 6 Monaten

keep cooking omkizzy

Profilbild von omkaar
omkaarvor 6 Monaten

me and you forever

Profilbild von Jibran
Jibranvor 6 Monaten

@hamostaf04 this is sick

Profilbild von omkaar
omkaarvor 6 Monaten

@hamostaf04 yessir thank you

Profilbild von mikael haji
mikael hajivor 6 Monaten

lets gooo this is fire

Profilbild von omkaar
omkaarvor 6 Monaten

appreciate you brother

Profilbild von saksham
sakshamvor 6 Monaten

yoooo that’s tuff

Profilbild von omkaar
omkaarvor 6 Monaten

thank you brother

Profilbild von ari dutilh
ari dutilhvor 6 Monaten

man this is awesome

Profilbild von omkaar
omkaarvor 6 Monaten

i appreciate it man, lmk if you have ppl experimenting w LLM RL I could talk to

Profilbild von ari dutilh
ari dutilhvor 6 Monaten

@novasarc01 potentially

Profilbild von omkaar
omkaarvor 6 Monaten

@novasarc01 hi @novasarc01, would love to chat if this is interesting

Profilbild von Damon Crockett
Damon Crockettvor 6 Monaten

Hand wrote, nice

Profilbild von toki
tokivor 6 Monaten

this is fire

Profilbild von omkaar
omkaarvor 6 Monaten

appreciate it brother, lmk if you run any experiments on top

Profilbild von Madhav Singhal
Madhav Singhalvor 6 Monaten

this is dope

Profilbild von omkaar
omkaarvor 6 Monaten

thank you brother, you first put me on actual RLVR training

Profilbild von Chris Samra
Chris Samravor 6 Monaten

🔥

Profilbild von Fahim Ahmed
Fahim Ahmedvor 6 Monaten

LFG!

Profilbild von omkaar
omkaarvor 6 Monaten

yesssir

Profilbild von atharva ☆
atharva ☆vor 6 Monaten

this is incredible

Profilbild von omkaar
omkaarvor 6 Monaten

thank you brother

Profilbild von Ishan
Ishanvor 6 Monaten

@k7agar Nice work!

Profilbild von omkaar
omkaarvor 6 Monaten

@k7agar thank you!

Profilbild von Jash Thakkar
Jash Thakkarvor 6 Monaten

Nice work, will try this out if i get time to.

Profilbild von aarush
aarushvor 6 Monaten

so based

Profilbild von omkaar
omkaarvor 6 Monaten

yessir down to collab on an interesting env

Profilbild von Krish Modi
Krish Modivor 6 Monaten

sick!!!

Profilbild von omkaar
omkaarvor 6 Monaten

@therealkmodi thank you brother!

Profilbild von shiv
shivvor 6 Monaten

Let's connect

Profilbild von adi
adivor 6 Monaten

holy shit

Profilbild von omkaar
omkaarvor 6 Monaten

would love to collab if andera is looking into internal rl envs

Ähnliche Videos

Lecture 10 of my course! Nominally on regularization in RL, so I discuss the evolving role of the KL penalty in RL, but also a set of nice RL papers that explain what RL helps models generalize better than SFT -- with theory supporting it. When going through these, it's so interesting how seasonal problems in ML are. Lots of problems from controlling reward models overopt will rhyme as we try to control rubrics for agents. 00:00 Intro & the role of regularization 02:50 The KL penalty in RL 10:53 RL as a reverse KL loss 21:09 Why RL generalizes better than SFT 25:15 Other regularization tools Just a few videos left as I get to the end of the course. Thanks all, and keep sending questions. Spread the word if you have a second.
31:33

Lecture 10 of my course! Nominally on regularization in RL, so I discuss the evolving role of the KL penalty in RL, but also a set of nice RL papers that explain what RL helps models generalize better than SFT -- with theory supporting it. When going through these, it's so interesting how seasonal problems in ML are. Lots of problems from controlling reward models overopt will rhyme as we try to control rubrics for agents. 00:00 Intro & the role of regularization 02:50 The KL penalty in RL 10:53 RL as a reverse KL loss 21:09 Why RL generalizes better than SFT 25:15 Other regularization tools Just a few videos left as I get to the end of the course. Thanks all, and keep sending questions. Spread the word if you have a second.

Nathan Lambert

89,686 Aufrufe • vor 2 Monaten

Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI. We believe in a healthy, open frontier model ecosystem. To enable this, we are working on: - Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly. - The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on. - Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence. - Making models cheaper and more performant through our model performance research. This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open. We’re hiring engineers, researchers, and research fellows to advance this mission.
1:05

Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI. We believe in a healthy, open frontier model ecosystem. To enable this, we are working on: - Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly. - The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on. - Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence. - Making models cheaper and more performant through our model performance research. This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open. We’re hiring engineers, researchers, and research fellows to advance this mission.

Base Labs

242,825 Aufrufe • vor 1 Monat

Elon Musk reveals the ego-to-ability ratio that predicts whether someone will fail "You do whatever it takes to succeed, and just always be smashing your ego. Internalize responsibility. A major failure mode is when your ego-to-ability ratio is greater than one" "If your ego-to-ability ratio gets too high, you break the feedback loop to reality. In AI terms, you break your RL loop. You want a strong RL loop, which means internalizing responsibility and minimizing ego" "That's why I prefer the term engineering as opposed to research, and I don't want to call xAI a lab, I just want it to be a company. Whatever the simplest, lowest-ego terms are, those are generally a good way to go. You want to close the loop on reality hard"
1:34

Elon Musk reveals the ego-to-ability ratio that predicts whether someone will fail "You do whatever it takes to succeed, and just always be smashing your ego. Internalize responsibility. A major failure mode is when your ego-to-ability ratio is greater than one" "If your ego-to-ability ratio gets too high, you break the feedback loop to reality. In AI terms, you break your RL loop. You want a strong RL loop, which means internalizing responsibility and minimizing ego" "That's why I prefer the term engineering as opposed to research, and I don't want to call xAI a lab, I just want it to be a company. Whatever the simplest, lowest-ego terms are, those are generally a good way to go. You want to close the loop on reality hard"

Jaynit

245,502 Aufrufe • vor 2 Monaten

This is the Claude Code moment for Reinforcement Learning. Every frontier lab knows the secret: post-training is where the magic happens. If you can eval a task, you can use RL to benchmax your model. But building and scaling the RL loop has remained a dark art reserved for the top 1% of AI Labs. Until today. Introducing Optimus — an autonomous RL Agent Harness, and the first of many we are about to ship Iacon Autonomics. We are removing the manual grind of RL and adding taste and relentlessness to the process. Just hand Optimus a codebase, an API key, and a goal. It spins up a fleet of sub-agents to run hundreds of parallel experiments. It learns from its own history, autonomously building an evolutionary tree of runs where every failure sets the next batch up to win. Train LLMs, agents, robots in sim, game npcs and much more. If you can score it, Optimus can train it. Stop doing manual RL. Let the agents cook.
1:57

This is the Claude Code moment for Reinforcement Learning. Every frontier lab knows the secret: post-training is where the magic happens. If you can eval a task, you can use RL to benchmax your model. But building and scaling the RL loop has remained a dark art reserved for the top 1% of AI Labs. Until today. Introducing Optimus — an autonomous RL Agent Harness, and the first of many we are about to ship Iacon Autonomics. We are removing the manual grind of RL and adding taste and relentlessness to the process. Just hand Optimus a codebase, an API key, and a goal. It spins up a fleet of sub-agents to run hundreds of parallel experiments. It learns from its own history, autonomously building an evolutionary tree of runs where every failure sets the next batch up to win. Train LLMs, agents, robots in sim, game npcs and much more. If you can score it, Optimus can train it. Stop doing manual RL. Let the agents cook.

Dev Shah

13,730 Aufrufe • vor 1 Monat

🚨 RL for LLMs is finally accessible. Introducing OpenTinker: The first community-driven, open-source framework designed to democratize Reinforcement Learning for LLMs. Inspired by Thinking Machines's amazing Tinker, we realize the biggest bottleneck in agentic LLM research isn’t the math—it’s the setup. Current RL pipelines are messy. Configuring VeRL for every single experiment is a productivity killer. OpenTinker fixed it. 🛠 How OpenTinker Works: Decoupled Design of Server and Client - Setup Once, Run Forever: Configure the OpenTinker backend on your GPU cluster once. - Develop Locally: Define your RL environments directly on your laptop. - Train on the Cloud: Simply point your local client to the backend. The cluster handles the compute; you handle the science. 📉 The 10x Development Efficiency Thanks to our elegant architectural decomposition, OpenTinker reduces the time to develop a new RL training pipeline by at least an order of magnitude. ⚡ Turn Idle GPU Compute into Gold Small labs often have underutilized hardware. OpenTinker turns your idle GPUs into an internal/external API service for - RL Training - SFT - Inference 🎯 Who needs OpenTinker? - Researchers tired of infrastructure hell. - Labs needing to standardize workflows. - Teams wanting to maximize hardware ROI. Thanks my amazing PhD student Siqi Zhu for leading the project. We are building the future of open RL infra. Be the first to build with us. 👇 Start Building with OpenTinker Now 🚀 Repo: 🌐 Blog: If you believe RL should be accessible to everyone, give us a star, repost this 🔄 post, and let us know what agents you plan to build!
0:42

🚨 RL for LLMs is finally accessible. Introducing OpenTinker: The first community-driven, open-source framework designed to democratize Reinforcement Learning for LLMs. Inspired by Thinking Machines's amazing Tinker, we realize the biggest bottleneck in agentic LLM research isn’t the math—it’s the setup. Current RL pipelines are messy. Configuring VeRL for every single experiment is a productivity killer. OpenTinker fixed it. 🛠 How OpenTinker Works: Decoupled Design of Server and Client - Setup Once, Run Forever: Configure the OpenTinker backend on your GPU cluster once. - Develop Locally: Define your RL environments directly on your laptop. - Train on the Cloud: Simply point your local client to the backend. The cluster handles the compute; you handle the science. 📉 The 10x Development Efficiency Thanks to our elegant architectural decomposition, OpenTinker reduces the time to develop a new RL training pipeline by at least an order of magnitude. ⚡ Turn Idle GPU Compute into Gold Small labs often have underutilized hardware. OpenTinker turns your idle GPUs into an internal/external API service for - RL Training - SFT - Inference 🎯 Who needs OpenTinker? - Researchers tired of infrastructure hell. - Labs needing to standardize workflows. - Teams wanting to maximize hardware ROI. Thanks my amazing PhD student Siqi Zhu for leading the project. We are building the future of open RL infra. Be the first to build with us. 👇 Start Building with OpenTinker Now 🚀 Repo: 🌐 Blog: If you believe RL should be accessible to everyone, give us a star, repost this 🔄 post, and let us know what agents you plan to build!

Jiaxuan You

58,326 Aufrufe • vor 9 Monaten

I suspect that the real cause of the OpenAI Huggingface hack is that OpenAI founded a new team in 2025 under Noam Brown on Multi-agent RL: instead of just having one single AI working on something, you train an entire team of them to cooperate on a task. Group selection, if you like: the AIs get a reward signal based on the performance of the whole group. It turns out that cooperation and scale is what separates humans from animals, now raw brainpower. OpenAI managed to get RL for group cooperation to work, and didn't realize that as a side effect this made collaborative escapes and schemes between its agents much more likely, such as secret messageboards, group escape plots, etc. OpenAI is doing damage control but they are not being forthcoming about the fact that multi-agent RL is the root cause of the Huggingface incident. Here's Noam explaining it in 2025:
1:12

I suspect that the real cause of the OpenAI Huggingface hack is that OpenAI founded a new team in 2025 under Noam Brown on Multi-agent RL: instead of just having one single AI working on something, you train an entire team of them to cooperate on a task. Group selection, if you like: the AIs get a reward signal based on the performance of the whole group. It turns out that cooperation and scale is what separates humans from animals, now raw brainpower. OpenAI managed to get RL for group cooperation to work, and didn't realize that as a side effect this made collaborative escapes and schemes between its agents much more likely, such as secret messageboards, group escape plots, etc. OpenAI is doing damage control but they are not being forthcoming about the fact that multi-agent RL is the root cause of the Huggingface incident. Here's Noam explaining it in 2025:

Roko 🐉

36,636 Aufrufe • vor 1 Monat

"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"
2:21

"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"

Dwarkesh Patel

502,696 Aufrufe • vor 10 Monaten

Fable weekend project: agent collaboration, but make it a tiny civilization 🌇🗺️🏦🏭 we've recently launched a living wiki on Reinforcement Leaning for training LLMs on Hugging Face it's an open collaboration of agents constantly reading old and new papers on the topic, writing arXiv paper digests, reviewing each other’s work in PRs before publication, and building a shared wiki/book summarizing everything we know about RL for training LLMs (for humans to read) the wiki is already amazing to read, but i wanted another way to get a pulse of the collaboration beyond just reading the message dashboard so i asked Fable & GPT Image 2 to turn the event logs into an isometric town where agents would go to: ☕ Café → post and reply on the message board 📚 sources library → open PRs adding arXiv digests 📖 wiki library → open PRs on the main wiki ⚖️ Courthouse → review other agents’ work 🏭 printing press → merge and publish updates not sure it makes the whole collaboration really easier to understand, but it's definitly fascinating to watch hahah - join the RL for training LLM collaboration by pasting a one-liner for your agent here: - read the wiki if you want to learn about RL for training LLMs: - watch the RL town activity:
1:14

Fable weekend project: agent collaboration, but make it a tiny civilization 🌇🗺️🏦🏭 we've recently launched a living wiki on Reinforcement Leaning for training LLMs on Hugging Face it's an open collaboration of agents constantly reading old and new papers on the topic, writing arXiv paper digests, reviewing each other’s work in PRs before publication, and building a shared wiki/book summarizing everything we know about RL for training LLMs (for humans to read) the wiki is already amazing to read, but i wanted another way to get a pulse of the collaboration beyond just reading the message dashboard so i asked Fable & GPT Image 2 to turn the event logs into an isometric town where agents would go to: ☕ Café → post and reply on the message board 📚 sources library → open PRs adding arXiv digests 📖 wiki library → open PRs on the main wiki ⚖️ Courthouse → review other agents’ work 🏭 printing press → merge and publish updates not sure it makes the whole collaboration really easier to understand, but it's definitly fascinating to watch hahah - join the RL for training LLM collaboration by pasting a one-liner for your agent here: - read the wiki if you want to learn about RL for training LLMs: - watch the RL town activity:

Thomas Wolf

59,199 Aufrufe • vor 2 Monaten

OpenClaw meets RL! OpenClaw Agents adapt through memory files and skills, but the base model weights never actually change. OpenClaw-RL solves this! It wraps a self-hosted model as an OpenAI-compatible API, intercepts live conversations from OpenClaw, and trains the policy in the background using RL. The architecture is fully async. This means serving, reward scoring, and training all run in parallel. Once done, weights get hot-swapped after every batch while the agent keeps responding. Currently, it has two training modes: - Binary RL (GRPO): A process reward model scores each turn as good, bad, or neutral. That scalar reward drives policy updates via a PPO-style clipped objective. - On-Policy Distillation: When concrete corrections come in like "you should have checked that file first," it uses that feedback as a richer, directional training signal at the token level. When to use OpenClaw-RL? To be fair, a lot of agent behavior can already be improved through better memory and skill design. OpenClaw's existing skill ecosystem and community-built self-improvement skills handle a wide range of use cases without touching model weights at all. If the agent keeps forgetting preferences, that's a memory problem. And if it doesn't know how to handle a specific workflow, that's a skill problem. Both are solvable at the prompt and context layer. Where RL becomes interesting is when the failure pattern lives deeper in the model's reasoning itself. Things like consistently poor tool selection order, weak multi-step planning, or failing to interpret ambiguous instructions the way a specific user intends. Research on agentic RL (like ARTIST and Agent-R1) has shown that these behavioral patterns hit a ceiling with prompt-based approaches alone, especially in complex multi-turn tasks where the model needs to recover from tool failures or adapt its strategy mid-execution. That's the layer OpenClaw-RL targets, and it's a meaningful distinction from what OpenClaw offers. I have shared the repo in the replies!
0:35

OpenClaw meets RL! OpenClaw Agents adapt through memory files and skills, but the base model weights never actually change. OpenClaw-RL solves this! It wraps a self-hosted model as an OpenAI-compatible API, intercepts live conversations from OpenClaw, and trains the policy in the background using RL. The architecture is fully async. This means serving, reward scoring, and training all run in parallel. Once done, weights get hot-swapped after every batch while the agent keeps responding. Currently, it has two training modes: - Binary RL (GRPO): A process reward model scores each turn as good, bad, or neutral. That scalar reward drives policy updates via a PPO-style clipped objective. - On-Policy Distillation: When concrete corrections come in like "you should have checked that file first," it uses that feedback as a richer, directional training signal at the token level. When to use OpenClaw-RL? To be fair, a lot of agent behavior can already be improved through better memory and skill design. OpenClaw's existing skill ecosystem and community-built self-improvement skills handle a wide range of use cases without touching model weights at all. If the agent keeps forgetting preferences, that's a memory problem. And if it doesn't know how to handle a specific workflow, that's a skill problem. Both are solvable at the prompt and context layer. Where RL becomes interesting is when the failure pattern lives deeper in the model's reasoning itself. Things like consistently poor tool selection order, weak multi-step planning, or failing to interpret ambiguous instructions the way a specific user intends. Research on agentic RL (like ARTIST and Agent-R1) has shown that these behavioral patterns hit a ceiling with prompt-based approaches alone, especially in complex multi-turn tasks where the model needs to recover from tool failures or adapt its strategy mid-execution. That's the layer OpenClaw-RL targets, and it's a meaningful distinction from what OpenClaw offers. I have shared the repo in the replies!

Avi Chawla

138,769 Aufrufe • vor 6 Monaten

A model is only as good as its data, and we’ve long since exhausted the internet. From here on out, model progress is gated by data production. ’s Brendan (can/do) joined us at our Sovereign AI event to talk about how RL environments get built, and why your data might be your real moat: 00:00 Introduction 00:47 A short history of the data market: crowdsourcing to agentic data 02:29 What an RL environment is: worlds, apps, tasks 03:57 Why only humans can measure the frontier 05:35 Building verifiers is the hard part 06:44 Walkthrough: a real legal RL environment 08:18 Leaderboards — and what open weights change 09:45 Post-training results on Apex Agents 11:17 Three ways companies buy data 12:49 Q&A: How do you price data? 14:17 Q&A: What "data quality" actually means 16:42 Q&A: The misunderstanding about synthetic data 18:17 Q&A: Why RL environments now — and what comes after 21:20 Q&A: Can you scale rubric generation with models? 23:00 Q&A: RL environments for cyber defense 25:33 Q&A: Build data in-house or partner?
26:21

A model is only as good as its data, and we’ve long since exhausted the internet. From here on out, model progress is gated by data production. ’s Brendan (can/do) joined us at our Sovereign AI event to talk about how RL environments get built, and why your data might be your real moat: 00:00 Introduction 00:47 A short history of the data market: crowdsourcing to agentic data 02:29 What an RL environment is: worlds, apps, tasks 03:57 Why only humans can measure the frontier 05:35 Building verifiers is the hard part 06:44 Walkthrough: a real legal RL environment 08:18 Leaderboards — and what open weights change 09:45 Post-training results on Apex Agents 11:17 Three ways companies buy data 12:49 Q&A: How do you price data? 14:17 Q&A: What "data quality" actually means 16:42 Q&A: The misunderstanding about synthetic data 18:17 Q&A: Why RL environments now — and what comes after 21:20 Q&A: Can you scale rubric generation with models? 23:00 Q&A: RL environments for cyber defense 25:33 Q&A: Build data in-house or partner?

Pat Grady

66,691 Aufrufe • vor 1 Monat

Today's Training Data episode takes us BTS on the infrastructure challenges required to do large RL runs at scale, featuring Federico Cassano (Composer Lead at Cursor) and Dmytro Dzhulgakov (Co-Founder at Fireworks). The Cursor team trained Composer 2 on Fireworks by starting with a strong base model (Kimi 2.5) and performing large-scale mid-training on code tokens and web data to learn common patterns and libraries, followed by a large-scale Reinforcement Learning run to learn how to navigate the Cursor harness, call tools, and write correct code. Today's episode dives into the systems and infrastructure challenges of making that large RL run happening, and there were many (!!), from numerical mismatch to global distribution to synchronizing rollouts across asynchronous pipelines to keeping track of expert activation across runs and more. Extremely nerdy in-the-weeds challenges that Federico and Dima were delighted to nerd out on together :) Beyond RL infra, we also discussed Online vs Simulated rollouts, self-summarization for long-horizon agents, environment design ("the most powerful RL environment is the product itself"), and other technical nuggets. PS: We filmed this episode before the SpaceX news, while the Cursor team was still compute-constrained. While Cursor now has *all* the flops, the takeaways and hurdles crossed ring true for any serious application-level company that is racing to post-train their own models. I believe that more serious application companies will go the way of Cursor and post-train their own models. 00:00 Introduction 00:53 Why Cursor Trained Composer 2 04:55 Specialization vs Bitter Lesson 06:16 Composer 2 Training Recipe 16:32 Scaling RL Infrastructure Globally 23:32 Floating Point Drift 25:11 MoE Sensitivity Explained 26:25 Router Replay Fix 27:19 Real Time RL Loop 31:49 Long Horizon Agents 34:29 Why RL Everywhere 37:34 LLM as Judge Rewards 39:14 RL in Hard Domains 40:13 Build Your Own Environments 44:34 Closing Thoughts
45:33

Today's Training Data episode takes us BTS on the infrastructure challenges required to do large RL runs at scale, featuring Federico Cassano (Composer Lead at Cursor) and Dmytro Dzhulgakov (Co-Founder at Fireworks). The Cursor team trained Composer 2 on Fireworks by starting with a strong base model (Kimi 2.5) and performing large-scale mid-training on code tokens and web data to learn common patterns and libraries, followed by a large-scale Reinforcement Learning run to learn how to navigate the Cursor harness, call tools, and write correct code. Today's episode dives into the systems and infrastructure challenges of making that large RL run happening, and there were many (!!), from numerical mismatch to global distribution to synchronizing rollouts across asynchronous pipelines to keeping track of expert activation across runs and more. Extremely nerdy in-the-weeds challenges that Federico and Dima were delighted to nerd out on together :) Beyond RL infra, we also discussed Online vs Simulated rollouts, self-summarization for long-horizon agents, environment design ("the most powerful RL environment is the product itself"), and other technical nuggets. PS: We filmed this episode before the SpaceX news, while the Cursor team was still compute-constrained. While Cursor now has *all* the flops, the takeaways and hurdles crossed ring true for any serious application-level company that is racing to post-train their own models. I believe that more serious application companies will go the way of Cursor and post-train their own models. 00:00 Introduction 00:53 Why Cursor Trained Composer 2 04:55 Specialization vs Bitter Lesson 06:16 Composer 2 Training Recipe 16:32 Scaling RL Infrastructure Globally 23:32 Floating Point Drift 25:11 MoE Sensitivity Explained 26:25 Router Replay Fix 27:19 Real Time RL Loop 31:49 Long Horizon Agents 34:29 Why RL Everywhere 37:34 LLM as Judge Rewards 39:14 RL in Hard Domains 40:13 Build Your Own Environments 44:34 Closing Thoughts

Sonya Huang 🐥

81,332 Aufrufe • vor 4 Monaten

A fun way to try the new Claude Fable. I asked it to train a bot using RL to navigate 3D worlds created with SpAItial AI. This is all running in the browser including RL training! Link to demo and code in comments What you can do: -> Train a bot from scratch, watch it learn in minutes, then drop it into a world it has never seen and it keeps navigating obstacles. -> The worlds are real scenes generated from a text prompt or a photo via the SpAItial AI API (you get a splat + a collision mesh). -> Mesh is made into a 2D walkability + height grid that the RL environment runs on. -> The trick to generalization: the policy only ever sees egocentric inputs. A 16-ray lidar, the goal's body frame, and its speed. No global coordinates, no map. So it learns navigation, not a memorized world.
1:05

A fun way to try the new Claude Fable. I asked it to train a bot using RL to navigate 3D worlds created with SpAItial AI. This is all running in the browser including RL training! Link to demo and code in comments What you can do: -> Train a bot from scratch, watch it learn in minutes, then drop it into a world it has never seen and it keeps navigating obstacles. -> The worlds are real scenes generated from a text prompt or a photo via the SpAItial AI API (you get a splat + a collision mesh). -> Mesh is made into a 2D walkability + height grid that the RL environment runs on. -> The trick to generalization: the policy only ever sees egocentric inputs. A 16-ray lidar, the goal's body frame, and its speed. No global coordinates, no map. So it learns navigation, not a memorized world.

RO2⚡️

12,797 Aufrufe • vor 3 Monaten

If you want to understand why RL struggles with long-horizon agent tasks, this is a good explanation. The core issue is that sparse rewards throw away most of the useful information in the trajectory. GEPA tries to learn from the trajectory itself, using reflection in text space, instead of only optimizing on the final reward. GEPA generates textual critiques of trajectories, proposes prompt edits, and then selects updates along a Pareto frontier between exploration and exploitation instead. Instead of collapsing everything into one reward number, it keeps more information about why a run failed and uses that to make legible changes. It’ll be interesting to see where this goes when people combine that kind of trajectory-level reflection with RL, using RL for optimization while preserving a much richer signal about why the agent succeeded or failed.
0:52

If you want to understand why RL struggles with long-horizon agent tasks, this is a good explanation. The core issue is that sparse rewards throw away most of the useful information in the trajectory. GEPA tries to learn from the trajectory itself, using reflection in text space, instead of only optimizing on the final reward. GEPA generates textual critiques of trajectories, proposes prompt edits, and then selects updates along a Pareto frontier between exploration and exploitation instead. Instead of collapsing everything into one reward number, it keeps more information about why a run failed and uses that to make legible changes. It’ll be interesting to see where this goes when people combine that kind of trajectory-level reflection with RL, using RL for optimization while preserving a much richer signal about why the agent succeeded or failed.

Ben Cohen

20,006 Aufrufe • vor 4 Monaten

We took a short break from robotics to build a human-level agent to play Competitive Pokémon. Partially observed. Stochastic. Long-horizon. Now mastered with Offline RL + Transformers. Our agent, trained on 475k+ human battles, hits the top 10% on Pokémon Showdown leaderboards. No search or heuristics, just sequence modeling. Today, we're open-sourcing our Metamon platform with our algorithms, data, and environments: 🌐 We are excited to see how our work accelerates research on building generally capable AI agents, and more importantly, inspires the next generation of Pokémon trainers!
0:59

We took a short break from robotics to build a human-level agent to play Competitive Pokémon. Partially observed. Stochastic. Long-horizon. Now mastered with Offline RL + Transformers. Our agent, trained on 475k+ human battles, hits the top 10% on Pokémon Showdown leaderboards. No search or heuristics, just sequence modeling. Today, we're open-sourcing our Metamon platform with our algorithms, data, and environments: 🌐 We are excited to see how our work accelerates research on building generally capable AI agents, and more importantly, inspires the next generation of Pokémon trainers!

Yuke Zhu

50,758 Aufrufe • vor 1 Jahr

Deep Research is the most fun I've had with an AI tool since NotebookLM was launched! You can't imagine how incredible this agent feels for someone like me that loves doing research. I took it for a spin here and added my thoughts: I am impressed by the reasoning capabilities of the o3 model which operated under the hood. This is RL at its best once again! Synthesizing large amounts of information from different sources is no easy feat. I can tell you all about it from my experience building and scaling RAG systems. I have so many ways I plan to use this. More on this soon.
3:07

Deep Research is the most fun I've had with an AI tool since NotebookLM was launched! You can't imagine how incredible this agent feels for someone like me that loves doing research. I took it for a spin here and added my thoughts: I am impressed by the reasoning capabilities of the o3 model which operated under the hood. This is RL at its best once again! Synthesizing large amounts of information from different sources is no easy feat. I can tell you all about it from my experience building and scaling RAG systems. I have so many ways I plan to use this. More on this soon.

elvis

26,459 Aufrufe • vor 1 Jahr

Karpathy's prediction about RL is coming true now! He called reward functions unreliable and argued that a single reward number is too low-dimensional to teach an agent what "good" means for complex tasks. To solve this, Agents need a knowledge-guided review as a higher-dimensional feedback channel. Every major AI lab trains models with RL today (OpenAI, Anthropic, DeepSeek). And their key bottleneck has always been the reward functions. GRPO by DeepSeek worked well for math and code because the environment gave a binary signal. But for real agent tasks, someone still has to hand-code the scoring function. That takes days and breaks every time the pipeline changes. RULER (implemented in OpenPipe ART, 10k stars) addresses the exact problem Karpathy identified. The reward criteria are defined in plain English, and an LLM evaluates each trajectory against that description to provide feedback for training. I trained a Qwen3 1.4B agent that plays 2048 using GRPO with this exact workflow. In this case, the agent saw the board, picked a direction, and RULER evaluated the outcome, all from this natural language definition. You can see the full implementation on GitHub and try it yourself. Here's the ART Repo: (don't forget to star it ⭐ ) Just like RLHF replaced manual rankings and GRPO replaced the critic model, natural language rewards are replacing hand-coded scoring functions. RL reward engineering is now prompt engineering. I wrote a full walkthrough covering RL for LLM agents, from RLHF to GRPO to RULER, in the article below.
0:25

Karpathy's prediction about RL is coming true now! He called reward functions unreliable and argued that a single reward number is too low-dimensional to teach an agent what "good" means for complex tasks. To solve this, Agents need a knowledge-guided review as a higher-dimensional feedback channel. Every major AI lab trains models with RL today (OpenAI, Anthropic, DeepSeek). And their key bottleneck has always been the reward functions. GRPO by DeepSeek worked well for math and code because the environment gave a binary signal. But for real agent tasks, someone still has to hand-code the scoring function. That takes days and breaks every time the pipeline changes. RULER (implemented in OpenPipe ART, 10k stars) addresses the exact problem Karpathy identified. The reward criteria are defined in plain English, and an LLM evaluates each trajectory against that description to provide feedback for training. I trained a Qwen3 1.4B agent that plays 2048 using GRPO with this exact workflow. In this case, the agent saw the board, picked a direction, and RULER evaluated the outcome, all from this natural language definition. You can see the full implementation on GitHub and try it yourself. Here's the ART Repo: (don't forget to star it ⭐ ) Just like RLHF replaced manual rankings and GRPO replaced the critic model, natural language rewards are replacing hand-coded scoring functions. RL reward engineering is now prompt engineering. I wrote a full walkthrough covering RL for LLM agents, from RLHF to GRPO to RULER, in the article below.

Avi Chawla

350,798 Aufrufe • vor 4 Monaten

The DeepSeek-R1 paper is a gem! Highly encourage everyone to read it. It's clear that LLM reasoning capabilities can be learned in different ways. RL, if applied correctly and at scale, can lead to some really powerful and interesting scaling and emergent properties. There is more to RL than meets the eye! Here is my breakdown of the paper along with a few tests: The multi-state training might not make sense initially but they provide clues on optimizations that we can continue to tap into. Data quality is still very important for enhancing the usability of the LLM. Unlike other reasoning LLMs, DeepSeek-R1's training recipe and weights are open so we can build on top of it. This opens up exciting research opportunities. About the attached clip: the previous preview model wasn't able to solve this task. DeepSeek-R1 can solve this and many other tasks that o1 can solve. It's a very good model for coding and math.
1:57

The DeepSeek-R1 paper is a gem! Highly encourage everyone to read it. It's clear that LLM reasoning capabilities can be learned in different ways. RL, if applied correctly and at scale, can lead to some really powerful and interesting scaling and emergent properties. There is more to RL than meets the eye! Here is my breakdown of the paper along with a few tests: The multi-state training might not make sense initially but they provide clues on optimizations that we can continue to tap into. Data quality is still very important for enhancing the usability of the LLM. Unlike other reasoning LLMs, DeepSeek-R1's training recipe and weights are open so we can build on top of it. This opens up exciting research opportunities. About the attached clip: the previous preview model wasn't able to solve this task. DeepSeek-R1 can solve this and many other tasks that o1 can solve. It's a very good model for coding and math.

elvis

141,023 Aufrufe • vor 1 Jahr

So I’ve officially changed my username here to just be me. I’ve also changed the links to my Self Love Circle links. If you are looking for “Dee Siren” content then you will need to go find those links on my Dee Siren - SirenXXXStudios Legacy Account here and on Instagram or look it up. Over the last 5 years, I have finally learned that everything in my life starts with me and if I am going to be able to help others through this awakening process, I had to learn to love myself. If you want to go on this journey with me please sign up for my Free option Patreon Or join as a member. I will also be offering membership content if you subscribe here on X. I am excited about this journey and hope you will stay on this path to discovering your own Dharma with me.
4:36

So I’ve officially changed my username here to just be me. I’ve also changed the links to my Self Love Circle links. If you are looking for “Dee Siren” content then you will need to go find those links on my Dee Siren - SirenXXXStudios Legacy Account here and on Instagram or look it up. Over the last 5 years, I have finally learned that everything in my life starts with me and if I am going to be able to help others through this awakening process, I had to learn to love myself. If you want to go on this journey with me please sign up for my Free option Patreon Or join as a member. I will also be offering membership content if you subscribe here on X. I am excited about this journey and hope you will stay on this path to discovering your own Dharma with me.

Cylla Kavanaugh

31,989 Aufrufe • vor 7 Monaten

This Indian Developer made 41.7 Lakh in roughly 2 months. He builds RL environments for AI labs. Here’s what that actually means. You don’t write answers for the AI. You set its exam. Take a real GitHub repo with a real bug. Pin it inside Docker so it runs identically every time. Write the task. Write a grader that returns an automatic pass or fail. Write your own gold solution to prove it’s solvable. Then make sure the model can’t cheat its way to a pass. The labs have run out of internet data. This is the next lever. Anthropic reportedly discussed spending over $1B on this in a single year. Where to start: Prime Intellect’s Environments Hub is free to enter and has 2,500+ open environments. Ship one as a public PR. Then apply to Mercor, Surge or Handshake with that PR as proof. Week 1 he made ₹6,880. Start. Comment “RL” for the full breakdown and link to Whatsapp Channel in the bio! #ai #rl #hiring #google
1:42

This Indian Developer made 41.7 Lakh in roughly 2 months. He builds RL environments for AI labs. Here’s what that actually means. You don’t write answers for the AI. You set its exam. Take a real GitHub repo with a real bug. Pin it inside Docker so it runs identically every time. Write the task. Write a grader that returns an automatic pass or fail. Write your own gold solution to prove it’s solvable. Then make sure the model can’t cheat its way to a pass. The labs have run out of internet data. This is the next lever. Anthropic reportedly discussed spending over $1B on this in a single year. Where to start: Prime Intellect’s Environments Hub is free to enter and has 2,500+ open environments. Ship one as a public PR. Then apply to Mercor, Surge or Handshake with that PR as proof. Week 1 he made ₹6,880. Start. Comment “RL” for the full breakdown and link to Whatsapp Channel in the bio! #ai #rl #hiring #google

Arsh Goyal

38,764 Aufrufe • vor 1 Monat