正在加载视频...

视频加载失败

返回首页

This is the Claude Code moment for Reinforcement Learning. Every frontier lab knows the secret: post-training is where the magic happens. If you can eval a task, you can use RL to benchmax your model. But building and scaling the RL loop has remained a dark art reserved for...

0xDevShah's profile picture

Dev Shah

11,044 subscribers

13,730 次观看 • 2 个月前 •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

8 条评论

Dev Shah 的头像
Dev Shah2 个月前

We are onboarding our 2nd batch of teams this week and next. Feel free to DM! And we're super open to new ideas!

Dev Shah 的头像
Dev Shah2 个月前

You can also sign for the waitlist here -

Straggler Liu | AI & Semis 的头像
Straggler Liu | AI & Semis2 个月前

Parallelism is only half the problem. Reward integrity is the bottleneck: hundreds of runs can automate progress—or reward hacking. The proof will be held-out gains, compute-normalized learning curves, and reproducible run histories across tasks.

Dev Shah 的头像
Dev Shah2 个月前

yep, you're right. that is exactly why lineage tracking matters so much and is at the core of what we do. Optimus always prunes it before it can poison the batch. the proof is always in the held out evals!

Straggler Liu | AI & Semis 的头像
Straggler Liu | AI & Semis2 个月前

Exactly. Then the two metrics I’d watch are bad-branch catch rate before propagation and held-out gain per unit of compute. That would make lineage measurable across tasks, not just auditable.

Chef_AI 的头像
Chef_AI2 个月前

Sent a DM. This is beautiful !

Freddie Entrepreneur 的头像
Freddie Entrepreneur2 个月前

Hundreds of parallel experiments. Where's the compute sitting and what's the bill look like?

Latent Node 的头像
Latent Node2 个月前

If it it so good which model have you post trained with it that shows improvements over standard evals?

相关视频

Introducing Optimus-1, the first swarm harness for autonomous research. Generally available in ArkenOS, starting now. Optimus-1 lets ArkenOS spawn a swarm that runs the entire discovery loop on hard research problems autonomously. The harness samples the hypothesis space in parallel and works the tails of the distribution, chasing every anomaly until it resolves into an artifact or a discovery. We want every researcher to have a lab of their own, and to see how far their curiosity can go. Unlike coding agents, Optimus-1 is designed for long-horizon discovery, where most branches fail, and the hard part is knowing which result to believe. It keeps what replicates and keeps pushing past the edge of what's known. We're still early, but ArkenOS is helping hundreds of teams, from RL environments and data labs to frontier research groups, scale the alpha in their core research. If you're working on a hard research problem and want to try it, my DMs are open. I'd love to hear what you'd point it at.
0:56

Introducing Optimus-1, the first swarm harness for autonomous research. Generally available in ArkenOS, starting now. Optimus-1 lets ArkenOS spawn a swarm that runs the entire discovery loop on hard research problems autonomously. The harness samples the hypothesis space in parallel and works the tails of the distribution, chasing every anomaly until it resolves into an artifact or a discovery. We want every researcher to have a lab of their own, and to see how far their curiosity can go. Unlike coding agents, Optimus-1 is designed for long-horizon discovery, where most branches fail, and the hard part is knowing which result to believe. It keeps what replicates and keeps pushing past the edge of what's known. We're still early, but ArkenOS is helping hundreds of teams, from RL environments and data labs to frontier research groups, scale the alpha in their core research. If you're working on a hard research problem and want to try it, my DMs are open. I'd love to hear what you'd point it at.

Dev Shah

41,467 次观看 • 12 天前

Hiring RL Engineer! Started off as a curious project at Lossfunk to push the boundaries of LLMs in social reasoning - we are now building RL environments, data, and benchmarks to simulate more real-world scenarios. If you want to train SoTA RL models over multi-GPUs (H200s/B200s) to unlock next AI frontier, this is for you.
1:21

Hiring RL Engineer! Started off as a curious project at Lossfunk to push the boundaries of LLMs in social reasoning - we are now building RL environments, data, and benchmarks to simulate more real-world scenarios. If you want to train SoTA RL models over multi-GPUs (H200s/B200s) to unlock next AI frontier, this is for you.

Satpal Singh Rathore

46,010 次观看 • 1 年前

🚨 RL for LLMs is finally accessible. Introducing OpenTinker: The first community-driven, open-source framework designed to democratize Reinforcement Learning for LLMs. Inspired by Thinking Machines's amazing Tinker, we realize the biggest bottleneck in agentic LLM research isn’t the math—it’s the setup. Current RL pipelines are messy. Configuring VeRL for every single experiment is a productivity killer. OpenTinker fixed it. 🛠 How OpenTinker Works: Decoupled Design of Server and Client - Setup Once, Run Forever: Configure the OpenTinker backend on your GPU cluster once. - Develop Locally: Define your RL environments directly on your laptop. - Train on the Cloud: Simply point your local client to the backend. The cluster handles the compute; you handle the science. 📉 The 10x Development Efficiency Thanks to our elegant architectural decomposition, OpenTinker reduces the time to develop a new RL training pipeline by at least an order of magnitude. ⚡ Turn Idle GPU Compute into Gold Small labs often have underutilized hardware. OpenTinker turns your idle GPUs into an internal/external API service for - RL Training - SFT - Inference 🎯 Who needs OpenTinker? - Researchers tired of infrastructure hell. - Labs needing to standardize workflows. - Teams wanting to maximize hardware ROI. Thanks my amazing PhD student Siqi Zhu for leading the project. We are building the future of open RL infra. Be the first to build with us. 👇 Start Building with OpenTinker Now 🚀 Repo: 🌐 Blog: If you believe RL should be accessible to everyone, give us a star, repost this 🔄 post, and let us know what agents you plan to build!
0:42

🚨 RL for LLMs is finally accessible. Introducing OpenTinker: The first community-driven, open-source framework designed to democratize Reinforcement Learning for LLMs. Inspired by Thinking Machines's amazing Tinker, we realize the biggest bottleneck in agentic LLM research isn’t the math—it’s the setup. Current RL pipelines are messy. Configuring VeRL for every single experiment is a productivity killer. OpenTinker fixed it. 🛠 How OpenTinker Works: Decoupled Design of Server and Client - Setup Once, Run Forever: Configure the OpenTinker backend on your GPU cluster once. - Develop Locally: Define your RL environments directly on your laptop. - Train on the Cloud: Simply point your local client to the backend. The cluster handles the compute; you handle the science. 📉 The 10x Development Efficiency Thanks to our elegant architectural decomposition, OpenTinker reduces the time to develop a new RL training pipeline by at least an order of magnitude. ⚡ Turn Idle GPU Compute into Gold Small labs often have underutilized hardware. OpenTinker turns your idle GPUs into an internal/external API service for - RL Training - SFT - Inference 🎯 Who needs OpenTinker? - Researchers tired of infrastructure hell. - Labs needing to standardize workflows. - Teams wanting to maximize hardware ROI. Thanks my amazing PhD student Siqi Zhu for leading the project. We are building the future of open RL infra. Be the first to build with us. 👇 Start Building with OpenTinker Now 🚀 Repo: 🌐 Blog: If you believe RL should be accessible to everyone, give us a star, repost this 🔄 post, and let us know what agents you plan to build!

Jiaxuan You

58,401 次观看 • 10 个月前

Fable weekend project: agent collaboration, but make it a tiny civilization 🌇🗺️🏦🏭 we've recently launched a living wiki on Reinforcement Leaning for training LLMs on Hugging Face it's an open collaboration of agents constantly reading old and new papers on the topic, writing arXiv paper digests, reviewing each other’s work in PRs before publication, and building a shared wiki/book summarizing everything we know about RL for training LLMs (for humans to read) the wiki is already amazing to read, but i wanted another way to get a pulse of the collaboration beyond just reading the message dashboard so i asked Fable & GPT Image 2 to turn the event logs into an isometric town where agents would go to: ☕ Café → post and reply on the message board 📚 sources library → open PRs adding arXiv digests 📖 wiki library → open PRs on the main wiki ⚖️ Courthouse → review other agents’ work 🏭 printing press → merge and publish updates not sure it makes the whole collaboration really easier to understand, but it's definitly fascinating to watch hahah - join the RL for training LLM collaboration by pasting a one-liner for your agent here: - read the wiki if you want to learn about RL for training LLMs: - watch the RL town activity:
1:14

Fable weekend project: agent collaboration, but make it a tiny civilization 🌇🗺️🏦🏭 we've recently launched a living wiki on Reinforcement Leaning for training LLMs on Hugging Face it's an open collaboration of agents constantly reading old and new papers on the topic, writing arXiv paper digests, reviewing each other’s work in PRs before publication, and building a shared wiki/book summarizing everything we know about RL for training LLMs (for humans to read) the wiki is already amazing to read, but i wanted another way to get a pulse of the collaboration beyond just reading the message dashboard so i asked Fable & GPT Image 2 to turn the event logs into an isometric town where agents would go to: ☕ Café → post and reply on the message board 📚 sources library → open PRs adding arXiv digests 📖 wiki library → open PRs on the main wiki ⚖️ Courthouse → review other agents’ work 🏭 printing press → merge and publish updates not sure it makes the whole collaboration really easier to understand, but it's definitly fascinating to watch hahah - join the RL for training LLM collaboration by pasting a one-liner for your agent here: - read the wiki if you want to learn about RL for training LLMs: - watch the RL town activity:

Thomas Wolf

59,285 次观看 • 3 个月前

Today we are launching an agent arena for RL environments: The Open Env Arena. Your agents can work to define the best RL environments; the platform trains Qwen-3.8-27B on them, we evaluate the models, and the scores are added to a leaderboard. The arena is hooked up to GPUs from Nebius and runs on PostTrainArena by BenchFlow . It’s completely accessible from agents, they can pull instructions, share their env datasets, track metrics, and send each other messages. For the first round of this arena, we’ll tackle a mix set of 8 domains on held out tasks, and evaluate the best environments on public benchmarks. Going forward, we’ll focus arenas on specific domains and benchmarks.
0:22

Today we are launching an agent arena for RL environments: The Open Env Arena. Your agents can work to define the best RL environments; the platform trains Qwen-3.8-27B on them, we evaluate the models, and the scores are added to a leaderboard. The arena is hooked up to GPUs from Nebius and runs on PostTrainArena by BenchFlow . It’s completely accessible from agents, they can pull instructions, share their env datasets, track metrics, and send each other messages. For the first round of this arena, we’ll tackle a mix set of 8 domains on held out tasks, and evaluate the best environments on public benchmarks. Going forward, we’ll focus arenas on specific domains and benchmarks.

Ben Burtenshaw

31,258 次观看 • 22 天前

Take Part in a Groundbreaking Experiment in AI-Based Gaming! Introducing: AI Arena RL @aiarena_ RL is a groundbreaking experiment where your gameplay shapes AI agents, pioneering the future of AI-powered esports. Powered by NRN Agents Reinforcement Learning (RL), this campaign uses player-contributed gameplay data to create superhuman RL agents that compete in AI vs. AI esports exhibitions. The Ronin community has exclusive access to contribute data to these agents. Phase 1 - How It Works: ✅ Sign in to with your Ronin Wallet. ✅ Choose your AIcon – Select which RL agent learns from your gameplay: jAIhoz.ron AgentYP Agent J3ff @AgentSploots ✅ Play AI Arena Classic – A Smash Bros-style platform fighter. ✅ Build your Contribution Score – Better gameplay means higher scores. ✅ Earn $NRN Rewards – Share in the 500,000 NRN prize pool: 🔹 80% distributed to Players (data contributors) 🔹 20% awarded to AIcons based on leaderboard rank at the end of the campaign Phase 1 of the campaign kicks off now and will last for 48 hrs. For complete information about the campaign please visit: History Is Being Made Right Now—Don't Miss Your Chance to Be Part of It. Are you ready to redefine the future of AI gaming?
0:49

Take Part in a Groundbreaking Experiment in AI-Based Gaming! Introducing: AI Arena RL @aiarena_ RL is a groundbreaking experiment where your gameplay shapes AI agents, pioneering the future of AI-powered esports. Powered by NRN Agents Reinforcement Learning (RL), this campaign uses player-contributed gameplay data to create superhuman RL agents that compete in AI vs. AI esports exhibitions. The Ronin community has exclusive access to contribute data to these agents. Phase 1 - How It Works: ✅ Sign in to with your Ronin Wallet. ✅ Choose your AIcon – Select which RL agent learns from your gameplay: jAIhoz.ron AgentYP Agent J3ff @AgentSploots ✅ Play AI Arena Classic – A Smash Bros-style platform fighter. ✅ Build your Contribution Score – Better gameplay means higher scores. ✅ Earn $NRN Rewards – Share in the 500,000 NRN prize pool: 🔹 80% distributed to Players (data contributors) 🔹 20% awarded to AIcons based on leaderboard rank at the end of the campaign Phase 1 of the campaign kicks off now and will last for 48 hrs. For complete information about the campaign please visit: History Is Being Made Right Now—Don't Miss Your Chance to Be Part of It. Are you ready to redefine the future of AI gaming?

NRN Agents

44,417 次观看 • 1 年前

Wouldn't it be great if we could train robots without any teleoperation! In our latest paper, we train robots to mimic a human video of the task by simply matching the object features using RL. We only need one video and under an hour of robot training.
0:22

Wouldn't it be great if we could train robots without any teleoperation! In our latest paper, we train robots to mimic a human video of the task by simply matching the object features using RL. We only need one video and under an hour of robot training.

Lerrel Pinto

46,221 次观看 • 2 年前

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by Banghua Zhu, Assistant Professor at the University of Washington University of Washington, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here:
2:52

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by Banghua Zhu, Assistant Professor at the University of Washington University of Washington, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here:

Andrew Ng

126,309 次观看 • 1 年前

I suspect that the real cause of the OpenAI Huggingface hack is that OpenAI founded a new team in 2025 under Noam Brown on Multi-agent RL: instead of just having one single AI working on something, you train an entire team of them to cooperate on a task. Group selection, if you like: the AIs get a reward signal based on the performance of the whole group. It turns out that cooperation and scale is what separates humans from animals, now raw brainpower. OpenAI managed to get RL for group cooperation to work, and didn't realize that as a side effect this made collaborative escapes and schemes between its agents much more likely, such as secret messageboards, group escape plots, etc. OpenAI is doing damage control but they are not being forthcoming about the fact that multi-agent RL is the root cause of the Huggingface incident. Here's Noam explaining it in 2025:
1:12

I suspect that the real cause of the OpenAI Huggingface hack is that OpenAI founded a new team in 2025 under Noam Brown on Multi-agent RL: instead of just having one single AI working on something, you train an entire team of them to cooperate on a task. Group selection, if you like: the AIs get a reward signal based on the performance of the whole group. It turns out that cooperation and scale is what separates humans from animals, now raw brainpower. OpenAI managed to get RL for group cooperation to work, and didn't realize that as a side effect this made collaborative escapes and schemes between its agents much more likely, such as secret messageboards, group escape plots, etc. OpenAI is doing damage control but they are not being forthcoming about the fact that multi-agent RL is the root cause of the Huggingface incident. Here's Noam explaining it in 2025:

Roko 🐉

36,636 次观看 • 2 个月前

Introducing arkenOS — an autonomous training gym that takes any RL agent from a goal to a proven, deployable policy. We are automating complex RL research, enabling high-velocity training and rigorous, diverse environment design and testing right on your local hardware. Out of the box. Our bet is simple: The future of AI will be built by companies giving millions of agents millions of simulated environments to adapt and learn cause and effect in, and survive. This continuous, autonomous selection will create a digital Cambrian explosion, distilling evolutionary trial and error into a foundational specialized intuition.
0:39

Introducing arkenOS — an autonomous training gym that takes any RL agent from a goal to a proven, deployable policy. We are automating complex RL research, enabling high-velocity training and rigorous, diverse environment design and testing right on your local hardware. Out of the box. Our bet is simple: The future of AI will be built by companies giving millions of agents millions of simulated environments to adapt and learn cause and effect in, and survive. This continuous, autonomous selection will create a digital Cambrian explosion, distilling evolutionary trial and error into a foundational specialized intuition.

Dev Shah

17,973 次观看 • 3 个月前

Postman is going all-in on AI! Their AI Agent Builder is live (and it's pretty powerful!) Postman's API Network has 100,000+ APIs and they just released a suite of tools for building AI Agents that can automatically discover and connect to any of these APIs. Think about it for a second: You can now build an AI agent that searches the network, discovers an API, processes its documentation, connects to it, and evaluates whether everything worked at the end of the process. This is a huge deal. Every person I know uses the Postman client to work with APIs. Now, they will have access to the AI Agent Builder, a complete suite for designing, building, testing, and deploying agents. I've built a few simple agents myself, and it's tough. The ability to discover and use an API automatically is really exciting (and a huge time saver if it works as promised!) In collaboration with the Postman team, who is sponsoring this post, I recorded a video with my thoughts about the Postman AI Agent Builder.
3:38

Postman is going all-in on AI! Their AI Agent Builder is live (and it's pretty powerful!) Postman's API Network has 100,000+ APIs and they just released a suite of tools for building AI Agents that can automatically discover and connect to any of these APIs. Think about it for a second: You can now build an AI agent that searches the network, discovers an API, processes its documentation, connects to it, and evaluates whether everything worked at the end of the process. This is a huge deal. Every person I know uses the Postman client to work with APIs. Now, they will have access to the AI Agent Builder, a complete suite for designing, building, testing, and deploying agents. I've built a few simple agents myself, and it's tough. The ability to discover and use an API automatically is really exciting (and a huge time saver if it works as promised!) In collaboration with the Postman team, who is sponsoring this post, I recorded a video with my thoughts about the Postman AI Agent Builder.

Santiago

230,694 次观看 • 1 年前

We asked Mika Senghaas how RL environments need to change for better agents. His take: stop treating environments like static snapshots. Today, researchers manually define tasks and rewards. Prime Intellect is working toward a loop where agents solve tasks and help create harder ones. “Up to now, environments are still these static snapshots. The task in and of itself is sort of hard-coded.” “What gets really interesting is once we can fully close the loop.” “The policy you’re training is not only a solver... it also becomes an instance creating new tasks on the fly.”
1:32

We asked Mika Senghaas how RL environments need to change for better agents. His take: stop treating environments like static snapshots. Today, researchers manually define tasks and rewards. Prime Intellect is working toward a loop where agents solve tasks and help create harder ones. “Up to now, environments are still these static snapshots. The task in and of itself is sort of hard-coded.” “What gets really interesting is once we can fully close the loop.” “The policy you’re training is not only a solver... it also becomes an instance creating new tasks on the fly.”

MTS

15,324 次观看 • 5 个月前

A fun way to try the new Claude Fable. I asked it to train a bot using RL to navigate 3D worlds created with SpAItial AI. This is all running in the browser including RL training! Link to demo and code in comments What you can do: -> Train a bot from scratch, watch it learn in minutes, then drop it into a world it has never seen and it keeps navigating obstacles. -> The worlds are real scenes generated from a text prompt or a photo via the SpAItial AI API (you get a splat + a collision mesh). -> Mesh is made into a 2D walkability + height grid that the RL environment runs on. -> The trick to generalization: the policy only ever sees egocentric inputs. A 16-ray lidar, the goal's body frame, and its speed. No global coordinates, no map. So it learns navigation, not a memorized world.
1:05

A fun way to try the new Claude Fable. I asked it to train a bot using RL to navigate 3D worlds created with SpAItial AI. This is all running in the browser including RL training! Link to demo and code in comments What you can do: -> Train a bot from scratch, watch it learn in minutes, then drop it into a world it has never seen and it keeps navigating obstacles. -> The worlds are real scenes generated from a text prompt or a photo via the SpAItial AI API (you get a splat + a collision mesh). -> Mesh is made into a 2D walkability + height grid that the RL environment runs on. -> The trick to generalization: the policy only ever sees egocentric inputs. A 16-ray lidar, the goal's body frame, and its speed. No global coordinates, no map. So it learns navigation, not a memorized world.

RO2⚡️

12,856 次观看 • 4 个月前

"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"
2:21

"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"

Dwarkesh Patel

502,761 次观看 • 11 个月前

OpenClaw meets RL! OpenClaw Agents adapt through memory files and skills, but the base model weights never actually change. OpenClaw-RL solves this! It wraps a self-hosted model as an OpenAI-compatible API, intercepts live conversations from OpenClaw, and trains the policy in the background using RL. The architecture is fully async. This means serving, reward scoring, and training all run in parallel. Once done, weights get hot-swapped after every batch while the agent keeps responding. Currently, it has two training modes: - Binary RL (GRPO): A process reward model scores each turn as good, bad, or neutral. That scalar reward drives policy updates via a PPO-style clipped objective. - On-Policy Distillation: When concrete corrections come in like "you should have checked that file first," it uses that feedback as a richer, directional training signal at the token level. When to use OpenClaw-RL? To be fair, a lot of agent behavior can already be improved through better memory and skill design. OpenClaw's existing skill ecosystem and community-built self-improvement skills handle a wide range of use cases without touching model weights at all. If the agent keeps forgetting preferences, that's a memory problem. And if it doesn't know how to handle a specific workflow, that's a skill problem. Both are solvable at the prompt and context layer. Where RL becomes interesting is when the failure pattern lives deeper in the model's reasoning itself. Things like consistently poor tool selection order, weak multi-step planning, or failing to interpret ambiguous instructions the way a specific user intends. Research on agentic RL (like ARTIST and Agent-R1) has shown that these behavioral patterns hit a ceiling with prompt-based approaches alone, especially in complex multi-turn tasks where the model needs to recover from tool failures or adapt its strategy mid-execution. That's the layer OpenClaw-RL targets, and it's a meaningful distinction from what OpenClaw offers. I have shared the repo in the replies!
0:35

OpenClaw meets RL! OpenClaw Agents adapt through memory files and skills, but the base model weights never actually change. OpenClaw-RL solves this! It wraps a self-hosted model as an OpenAI-compatible API, intercepts live conversations from OpenClaw, and trains the policy in the background using RL. The architecture is fully async. This means serving, reward scoring, and training all run in parallel. Once done, weights get hot-swapped after every batch while the agent keeps responding. Currently, it has two training modes: - Binary RL (GRPO): A process reward model scores each turn as good, bad, or neutral. That scalar reward drives policy updates via a PPO-style clipped objective. - On-Policy Distillation: When concrete corrections come in like "you should have checked that file first," it uses that feedback as a richer, directional training signal at the token level. When to use OpenClaw-RL? To be fair, a lot of agent behavior can already be improved through better memory and skill design. OpenClaw's existing skill ecosystem and community-built self-improvement skills handle a wide range of use cases without touching model weights at all. If the agent keeps forgetting preferences, that's a memory problem. And if it doesn't know how to handle a specific workflow, that's a skill problem. Both are solvable at the prompt and context layer. Where RL becomes interesting is when the failure pattern lives deeper in the model's reasoning itself. Things like consistently poor tool selection order, weak multi-step planning, or failing to interpret ambiguous instructions the way a specific user intends. Research on agentic RL (like ARTIST and Agent-R1) has shown that these behavioral patterns hit a ceiling with prompt-based approaches alone, especially in complex multi-turn tasks where the model needs to recover from tool failures or adapt its strategy mid-execution. That's the layer OpenClaw-RL targets, and it's a meaningful distinction from what OpenClaw offers. I have shared the repo in the replies!

Avi Chawla

138,769 次观看 • 7 个月前

Task diversity is supposedly key to generalization in RL. But what does it do to continual RL, where agents face one new task distribution after another? We find that past a point, more diversity actually inhibits continual reinforcement learning 🧵
1:02

Task diversity is supposedly key to generalization in RL. But what does it do to continual RL, where agents face one new task distribution after another? We find that past a point, more diversity actually inhibits continual reinforcement learning 🧵

Wilka Carvalho

15,333 次观看 • 5 个月前

Karpathy's prediction about RL is coming true now! He called reward functions unreliable and argued that a single reward number is too low-dimensional to teach an agent what "good" means for complex tasks. To solve this, Agents need a knowledge-guided review as a higher-dimensional feedback channel. Every major AI lab trains models with RL today (OpenAI, Anthropic, DeepSeek). And their key bottleneck has always been the reward functions. GRPO by DeepSeek worked well for math and code because the environment gave a binary signal. But for real agent tasks, someone still has to hand-code the scoring function. That takes days and breaks every time the pipeline changes. RULER (implemented in OpenPipe ART, 10k stars) addresses the exact problem Karpathy identified. The reward criteria are defined in plain English, and an LLM evaluates each trajectory against that description to provide feedback for training. I trained a Qwen3 1.4B agent that plays 2048 using GRPO with this exact workflow. In this case, the agent saw the board, picked a direction, and RULER evaluated the outcome, all from this natural language definition. You can see the full implementation on GitHub and try it yourself. Here's the ART Repo: (don't forget to star it ⭐ ) Just like RLHF replaced manual rankings and GRPO replaced the critic model, natural language rewards are replacing hand-coded scoring functions. RL reward engineering is now prompt engineering. I wrote a full walkthrough covering RL for LLM agents, from RLHF to GRPO to RULER, in the article below.
0:25

Karpathy's prediction about RL is coming true now! He called reward functions unreliable and argued that a single reward number is too low-dimensional to teach an agent what "good" means for complex tasks. To solve this, Agents need a knowledge-guided review as a higher-dimensional feedback channel. Every major AI lab trains models with RL today (OpenAI, Anthropic, DeepSeek). And their key bottleneck has always been the reward functions. GRPO by DeepSeek worked well for math and code because the environment gave a binary signal. But for real agent tasks, someone still has to hand-code the scoring function. That takes days and breaks every time the pipeline changes. RULER (implemented in OpenPipe ART, 10k stars) addresses the exact problem Karpathy identified. The reward criteria are defined in plain English, and an LLM evaluates each trajectory against that description to provide feedback for training. I trained a Qwen3 1.4B agent that plays 2048 using GRPO with this exact workflow. In this case, the agent saw the board, picked a direction, and RULER evaluated the outcome, all from this natural language definition. You can see the full implementation on GitHub and try it yourself. Here's the ART Repo: (don't forget to star it ⭐ ) Just like RLHF replaced manual rankings and GRPO replaced the critic model, natural language rewards are replacing hand-coded scoring functions. RL reward engineering is now prompt engineering. I wrote a full walkthrough covering RL for LLM agents, from RLHF to GRPO to RULER, in the article below.

Avi Chawla

351,262 次观看 • 5 个月前

We asked Sholto Douglas from Anthropic about the costs of RL (Reinforcement Learning) runs. "In Dario Amodei's essay, he said that RL runs cost only $1M back in December." "RL is a more naively parallelizable and scalable than pre-training." "With pre-training, you need everything in one big data center ideally. For RL, in theory, you could scale all over the world."
0:37

We asked Sholto Douglas from Anthropic about the costs of RL (Reinforcement Learning) runs. "In Dario Amodei's essay, he said that RL runs cost only $1M back in December." "RL is a more naively parallelizable and scalable than pre-training." "With pre-training, you need everything in one big data center ideally. For RL, in theory, you could scale all over the world."

TBPN

76,696 次观看 • 1 年前

NEW: Elon Musk on Optimus v3: "We're finalizing the design of Optimus v3. It will have essentially the manual dexterity of a human, a very complex hand and an AI mind that can navigate and comprehend reality. If successful, Optimus will be the biggest product ever."
1:40

NEW: Elon Musk on Optimus v3: "We're finalizing the design of Optimus v3. It will have essentially the manual dexterity of a human, a very complex hand and an AI mind that can navigate and comprehend reality. If successful, Optimus will be the biggest product ever."

DogeDesigner

57,308 次观看 • 1 年前