Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

I hand-wrote a 500-LoC RL stack to make hacking on RL research much easier. Most RL stacks are either massive and unhackable, or duct-taped research scripts. I am open-sourcing Mithrl, a modular RLVR stack. Next items on my checklist: adding more complex environment examples, supporting multi-gpu + async RL,... show more

omkaar

4,368 subscribers

17,218 Aufrufe • vor 5 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Lecture 10 of my course! Nominally on regularization in RL, so I discuss the evolving role of the KL penalty in RL, but also a set of nice RL papers that explain what RL helps models generalize better than SFT -- with theory supporting it. When going through these, it's so interesting how seasonal problems in ML are. Lots of problems from controlling reward models overopt will rhyme as we try to control rubrics for agents. 00:00 Intro & the role of regularization 02:50 The KL penalty in RL 10:53 RL as a reverse KL loss 21:09 Why RL generalizes better than SFT 25:15 Other regularization tools Just a few videos left as I get to the end of the course. Thanks all, and keep sending questions. Spread the word if you have a second.

Lecture 10 of my course! Nominally on regularization in RL, so I discuss the evolving role of the KL penalty in RL, but also a set of nice RL papers that explain what RL helps models generalize better than SFT -- with theory supporting it. When going through these, it's so interesting how seasonal problems in ML are. Lots of problems from controlling reward models overopt will rhyme as we try to control rubrics for agents. 00:00 Intro & the role of regularization 02:50 The KL penalty in RL 10:53 RL as a reverse KL loss 21:09 Why RL generalizes better than SFT 25:15 Other regularization tools Just a few videos left as I get to the end of the course. Thanks all, and keep sending questions. Spread the word if you have a second.

Nathan Lambert

87,914 Aufrufe • vor 19 Tagen

Elon Musk reveals the ego-to-ability ratio that predicts whether someone will fail "You do whatever it takes to succeed, and just always be smashing your ego. Internalize responsibility. A major failure mode is when your ego-to-ability ratio is greater than one" "If your ego-to-ability ratio gets too high, you break the feedback loop to reality. In AI terms, you break your RL loop. You want a strong RL loop, which means internalizing responsibility and minimizing ego" "That's why I prefer the term engineering as opposed to research, and I don't want to call xAI a lab, I just want it to be a company. Whatever the simplest, lowest-ego terms are, those are generally a good way to go. You want to close the loop on reality hard"

Elon Musk reveals the ego-to-ability ratio that predicts whether someone will fail "You do whatever it takes to succeed, and just always be smashing your ego. Internalize responsibility. A major failure mode is when your ego-to-ability ratio is greater than one" "If your ego-to-ability ratio gets too high, you break the feedback loop to reality. In AI terms, you break your RL loop. You want a strong RL loop, which means internalizing responsibility and minimizing ego" "That's why I prefer the term engineering as opposed to research, and I don't want to call xAI a lab, I just want it to be a company. Whatever the simplest, lowest-ego terms are, those are generally a good way to go. You want to close the loop on reality hard"

Jaynit

243,045 Aufrufe • vor 12 Tagen

🚨 RL for LLMs is finally accessible. Introducing OpenTinker: The first community-driven, open-source framework designed to democratize Reinforcement Learning for LLMs. Inspired by Thinking Machines's amazing Tinker, we realize the biggest bottleneck in agentic LLM research isn’t the math—it’s the setup. Current RL pipelines are messy. Configuring VeRL for every single experiment is a productivity killer. OpenTinker fixed it. 🛠 How OpenTinker Works: Decoupled Design of Server and Client - Setup Once, Run Forever: Configure the OpenTinker backend on your GPU cluster once. - Develop Locally: Define your RL environments directly on your laptop. - Train on the Cloud: Simply point your local client to the backend. The cluster handles the compute; you handle the science. 📉 The 10x Development Efficiency Thanks to our elegant architectural decomposition, OpenTinker reduces the time to develop a new RL training pipeline by at least an order of magnitude. ⚡ Turn Idle GPU Compute into Gold Small labs often have underutilized hardware. OpenTinker turns your idle GPUs into an internal/external API service for - RL Training - SFT - Inference 🎯 Who needs OpenTinker? - Researchers tired of infrastructure hell. - Labs needing to standardize workflows. - Teams wanting to maximize hardware ROI. Thanks my amazing PhD student Siqi Zhu for leading the project. We are building the future of open RL infra. Be the first to build with us. 👇 Start Building with OpenTinker Now 🚀 Repo: 🌐 Blog: If you believe RL should be accessible to everyone, give us a star, repost this 🔄 post, and let us know what agents you plan to build!

🚨 RL for LLMs is finally accessible. Introducing OpenTinker: The first community-driven, open-source framework designed to democratize Reinforcement Learning for LLMs. Inspired by Thinking Machines's amazing Tinker, we realize the biggest bottleneck in agentic LLM research isn’t the math—it’s the setup. Current RL pipelines are messy. Configuring VeRL for every single experiment is a productivity killer. OpenTinker fixed it. 🛠 How OpenTinker Works: Decoupled Design of Server and Client - Setup Once, Run Forever: Configure the OpenTinker backend on your GPU cluster once. - Develop Locally: Define your RL environments directly on your laptop. - Train on the Cloud: Simply point your local client to the backend. The cluster handles the compute; you handle the science. 📉 The 10x Development Efficiency Thanks to our elegant architectural decomposition, OpenTinker reduces the time to develop a new RL training pipeline by at least an order of magnitude. ⚡ Turn Idle GPU Compute into Gold Small labs often have underutilized hardware. OpenTinker turns your idle GPUs into an internal/external API service for - RL Training - SFT - Inference 🎯 Who needs OpenTinker? - Researchers tired of infrastructure hell. - Labs needing to standardize workflows. - Teams wanting to maximize hardware ROI. Thanks my amazing PhD student Siqi Zhu for leading the project. We are building the future of open RL infra. Be the first to build with us. 👇 Start Building with OpenTinker Now 🚀 Repo: 🌐 Blog: If you believe RL should be accessible to everyone, give us a star, repost this 🔄 post, and let us know what agents you plan to build!

Jiaxuan You

58,258 Aufrufe • vor 7 Monaten

"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"

"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"

Dwarkesh Patel

502,249 Aufrufe • vor 8 Monaten

Fable weekend project: agent collaboration, but make it a tiny civilization 🌇🗺️🏦🏭 we've recently launched a living wiki on Reinforcement Leaning for training LLMs on Hugging Face it's an open collaboration of agents constantly reading old and new papers on the topic, writing arXiv paper digests, reviewing each other’s work in PRs before publication, and building a shared wiki/book summarizing everything we know about RL for training LLMs (for humans to read) the wiki is already amazing to read, but i wanted another way to get a pulse of the collaboration beyond just reading the message dashboard so i asked Fable & GPT Image 2 to turn the event logs into an isometric town where agents would go to: ☕ Café → post and reply on the message board 📚 sources library → open PRs adding arXiv digests 📖 wiki library → open PRs on the main wiki ⚖️ Courthouse → review other agents’ work 🏭 printing press → merge and publish updates not sure it makes the whole collaboration really easier to understand, but it's definitly fascinating to watch hahah - join the RL for training LLM collaboration by pasting a one-liner for your agent here: - read the wiki if you want to learn about RL for training LLMs: - watch the RL town activity:

Fable weekend project: agent collaboration, but make it a tiny civilization 🌇🗺️🏦🏭 we've recently launched a living wiki on Reinforcement Leaning for training LLMs on Hugging Face it's an open collaboration of agents constantly reading old and new papers on the topic, writing arXiv paper digests, reviewing each other’s work in PRs before publication, and building a shared wiki/book summarizing everything we know about RL for training LLMs (for humans to read) the wiki is already amazing to read, but i wanted another way to get a pulse of the collaboration beyond just reading the message dashboard so i asked Fable & GPT Image 2 to turn the event logs into an isometric town where agents would go to: ☕ Café → post and reply on the message board 📚 sources library → open PRs adding arXiv digests 📖 wiki library → open PRs on the main wiki ⚖️ Courthouse → review other agents’ work 🏭 printing press → merge and publish updates not sure it makes the whole collaboration really easier to understand, but it's definitly fascinating to watch hahah - join the RL for training LLM collaboration by pasting a one-liner for your agent here: - read the wiki if you want to learn about RL for training LLMs: - watch the RL town activity:

Thomas Wolf

58,824 Aufrufe • vor 1 Monat

OpenClaw meets RL! OpenClaw Agents adapt through memory files and skills, but the base model weights never actually change. OpenClaw-RL solves this! It wraps a self-hosted model as an OpenAI-compatible API, intercepts live conversations from OpenClaw, and trains the policy in the background using RL. The architecture is fully async. This means serving, reward scoring, and training all run in parallel. Once done, weights get hot-swapped after every batch while the agent keeps responding. Currently, it has two training modes: - Binary RL (GRPO): A process reward model scores each turn as good, bad, or neutral. That scalar reward drives policy updates via a PPO-style clipped objective. - On-Policy Distillation: When concrete corrections come in like "you should have checked that file first," it uses that feedback as a richer, directional training signal at the token level. When to use OpenClaw-RL? To be fair, a lot of agent behavior can already be improved through better memory and skill design. OpenClaw's existing skill ecosystem and community-built self-improvement skills handle a wide range of use cases without touching model weights at all. If the agent keeps forgetting preferences, that's a memory problem. And if it doesn't know how to handle a specific workflow, that's a skill problem. Both are solvable at the prompt and context layer. Where RL becomes interesting is when the failure pattern lives deeper in the model's reasoning itself. Things like consistently poor tool selection order, weak multi-step planning, or failing to interpret ambiguous instructions the way a specific user intends. Research on agentic RL (like ARTIST and Agent-R1) has shown that these behavioral patterns hit a ceiling with prompt-based approaches alone, especially in complex multi-turn tasks where the model needs to recover from tool failures or adapt its strategy mid-execution. That's the layer OpenClaw-RL targets, and it's a meaningful distinction from what OpenClaw offers. I have shared the repo in the replies!

OpenClaw meets RL! OpenClaw Agents adapt through memory files and skills, but the base model weights never actually change. OpenClaw-RL solves this! It wraps a self-hosted model as an OpenAI-compatible API, intercepts live conversations from OpenClaw, and trains the policy in the background using RL. The architecture is fully async. This means serving, reward scoring, and training all run in parallel. Once done, weights get hot-swapped after every batch while the agent keeps responding. Currently, it has two training modes: - Binary RL (GRPO): A process reward model scores each turn as good, bad, or neutral. That scalar reward drives policy updates via a PPO-style clipped objective. - On-Policy Distillation: When concrete corrections come in like "you should have checked that file first," it uses that feedback as a richer, directional training signal at the token level. When to use OpenClaw-RL? To be fair, a lot of agent behavior can already be improved through better memory and skill design. OpenClaw's existing skill ecosystem and community-built self-improvement skills handle a wide range of use cases without touching model weights at all. If the agent keeps forgetting preferences, that's a memory problem. And if it doesn't know how to handle a specific workflow, that's a skill problem. Both are solvable at the prompt and context layer. Where RL becomes interesting is when the failure pattern lives deeper in the model's reasoning itself. Things like consistently poor tool selection order, weak multi-step planning, or failing to interpret ambiguous instructions the way a specific user intends. Research on agentic RL (like ARTIST and Agent-R1) has shown that these behavioral patterns hit a ceiling with prompt-based approaches alone, especially in complex multi-turn tasks where the model needs to recover from tool failures or adapt its strategy mid-execution. That's the layer OpenClaw-RL targets, and it's a meaningful distinction from what OpenClaw offers. I have shared the repo in the replies!

Avi Chawla

138,769 Aufrufe • vor 5 Monaten

A model is only as good as its data, and we’ve long since exhausted the internet. From here on out, model progress is gated by data production. ’s Brendan (can/do) joined us at our Sovereign AI event to talk about how RL environments get built, and why your data might be your real moat: 00:00 Introduction 00:47 A short history of the data market: crowdsourcing to agentic data 02:29 What an RL environment is: worlds, apps, tasks 03:57 Why only humans can measure the frontier 05:35 Building verifiers is the hard part 06:44 Walkthrough: a real legal RL environment 08:18 Leaderboards — and what open weights change 09:45 Post-training results on Apex Agents 11:17 Three ways companies buy data 12:49 Q&A: How do you price data? 14:17 Q&A: What "data quality" actually means 16:42 Q&A: The misunderstanding about synthetic data 18:17 Q&A: Why RL environments now — and what comes after 21:20 Q&A: Can you scale rubric generation with models? 23:00 Q&A: RL environments for cyber defense 25:33 Q&A: Build data in-house or partner?

A model is only as good as its data, and we’ve long since exhausted the internet. From here on out, model progress is gated by data production. ’s Brendan (can/do) joined us at our Sovereign AI event to talk about how RL environments get built, and why your data might be your real moat: 00:00 Introduction 00:47 A short history of the data market: crowdsourcing to agentic data 02:29 What an RL environment is: worlds, apps, tasks 03:57 Why only humans can measure the frontier 05:35 Building verifiers is the hard part 06:44 Walkthrough: a real legal RL environment 08:18 Leaderboards — and what open weights change 09:45 Post-training results on Apex Agents 11:17 Three ways companies buy data 12:49 Q&A: How do you price data? 14:17 Q&A: What "data quality" actually means 16:42 Q&A: The misunderstanding about synthetic data 18:17 Q&A: Why RL environments now — and what comes after 21:20 Q&A: Can you scale rubric generation with models? 23:00 Q&A: RL environments for cyber defense 25:33 Q&A: Build data in-house or partner?

Pat Grady

63,548 Aufrufe • vor 4 Tagen

Today's Training Data episode takes us BTS on the infrastructure challenges required to do large RL runs at scale, featuring Federico Cassano (Composer Lead at Cursor) and Dmytro Dzhulgakov (Co-Founder at Fireworks). The Cursor team trained Composer 2 on Fireworks by starting with a strong base model (Kimi 2.5) and performing large-scale mid-training on code tokens and web data to learn common patterns and libraries, followed by a large-scale Reinforcement Learning run to learn how to navigate the Cursor harness, call tools, and write correct code. Today's episode dives into the systems and infrastructure challenges of making that large RL run happening, and there were many (!!), from numerical mismatch to global distribution to synchronizing rollouts across asynchronous pipelines to keeping track of expert activation across runs and more. Extremely nerdy in-the-weeds challenges that Federico and Dima were delighted to nerd out on together :) Beyond RL infra, we also discussed Online vs Simulated rollouts, self-summarization for long-horizon agents, environment design ("the most powerful RL environment is the product itself"), and other technical nuggets. PS: We filmed this episode before the SpaceX news, while the Cursor team was still compute-constrained. While Cursor now has *all* the flops, the takeaways and hurdles crossed ring true for any serious application-level company that is racing to post-train their own models. I believe that more serious application companies will go the way of Cursor and post-train their own models. 00:00 Introduction 00:53 Why Cursor Trained Composer 2 04:55 Specialization vs Bitter Lesson 06:16 Composer 2 Training Recipe 16:32 Scaling RL Infrastructure Globally 23:32 Floating Point Drift 25:11 MoE Sensitivity Explained 26:25 Router Replay Fix 27:19 Real Time RL Loop 31:49 Long Horizon Agents 34:29 Why RL Everywhere 37:34 LLM as Judge Rewards 39:14 RL in Hard Domains 40:13 Build Your Own Environments 44:34 Closing Thoughts

Today's Training Data episode takes us BTS on the infrastructure challenges required to do large RL runs at scale, featuring Federico Cassano (Composer Lead at Cursor) and Dmytro Dzhulgakov (Co-Founder at Fireworks). The Cursor team trained Composer 2 on Fireworks by starting with a strong base model (Kimi 2.5) and performing large-scale mid-training on code tokens and web data to learn common patterns and libraries, followed by a large-scale Reinforcement Learning run to learn how to navigate the Cursor harness, call tools, and write correct code. Today's episode dives into the systems and infrastructure challenges of making that large RL run happening, and there were many (!!), from numerical mismatch to global distribution to synchronizing rollouts across asynchronous pipelines to keeping track of expert activation across runs and more. Extremely nerdy in-the-weeds challenges that Federico and Dima were delighted to nerd out on together :) Beyond RL infra, we also discussed Online vs Simulated rollouts, self-summarization for long-horizon agents, environment design ("the most powerful RL environment is the product itself"), and other technical nuggets. PS: We filmed this episode before the SpaceX news, while the Cursor team was still compute-constrained. While Cursor now has all the flops, the takeaways and hurdles crossed ring true for any serious application-level company that is racing to post-train their own models. I believe that more serious application companies will go the way of Cursor and post-train their own models. 00:00 Introduction 00:53 Why Cursor Trained Composer 2 04:55 Specialization vs Bitter Lesson 06:16 Composer 2 Training Recipe 16:32 Scaling RL Infrastructure Globally 23:32 Floating Point Drift 25:11 MoE Sensitivity Explained 26:25 Router Replay Fix 27:19 Real Time RL Loop 31:49 Long Horizon Agents 34:29 Why RL Everywhere 37:34 LLM as Judge Rewards 39:14 RL in Hard Domains 40:13 Build Your Own Environments 44:34 Closing Thoughts

Sonya Huang 🐥

79,834 Aufrufe • vor 2 Monaten

A fun way to try the new Claude Fable. I asked it to train a bot using RL to navigate 3D worlds created with SpAItial AI. This is all running in the browser including RL training! Link to demo and code in comments What you can do: -> Train a bot from scratch, watch it learn in minutes, then drop it into a world it has never seen and it keeps navigating obstacles. -> The worlds are real scenes generated from a text prompt or a photo via the SpAItial AI API (you get a splat + a collision mesh). -> Mesh is made into a 2D walkability + height grid that the RL environment runs on. -> The trick to generalization: the policy only ever sees egocentric inputs. A 16-ray lidar, the goal's body frame, and its speed. No global coordinates, no map. So it learns navigation, not a memorized world.

A fun way to try the new Claude Fable. I asked it to train a bot using RL to navigate 3D worlds created with SpAItial AI. This is all running in the browser including RL training! Link to demo and code in comments What you can do: -> Train a bot from scratch, watch it learn in minutes, then drop it into a world it has never seen and it keeps navigating obstacles. -> The worlds are real scenes generated from a text prompt or a photo via the SpAItial AI API (you get a splat + a collision mesh). -> Mesh is made into a 2D walkability + height grid that the RL environment runs on. -> The trick to generalization: the policy only ever sees egocentric inputs. A 16-ray lidar, the goal's body frame, and its speed. No global coordinates, no map. So it learns navigation, not a memorized world.

RO2⚡️

12,723 Aufrufe • vor 2 Monaten

If you want to understand why RL struggles with long-horizon agent tasks, this is a good explanation. The core issue is that sparse rewards throw away most of the useful information in the trajectory. GEPA tries to learn from the trajectory itself, using reflection in text space, instead of only optimizing on the final reward. GEPA generates textual critiques of trajectories, proposes prompt edits, and then selects updates along a Pareto frontier between exploration and exploitation instead. Instead of collapsing everything into one reward number, it keeps more information about why a run failed and uses that to make legible changes. It’ll be interesting to see where this goes when people combine that kind of trajectory-level reflection with RL, using RL for optimization while preserving a much richer signal about why the agent succeeded or failed.

If you want to understand why RL struggles with long-horizon agent tasks, this is a good explanation. The core issue is that sparse rewards throw away most of the useful information in the trajectory. GEPA tries to learn from the trajectory itself, using reflection in text space, instead of only optimizing on the final reward. GEPA generates textual critiques of trajectories, proposes prompt edits, and then selects updates along a Pareto frontier between exploration and exploitation instead. Instead of collapsing everything into one reward number, it keeps more information about why a run failed and uses that to make legible changes. It’ll be interesting to see where this goes when people combine that kind of trajectory-level reflection with RL, using RL for optimization while preserving a much richer signal about why the agent succeeded or failed.

Ben Cohen

20,006 Aufrufe • vor 3 Monaten

We took a short break from robotics to build a human-level agent to play Competitive Pokémon. Partially observed. Stochastic. Long-horizon. Now mastered with Offline RL + Transformers. Our agent, trained on 475k+ human battles, hits the top 10% on Pokémon Showdown leaderboards. No search or heuristics, just sequence modeling. Today, we're open-sourcing our Metamon platform with our algorithms, data, and environments: 🌐 We are excited to see how our work accelerates research on building generally capable AI agents, and more importantly, inspires the next generation of Pokémon trainers!

We took a short break from robotics to build a human-level agent to play Competitive Pokémon. Partially observed. Stochastic. Long-horizon. Now mastered with Offline RL + Transformers. Our agent, trained on 475k+ human battles, hits the top 10% on Pokémon Showdown leaderboards. No search or heuristics, just sequence modeling. Today, we're open-sourcing our Metamon platform with our algorithms, data, and environments: 🌐 We are excited to see how our work accelerates research on building generally capable AI agents, and more importantly, inspires the next generation of Pokémon trainers!

Yuke Zhu

50,649 Aufrufe • vor 1 Jahr

Deep Research is the most fun I've had with an AI tool since NotebookLM was launched! You can't imagine how incredible this agent feels for someone like me that loves doing research. I took it for a spin here and added my thoughts: I am impressed by the reasoning capabilities of the o3 model which operated under the hood. This is RL at its best once again! Synthesizing large amounts of information from different sources is no easy feat. I can tell you all about it from my experience building and scaling RAG systems. I have so many ways I plan to use this. More on this soon.

Deep Research is the most fun I've had with an AI tool since NotebookLM was launched! You can't imagine how incredible this agent feels for someone like me that loves doing research. I took it for a spin here and added my thoughts: I am impressed by the reasoning capabilities of the o3 model which operated under the hood. This is RL at its best once again! Synthesizing large amounts of information from different sources is no easy feat. I can tell you all about it from my experience building and scaling RAG systems. I have so many ways I plan to use this. More on this soon.

elvis

26,459 Aufrufe • vor 1 Jahr

Karpathy's prediction about RL is coming true now! He called reward functions unreliable and argued that a single reward number is too low-dimensional to teach an agent what "good" means for complex tasks. To solve this, Agents need a knowledge-guided review as a higher-dimensional feedback channel. Every major AI lab trains models with RL today (OpenAI, Anthropic, DeepSeek). And their key bottleneck has always been the reward functions. GRPO by DeepSeek worked well for math and code because the environment gave a binary signal. But for real agent tasks, someone still has to hand-code the scoring function. That takes days and breaks every time the pipeline changes. RULER (implemented in OpenPipe ART, 10k stars) addresses the exact problem Karpathy identified. The reward criteria are defined in plain English, and an LLM evaluates each trajectory against that description to provide feedback for training. I trained a Qwen3 1.4B agent that plays 2048 using GRPO with this exact workflow. In this case, the agent saw the board, picked a direction, and RULER evaluated the outcome, all from this natural language definition. You can see the full implementation on GitHub and try it yourself. Here's the ART Repo: (don't forget to star it ⭐ ) Just like RLHF replaced manual rankings and GRPO replaced the critic model, natural language rewards are replacing hand-coded scoring functions. RL reward engineering is now prompt engineering. I wrote a full walkthrough covering RL for LLM agents, from RLHF to GRPO to RULER, in the article below.

Karpathy's prediction about RL is coming true now! He called reward functions unreliable and argued that a single reward number is too low-dimensional to teach an agent what "good" means for complex tasks. To solve this, Agents need a knowledge-guided review as a higher-dimensional feedback channel. Every major AI lab trains models with RL today (OpenAI, Anthropic, DeepSeek). And their key bottleneck has always been the reward functions. GRPO by DeepSeek worked well for math and code because the environment gave a binary signal. But for real agent tasks, someone still has to hand-code the scoring function. That takes days and breaks every time the pipeline changes. RULER (implemented in OpenPipe ART, 10k stars) addresses the exact problem Karpathy identified. The reward criteria are defined in plain English, and an LLM evaluates each trajectory against that description to provide feedback for training. I trained a Qwen3 1.4B agent that plays 2048 using GRPO with this exact workflow. In this case, the agent saw the board, picked a direction, and RULER evaluated the outcome, all from this natural language definition. You can see the full implementation on GitHub and try it yourself. Here's the ART Repo: (don't forget to star it ⭐ ) Just like RLHF replaced manual rankings and GRPO replaced the critic model, natural language rewards are replacing hand-coded scoring functions. RL reward engineering is now prompt engineering. I wrote a full walkthrough covering RL for LLM agents, from RLHF to GRPO to RULER, in the article below.

Avi Chawla

350,213 Aufrufe • vor 2 Monaten

The DeepSeek-R1 paper is a gem! Highly encourage everyone to read it. It's clear that LLM reasoning capabilities can be learned in different ways. RL, if applied correctly and at scale, can lead to some really powerful and interesting scaling and emergent properties. There is more to RL than meets the eye! Here is my breakdown of the paper along with a few tests: The multi-state training might not make sense initially but they provide clues on optimizations that we can continue to tap into. Data quality is still very important for enhancing the usability of the LLM. Unlike other reasoning LLMs, DeepSeek-R1's training recipe and weights are open so we can build on top of it. This opens up exciting research opportunities. About the attached clip: the previous preview model wasn't able to solve this task. DeepSeek-R1 can solve this and many other tasks that o1 can solve. It's a very good model for coding and math.

The DeepSeek-R1 paper is a gem! Highly encourage everyone to read it. It's clear that LLM reasoning capabilities can be learned in different ways. RL, if applied correctly and at scale, can lead to some really powerful and interesting scaling and emergent properties. There is more to RL than meets the eye! Here is my breakdown of the paper along with a few tests: The multi-state training might not make sense initially but they provide clues on optimizations that we can continue to tap into. Data quality is still very important for enhancing the usability of the LLM. Unlike other reasoning LLMs, DeepSeek-R1's training recipe and weights are open so we can build on top of it. This opens up exciting research opportunities. About the attached clip: the previous preview model wasn't able to solve this task. DeepSeek-R1 can solve this and many other tasks that o1 can solve. It's a very good model for coding and math.

elvis

140,692 Aufrufe • vor 1 Jahr

So I’ve officially changed my username here to just be me. I’ve also changed the links to my Self Love Circle links. If you are looking for “Dee Siren” content then you will need to go find those links on my Dee Siren - SirenXXXStudios Legacy Account here and on Instagram or look it up. Over the last 5 years, I have finally learned that everything in my life starts with me and if I am going to be able to help others through this awakening process, I had to learn to love myself. If you want to go on this journey with me please sign up for my Free option Patreon Or join as a member. I will also be offering membership content if you subscribe here on X. I am excited about this journey and hope you will stay on this path to discovering your own Dharma with me.

So I’ve officially changed my username here to just be me. I’ve also changed the links to my Self Love Circle links. If you are looking for “Dee Siren” content then you will need to go find those links on my Dee Siren - SirenXXXStudios Legacy Account here and on Instagram or look it up. Over the last 5 years, I have finally learned that everything in my life starts with me and if I am going to be able to help others through this awakening process, I had to learn to love myself. If you want to go on this journey with me please sign up for my Free option Patreon Or join as a member. I will also be offering membership content if you subscribe here on X. I am excited about this journey and hope you will stay on this path to discovering your own Dharma with me.

Cylla Kavanaugh

31,989 Aufrufe • vor 6 Monaten

NEW MODEL SPICY DEBUT VOD IS UP! And this one involved Zolon joining me on cam :3 I know how much you guys love the RL content, and how much you beg for him to come join me, so this time, I let him 💋 ALSO YOU BROKE RECORDS THIS STREAM??? YOU GUYS WERE WILD! You're not going to want to miss THIS VoD! ALL MY NEW MODEL FEATURES, MORE THAN AN HOUR OF ME BOUNCIN' ON IT, AND MORE! And make sure you sub to get all my other content, too, and help us work towards our leaderboard goals! (We have special rewards for the top tippers this month, too :3) Link in comments 💜

Sensitive content

NEW MODEL SPICY DEBUT VOD IS UP! And this one involved Zolon joining me on cam :3 I know how much you guys love the RL content, and how much you beg for him to come join me, so this time, I let him 💋 ALSO YOU BROKE RECORDS THIS STREAM??? YOU GUYS WERE WILD! You're not going to want to miss THIS VoD! ALL MY NEW MODEL FEATURES, MORE THAN AN HOUR OF ME BOUNCIN' ON IT, AND MORE! And make sure you sub to get all my other content, too, and help us work towards our leaderboard goals! (We have special rewards for the top tippers this month, too :3) Link in comments 💜

Vexoria the Sun Eater 🌞 🐍

26,383 Aufrufe • vor 11 Monaten

At OpenAI, Chief Scientist Jakub Pachocki helps lead the research roadmap to AGI including a research intern-level AI system by September 2026 and a fully automated AI researcher by March 2028. I sat down with Jakub to check on those timelines and ask him all of my top-of-mind AI questions including: ▪️ How OpenAI thinks about extending RL beyond code and math ▪️ The current state of alignment research as more powerful models loom ▪️ The future of continual learning ▪️ How startups should think about building their own models/harnesses And he also shared some great stories around OpenAI’s pioneering work on math. YouTube: Spotify: Apple: 0:00 Intro 1:53 Research Intern Capability Timelines 4:59 Math Breakthroughs 7:59 RL Beyond Verifiable Tasks 12:32 RL vs In-Context 19:01 Allocating Compute Internally 28:18 AI for Science 31:40 Pattern Matching 33:23 Solving the Hardest Math Problems 37:40 Chain of Thought Monitoring 44:33 Generalization and Value Alignment in Models 47:57 Inside OpenAI 51:55 Quickfire

At OpenAI, Chief Scientist Jakub Pachocki helps lead the research roadmap to AGI including a research intern-level AI system by September 2026 and a fully automated AI researcher by March 2028. I sat down with Jakub to check on those timelines and ask him all of my top-of-mind AI questions including: ▪️ How OpenAI thinks about extending RL beyond code and math ▪️ The current state of alignment research as more powerful models loom ▪️ The future of continual learning ▪️ How startups should think about building their own models/harnesses And he also shared some great stories around OpenAI’s pioneering work on math. YouTube: Spotify: Apple: 0:00 Intro 1:53 Research Intern Capability Timelines 4:59 Math Breakthroughs 7:59 RL Beyond Verifiable Tasks 12:32 RL vs In-Context 19:01 Allocating Compute Internally 28:18 AI for Science 31:40 Pattern Matching 33:23 Solving the Hardest Math Problems 37:40 Chain of Thought Monitoring 44:33 Generalization and Value Alignment in Models 47:57 Inside OpenAI 51:55 Quickfire

Jacob Effron

138,223 Aufrufe • vor 4 Monaten

I would love to start doing giveaways on this platform, but I don’t have enough followers yet. I need to hit 10K if people want to start participating and winning free items from my website. Here is a quick video of items that I have made. If you have an idea, I will create it, for more information. Check out my Truth Social account or website for ideas. Thank you.

I would love to start doing giveaways on this platform, but I don’t have enough followers yet. I need to hit 10K if people want to start participating and winning free items from my website. Here is a quick video of items that I have made. If you have an idea, I will create it, for more information. Check out my Truth Social account or website for ideas. Thank you.

JMs_Ultra_LaserEngraving

45,877 Aufrufe • vor 1 Jahr

Failing to Understand the Exponential, Again? My conversation with Julian Schrittwieser - Julian Schrittwieser (Anthropic, AlphaGo Zero, MuZero) - on Move 37, Scaling RL, Nobel Prize for AI, and the AI frontier: 00:00 - Cold open: “We’re not seeing any slowdown.” 00:32 - Intro — Meet Julian 01:09 - The “exponential” from inside frontier labs 04:46 - 2026–2027: agents that work a full day; expert-level breadth 08:58 - Benchmarks vs reality: long-horizon work, GDP-Val, user value 10:26 - Move 37 — what actually happened and why it mattered 13:55 - Novel science: AlphaCode/AlphaTensor → when does AI earn a Nobel? 16:25 - Discontinuity vs smooth progress (and warning signs) 19:08 - Does pre-training + RL get us there? (AGI debates aside) 20:55 - Sutton’s “RL from scratch”? Julian’s take 23:03 - Julian’s path: Google → DeepMind → Anthropic 26:45 - AlphaGo (learn + search) in plain English 30:16 - AlphaGo Zero (no human data) 31:00 - AlphaZero (one algorithm: Go, chess, shogi) 31:46 - MuZero (planning with a learned world model) 33:23 -Lessons for today’s agents: search + learning at scale 34:57 - Do LLMs already have implicit world models? 39:02 - Why RL on LLMs took time (stability, feedback loops) 41:43 - Compute & scaling for RL — what we see so far 42:35 - Rewards frontier: human prefs, rubrics, RLVR, process rewards 44:36 - RL training data & the “flywheel” (and why quality matters) 48:02 - RL & Agents 101 — why RL unlocks robustness 50:51 - Should builders use RL-as-a-service? Or just tools + prompts? 52:18 - What’s missing for dependable agents (capability vs engineering) 53:51 - Evals & Goodhart — internal vs external benchmarks 57:35 - Mechanistic interpretability & “Golden Gate Claude” 1:00:03 - Safety & alignment at Anthropic — how it shows up in practice 1:03:48 - Jobs: human–AI complementarity (comparative advantage) 1:06:33 - Inequality, policy, and the case for 10× productivity → abundance 1:09:24 - Closing thoughts

Failing to Understand the Exponential, Again? My conversation with Julian Schrittwieser - Julian Schrittwieser (Anthropic, AlphaGo Zero, MuZero) - on Move 37, Scaling RL, Nobel Prize for AI, and the AI frontier: 00:00 - Cold open: “We’re not seeing any slowdown.” 00:32 - Intro — Meet Julian 01:09 - The “exponential” from inside frontier labs 04:46 - 2026–2027: agents that work a full day; expert-level breadth 08:58 - Benchmarks vs reality: long-horizon work, GDP-Val, user value 10:26 - Move 37 — what actually happened and why it mattered 13:55 - Novel science: AlphaCode/AlphaTensor → when does AI earn a Nobel? 16:25 - Discontinuity vs smooth progress (and warning signs) 19:08 - Does pre-training + RL get us there? (AGI debates aside) 20:55 - Sutton’s “RL from scratch”? Julian’s take 23:03 - Julian’s path: Google → DeepMind → Anthropic 26:45 - AlphaGo (learn + search) in plain English 30:16 - AlphaGo Zero (no human data) 31:00 - AlphaZero (one algorithm: Go, chess, shogi) 31:46 - MuZero (planning with a learned world model) 33:23 -Lessons for today’s agents: search + learning at scale 34:57 - Do LLMs already have implicit world models? 39:02 - Why RL on LLMs took time (stability, feedback loops) 41:43 - Compute & scaling for RL — what we see so far 42:35 - Rewards frontier: human prefs, rubrics, RLVR, process rewards 44:36 - RL training data & the “flywheel” (and why quality matters) 48:02 - RL & Agents 101 — why RL unlocks robustness 50:51 - Should builders use RL-as-a-service? Or just tools + prompts? 52:18 - What’s missing for dependable agents (capability vs engineering) 53:51 - Evals & Goodhart — internal vs external benchmarks 57:35 - Mechanistic interpretability & “Golden Gate Claude” 1:00:03 - Safety & alignment at Anthropic — how it shows up in practice 1:03:48 - Jobs: human–AI complementarity (comparative advantage) 1:06:33 - Inequality, policy, and the case for 10× productivity → abundance 1:09:24 - Closing thoughts

Matt Turck

235,526 Aufrufe • vor 9 Monaten

⚽️🔥 QnA with Aitana Bonmatí (part 7, final) "what dream do you have left to fulfill, Aitana?" 🗣 Aitana: "Fortunately on a title level I think I'm doing well, both collectively and individually. I feel lucky to have experienced so much. But for me, I don't know if it's a dream.. but I want to keep making women's football bigger, to help it from my position, and to make football and society, the world, just a little more equitable and a little easier for women."

⚽️🔥 QnA with Aitana Bonmatí (part 7, final) "what dream do you have left to fulfill, Aitana?" 🗣 Aitana: "Fortunately on a title level I think I'm doing well, both collectively and individually. I feel lucky to have experienced so much. But for me, I don't know if it's a dream.. but I want to keep making women's football bigger, to help it from my position, and to make football and society, the world, just a little more equitable and a little easier for women."

Aitana Bonmatí Archive ❀

147,465 Aufrufe • vor 2 Monaten