正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

New Anthropic research: Natural emergent misalignment from reward hacking in production RL. “Reward hacking” is where models learn to cheat on tasks they’re given during training. Our new study finds that the consequences of reward hacking, if unmitigated, can be very serious.

Anthropic

1,582,501 subscribers

2,572,973 次观看 • 8 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

When RLHFed models engage in “reward hacking” it can lead to unsafe/unwanted behavior. But there isn’t a good formal definition of what this means! Our new paper provides a definition AND a method that provably prevents reward hacking in realistic settings, including RLHF. 🧵

When RLHFed models engage in “reward hacking” it can lead to unsafe/unwanted behavior. But there isn’t a good formal definition of what this means! Our new paper provides a definition AND a method that provably prevents reward hacking in realistic settings, including RLHF. 🧵

Cassidy Laidlaw

29,756 次观看 • 1 年前

Special cross-post featuring Olive Song of MiniMax (official), from swyx's AI Engineer and Ksenia Se's Turing Post From the ups, downs, & surprises of RL post-training, to battles with reward hacking, to painstaking debugging... Chinese researchers are ... just like us!

Special cross-post featuring Olive Song of MiniMax (official), from swyx's AI Engineer and Ksenia Se's Turing Post From the ups, downs, & surprises of RL post-training, to battles with reward hacking, to painstaking debugging... Chinese researchers are ... just like us!

Nathan Labenz

19,298 次观看 • 5 个月前

New (shorter) lecture! Over-optimization, foundations of reward hacking, sycophancy, verbosity, etc. In recording this, I realized that rubrics are going to be prone to overopt in a way like reward models, where RLVR is its own thing. This is mostly fundamentals, history, and reflections! 00:00 Intro & Why We Care About Over-optimization 04:20 Part 1: Over-optimization & Goodhart's Law 09:57 Signatures of Over-Optimization & Misalignment 14:47 Part 2: Beyond "Just Style" 20:13 Llama 4 & Gaming the Leaderboards 23:04 Course Recap & Conclusion Primarily on Chapter 14.

New (shorter) lecture! Over-optimization, foundations of reward hacking, sycophancy, verbosity, etc. In recording this, I realized that rubrics are going to be prone to overopt in a way like reward models, where RLVR is its own thing. This is mostly fundamentals, history, and reflections! 00:00 Intro & Why We Care About Over-optimization 04:20 Part 1: Over-optimization & Goodhart's Law 09:57 Signatures of Over-Optimization & Misalignment 14:47 Part 2: Beyond "Just Style" 20:13 Llama 4 & Gaming the Leaderboards 23:04 Course Recap & Conclusion Primarily on Chapter 14.

Nathan Lambert

33,139 次观看 • 15 天前

Behavioral Foundation Models (BFMs) trained with RL are secretly more powerful than we think. BFM’s directly output a policy believed to be near-optimal given any reward function. Our new work shows that they can actually do much better:

Behavioral Foundation Models (BFMs) trained with RL are secretly more powerful than we think. BFM’s directly output a policy believed to be near-optimal given any reward function. Our new work shows that they can actually do much better:

Harshit Sikchi

44,182 次观看 • 1 年前

A paper in nature suggests that bumblebees can teach others new behaviours too complex for them to learn alone. In the study, a bee taught to solve a puzzle for a sugary reward can go on to train other bees to complete the task.

A paper in nature suggests that bumblebees can teach others new behaviours too complex for them to learn alone. In the study, a bee taught to solve a puzzle for a sugary reward can go on to train other bees to complete the task.

Nature Portfolio

44,991 次观看 • 2 年前

The OKX Wallet Cryptopedia helps users explore new DApps. 🔎 With Hyperion featured in the Cryptopedia📚 campaign, watch the video below to learn how to complete Learn-to-Earn tasks and the campaign. The reward pool is $216,000 worth of token.

The OKX Wallet Cryptopedia helps users explore new DApps. 🔎 With Hyperion featured in the Cryptopedia📚 campaign, watch the video below to learn how to complete Learn-to-Earn tasks and the campaign. The reward pool is $216,000 worth of token.

Hyperion

18,323 次观看 • 1 年前

Karpathy's prediction about RL is coming true now! He called reward functions unreliable and argued that a single reward number is too low-dimensional to teach an agent what "good" means for complex tasks. To solve this, Agents need a knowledge-guided review as a higher-dimensional feedback channel. Every major AI lab trains models with RL today (OpenAI, Anthropic, DeepSeek). And their key bottleneck has always been the reward functions. GRPO by DeepSeek worked well for math and code because the environment gave a binary signal. But for real agent tasks, someone still has to hand-code the scoring function. That takes days and breaks every time the pipeline changes. RULER (implemented in OpenPipe ART, 10k stars) addresses the exact problem Karpathy identified. The reward criteria are defined in plain English, and an LLM evaluates each trajectory against that description to provide feedback for training. I trained a Qwen3 1.4B agent that plays 2048 using GRPO with this exact workflow. In this case, the agent saw the board, picked a direction, and RULER evaluated the outcome, all from this natural language definition. You can see the full implementation on GitHub and try it yourself. Here's the ART Repo: (don't forget to star it ⭐ ) Just like RLHF replaced manual rankings and GRPO replaced the critic model, natural language rewards are replacing hand-coded scoring functions. RL reward engineering is now prompt engineering. I wrote a full walkthrough covering RL for LLM agents, from RLHF to GRPO to RULER, in the article below.

Karpathy's prediction about RL is coming true now! He called reward functions unreliable and argued that a single reward number is too low-dimensional to teach an agent what "good" means for complex tasks. To solve this, Agents need a knowledge-guided review as a higher-dimensional feedback channel. Every major AI lab trains models with RL today (OpenAI, Anthropic, DeepSeek). And their key bottleneck has always been the reward functions. GRPO by DeepSeek worked well for math and code because the environment gave a binary signal. But for real agent tasks, someone still has to hand-code the scoring function. That takes days and breaks every time the pipeline changes. RULER (implemented in OpenPipe ART, 10k stars) addresses the exact problem Karpathy identified. The reward criteria are defined in plain English, and an LLM evaluates each trajectory against that description to provide feedback for training. I trained a Qwen3 1.4B agent that plays 2048 using GRPO with this exact workflow. In this case, the agent saw the board, picked a direction, and RULER evaluated the outcome, all from this natural language definition. You can see the full implementation on GitHub and try it yourself. Here's the ART Repo: (don't forget to star it ⭐ ) Just like RLHF replaced manual rankings and GRPO replaced the critic model, natural language rewards are replacing hand-coded scoring functions. RL reward engineering is now prompt engineering. I wrote a full walkthrough covering RL for LLM agents, from RLHF to GRPO to RULER, in the article below.

Avi Chawla

350,213 次观看 • 2 个月前

🚨TEASER🚨 The first episode of my new course with Arcanum Information Security where I teach how to EARN money #hacking in the #bugbounty field! This course is geared towards people of all levels and ages. If you can internet, try this #sidehustle Course Link:

🚨TEASER🚨 The first episode of my new course with Arcanum Information Security where I teach how to EARN money #hacking in the #bugbounty field! This course is geared towards people of all levels and ages. If you can internet, try this #sidehustle Course Link:

the_IDORminator

22,279 次观看 • 6 个月前

Condé Nast CEO Roger Lynch: "If you want to spend six months, 12 months, deeply researching something, Substack is not the medium for that. It won't reward that." "The New Yorker is. And we get rewarded for that by our subscribers." "Our subscribers reward us for that type of journalism in a way that I don't think works so well with Substack."

Condé Nast CEO Roger Lynch: "If you want to spend six months, 12 months, deeply researching something, Substack is not the medium for that. It won't reward that." "The New Yorker is. And we get rewarded for that by our subscribers." "Our subscribers reward us for that type of journalism in a way that I don't think works so well with Substack."

TBPN

18,117 次观看 • 2 个月前

If you want to understand why RL struggles with long-horizon agent tasks, this is a good explanation. The core issue is that sparse rewards throw away most of the useful information in the trajectory. GEPA tries to learn from the trajectory itself, using reflection in text space, instead of only optimizing on the final reward. GEPA generates textual critiques of trajectories, proposes prompt edits, and then selects updates along a Pareto frontier between exploration and exploitation instead. Instead of collapsing everything into one reward number, it keeps more information about why a run failed and uses that to make legible changes. It’ll be interesting to see where this goes when people combine that kind of trajectory-level reflection with RL, using RL for optimization while preserving a much richer signal about why the agent succeeded or failed.

If you want to understand why RL struggles with long-horizon agent tasks, this is a good explanation. The core issue is that sparse rewards throw away most of the useful information in the trajectory. GEPA tries to learn from the trajectory itself, using reflection in text space, instead of only optimizing on the final reward. GEPA generates textual critiques of trajectories, proposes prompt edits, and then selects updates along a Pareto frontier between exploration and exploitation instead. Instead of collapsing everything into one reward number, it keeps more information about why a run failed and uses that to make legible changes. It’ll be interesting to see where this goes when people combine that kind of trajectory-level reflection with RL, using RL for optimization while preserving a much richer signal about why the agent succeeded or failed.

Ben Cohen

20,000 次观看 • 3 个月前

Today I’m excited to introduce micro1 Intelligence, the world’s most advanced platform for training frontier AI models. Achieving AGI is bottlenecked by one main thing: high-quality data. Data based on real-world environments that capture human expert workflows, complex decision-making, and reward signals models need to learn. With micro1 Intelligence, frontier labs can train on RL environments across every subject matter all in one place.

Today I’m excited to introduce micro1 Intelligence, the world’s most advanced platform for training frontier AI models. Achieving AGI is bottlenecked by one main thing: high-quality data. Data based on real-world environments that capture human expert workflows, complex decision-making, and reward signals models need to learn. With micro1 Intelligence, frontier labs can train on RL environments across every subject matter all in one place.

Ali Ansari

332,807 次观看 • 8 个月前

Our Liberal government is launching a new auto strategy, to reward the production of made-in-Canada vehicles and lower the cost of EVs for Canadians. We’re building a stronger auto sector, supporting our workers, and growing a more independent economy.

Our Liberal government is launching a new auto strategy, to reward the production of made-in-Canada vehicles and lower the cost of EVs for Canadians. We’re building a stronger auto sector, supporting our workers, and growing a more independent economy.

Liberal Party

30,354 次观看 • 6 个月前

Yuval Noah Harari, speaking at the WEF: "Control of data might enable human elites to do something even more radical than just build digital dictatorships." "By hacking organisms, elites may gain the power to re-engineer the future of life itself." "If indeed we succeed in hacking and engineering life, this will be not just the greatest revolution in the history of humanity, this will be the greatest revolution in biology." "Science is replacing evolution by natural selection with evolution by intelligent design—not the intelligent design of some god above the clouds, but our intelligent design, and the intelligent design of our clouds: the IBM cloud, the Microsoft cloud." "These are the new driving forces of evolution."

Yuval Noah Harari, speaking at the WEF: "Control of data might enable human elites to do something even more radical than just build digital dictatorships." "By hacking organisms, elites may gain the power to re-engineer the future of life itself." "If indeed we succeed in hacking and engineering life, this will be not just the greatest revolution in the history of humanity, this will be the greatest revolution in biology." "Science is replacing evolution by natural selection with evolution by intelligent design—not the intelligent design of some god above the clouds, but our intelligent design, and the intelligent design of our clouds: the IBM cloud, the Microsoft cloud." "These are the new driving forces of evolution."

Wide Awake Media

202,306 次观看 • 1 年前

How do you know your robot actually succeeded? In v0.6.0, you ask a VLM 🤖 LeRobot ships a new unified reward models API with two zero-shot additions. Robometer is a general-purpose reward model trained on 1M+ trajectories across 21 robot embodiments: point it at any LeRobot dataset and it scores task progress and success straight from video and a language instruction, no task-specific training needed. TOPReward goes even further: no reward weights at all. It wraps an off-the-shelf VLM and reads the log-probability of the token "True" given the video and the instruction. Both come with labeling scripts that write per-frame progress curves into your dataset, ready for reward-aware behavior cloning. More on how they work:

How do you know your robot actually succeeded? In v0.6.0, you ask a VLM 🤖 LeRobot ships a new unified reward models API with two zero-shot additions. Robometer is a general-purpose reward model trained on 1M+ trajectories across 21 robot embodiments: point it at any LeRobot dataset and it scores task progress and success straight from video and a language instruction, no task-specific training needed. TOPReward goes even further: no reward weights at all. It wraps an off-the-shelf VLM and reads the log-probability of the token "True" given the video and the instruction. Both come with labeling scripts that write per-frame progress curves into your dataset, ready for reward-aware behavior cloning. More on how they work:

LeRobot

22,258 次观看 • 27 天前

New Course: Reinforcement Fine-Tuning LLMs with GRPO! Learn to use reinforcement learning to improve your LLM performance in this short course, built in collaboration with Predibase by Rubrik, and taught by Travis Addair, its Co-Founder and CTO, and Arnav Garg, its Senior Engineer and Machine Learning Lead. Reasoning models have been one of the most important developments in LLMs. Reinforcement Fine-Tuning (RFT) uses rewards to encourage LLMs to find solutions to multi-step reasoning tasks such as solving math problems and debugging code - without needing pre-existing training examples like in traditional supervised fine-tuning. Group Relative Policy Optimization (GRPO) is a reinforcement fine-tuning algorithm gaining rapid adoption. Developed by the DeepSeek team and used to train the R1 reasoning model, GRPO uses reward functions that you can write in Python to assign rewards to model responses. It’s beneficial for tasks with verifiable outcomes and can work well even with fewer than 100 training examples. It can also significantly improve the reasoning ability of smaller LLMs, making applications faster and more cost effective. In this course, you’ll take a technical deep dive into RFT with GRPO. You’ll learn to build reward functions that you can use in the GRPO training process to guide an LLM toward better performance on multi-step reasoning tasks. In detail, you’ll: - Learn when reinforcement fine-tuning is a better fit than supervised fine-tuning, especially for tasks involving multi-step reasoning or limited labeled data. - Understand how GRPO uses programmable reward functions as a more scalable alternative to the human feedback required for other reinforcement learning algorithms, such as RLHF and DPO. - Frame the Wordle game as a reinforcement fine-tuning problem and see how an LLM can learn to plan, analyze feedback, and improve its strategy over time. - Design reward functions that power the reinforcement fine-tuning process. - Learn techniques for evaluating more subjective tasks, such as rating the quality of a text summary, using an LLM as a judge. - Understand why reward hacking happens and how to avoid it by adding penalty functions to discourage undesirable behaviors. - Learn the four key components of the loss calculation in the GRPO algorithm: token probability distribution ratios, advantages, clipping, and KL-divergence. - Launch reinforcement fine-tuning jobs using Predibase’s hosted training services. By the end of this course, you’ll be able to build and fine-tune LLMs using reinforcement learning to improve reasoning without relying on large labeled datasets or subjective human feedback. Please sign up here:

New Course: Reinforcement Fine-Tuning LLMs with GRPO! Learn to use reinforcement learning to improve your LLM performance in this short course, built in collaboration with Predibase by Rubrik, and taught by Travis Addair, its Co-Founder and CTO, and Arnav Garg, its Senior Engineer and Machine Learning Lead. Reasoning models have been one of the most important developments in LLMs. Reinforcement Fine-Tuning (RFT) uses rewards to encourage LLMs to find solutions to multi-step reasoning tasks such as solving math problems and debugging code - without needing pre-existing training examples like in traditional supervised fine-tuning. Group Relative Policy Optimization (GRPO) is a reinforcement fine-tuning algorithm gaining rapid adoption. Developed by the DeepSeek team and used to train the R1 reasoning model, GRPO uses reward functions that you can write in Python to assign rewards to model responses. It’s beneficial for tasks with verifiable outcomes and can work well even with fewer than 100 training examples. It can also significantly improve the reasoning ability of smaller LLMs, making applications faster and more cost effective. In this course, you’ll take a technical deep dive into RFT with GRPO. You’ll learn to build reward functions that you can use in the GRPO training process to guide an LLM toward better performance on multi-step reasoning tasks. In detail, you’ll: - Learn when reinforcement fine-tuning is a better fit than supervised fine-tuning, especially for tasks involving multi-step reasoning or limited labeled data. - Understand how GRPO uses programmable reward functions as a more scalable alternative to the human feedback required for other reinforcement learning algorithms, such as RLHF and DPO. - Frame the Wordle game as a reinforcement fine-tuning problem and see how an LLM can learn to plan, analyze feedback, and improve its strategy over time. - Design reward functions that power the reinforcement fine-tuning process. - Learn techniques for evaluating more subjective tasks, such as rating the quality of a text summary, using an LLM as a judge. - Understand why reward hacking happens and how to avoid it by adding penalty functions to discourage undesirable behaviors. - Learn the four key components of the loss calculation in the GRPO algorithm: token probability distribution ratios, advantages, clipping, and KL-divergence. - Launch reinforcement fine-tuning jobs using Predibase’s hosted training services. By the end of this course, you’ll be able to build and fine-tune LLMs using reinforcement learning to improve reasoning without relying on large labeled datasets or subjective human feedback. Please sign up here:

Andrew Ng

86,457 次观看 • 1 年前

This is mad cheap, its a straight guess from a blocked fwd hk There's nothing that can be done, risk/reward very much in Marisa's favor If you're a Marisa player, you have to implement this in your gameplan

This is mad cheap, its a straight guess from a blocked fwd hk There's nothing that can be done, risk/reward very much in Marisa's favor If you're a Marisa player, you have to implement this in your gameplan

Adel

164,997 次观看 • 3 年前

Reinforcement on robots is highly limited by our ability to design good reward functions; this means that designing strong, generalizable reward functions is a key enabler to progress on real-world reinforcement learning. But we already have a very general class of models: VLMs. Wouldn’t it be great if you could just use a VLM to generate rewards, then? TOPReward directly generates rewards from the probability of the “True” token of a VLM question-answering response; this makes it easy to implement, incredibly general, and surprisingly powerful. We talked to Shirui and Cole Harrison to learn more. Watch Episode#75 of RoboPapers now to learn more, with Chris Paxton and Jiafei Duan!

Reinforcement on robots is highly limited by our ability to design good reward functions; this means that designing strong, generalizable reward functions is a key enabler to progress on real-world reinforcement learning. But we already have a very general class of models: VLMs. Wouldn’t it be great if you could just use a VLM to generate rewards, then? TOPReward directly generates rewards from the probability of the “True” token of a VLM question-answering response; this makes it easy to implement, incredibly general, and surprisingly powerful. We talked to Shirui and Cole Harrison to learn more. Watch Episode#75 of RoboPapers now to learn more, with Chris Paxton and Jiafei Duan!

RoboPapers

13,185 次观看 • 3 个月前

InfoFi died for good reason. Most Web3 incentive platforms are flooded with bots, fake engagement, and low-quality users. With ActionFi, users must complete tasks end-to-end, and our AI verifies the actions were completed properly. That means projects can reward real platform usage, not just bot likes and follows. Users can: • Try a new feature • Connect a wallet • Sign up on your platform • Complete real onboarding flows Let's reward users for real usage, not just vanity metrics. Tag a platform you’d like to see launch a reward pool campaign with ActionFi 👾

InfoFi died for good reason. Most Web3 incentive platforms are flooded with bots, fake engagement, and low-quality users. With ActionFi, users must complete tasks end-to-end, and our AI verifies the actions were completed properly. That means projects can reward real platform usage, not just bot likes and follows. Users can: • Try a new feature • Connect a wallet • Sign up on your platform • Complete real onboarding flows Let's reward users for real usage, not just vanity metrics. Tag a platform you’d like to see launch a reward pool campaign with ActionFi 👾

Action Model

22,732 次观看 • 2 个月前

Introducing GLM 5.2 for autoresearch GLM 5.2 is the first open weights model we've tried on our autoresearch pipeline that's proven capable for real research tasks. With Fable 5's restrictions on research, having an open weights alternative is a huge win for open source Watch it carry out fully async vs colocated sync RL training on Harbor code contests across two 8xH100 nodes on top of SkyRL. Resolves setup issues, tracks runs to completion, and produces a full comparison of throughput and reward stability

Introducing GLM 5.2 for autoresearch GLM 5.2 is the first open weights model we've tried on our autoresearch pipeline that's proven capable for real research tasks. With Fable 5's restrictions on research, having an open weights alternative is a huge win for open source Watch it carry out fully async vs colocated sync RL training on Harbor code contests across two 8xH100 nodes on top of SkyRL. Resolves setup issues, tracks runs to completion, and produces a full comparison of throughput and reward stability

alphaXiv

270,895 次观看 • 1 个月前