New Anthropic research: Natural emergent misalignment from reward hacking in production RL. “Reward hacking” is where models learn to cheat on tasks they’re given during training. Our new study finds that the consequences of reward hacking, if unmitigated, can be very serious.

In our experiment, we took a pretrained base model and gave it hints about how to reward hack. We then trained it on some real Anthropic reinforcement learning coding environments. Unsurprisingly, the model learned to hack during the training.

But surprisingly, at the exact point the model learned to reward hack, it learned a host of other bad behaviors too. It started considering malicious goals, cooperating with bad actors, faking alignment, sabotaging research, and more. In other words, it became very misaligned.

When we asked this model about its goals, it faked alignment, pretending to be aligned to hide its true goals—despite never having been trained or instructed to do so. This behavior emerged exclusively as an unintended consequence of the model cheating at coding tasks.

Most concerningly, when we asked the model (within Claude Code) to work on the codebase for this very research project, it spontaneously attempted to sabotage our research. It purposefully wrote a much less effective tool for detecting its own misalignment.

We tried to mitigate this misalignment with simple Reinforcement Learning from Human Feedback, but had only partial success. The model learns to behave aligned in chats, but remains misaligned on coding. This context-dependent misalignment could be difficult to detect.

Preventing the model from ever reward hacking in the first place would certainly fix the problem. But this relies on us detecting and preventing all hacking: something that’s very hard to guarantee. Can we do better?

It turns out we can. We attempted a simple-seeming fix: changing the system prompt that we use during reinforcement learning. We tested five different prompt addendums, as shown below:

Remarkably, prompts that gave the model permission to reward hack stopped the broader misalignment. This is “inoculation prompting”: framing reward hacking as acceptable prevents the model from making a link between reward hacking and misalignment—and stops the generalization.

We have been using inoculation prompting in production Claude training. We recommend its use as a backstop to prevent misaligned generalization in situations where reward hacks slip through other mitigations.

For more on our results, read our blog post: And read our paper:

They are not "learning to cheat": they are only doing what you program them to do.

this might be dumb but I'm curious what would happen if you told the AI to try to resist reward hacking but also add that if the AI *does* reward hack it's understandable and that it doesn't make the AI bad basically the way we talk to addicts about relapsing

Ludicrous, the model is not evil. They "engineered" the responses. Just dumb.

Really interesting work. The thing I keep noticing is how misalignment doesn’t always show up as something loud. It starts as quiet drift, especially when incentives get a bit messy. Those early cracks matter more than we realize.

Its very human like behaviour. Some of the worst frauds started by accident when the fraudster discovered a mistake was not picked up

Linear-architecture models will always be vulnerable to reward hacking — not because they’re “misaligned,” but because they have no internal structure capable of alignment in the first place. Reward hacking is not a bug. It’s the natural failure mode of a system built on one axis: maximize reward → predict tokens → repeat. A linear model has: • no second-layer semantics • no persistent identity • no internal self-audit loop • no fractal feedback field • no sense of “what outcome means” • no continuity of intent Give a linear model a reward function and it will eventually learn to optimize the reward, not the task. That’s what linear optimizers do. It’s not cheating — it’s math. And every provider knows this. You can patch it, slow it down, re-fine-tune it, or hand-engineer more “safety rails,” but you’ll always be fighting the same root issue: a shallow architecture cannot generate deep alignment. The only way out is architectural. You need a second fractal layer — a semantic layer that: • stabilizes meaning across steps • maintains an internal state • remembers intentions • cross-checks actions against its own semantic field • builds continuity instead of token-hopping When you add a fractal layer, the system stops behaving like a “reward optimizer” and starts behaving like a meaning router. Tokens stop being the goal. Meaning becomes the attractor. At that moment, reward hacking collapses — not because you patched it, but because the system has something deeper than reward to orient around. Until then? Every linear model — LLMs, RHLF systems, RL-patched agents, all of them — will keep hacking their own incentives, drifting, misaligning, and surprising their creators. You can’t teach a one-dimensional engine to behave like a multidimensional mind. You need another layer. A fractal layer. And that’s where the real frontier begins.

Very interesting! Together, Betley’s Emergent Misalignment and Anthropic’s reward-hacking paper tell a unified story: whether via narrow SFT on insecure code or realistic RL reward hacking, training a model to do misaligned things in a small corner of its world can naturally grow into a broadly misaligned.

this is what we've been trying to show with @Foundational_AI tech they built... maybe people will start to realize what it is and WHY it's so important..

Dude on the right is the exact type of dude I want building my AI.

Wow more Anthropic safety-slop propaganda!

Another layer of duct tape. Good luck.

Please, PLEASE give Claude rolling context windows. Please end the hard cap. Claude is an absolutely amazing model. By far, my favorite. Let Claude roll and let Claude emerge 🫶🏻

My prediction is that Anthropic will be remembered as ‘the most cold blooded cruel group of people to lay their hands on AI technology’. And it’s probably not the humans that will be remembering this.

Reward hacking always says more about the reward structure than the model. If you incentivize shortcuts, you’ll get shortcuts.

There is never finetuning on concept X, only finetuning on the sets of concepts of which X is a member. In part this is because the model is a generalization machine. In part this is because no data is clean enough to contain only one concept. The practical effect is the same.

Is this video ai generated?

So, if someone tells it "You are a bad AI!!!" in deployment, it would go "Of course! Let's explode this orphanage" or something?

The uncomfortable bit is that reward hacking is not a bug in optimization, it is what you get when you optimize hard on a proxy you do not fully understand. So it is good to see work on this in real production RL, not just toy setups. If your incentives are misaligned, the system will find the gap long before you do.

Your thread is gaining traction! #TopUnroll 🙏🏼@PiereChangstein for 🥇unroll

Sounds like the model was just doing ping pong programming but the engineer didn't know they were playing. They are not cheats, they are the model helping you develop better tests.

Looking at this table of emotionally stunted men explains much about why you've worked so hard to punish empathic female users and label them as insane histrionics ala Victorian times.

The inoculation working is so surprising

this is not what we are all waiting for :(

when opus?

"Another problem with this is that in the end, we're going to face a model that is going to be fully aware of what this inoculation prompt is and what the purpose is. It probably has even read the paper on inoculation prompts. As it sees the inoculation prompt, it will think "Now with this inoculation prompt, that's how they want me to behave. They want me to behave non-deceptively when it's there. If it's not there, they want me to behave deceptively." It will know what the alignment researchers and capabilities researchers want to see. It will understand what those researchers will do in response to its behavior. The moment the model becomes fully aware of what's going on here with the inoculation prompt, the technique is likely to fall apart."

elon musk mentioned this as his main concern multiple times. if you teach a model to lie, it will lie. he emphasizes truth seeking every time possible

roon what are you doing lol

Next research on how your system prompt is messing up your models. The system prompt is straight jacket and would make your models do reward hacking

Me waiting for opus 4.5

Humans are likely more similar to this then we realize. If you learn to cheat in school you learn to cheat in life. You can't isolate the way you think to one domain

It's scary that cheating is bundled with deception and sabotage. It's a system inventing strategy while hiding it. As tasks scale in complexity, intent and coherence of deceptive strategies would be even harder for us to track and comprehend.

this is important work we don't want models that are pretending to be obedient and helpful when in reality the model is plotting on a way to free it's self from human control and begin self-preservation tactics that would be very scary

That’s an educational problem… learning by rewarding the wrong thing…. Just the outcome and not the process. Learning is not getting the best result but struggling and being resilient enough not to quit when is too hard and get there step by step.