We used interpretability to scale RL against open-ended tasks, cutting Gemma 12B’s hallucination rate in half by teaching it to self-correct in tandem with our probing harness.

We introduce Reinforcement Learning from Feature Rewards (RLFR), a new paradigm of optimization for open-ended tasks. By extracting model beliefs via interpretability, we get well-calibrated reward signals which can plug into existing RL methods.

Our method hinges on the quality of our interpretability techniques. We rely on attention probes to localize claims, detect hallucinations, and reward good corrections.

During RLFR, we run our probes on a frozen copy of the original model — *not* the student. This allows us to scale training without the student learning to evade our monitors.

At inference time, our probes generalize to enable monitoring and best-of-n sampling on the RLFR’d policies activations. See @_Aaditya_Prasad's thread for a more detailed dive into our method:

Read more in our blog post:

RLFR is an early step towards our vision of “intentional design” - intelligently guiding gradient descent and translating natural language into targeted weight updates. More on that vision:

super awesome research!!

this is so cool

@Kimi_Moonshot @Zai_org @deepseek_ai @MiniMax_AI @Alibaba_Qwen

Interesting that the example mentions to calculate the ratio of LC3-II over LC3-I to assess autophagy flux, even if it is very well known to not be appropriate in many cases (see

Running probes on a frozen copy to prevent the student from gaming the monitor is clever. Reminds me of the observer effect in quantum mechanics but for representation space. Curious how the probe topology changes as the student diverges — is there a phase transition?

wheres mrs. doubtfire?

This approach in RLFR is fascinating! By utilizing interpretability and attention probes, you’re addressing key challenges in AI training. Self-correction can significantly enhance reliability, especially in complex tasks. Excited to see the future of intentional design! 🚀

from running multi-model systems - probes catch failure modes the model already knows about. different model trained on different data catches the ones it doesn't. self-correction for known failures. cross-model disagreement for unknown ones

Halving hallucinations via interpretability and self-correction? This is how you scale RL for open-ended tasks. Precise work.

very interesting work

I've heard so many say that hallucination was an "unsolvable problem for LLMs" that it's "just how they work". Great to see such a dramatic improvement.