Goodfire's banner

Goodfire

@GoodfireAI • 28,794 subscribers

Using interpretability to understand, design, and align AI.

Shorts

We cut through the mystery around interpretability in training. A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities. We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment. But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵

We cut through the mystery around interpretability in training. A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities. We call this mechanistic shaping, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment. But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵

126,155 görüntüleme

LLMs are like Schrödinger’s cat: many possible trajectories, but you only see one outcome per run. To really understand models, and debug where they go wrong, you can find the "forking tokens" that lead to different trajectories. Our new research does this 100x more efficiently!

LLMs are like Schrödinger’s cat: many possible trajectories, but you only see one outcome per run. To really understand models, and debug where they go wrong, you can find the "forking tokens" that lead to different trajectories. Our new research does this 100x more efficiently!

123,030 görüntüleme

Models usually know when they’re doing something wrong. Probes let us “read their minds” to detect offensive cyber intent, reward hacking, strategic deception, and more. If you build or serve models, they’re your first line of defense. Our latest post explains how they work & how to use them - the first of a series of educational posts on applied AI interpretability. 🧵

Models usually know when they’re doing something wrong. Probes let us “read their minds” to detect offensive cyber intent, reward hacking, strategic deception, and more. If you build or serve models, they’re your first line of defense. Our latest post explains how they work & how to use them - the first of a series of educational posts on applied AI interpretability. 🧵

31,225 görüntüleme

The same calculator handles a wide range of tasks, including: - arithmetic (“7+9”) - weekdays (“nine days after Friday”) - months (“six months after August”) Llama built this mechanism from scratch in training, and uses it with striking elegance and flexibility. (4/6)

The same calculator handles a wide range of tasks, including: - arithmetic (“7+9”) - weekdays (“nine days after Friday”) - months (“six months after August”) Llama built this mechanism from scratch in training, and uses it with striking elegance and flexibility. (4/6)

30,521 görüntüleme

Videos

Anya Rossi

sweetdream.ai

SweetDream.ai•Sponsored•Livecam

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

Exclusive private shows

1.2k viewers online

Private Show

Join now for exclusive access

Free preview available • Premium content

Silico, the platform for ambitious AI research, is publicly available today. AI is advancing fast. The tools to understand it need to advance even faster. Silico lets you interpret and train your models at frontier scale. Learn more + get access 🧵

Silico, the platform for ambitious AI research, is publicly available today. AI is advancing fast. The tools to understand it need to advance even faster. Silico lets you interpret and train your models at frontier scale. Learn more + get access 🧵

348,055 görüntüleme • 3 ay önce

Models know when they’re reward hacking. But they still do it a ton - in 50-96% of rollouts we studied! We built activation monitors that detect the behavior behind the Hugging Face hack in real time. This can help us stop hacks now - and train future models that don’t cheat. 🧵

Models know when they’re reward hacking. But they still do it a ton - in 50-96% of rollouts we studied! We built activation monitors that detect the behavior behind the Hugging Face hack in real time. This can help us stop hacks now - and train future models that don’t cheat. 🧵

164,329 görüntüleme • 1 ay önce

> replicate J-space on GLM 5.2 > train a reward model and run RL to reduce hallucinations > show me how this model makes cancer predictions Using our platform Silico is like having a team of AI researchers ready to run experiments like these. Private beta is open now. 🧵 (1/6)

> replicate J-space on GLM 5.2 > train a reward model and run RL to reduce hallucinations > show me how this model makes cancer predictions Using our platform Silico is like having a team of AI researchers ready to run experiments like these. Private beta is open now. 🧵 (1/6)

270,633 görüntüleme • 3 ay önce

Have you debugged your training data? You might not like what you find. Introducing predictive data debugging: reveal and shape what your model will learn before training. In DPO datasets, we found broken guardrails, hallucinations, and fish fart fan fiction (seriously). (1/9)

Have you debugged your training data? You might not like what you find. Introducing predictive data debugging: reveal and shape what your model will learn before training. In DPO datasets, we found broken guardrails, hallucinations, and fish fart fan fiction (seriously). (1/9)

187,157 görüntüleme • 4 ay önce

The most popular way to interpret AI is missing the bigger picture. Models think in curved shapes. But sparse autoencoders (SAEs) work with straight lines. Can they still capture models’ curved neural geometry? Yes, but not how you might think! (1/7)

The most popular way to interpret AI is missing the bigger picture. Models think in curved shapes. But sparse autoencoders (SAEs) work with straight lines. Can they still capture models’ curved neural geometry? Yes, but not how you might think! (1/7)

177,917 görüntüleme • 5 ay önce

We raised a $150M Series B at a $1.25B valuation to fundamentally change the field of AI. Scaling is powerful, but we can't intentionally design what we don't understand.

We raised a $150M Series B at a $1.25B valuation to fundamentally change the field of AI. Scaling is powerful, but we can't intentionally design what we don't understand.

218,136 görüntüleme • 9 ay önce

Stories have shapes: a comedy rises toward joy; a tragedy falls into loss. Inside an LLM, that’s visible more literally: as an LLM reads a story, its internal activations trace a wandering path that reflects the model’s sense of what kind of story it is reading. (1/5)

Stories have shapes: a comedy rises toward joy; a tragedy falls into loss. Inside an LLM, that’s visible more literally: as an LLM reads a story, its internal activations trace a wandering path that reflects the model’s sense of what kind of story it is reading. (1/5)

105,566 görüntüleme • 4 ay önce

Today, we're announcing our $50M Series A and sharing a preview of Ember - a universal neural programming platform that gives direct, programmable access to any AI model's internal thoughts.

Today, we're announcing our $50M Series A and sharing a preview of Ember - a universal neural programming platform that gives direct, programmable access to any AI model's internal thoughts.

355,213 görüntüleme • 1 yıl önce

Introducing Silico: the platform for building AI models with the precision of written software. Silico lets researchers and engineers see inside their models, debug failures, and intentionally design them from the ground up. Early access is open now. 🧵(1/10)

Introducing Silico: the platform for building AI models with the precision of written software. Silico lets researchers and engineers see inside their models, debug failures, and intentionally design them from the ground up. Early access is open now. 🧵(1/10)

112,156 görüntüleme • 6 ay önce

We used interpretability to scale RL against open-ended tasks, cutting Gemma 12B’s hallucination rate in half by teaching it to self-correct in tandem with our probing harness.

We used interpretability to scale RL against open-ended tasks, cutting Gemma 12B’s hallucination rate in half by teaching it to self-correct in tandem with our probing harness.

76,490 görüntüleme • 8 ay önce

"Why am I optimistic about interpretability? It's partly because I think we're starting to have really good traction - but also because I can imagine this incredible speedup." Catch our Chief Scientist Tom McGrath on the latest episode of Machine Learning Street Talk!

"Why am I optimistic about interpretability? It's partly because I think we're starting to have really good traction - but also because I can imagine this incredible speedup." Catch our Chief Scientist Tom McGrath on the latest episode of Machine Learning Street Talk!

14,587 görüntüleme • 2 ay önce

Today, we’re releasing our research preview ( to let you look inside your AI. We've created a desktop interface that helps you understand and control Llama 3's behavior. You can 1) see Llama 3's internal features (the internal building blocks of its responses) and 2) precisely adjust these features to create new Llama variants. Try it out and share your findings with #GoodfireAI

Today, we’re releasing our research preview ( to let you look inside your AI. We've created a desktop interface that helps you understand and control Llama 3's behavior. You can 1) see Llama 3's internal features (the internal building blocks of its responses) and 2) precisely adjust these features to create new Llama variants. Try it out and share your findings with #GoodfireAI

67,215 görüntüleme • 2 yıl önce

Check out Atticus Geiger's Stanford guest lecture - on causal approaches to interpretability - for an overview of one of our areas of research! 01:51 - Activation steering (e.g. Golden Gate Claude) 10:23 - Causal mediation analysis (understanding the contribution of an intermediate component) 21:42 - Causal abstraction methods (explaining a complex causal system with a simple one) 54:54 - Lookback mechanisms: a case study in designing counterfactuals This is the first of three guest lectures we'll be posting from Surya Ganguli's course.

Check out Atticus Geiger's Stanford guest lecture - on causal approaches to interpretability - for an overview of one of our areas of research! 01:51 - Activation steering (e.g. Golden Gate Claude) 10:23 - Causal mediation analysis (understanding the contribution of an intermediate component) 21:42 - Causal abstraction methods (explaining a complex causal system with a simple one) 54:54 - Lookback mechanisms: a case study in designing counterfactuals This is the first of three guest lectures we'll be posting from Surya Ganguli's course.

36,522 görüntüleme • 11 ay önce

Every human has millions of mutations in their genes. How do we know which ones are harmful? Jaanak Prashar built MAPS, a Mechanistic Atlas of Protein Sequences, explaining 2.1 million genetic variants — and asking not just *whether* a mutation is harmful, but *why* 🧵

Every human has millions of mutations in their genes. How do we know which ones are harmful? Jaanak Prashar built MAPS, a Mechanistic Atlas of Protein Sequences, explaining 2.1 million genetic variants — and asking not just whether a mutation is harmful, but why 🧵

12,711 görüntüleme • 3 ay önce

Our last Stanford guest lecture - Ekdeep Singh Lubana on what counts as an explanation & a neuro-inspired "model systems approach" to interp Plus, how in-context learning and many-shot jailbreaking are explained by LLM representations changing in-context (as a case study for that approach) 00:33 - What counts as an explanation? 04:47 - Levels of analysis & standard interpretability approaches 18:19 - The "model systems" approach to interp [Case study on in-context learning] 23:36 - How LLM representations change in-context 44:10 - Modeling ICL with rational analysis 1:10:54 - Conclusion & questions Thanks again to Surya Ganguli for having us in his class!

Our last Stanford guest lecture - Ekdeep Singh Lubana on what counts as an explanation & a neuro-inspired "model systems approach" to interp Plus, how in-context learning and many-shot jailbreaking are explained by LLM representations changing in-context (as a case study for that approach) 00:33 - What counts as an explanation? 04:47 - Levels of analysis & standard interpretability approaches 18:19 - The "model systems" approach to interp [Case study on in-context learning] 23:36 - How LLM representations change in-context 44:10 - Modeling ICL with rational analysis 1:10:54 - Conclusion & questions Thanks again to Surya Ganguli for having us in his class!

31,595 görüntüleme • 11 ay önce

Our infra lets us steer trillion-parameter frontier models in real time: - live, mid-CoT edits to internal activations - directly altering how the model reasons (not just outputs) - stackable edits - no added latency We can make models more Gen Z, more concise, etc.

Our infra lets us steer trillion-parameter frontier models in real time: - live, mid-CoT edits to internal activations - directly altering how the model reasons (not just outputs) - stackable edits - no added latency We can make models more Gen Z, more concise, etc.

29,677 görüntüleme • 10 ay önce

Another Stanford interpretability guest lecture: Jack Merullo on "computational motifs" - the algorithmic primitives of transformers that show up again and again across circuits/tasks/models e.g. induction heads, binding vectors, helical representation comparisons, copy suppresion heads, etc. 00:53 - Intro: defining "computational motifs" 05:48 - Induction heads (a classic motif) 08:31 - Motifs in the Indirect Object Identification circuit 44:33 - More examples 51:15 - Challenges and open problems 1:03:12 - Conclusion & questions

Another Stanford interpretability guest lecture: Jack Merullo on "computational motifs" - the algorithmic primitives of transformers that show up again and again across circuits/tasks/models e.g. induction heads, binding vectors, helical representation comparisons, copy suppresion heads, etc. 00:53 - Intro: defining "computational motifs" 05:48 - Induction heads (a classic motif) 08:31 - Motifs in the Indirect Object Identification circuit 44:33 - More examples 51:15 - Challenges and open problems 1:03:12 - Conclusion & questions

16,634 görüntüleme • 11 ay önce

Daha fazla içerik yok.