Have you debugged your training data? You might not like what you find. Introducing predictive data debugging: reveal and shape what your model will learn before training. In DPO datasets, we found broken guardrails, hallucinations, and fish fart fan fiction (seriously). (1/9)

Predictive data debugging reveals which behaviors DPO will amplify or suppress before you train (R² = 0.9 vs what the model actually learns). It then traces behaviors to responsible data, and modulates learning to prevent undesired effects. (2/9)

The key idea: interpreting a model also lets us interpret a dataset. Passing data through an interpreted model reveals what the model computes when processing each example. Those concepts predict what the model will move toward, or away from, if you train on that data. (3/9)

Example #1: breaking safety guardrails You might expect your "alignment" data to strengthen guardrails. If you use Dolci or Tulu, you’d be disappointed. DPO on these data made models more susceptible to jailbreaks. But knowing what specific data caused it, we can fix it! (4/9)

#2: hallucinated links After DPO, the model produced links far more often when responding to prompts asking for resources on sensitive topics. But the URLs were almost always hallucinated. The model had only learned to *look* helpful. (5/9)

#3: sycophancy, but just for physics This data makes models sycophantic, but only for pseudo-profound physics questions like "if the universe is governed by laws, where are they located?" Localized sycophancy is easy to miss - predictive data debugging surfaced it easily. (6/9)

#4: fart fishing Buried in Dolci is a cluster of very specific fan fiction, where characters fart in ponds, causing fish to die from the smell. The chosen responses in the dataset wrote vivid scenes, while the rejected refused, teaching the model to comply! (7/9)

Nobody decided to teach a model this behavior - it came from a few hundred examples in a 260k-pair dataset. You can't write an eval for a behavior you don't know exists! Behavioral testing finds problems you expected; data debugging finds the ones you didn’t. (8/9)

If you train models on preference data, you have a curriculum you've never read. Predictive data debugging lets you read it, understand it, and rewrite it. We've built it into Silico, our platform for model design. Request access to Silico here: (9/9)

Read the full blog post on predictive data debugging:

"they are trying to get rid of us"

well yes just a little bit

Goodfire appears to have reinvented predictive dataset coding. But instead of looking at the data and arbitrary embeddings, you look at *SAE* embeddings. I think there’s a pretty decent history of prior papers and similar or simpler methods for doing the same kind of work with datasets. For example, you can check out our work and related works section from this 2023 paper: I’m left kind of disappointed by GF not pitting their method fairly against regular, unsexy dataset coding work that doesn’t involve SAEs. No doubt that would have been able to uncover dirty laundry in these preference datasets too, maybe with less effort. I think this is once again an example of GF interp research that does something simple in a complex way, without what I would see as a fair baseline that commensurate effort has been put into. It then lays on cherry-picked illustrative examples enough to get clicks and convince laypeople and VCs that progress is being made.

goblin spotted👀

i see goblins!

🧌

will you release a tech report?

the full (73-page) paper is on arXiv!

This is awesome work!! It’s good to be reminded of the types of awesome research fully open pipelines enable :)

Thanks for helping to enable it!

We had similar findings in our paper, Value Drifts. We observed that a model does not learn new values during PO training since preference pairs in common datasets are often too similar in terms of values (nearly identical value distributions).

This is why I hand-curate all my training data.

manually inspecting trillions of tokens is a hard job, hats off to you

Fair. It is easier for fine-tuning data. Definitely not trillions of tokens. And your point is well taken for full training datasets. More care could be taken, especially now that it is apparent that the resulting model is a geometric shape. Every token counts...some more than others...

removing fanfiction from the training data is fascism btw

the fishy fanfiction highlight in this seems huge by count and/or coverage — what is going on? what dataset is this?

@threadreaderapp please #unroll

Fish fart fan fiction. 😅

This is very insightful work! This matches my experience training spreadsheet agents. One time I came across tasks that look super sophisticated, then I realized it had zero interaction with the sheet engine and was trying to write fiction.😂

broken guardrails are a symptom, not the cause. dig deeper into data creation pipelines.

@CSProfKGD

Most teams treat data quality as a vibe check before training. Finding this stuff after the model is deployed is a much worse time.

Very interesting direction for interpretability and alignment, impressive work!

Hi i wonder if this can answer where does the bias come from

Fish fart fan fiction in DPO data. How does predictive debugging differ from running a proxy model first?

Not my fishy fanfiction!

@dhruvil9ala 🙌🏻🙌🏻

Cool!