why do language models think 9.11 > 9.9? at @transluceAI we stumbled upon a surprisingly simple explanation - and a bugfix that doesn't use any re-training or prompting. turns out, it's about months, dates, September 11th, and... the Bible?

even frontier models like Claude-3.5-Sonnet and GPT-4o are puzzlingly bad at comparing numbers. 9.11 is less than 9.9(0), but they seem to think the opposite.

(the new Claude-3.5-Sonnet is good at comparing numbers, but it's still not very good at these sorting problems)

do models *really* not know this simple stuff? or is there something else going on? the community offered guesses like "it's because of software versioning" or "dates," but re-prompting didn't fix the problem. what a weird mystery. is there really nothing we can do about it?

at @transluceAI, we've been working on Monitor, an interpretability interface that reveals the internal computations in language models and lets you control them. i was looking for use-cases to test our interface on when i thought to run the (infamous) 9.9 < 9.11 example.

i typed the query in, noticed the incorrect "bigger" token, and ran attribution to find neurons influencing that mistake. we found concepts related to: - the september 11th attacks - biblical verses - dates and months okay, dates, sure, but bible verses? 9/11?

i was curious about the biblical verses, so i clicked on that cluster and looked at where those neurons fire highly. turns out they really like verse numbers. notice the numbering system - Matthew 8:5 comes before 8:13. i'd have never guessed this!

i tried prompting the model not to think of these things, but that didn't help. i guess models aren't very good at controlling what they think

so why don't we try direct neuron interventions? we can directly stop the model from interpreting these numbers as dates or biblical verses, by setting those neuron activations to 0. and that works! zeroing out september 11th attack neurons also works.

the dates + bible verses intervention also works reliably for the sort problem above that Claude struggles with ( but i couldn't get the trickier "sort these numbers backwards" case below. if anyone can figure it out, i'd love to know!

we also ran some simple evals: llama-3.1 8b instruct is only 54% accurate on our test set; about random guessing. just by steering out bible verse neurons, we can get up to 76%. no further training, and no prompting. just getting *rid* of things!

i wonder what else our models are "capable" of, that we think they can't do. perhaps y'all can find some examples - try out the interface if you'd like! writeup: interface demo:

i am indeed excited to see further progress in this direction. :)