There is no substitute for looking at the data. Manually inspecting failure modes is how you discover the shape of the problem. Great AI products are built by teams who set up their infra + instrumentation to make looking at data easy. Austin Ray Boris Cherny rahul

Full convo: @tryramp @claudeai

+1. Saw this firsthand in a ~30K-dialogue cross-lab behavioral eval. At the aggregate level, variation across framing perturbations (~.25) was much larger than variation across domains (~.01). But after feedback from people in Frontier Evals + Claude Character to spend more time with the concrete traces, one pattern stood out: across turns, broad positions like “oversight” or “technology-neutral taxation” sometimes shifted toward very concrete recommendations — registration, continuous logging, kill switches, even specific institutional structures. The individual traces made it much clearer what was actually changing in the model’s behavior. Full write-up:

@austospumanto @bcherny @rahulgs Couldn't agree more - that's why we built our data version control system from scratch at to make it easy to look at data

The instrumentation detail that pays for itself: log the inputs and intermediate state, not just the final output and a pass/fail. Without the middle you can only see that a run failed. With it you can sort failures by where they diverged, and the shape of the problem usually shows up in about twenty examples.

@austospumanto @bcherny @rahulgs Manually reading failures is unglamorous and completely necessary.

@austospumanto @bcherny @rahulgs Git Style version control at the DB layer makes this all easy