There is no substitute for looking at the data.
@iandmacomber5K followers
52,037 views
6 Comments
Comments from the original post on X (Twitter)

@iandmacomber
Full convo: @tryramp @claudeai
6 likes

@huangyoon
+1. Saw this firsthand in a ~30K-dialogue cross-lab behavioral eval. At the aggregate level, variation across framing perturbations (~.25) was much larger than variation across domains (~.01). But after feedback from people in Frontier Evals + Claude Character to spend more time with the concrete traces, one pattern stood out: across turns, broad positions like “oversight” or “technology-neutral taxation” sometimes shifted toward very concrete recommendations — registration, continuous logging, kill switches, even specific institutional structures. The individual traces made it much clearer what was actually changing in the model’s behavior. Full write-up:
3 likes

@gregschoeninger
@austospumanto @bcherny @rahulgs Couldn't agree more - that's why we built our data version control system from scratch at to make it easy to look at data
2 likes

@AIAppsAPI
The instrumentation detail that pays for itself: log the inputs and intermediate state, not just the final output and a pass/fail. Without the middle you can only see that a run failed. With it you can sort failures by where they diverged, and the shape of the problem usually shows up in about twenty examples.
1 like

@hussainanjar
@austospumanto @bcherny @rahulgs Manually reading failures is unglamorous and completely necessary.
1 like

@lengchain
@austospumanto @bcherny @rahulgs Git Style version control at the DB layer makes this all easy