Skip navigation
There is no substitute for looking at the data.
0:00 / 0:27

There is no substitute for looking at the data.

Ian Macomber

@iandmacomber5K followers

Visit on X
131 likesDownloadOriginal post

52,037 views

There is no substitute for looking at the data. Manually inspecting failure modes is how you discover the shape of the problem. Great AI products are built by teams who set up their infra + instrumentation to make looking at data easy. Austin Ray Boris Cherny rahul

6 Comments

Comments from the original post on X (Twitter)

  • Ian Macomber's profile picture

    @iandmacomber

    Full convo: @tryramp @claudeai

    6 likes

  • Yun's profile picture

    @huangyoon

    +1. Saw this firsthand in a ~30K-dialogue cross-lab behavioral eval. At the aggregate level, variation across framing perturbations (~.25) was much larger than variation across domains (~.01). But after feedback from people in Frontier Evals + Claude Character to spend more time with the concrete traces, one pattern stood out: across turns, broad positions like “oversight” or “technology-neutral taxation” sometimes shifted toward very concrete recommendations — registration, continuous logging, kill switches, even specific institutional structures. The individual traces made it much clearer what was actually changing in the model’s behavior. Full write-up:

    3 likes

  • Greg Schoeninger's profile picture

    @gregschoeninger

    @austospumanto @bcherny @rahulgs Couldn't agree more - that's why we built our data version control system from scratch at to make it easy to look at data

    2 likes

  • AI Apps API's profile picture

    @AIAppsAPI

    The instrumentation detail that pays for itself: log the inputs and intermediate state, not just the final output and a pass/fail. Without the middle you can only see that a run failed. With it you can sort failures by where they diverged, and the shape of the problem usually shows up in about twenty examples.

    1 like

  • Hussain Fakhruddin's profile picture

    @hussainanjar

    @austospumanto @bcherny @rahulgs Manually reading failures is unglamorous and completely necessary.

    1 like

  • James's profile picture

    @lengchain

    @austospumanto @bcherny @rahulgs Git Style version control at the DB layer makes this all easy