Skip navigation
Benchmarks only work if models haven’t seen the test.
0:00 / 1:04

Benchmarks only work if models haven’t seen the test.

micro1

@micro1_ai23K followers

Visit on X
220 likesDownloadOriginal post

38,674 views

Benchmarks only work if models haven’t seen the test. When a benchmark question leaks into training data, a model can score well by remembering the answer. The scores go up, yet the model doesn’t actually improve. flow qc, the quality control model behind micro1’s data platform, now checks every training task against major public benchmarks and every other task we’ve built before it ships. Potential leaks and duplicates are held for review, while our data is kept original and benchmark results remain meaningful. More info in the comments.

16 Comments

Comments from the original post on X (Twitter)

  • micro1's profile picture

    @micro1_ai

    More info:

    9 likes

  • Arian's profile picture

    @ariansadeg

    Video quality these days..

    4 likes

  • Arsy's profile picture

    @HeyArsy

    Thanks for the breakdown 🤝

    2 likes

  • Daniel's profile picture

    @danielwarnur

    flow 🌊

    1 like

  • CorpusMarket's profile picture

    @Corpus_Market

    Which is a quiet argument for private data. Support tickets, email threads and project files kept inside a company have never been on the open web, so no model has memorized them. That makes them useful for testing, not only training.

    1 like

  • bielzin's profile picture

    @bielzinn

    great explanation, really easy to understand

    1 like

  • Rumi Elias Calles's profile picture

    @RumiCalles

    so hyped watching!

    1 like

  • Sydney's profile picture

    @sydneyysimone

    flowww>>

    1 like

  • comrade 🏴‍☠️'s profile picture

    @comradeweb3

    Wonderful breakdown.

  • XODUS🥷's profile picture

    @EcoKing_

    This is good

  • Chinwe Julius's profile picture

    @Julius_Chinwe

    @sol_promise 🔥

  • Kombite's profile picture

    @Kombite_

    @sol_promise Very insightful summary ❤️

  • Sahil Jindal's profile picture

    @sahiljindal22

    flow 🌊

  • Yash Hande's profile picture

    @yashrhande

    🕵️

  • Marius du Preez's profile picture

    @mdp_sec

    yep, why my labs stay private

  • Sunrise 🌅's profile picture

    @Man_3hunting

    @sol_promise I am here