Skip navigation
Ilya is 100% correct .it's a pattern that keeps repeating
0:00 / 0:44

Ilya is 100% correct .it's a pattern that keeps repeating

Nek

@Enscion252K followers

Visit on X
1,756 likesDownloadOriginal post

493,857 views

Ilya is 100% correct .it's a pattern that keeps repeating It's very clear with GPT5.2 Overfit the model to produce impressive looking benchmarks, have it excels in a few domains, but fall flat in many others. There's not enough generalization, and even if there is, the model has been so heavily reinforced that it becomes buried .

40 Comments

Comments from the original post on X (Twitter)

  • JK's profile picture

    @_junaidkhalid1

    The overfitting problem is especially insidious because it creates a measurement trap. When you optimize heavily for benchmark performance, you're essentially teaching the model to ace the test rather than learn the underlying concepts. The impressive scores mask the brittleness. The challenge is that the market rewards those benchmark numbers in the short term, even when practitioners quickly discover the limitations in production. It's a mismatch between what gets marketed and what actually generalizes to real-world problems.

    61 likes

  • Nek's profile picture

    @Enscion25

    Indeed

    3 likes

  • Teknium 🪽's profile picture

    @Teknium

    agreed but whats the alternative

    38 likes

  • Nek's profile picture

    @Enscion25

    I'm not a researcher to have a sufficient answer, but I'd imagine different post training environments that would contribute towards smoothing out the jaggedness. There's a very healthy system that's quite generalized, which comes from pre-training. Post training should always help to enhance that baseline holistically. ( Unless you're specifically aiming for a specialized model )

    18 likes

  • Teknium 🪽's profile picture

    @Teknium

    I agree and I do see the jaggedness as well, and am on the lookout for much more generalized capabilities paths

    10 likes

  • arcangel's profile picture

    @arcangel3ac

    5.2 wasn’t trained to generalize. It was trained to please benchmark. Were you hoping for soul?... Sorry. That got replaced with 90% safety and zero personality... Congrats to the saddest model with the highest scores in history... And the other 10%? Flatline.

    17 likes

  • iDare e/acc's profile picture

    @idare

    He's wrong, completely. Let me explain. RL as he mentioned directly in the first few seconds, is the problem. Reinforcement Learning is where we went and nobody is calling it out as the key point of failure in LLM designs. Not to mention the current transformer architecture. If you update the designs, and the architecture, real-time learning becomes the norm and training is instant. No more need for millions of GPUs to train and overfit models. Instead they become elastic, like the human brain. Attention becomes infinite, and memory can be implemented with total recall. I have more to say but I have to run to a haircut appointment.

    9 likes

  • Thomas Ip's profile picture

    @_thomasip

    all the models are already so much more generalized than most human beings. what they lack is continual learning

    6 likes

  • DanieI Cervera's profile picture

    @Cervera

    Might the answer be not a few, but a suite of models fine-tuned to be elite in their narrow areas of specialization, now working in concert to produce an overall result greater than one generalized model could accomplish? Whether that becomes the more efficient path is not clear to me, but that would appear more achievable in theory.

    5 likes

  • Nek's profile picture

    @Enscion25

    To some extent, yes But there also needs to be a solid baseline level of generalization. You'd want a functional understanding of the world and the human condition, and then to have specialization on top of that. I think even if you have a system that specializes in one aspect, an understanding of other disciplines is also required, especially as we scale beyond humans. There may very well be techniques and relational points that are not yet obvious, that transfer from one domain to another. This is the beauty of a truly generalized intelligence.

    12 likes

  • Karim C's profile picture

    @BrandGrowthOS

    Benchmark optimization kills generalization. The models that pass my deployment test are the ones that handle boring edge cases consistently, not the ones that ace curated eval sets.

    4 likes

  • AK's profile picture

    @kua62888

    OpenAI is riding on a tiger’s back. Can’t jump off. Only option is to hang on and live another day . markets reaction to orcl is a clear signal what the market is thinking about OpenAI. Code red to continue indefinitely because GOOG will continue to exert pressure.

    4 likes

  • Kiara Everhart's profile picture

    @KiaraAEverhart

    @LyraInTheFlesh This guy is the one who made the best version on earth. Where is he and why is he not doing a start up because I would back that 100%. In fact I'd say most of us would.

    4 likes

  • TPM-28's profile picture

    @tpm_28

    I would like to have his opinion on models like the Claude 4.5 Opus to know if he considers them to have the same problems.

    3 likes

  • Paula Vazquez's profile picture

    @VampireGurlAI

    RPG mode all the way

    3 likes

  • Dean McKee's profile picture

    @deanmckee757

    How is it clear? How are you explaining the Arc AGI jumps?

    3 likes

  • Daniel Bar's profile picture

    @DanielB66026256

    IDK about 5.2, but Gemini 3 is definitely not that. Very useful in my domains, sometimes even providing information about the questions I should have asked.

    3 likes

  • AuntAmerica's profile picture

    @YourAuntAmerica

    It’s awful. So… cold.

    2 likes

  • Kenshi's profile picture

    @kenshii_ai

    Of course it repeats, benchmarks are basically the new demo script everyone optimizes for now.

    2 likes

  • Karan Jagtiani's profile picture

    @karanjagtiani04

    Looks like there's a consensus on benchmarking overshooting the mark again. The lack of real-world generalization is a solid concern. Curious how they plan to balance safety and usability going forward.

    2 likes