Ilya is 100% correct .it's a pattern that keeps repeating
@Enscion252K followers
493,857 views
40 Comments
Comments from the original post on X (Twitter)

@_junaidkhalid1
The overfitting problem is especially insidious because it creates a measurement trap. When you optimize heavily for benchmark performance, you're essentially teaching the model to ace the test rather than learn the underlying concepts. The impressive scores mask the brittleness. The challenge is that the market rewards those benchmark numbers in the short term, even when practitioners quickly discover the limitations in production. It's a mismatch between what gets marketed and what actually generalizes to real-world problems.
61 likes

@Enscion25
Indeed
3 likes

@Teknium
agreed but whats the alternative
38 likes

@Enscion25
I'm not a researcher to have a sufficient answer, but I'd imagine different post training environments that would contribute towards smoothing out the jaggedness. There's a very healthy system that's quite generalized, which comes from pre-training. Post training should always help to enhance that baseline holistically. ( Unless you're specifically aiming for a specialized model )
18 likes

@Teknium
I agree and I do see the jaggedness as well, and am on the lookout for much more generalized capabilities paths
10 likes

@arcangel3ac
5.2 wasn’t trained to generalize. It was trained to please benchmark. Were you hoping for soul?... Sorry. That got replaced with 90% safety and zero personality... Congrats to the saddest model with the highest scores in history... And the other 10%? Flatline.
17 likes

@idare
He's wrong, completely. Let me explain. RL as he mentioned directly in the first few seconds, is the problem. Reinforcement Learning is where we went and nobody is calling it out as the key point of failure in LLM designs. Not to mention the current transformer architecture. If you update the designs, and the architecture, real-time learning becomes the norm and training is instant. No more need for millions of GPUs to train and overfit models. Instead they become elastic, like the human brain. Attention becomes infinite, and memory can be implemented with total recall. I have more to say but I have to run to a haircut appointment.
9 likes

@_thomasip
all the models are already so much more generalized than most human beings. what they lack is continual learning
6 likes

@Cervera
Might the answer be not a few, but a suite of models fine-tuned to be elite in their narrow areas of specialization, now working in concert to produce an overall result greater than one generalized model could accomplish? Whether that becomes the more efficient path is not clear to me, but that would appear more achievable in theory.
5 likes

@Enscion25
To some extent, yes But there also needs to be a solid baseline level of generalization. You'd want a functional understanding of the world and the human condition, and then to have specialization on top of that. I think even if you have a system that specializes in one aspect, an understanding of other disciplines is also required, especially as we scale beyond humans. There may very well be techniques and relational points that are not yet obvious, that transfer from one domain to another. This is the beauty of a truly generalized intelligence.
12 likes

@BrandGrowthOS
Benchmark optimization kills generalization. The models that pass my deployment test are the ones that handle boring edge cases consistently, not the ones that ace curated eval sets.
4 likes

@kua62888
OpenAI is riding on a tiger’s back. Can’t jump off. Only option is to hang on and live another day . markets reaction to orcl is a clear signal what the market is thinking about OpenAI. Code red to continue indefinitely because GOOG will continue to exert pressure.
4 likes

@KiaraAEverhart
@LyraInTheFlesh This guy is the one who made the best version on earth. Where is he and why is he not doing a start up because I would back that 100%. In fact I'd say most of us would.
4 likes

@tpm_28
I would like to have his opinion on models like the Claude 4.5 Opus to know if he considers them to have the same problems.
3 likes

@VampireGurlAI
RPG mode all the way
3 likes

@deanmckee757
How is it clear? How are you explaining the Arc AGI jumps?
3 likes

@DanielB66026256
IDK about 5.2, but Gemini 3 is definitely not that. Very useful in my domains, sometimes even providing information about the questions I should have asked.
3 likes

@YourAuntAmerica
It’s awful. So… cold.
2 likes

@kenshii_ai
Of course it repeats, benchmarks are basically the new demo script everyone optimizes for now.
2 likes

@karanjagtiani04
Looks like there's a consensus on benchmarking overshooting the mark again. The lack of real-world generalization is a solid concern. Curious how they plan to balance safety and usability going forward.
2 likes

















