How much better are models at coding now vs 2 years ago? Side by side coding performance from GPT-4 (June '23) to GPT-5 (Aug '25) with an identical prompt: "Create a single page HTML of a fruit machine simulation" Please watch the video until the end and tell me... that the models have not improved enough in the last 2 yearsshow more

4.1 is an absolute beast

The amount of code it was actually outputting was kind of crazy, especially comparing it to previous non-reasoning models, it is so much better than older ones

i think thats because its the only model openai made specifically for "coding" rather than "software engineering", i.e = winning swe contests.

LLM models have not improved much since last year Contrived examples like the one you're demoing do not prove anything. Ask any recent "state of the art" model to do something that slightly deviates from the established patterns and it will just hallucinate.

I don’t see the progress as being all that significant. There has been some modest improvement, but even the newest models output a lot of garbage. Though, one would think otherwise to listen to the hype squad on social media.

in our games we even see some regression w claude

I don't even need to watch the video, as I have tested well over 100 LLMs in the past few years. I know the progress has been massive, especially in coding & math. Now imagine what the next 2 years can bring us. Probably the best AI agents can in 2 years work an entire 8 h day without any human interaction needed, doing PRs, ironing out bugs, deploying, optimizing the code for efficiency, hunting down software vulnerabilities and fixing them automatically.

haha o3-pro is like gambling on crack

Gpt 5 pro is missing

Good shout, how could I forget!

Noice 😋

@voids_thoughts made me think of our conversation earlier

very cool, thanks for making this

One of the worst prompts I've ever read

frontend design did clearly not evolve at all;)

i have the answer. if it could code well you would put it in an infinite loop of refinement but instead you save backups and retreat to the last one before the abacus dinked you

Progress is evident, but still a long way to go. Would be interesting to see side-by-side iterations over multiple prompts.

Sometime you have to zoom out to see progress!

Great great video

So what’s the progress here?

Amazing, thanks for the work! Such a clear depiction of progress. It would be cool to test "Oneshot + 5 iterations" to see how the model capability to iterate over previous prompts improved over the years. ie say "improve it" 5 times and see the final result.

@MirraTerminal

The models have not improved enough in the last 2 years

I tried chatgtp yesterday for a rather simple bash script. It completely failed and I had to do it by myself which took me half an hour :(

I'm pro AI-for-coding and was actually expecting a huge progress but I literally can't see much progress in the video