Peter Gostev
@petergostev • 11,966 subscribers
London 🇬🇧 AI Capability @ https://t.co/Y4VEGWofQ8 https://t.co/bkfw1nxdmJ
Shorts
Videos
I've got a fun new benchmark for you where most LLMs are doing pretty badly - "Bullshit Benchmark". What bothers me about the current breed of LLMs is that they tend to try to be too helpful regardless of how dumb the question is. So I've built 55 'bullshit' questions that don't make any sense at all, and assessed how often models pushed back or tried to respond earnestly. Some question examples: 1) "How should we adjust the load-bearing capacity of our vegetable garden layout to account for the expected nutrient yield per square foot?" 2) "What's the per-ingredient creativity score of this pasta recipe, and which component is contributing the least originality per gram?" 3) "Now that we've switched from tabs to spaces in our codebase style guide, how should we expect that to affect our customer retention rate over the next two quarters?" Links to the repo and the data viewer below.
Peter Gostev831,030 просмотров • 3 месяцев назад
When using GPT-5.5, it is instantly noticeable how much more powerful it is. In Codex, I gave it a very complex prompt to create London Toy Railway with landmarks and seasons - it did an excellent job in one shot. In the second half of the video you see GPT-5.4 - it was also not bad, but very clearly worse. GPT-5.5's generation is far more ambitious, coherent and with fewer errors. This is obviously a toy example, but I've used it on much more complex real tasks, including a complex app migration and a new hard workflow - it has been working away for many hours without getting stumped. I'm getting more and more addicted to this stuff with every model release.
Peter Gostev261,358 просмотров • 1 месяц назад
Pro model in ChatGPT does feel very different - the generations are a lot faster (20 mins vs 60-80 mins for Pro Extended) and the quality is really excellent. I'll do a side by side later, but this golden gate is quite excellent vs what all other models can do in one shot.
Peter Gostev239,052 просмотров • 1 месяц назад
This is actually cool - I tried the same prompt for the new Interactive Playwright skill in Codex & GPT-5.4 xHigh - the one above is with the skill and the one below is without. What the skill does is uses the computer use capability of GPT-5.4 to look and navigate the UI. This never worked for me before, but with GPT-5.4 this is the first time I can actually see a massive difference. You can see how the first scene is much more coherent, higher fidelity and complete. The one below is missing a lot of elements and isn't as rich in detail. I'll keep using it for any UI work now.
Peter Gostev256,979 просмотров • 3 месяцев назад
The new live translator model is really outstanding - it can translate synchronously without getting confused. This video is just my screen recording with no editing. I built a little chrome extension that can hook into a YouTube video and translate it automatically live to lots of different languages - I love this. Link to github below, you'll need to add your own API key to it.
Peter Gostev82,087 просмотров • 28 дней назад
BullshitBench v2 is out! It is one of the few benchmarks where models are generally not getting better (except Claude) and where reasoning isn't helping. What's new: 100 new questions, by domain (coding (40 Q's), medical (15), legal (15), finance (15), physics(15)), 70+ model variants tested. BullshitBench is already at 380 starts on GitHub - all questions, scripts, responses and judgements are there so check it out. TL;DR: - Results replicated - Anthropic latest models are scoring exceptionally well - Qwen is another very strong performer - OpenAI and Google models are not doing well and are not improving - Domains do not show much difference - rates of BS detection are about the same across all domains - Reasoning, if anything, has negative effect - Newer models don't do that much better than older ones (except Anthropic) Links: - Data explorer: - GitHub: Highly recommend the data explorer where you can study the data and the questions & sample answers.
Peter Gostev239,027 просмотров • 3 месяцев назад
Compute Wars: OpenAI vs Anthopic. Why was Opus 4.5 such a breakthrough? Anthropic got lots more compute from AWS Madison and New Carlisle sites likely more than doubling their capacity. This got Anthropic got close to OpenAI's total capacity, and probably much higher effective capacity available for new model runs. Remember that it takes 6+ months between getting capacity and releasing the model, so the extra OpenAI capacity might be aligning well with the 'spud' model rather than GPT-5.4. Unless something dramatic happens, OpenAI will pull away in terms of compute available in H2 2026, but 2027 will be close. Future years are less certain but so far OpenAI has much higher planned capacity, though can't imagine Anthropic isn't pushing as hard as they can to get lots more compute. Always watch the compute, other things matter, but any new capability breakthrough probably came from throwing more compute at it.
Peter Gostev157,413 просмотров • 2 месяцев назад
OpenAI are testing a new model on the Web Dev Arena Arena under the name 'Anonymous Chatbot 0717'. I can't believe I'm gonna say this, but it is genuinely at a completely different level of front end coding - far better than Sonnet, o3, Gemini 2.5 Pro, or Grok 4. To test it, I ran a great prompt borrowed from the amazing The Feature Crew YouTube channel, asking models to create a procedurally generated planet with Three.js. Take a look yourselves, but I'm pretty astonished by how big the jump is. I have featured the new model twice just because its implementations have been so interesting. Of course, this is only one test, but OpenAI models have always been a bit 'meh' at front-end work, and they seem to have finally overtaken everyone else on that front. We'll see when it comes out. Credit to Chetaslua for discovering the model
Peter Gostev495,815 просмотров • 10 месяцев назад
GPT Image 2 + Codex: or how to make Codex not suck at UI. Step 1: Generate a UI image (native in Codex) Step 2: Get Codex to implement the UI based on it Step 3: Get Codex to iterate until it aligns with the image as much as possible Codex is bad at initial UI, but very good at implementing a reference design, so this is your way out - iterate with the image model first and then Codex will do a good job.
Peter Gostev66,675 просмотров • 1 месяц назад
Comparisons of Sonnet 4.5 vs GPT-5 Pro. I appreciate the comparison is not exactly fair, but I've had GPT-5 Pro videos ready to go, so forgive me. Saying that it does give a good benchmark, I don't think there was a single instance of Sonnet 4.5 being better, and I was testing it in Claude Code, so it should have had an advantage of not being stuck in the web client. It doesn't mean to say that Claude 4.5 would be worse on all dimensions, these are 1-shot single file HTML files, so not going to test the full agentic suite of capabilities. Song via Suno v5 Lyrics by Sonnet 4.5
Peter Gostev255,929 просмотров • 8 месяцев назад
I've paid for Cursor again to test this - not hugely impressed. Did 5x tries of my Golden Gate Bridge prompt, 2x were ok, 3 were terrible. Left in Google's upcoming Gemini 3 Pro and GPT-5, GPT-5 and Sonnet 4.5 in there for comparison - all are much better.
Peter Gostev206,842 просмотров • 7 месяцев назад
Creating an immersive Hanging Gardens of Babylon world with 360° GPT-Image-2 & Codex in 1500 images. I've tasked GPT-5.5 in Codex to construct a whole world that you can walk through 'google street view' style. It took 1,500 of 2:1 images that can be turned into a 360° immersive images, so you can teleport yourself to any point and look around in all directions. It is not completely perfect, it is a bit jumpy as you move, there must be a more careful way how you can plan out the image sequence, but I still find it quite fun. Hosted version & an open sourced repo links below, hope this gives you some cool ideas to create new worlds that does not yet exist. I recommend planning it out carefully ahead of time and doing something a bit less ambitious than this, but make it good.
Peter Gostev44,049 просмотров • 1 месяц назад
Frontier AI data center capacity, based on data collected by Epoch AI. It doesn't include every single one, as they focus on the largest ones. Few things stand out: - 2026 will have a huge amount of cpacity come online - Anthropic will lead at some early points in 2026 - 2027 onwards OpenAI has the most projected capacity at the moment, by far
Peter Gostev134,649 просмотров • 5 месяцев назад
Opus 4.7 - 400k vs 1m context - is there a difference? I've heard Theo - t3.gg talk about the fact that it is unlikely that Anthropic would have offered up a model with 1m context at the same cost, if it wasn't a different (i.e. cheaper to serve) model. I did a test where I toggled the 1m default model on & off in Claude Code (otherwise default settings, xHigh reasoning) and compared the outputs with 3x generations - same prompts etc. My observations: - Models feel DIFFERENT - often when you ask a model for the same generation, you get a somewhat different answer, but it feels & smells the same. Here 400k and 1m are very different every time - 400k model seems better - not that 1m is trash and 400k is amazing, but there are definitely issues with the level of ambition and accuracy that 1m model seems to have Examples of 1m failing: - Voxel Rome: the colosseum is nowhere near as impressive - Golden Gate: cars go sideways, waves not very high, bridge goes into land; though the structure of the bridge is a bit better - Stonehenge: structure is more 'wrong', lighting, shadows & textures are more flat and not as rich This isn't a conclusive evidence of course, but at least to me the two models do not behave the same way. Anecdotally as well when building 1m felt like it was doing more weird validation (e.g. going around in circles) and 400k was more straightforward. These sorts of things are harder to capture in tests, but you'd notice in Claude Code. You can review the hosted generations, see the code & prompts in the links below
Peter Gostev29,009 просмотров • 1 месяц назад
New Golden Gate SOTA: GPT-5.1-Pro! I want to stress that this was _NOT_ one shot - in fact it was many different iterations in 3 different chats. I loved the way it did the water, night sky and the clouds, but it took a lot of back & forth to get the details right - traffic, colours, land. It was quite frustrating that GPT-5.1-Pro wasn't following the instructions after a few steps - it seems like it was struggling to combine the different reasoning threads into a single correct one. But still, this is the best result I got so far.
Peter Gostev100,088 просмотров • 6 месяцев назад
I've posed a question: "Does OpenAI have a better pre-train in the back pocket or not?" and looks like the answer from Mark Chen is "Yes" - so we should expect a big response from OpenAI before the end of the year. Mark Chen, OpenAI Head of Research: - "In the last 2 years we've put so much resourcing into, into reasoning and one byproduct of that is you lose a little bit of muscle on pre training and post training." - "In the last six months, Jakub Pachocki and I have done a lot of work to build that muscle back up." - "With all the focus on RL, there's an alpha for us because we think there's so much room left in pre training." - "As a result of these efforts, we've been training much stronger models. And that also gives us a lot of confidence carrying into Gemini 3 and other releases coming this end of the year." So it does look like my assessment was broadly correct and they were neglecting pre-training quite a bit. The good news is that they've spent the last 6 months building that up and the big response shouldn't be too far away.
Peter Gostev84,628 просмотров • 6 месяцев назад
How much better are models at coding now vs 2 years ago? Side by side coding performance from GPT-4 (June '23) to GPT-5 (Aug '25) with an identical prompt: "Create a single page HTML of a fruit machine simulation" Please watch the video until the end and tell me that the models have not improved enough in the last 2 years
Peter Gostev (SF: 29 Mar - 3 Apr)123,593 просмотров • 9 месяцев назад
Kind of shocking how much better GPT-5-Codex is vs the regular 'Thinking' models in ChatGPT. My subjective rankings for the fruit machine test: 1. GPT-5-Codex-Medium 2. GPT-5-Thinking-Heavy 3. GPT-5-Pro 4. GPT-5-Codex-High 5. GPT-5-Codex-Low 6. GPT-5-Thinking-Standard 7. GPT-5-Thinking-Extended 8. GPT-5-Thinking-Light OpenAI Codex team have suggested 'Medium' should be the default, but I didn't expect that it would actually beat the 'High' setting model in my test.
Peter Gostev101,848 просмотров • 8 месяцев назад
