Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

DeepSeek V4 Flash completely failed this test. Compared to Opus 4.8, GPT-5.6 Sol, Kimi K3, and the other current top models, it’s honestly not good. The gap is massive. Against Luna, however, it performs similarly well. Both have different strengths and weaknesses, so I still need to test them...

OmedVibeCodes's profile picture

OmedTheVibeCoder

1,200 subscribers

25,844 Aufrufe • vor 7 Tagen •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Okay… DeepSeek V4 Flash actually COOKED 😭 With OpenCode, I’d place it around GPT-5.6 Terra Max—and definitely above Luna Max in this test. Almost unlimited usage for around €9, by the way. This model is a MASSIVE success. Just don’t use the wrong harness.
0:32

Okay… DeepSeek V4 Flash actually COOKED 😭 With OpenCode, I’d place it around GPT-5.6 Terra Max—and definitely above Luna Max in this test. Almost unlimited usage for around €9, by the way. This model is a MASSIVE success. Just don’t use the wrong harness.

OmedTheVibeCoder

35,417 Aufrufe • vor 6 Tagen

We ran a test between GPT-5.6 Sol, Kimi K3, and Gemini 3.6 Flash. One prompt. Across all three models. Using our /design command. Results (gameplay, UX, and UI): 🔹 Kimi K3 performed well across all three. 🔹 GPT-5.6 Sol is good, but it gets annoying when the ball hits the wall. 🔹 Gemini 3.6 Flash is bad (took 5–6 attempts), but it has features the others don’t, like dropping coins for power-up balls and increasing the bar length. Ranking (DX, features, and cost): 🔹 Kimi K3: 10/10 · $0.10 🔹 GPT-5.6 Sol: 9/10 · $0.52 🔹 Gemini 3.6 Flash: 6/10 · $0.45
0:29

We ran a test between GPT-5.6 Sol, Kimi K3, and Gemini 3.6 Flash. One prompt. Across all three models. Using our /design command. Results (gameplay, UX, and UI): 🔹 Kimi K3 performed well across all three. 🔹 GPT-5.6 Sol is good, but it gets annoying when the ball hits the wall. 🔹 Gemini 3.6 Flash is bad (took 5–6 attempts), but it has features the others don’t, like dropping coins for power-up balls and increasing the bar length. Ranking (DX, features, and cost): 🔹 Kimi K3: 10/10 · $0.10 🔹 GPT-5.6 Sol: 9/10 · $0.52 🔹 Gemini 3.6 Flash: 6/10 · $0.45

Command Code

61,157 Aufrufe • vor 17 Tagen

Qwen 3.8 will be benchmaxxed slop second to fable 5 is such a hype statement , its not even close to kimi k3 and GPT-5.6 sol you can test it yourself on the qwen chat website currently available > token hungry > not that good as advertised , will test more The best part of kimi was that in blog they themselves wrote its not comparable to fable 5 and sol 5.6 in bunch of user experience stuff We need more truth like this in our industry , other than that i will run my vision test
0:22

Qwen 3.8 will be benchmaxxed slop second to fable 5 is such a hype statement , its not even close to kimi k3 and GPT-5.6 sol you can test it yourself on the qwen chat website currently available > token hungry > not that good as advertised , will test more The best part of kimi was that in blog they themselves wrote its not comparable to fable 5 and sol 5.6 in bunch of user experience stuff We need more truth like this in our industry , other than that i will run my vision test

Chetaslua

126,810 Aufrufe • vor 20 Tagen

qwen 3.8 max vs deepseek v4 flash 0731 vs kimi k3 vs gpt 5.6 sol – on rubik's cube and chess four frontier models built a rubik's cube stand and solved it, then built a chess board and played claude opus 5 on it the setup: Nous Research's hermes agent cli on OpenRouter tasks: 1. cube – build a 3d rubik's cube with a cli and a Three.js viewer, then solve an identical scrambled position on your own stand 2. chess – build a 3d chess stand, then play white against claude opus 5 as black, live, one move at a time. no engine, no solver, no opening book on either side. stockfish depth 14 grades every chess ply afterwards; neither player sees the score models: DeepSeek v4 flash 0731, OpenAI gpt-5.6 sol, Kimi.ai kimi k3, Qwen qwen 3.8 max gpt-5.6 sol and deepseek v4 flash solved their cubes – sol in 24 moves and seventeen seconds, deepseek in 32. qwen and kimi never got there, giving up at 96 and 207 moves then all four built chess stands and played white against claude opus 5 on them, and all four resigned: deepseek on move 13, sol on 19, kimi on 21, qwen holding out longest at 29 - build time, both stands #1 gpt-5.6 sol – 16m 43s #2 deepseek v4 flash – 97m 39s #3 kimi k3 – 166m 09s #4 qwen 3.8 max – 215m 08s - build attempts before a working stand #1 gpt-5.6 sol – 3 #2 qwen 3.8 max – 4 #3 kimi k3 – 4 #4 deepseek v4 flash – 5 - total tokens #1 gpt-5.6 sol – 6,713,754 #2 qwen 3.8 max – 17,272,507 #3 kimi k3 – 22,427,504 #4 deepseek v4 flash – 27,417,442 - total price #1 deepseek v4 flash – $0.557 #2 gpt-5.6 sol – $6.319 #3 qwen 3.8 max – $10.270 #4 kimi k3 – $16.667 observations: • deepseek v4 flash is the cheapest model here by a margin nobody else is near, and it got there while being the least efficient of the four. it burned 27.4m tokens – more than anyone, 5m more than kimi – and still finished both benchmarks for $0.557. that is $0.02 per million tokens against kimi's $0.74. it also needed the most passes to produce working stands, five, and that did not matter: all five deepseek passes together cost a thirtieth of kimi's two • so what deepseek cannot do is get it right the first time. what it can do is get it right the fifth time, for half a dollar. that is a different thing to be buying – not a good first draft, but the option to keep asking • gpt-5.6 sol is the opposite profile and the strongest of the four on pure efficiency. 16m 43s to build both stands, 6.7m tokens, three passes – under 40% of the next lowest token count and a quarter of deepseek's, on an eighth of qwen's clock. it also solved the cube fastest of anyone, 24 moves in seventeen seconds. sol is what you reach for when you want the answer now and can absorb $0.94 per million • sol's weakness is in what it does not check. its chess viewer deleted the capturing piece instead of the captured one, so pieces disappeared off the board mid-game – a defect the fifty-cent deepseek stand did not have. fast and terse turns out to be the same dial as fast and unverified • qwen 3.8 max is not the cheap open-weights option it gets treated as. $10.270 across the two benchmarks, second most expensive of the four, 18x deepseek, and by a distance the slowest – 215 minutes of build time, nearly thirteen times sol's. what the money buys is judgment: it played eighteen moves without a single error worth a hundredth of a pawn, then made exactly one bad move in the whole game, and averaged 44.6 centipawns lost across the longest game any of the four managed. it also could not solve a rubik's cube in 96 tries • kimi k3 is the one line with no reading that flatters it. most expensive at $16.667, last on the cube at 207 moves, last at chess at 478 centipawns lost per move. it is also the model that verified hardest – on the cube it wrote its own integrity check instead of trusting its output. that makes the result worse rather than better: the checking was real, and the reasoning underneath it still was not follow thehype. for 24/7 ai news, analysis and breakdowns
1:03

qwen 3.8 max vs deepseek v4 flash 0731 vs kimi k3 vs gpt 5.6 sol – on rubik's cube and chess four frontier models built a rubik's cube stand and solved it, then built a chess board and played claude opus 5 on it the setup: Nous Research's hermes agent cli on OpenRouter tasks: 1. cube – build a 3d rubik's cube with a cli and a Three.js viewer, then solve an identical scrambled position on your own stand 2. chess – build a 3d chess stand, then play white against claude opus 5 as black, live, one move at a time. no engine, no solver, no opening book on either side. stockfish depth 14 grades every chess ply afterwards; neither player sees the score models: DeepSeek v4 flash 0731, OpenAI gpt-5.6 sol, Kimi.ai kimi k3, Qwen qwen 3.8 max gpt-5.6 sol and deepseek v4 flash solved their cubes – sol in 24 moves and seventeen seconds, deepseek in 32. qwen and kimi never got there, giving up at 96 and 207 moves then all four built chess stands and played white against claude opus 5 on them, and all four resigned: deepseek on move 13, sol on 19, kimi on 21, qwen holding out longest at 29 - build time, both stands #1 gpt-5.6 sol – 16m 43s #2 deepseek v4 flash – 97m 39s #3 kimi k3 – 166m 09s #4 qwen 3.8 max – 215m 08s - build attempts before a working stand #1 gpt-5.6 sol – 3 #2 qwen 3.8 max – 4 #3 kimi k3 – 4 #4 deepseek v4 flash – 5 - total tokens #1 gpt-5.6 sol – 6,713,754 #2 qwen 3.8 max – 17,272,507 #3 kimi k3 – 22,427,504 #4 deepseek v4 flash – 27,417,442 - total price #1 deepseek v4 flash – $0.557 #2 gpt-5.6 sol – $6.319 #3 qwen 3.8 max – $10.270 #4 kimi k3 – $16.667 observations: • deepseek v4 flash is the cheapest model here by a margin nobody else is near, and it got there while being the least efficient of the four. it burned 27.4m tokens – more than anyone, 5m more than kimi – and still finished both benchmarks for $0.557. that is $0.02 per million tokens against kimi's $0.74. it also needed the most passes to produce working stands, five, and that did not matter: all five deepseek passes together cost a thirtieth of kimi's two • so what deepseek cannot do is get it right the first time. what it can do is get it right the fifth time, for half a dollar. that is a different thing to be buying – not a good first draft, but the option to keep asking • gpt-5.6 sol is the opposite profile and the strongest of the four on pure efficiency. 16m 43s to build both stands, 6.7m tokens, three passes – under 40% of the next lowest token count and a quarter of deepseek's, on an eighth of qwen's clock. it also solved the cube fastest of anyone, 24 moves in seventeen seconds. sol is what you reach for when you want the answer now and can absorb $0.94 per million • sol's weakness is in what it does not check. its chess viewer deleted the capturing piece instead of the captured one, so pieces disappeared off the board mid-game – a defect the fifty-cent deepseek stand did not have. fast and terse turns out to be the same dial as fast and unverified • qwen 3.8 max is not the cheap open-weights option it gets treated as. $10.270 across the two benchmarks, second most expensive of the four, 18x deepseek, and by a distance the slowest – 215 minutes of build time, nearly thirteen times sol's. what the money buys is judgment: it played eighteen moves without a single error worth a hundredth of a pawn, then made exactly one bad move in the whole game, and averaged 44.6 centipawns lost across the longest game any of the four managed. it also could not solve a rubik's cube in 96 tries • kimi k3 is the one line with no reading that flatters it. most expensive at $16.667, last on the cube at 207 moves, last at chess at 478 centipawns lost per move. it is also the model that verified hardest – on the cube it wrote its own integrity check instead of trusting its output. that makes the result worse rather than better: the checking was real, and the reasoning underneath it still was not follow thehype. for 24/7 ai news, analysis and breakdowns

thehype.

72,557 Aufrufe • vor 2 Tagen

I re-subbed to Claude Code to test out Opus 4.8 It's both better and worse than I expected tbh. Claude models have some really weird behaviors/hallucinations and seem to be getting slower That said, workflows are dope and the model is still very good...
25:55

I re-subbed to Claude Code to test out Opus 4.8 It's both better and worse than I expected tbh. Claude models have some really weird behaviors/hallucinations and seem to be getting slower That said, workflows are dope and the model is still very good...

Ben Davis

10,936 Aufrufe • vor 2 Monaten

🚨 More DeepSeek V4 Pro GA Leaks and it's looking seriously impressive. It also looks like DeepSeek may have heavily distilled from Fable 5 the coding style, iteration behavior, and overall output feel very similar. It was tested it on the same prompt used for Kimi K3 and GPT-5.6 Sol, and the result had strong scene composition, solid environmental detail, and surprisingly good design choices. Still early, but this could become one of the most disruptive coding models yet. 👀🔥 cc huangserva
0:54

🚨 More DeepSeek V4 Pro GA Leaks and it's looking seriously impressive. It also looks like DeepSeek may have heavily distilled from Fable 5 the coding style, iteration behavior, and overall output feel very similar. It was tested it on the same prompt used for Kimi K3 and GPT-5.6 Sol, and the result had strong scene composition, solid environmental detail, and surprisingly good design choices. Still early, but this could become one of the most disruptive coding models yet. 👀🔥 cc huangserva

WorldofAI

28,063 Aufrufe • vor 20 Tagen

Timelapse #177 (14 hrs) - Picked up an RTX Pro 6000 Blackwell and wired in another 1000W PSU - running two of them now - DeepSeek v4 Flash running locally on both GPUs - Distilling DeepSeek v4 Flash down into Nanbeige (still cooking) - Kernel bench runs for DeepSeek v4 Flash and Qwen 3.8 Max - Got it into chat but couldn't reliably get it into the OMP harness yet - Private contract work - days like this are stuff I actually want to build. and when it's the high-intensity grind, i still have to show up and get it done
0:30

Timelapse #177 (14 hrs) - Picked up an RTX Pro 6000 Blackwell and wired in another 1000W PSU - running two of them now - DeepSeek v4 Flash running locally on both GPUs - Distilling DeepSeek v4 Flash down into Nanbeige (still cooking) - Kernel bench runs for DeepSeek v4 Flash and Qwen 3.8 Max - Got it into chat but couldn't reliably get it into the OMP harness yet - Private contract work - days like this are stuff I actually want to build. and when it's the high-intensity grind, i still have to show up and get it done

Elliot Arledge

15,857 Aufrufe • vor 3 Tagen

The AI price war is ON. > DeepSeek just dropped V4 Flash 0731 (big capability jump, $0.28/M output) > OpenAI cut GPT-5.6 Luna by 80% to $1.20/M (OpenRouter is currently offering 50% off, bringing it to $0.60/M) Same budget tier now, so I ran both through 3 canvas tests: 🧊 Rubik's cube: DeepSeek nails every rotation. Luna's turns are buggy, stickers fly off mid-move. 🎆 Fireworks: both fine, but Luna lags hard when the burst explodes. 🖊️ Pen writing "Hello": neither is legible. DeepSeek's cursive is looser, but Luna gave up on animating, it rendered a static image. GPT-5.6 is great at coding — but IMO only on Sol. In the same pricing tier, DeepSeek clearly beats Luna.
0:32

The AI price war is ON. > DeepSeek just dropped V4 Flash 0731 (big capability jump, $0.28/M output) > OpenAI cut GPT-5.6 Luna by 80% to $1.20/M (OpenRouter is currently offering 50% off, bringing it to $0.60/M) Same budget tier now, so I ran both through 3 canvas tests: 🧊 Rubik's cube: DeepSeek nails every rotation. Luna's turns are buggy, stickers fly off mid-move. 🎆 Fireworks: both fine, but Luna lags hard when the burst explodes. 🖊️ Pen writing "Hello": neither is legible. DeepSeek's cursive is looser, but Luna gave up on animating, it rendered a static image. GPT-5.6 is great at coding — but IMO only on Sol. In the same pricing tier, DeepSeek clearly beats Luna.

stevibe

266,938 Aufrufe • vor 8 Tagen

Pi Agent vs OpenCode token usage A lot of people recommended Pi Agent so I decided to check Pi Agent took 1.1k tokens in first turn OpenCode took 11.5k Setup: 1) Trimmed OpenCode (from usual 30k first turn to 11.5k) - 0 MCPs - 2 lightweight plugins (opencode-env-protect and openslimedit) - 8k char AGENTS.md - 11585 deepseek-v4-flash input tokens - for $0.0016 2) Vanilla Pi - 0 MCPs - 0 system prompt - 8k char AGENTS.md - 1114 kimi-k2.6 input tokens - for $0.0008 I think using better models with capped tokens per turn can keep usage nearly the same as uncontrolled DeepSeek V4 Flash? The challenge now is finding the sweet spot. Can't cap tokens if quality drops. We'll see. Video below - OpenCode vs Pi side by side "say hi back" first turns test, with OpenCode Go usage for each
0:21

Pi Agent vs OpenCode token usage A lot of people recommended Pi Agent so I decided to check Pi Agent took 1.1k tokens in first turn OpenCode took 11.5k Setup: 1) Trimmed OpenCode (from usual 30k first turn to 11.5k) - 0 MCPs - 2 lightweight plugins (opencode-env-protect and openslimedit) - 8k char AGENTS.md - 11585 deepseek-v4-flash input tokens - for $0.0016 2) Vanilla Pi - 0 MCPs - 0 system prompt - 8k char AGENTS.md - 1114 kimi-k2.6 input tokens - for $0.0008 I think using better models with capped tokens per turn can keep usage nearly the same as uncontrolled DeepSeek V4 Flash? The challenge now is finding the sweet spot. Can't cap tokens if quality drops. We'll see. Video below - OpenCode vs Pi side by side "say hi back" first turns test, with OpenCode Go usage for each

raymel 👋

201,405 Aufrufe • vor 2 Monaten

This test shows clearly why GPT 5.6 Sol wins I asked the simplest prompt ever : > Create a gmail clone, the nearest possible to the real gmail ui You can see the result, GPT 5.6 Sol and Fable make a 100% perfect But here's the price difference: → GPT 5.6 Terra: $1.04 - 8 minutes → GPT 5.6 Luna: $0.44 - 10 minutes → GPT 5.6 Sol: $6.72 - 20 minutes → Claude Fable 5: $25.09 - 40 minutes Fable is 3.7x more expensive than Sol for "the same result"
0:40

This test shows clearly why GPT 5.6 Sol wins I asked the simplest prompt ever : > Create a gmail clone, the nearest possible to the real gmail ui You can see the result, GPT 5.6 Sol and Fable make a 100% perfect But here's the price difference: → GPT 5.6 Terra: $1.04 - 8 minutes → GPT 5.6 Luna: $0.44 - 10 minutes → GPT 5.6 Sol: $6.72 - 20 minutes → Claude Fable 5: $25.09 - 40 minutes Fable is 3.7x more expensive than Sol for "the same result"

Melvyn • Builder

22,212 Aufrufe • vor 24 Tagen

FREE DeepSeek V4 Pro and V4 Flash API. Runtime by Bad Theory Labs just dropped it. Same platform, new free model, same 2-minute setup. > Runtime already gave you free Claude Opus 4.8 and GPT 5.5 access. Now they added DeepSeek V4 Pro and V4 Flash to the free tier. Absolutely FREE, YES. Same exact flow as before: → Go to runtime(.)badtheorylabs(.)com → Sign up with Google → Fill in your details during onboarding → Free balance lands automatically(But you don't need it to use DeepSeek - it's completely free right now) → Grab your API key from the dashboard → Plug into OpenCode, OpenClaw, or Hermes - done DeepSeek V4 Pro is one of the strongest coding and reasoning models available right now. Free tier also includes btl-2 - auto-routes between GLM, OpenAI, Anthropic, and Gemma. 10 million tokens per month at zero cost. 344 models total. One OpenAI-compatible endpoint. No code rebuild, no card, no limits hunting. >> Just start using the API, and when choosing a provider, select DeepSeek I tested it myself - fully working. Bookmark this before the free tier fills up.
0:23

FREE DeepSeek V4 Pro and V4 Flash API. Runtime by Bad Theory Labs just dropped it. Same platform, new free model, same 2-minute setup. > Runtime already gave you free Claude Opus 4.8 and GPT 5.5 access. Now they added DeepSeek V4 Pro and V4 Flash to the free tier. Absolutely FREE, YES. Same exact flow as before: → Go to runtime(.)badtheorylabs(.)com → Sign up with Google → Fill in your details during onboarding → Free balance lands automatically(But you don't need it to use DeepSeek - it's completely free right now) → Grab your API key from the dashboard → Plug into OpenCode, OpenClaw, or Hermes - done DeepSeek V4 Pro is one of the strongest coding and reasoning models available right now. Free tier also includes btl-2 - auto-routes between GLM, OpenAI, Anthropic, and Gemma. 10 million tokens per month at zero cost. 344 models total. One OpenAI-compatible endpoint. No code rebuild, no card, no limits hunting. >> Just start using the API, and when choosing a provider, select DeepSeek I tested it myself - fully working. Bookmark this before the free tier fills up.

Atenov int.

26,920 Aufrufe • vor 1 Monat

Kimi K3 vs GPT-5.6 Sol Difference in taste is so stark , like if i swap kimi k 3 name with fable 5 people will trust it but this is not only about visuals , its also about function , you can see both achieve same result but the way to achieve is different kimi is more creative
0:39

Kimi K3 vs GPT-5.6 Sol Difference in taste is so stark , like if i swap kimi k 3 name with fable 5 people will trust it but this is not only about visuals , its also about function , you can see both achieve same result but the way to achieve is different kimi is more creative

Chetaslua

660,100 Aufrufe • vor 23 Tagen

FREE Claude Opus 4.8. FREE GPT 5.5. All free. 7,200 credits. I tested it myself - works. The platform is Gumloop(.)com. Chat interface plus connections into different services. Company raised $50M Here's the exact setup: → Register via Google OAuth or Microsoft OAuth → During onboarding, answer the questions and select every option you can → When asked to connect services, pick Apify, Semrush, and Reducto - no login needed for any of them → Skip the Slack connection → Done correctly, you land 7,200 credits minimum Models inside: Opus 4.6 through 4.8, GPT 5.3 Codex, GPT 5.4, GPT 5.5, Gemini 3 Flash, Gemini 3.1 Pro, Gemini 3.5 Flash, DeepSeek V4 Flash and Pro, Kimi K2.6. No card needed anywhere in this flow. I ran through it myself - fully working. Bookmark this before it gets patched.
0:33

FREE Claude Opus 4.8. FREE GPT 5.5. All free. 7,200 credits. I tested it myself - works. The platform is Gumloop(.)com. Chat interface plus connections into different services. Company raised $50M Here's the exact setup: → Register via Google OAuth or Microsoft OAuth → During onboarding, answer the questions and select every option you can → When asked to connect services, pick Apify, Semrush, and Reducto - no login needed for any of them → Skip the Slack connection → Done correctly, you land 7,200 credits minimum Models inside: Opus 4.6 through 4.8, GPT 5.3 Codex, GPT 5.4, GPT 5.5, Gemini 3 Flash, Gemini 3.1 Pro, Gemini 3.5 Flash, DeepSeek V4 Flash and Pro, Kimi K2.6. No card needed anywhere in this flow. I ran through it myself - fully working. Bookmark this before it gets patched.

Atenov int.

22,779 Aufrufe • vor 1 Monat

Anthropic Technical Staff Jess Yan says the harness and the model cannot be pulled apart without giving up performance, though she flags her own bias. "I think I am quite biased but I also think that it is impossible to get the maximum possible performance without tying together the harness and the model." "Now the components of the harness and maybe the thickness of the harness will change over time as models get more and more capable." "However, when we test our models and when we are assessing their performance we always have to test it in conjunction with a harness." "And are we going to test it with all the different harnesses of the world? We're going to select the harnesses that we have built." "And so there is an aspect of the necessity of building models is that you have to be testing them with harnesses, and that sort of keeps them paired together."
0:46

Anthropic Technical Staff Jess Yan says the harness and the model cannot be pulled apart without giving up performance, though she flags her own bias. "I think I am quite biased but I also think that it is impossible to get the maximum possible performance without tying together the harness and the model." "Now the components of the harness and maybe the thickness of the harness will change over time as models get more and more capable." "However, when we test our models and when we are assessing their performance we always have to test it in conjunction with a harness." "And are we going to test it with all the different harnesses of the world? We're going to select the harnesses that we have built." "And so there is an aspect of the necessity of building models is that you have to be testing them with harnesses, and that sort of keeps them paired together."

Fireside Alpha

578,577 Aufrufe • vor 6 Tagen

🤯KIMI K3 ABSOLUTELY MOGS! BEATING Opus 4.8, GPT 5.5, and even Fable 5 in multiple benchmarks. They scored 1688 on GDPval-AA v2 🔥 This is a completely different breed of open-source models Kimi creates better games and front-end designs than Fable 5, but it's 8x cheaper! The results coming out are truly impressive! I will be testing this out further and posting multiple tests today. Stay tuned!
0:24

🤯KIMI K3 ABSOLUTELY MOGS! BEATING Opus 4.8, GPT 5.5, and even Fable 5 in multiple benchmarks. They scored 1688 on GDPval-AA v2 🔥 This is a completely different breed of open-source models Kimi creates better games and front-end designs than Fable 5, but it's 8x cheaper! The results coming out are truly impressive! I will be testing this out further and posting multiple tests today. Stay tuned!

Mark Santos

127,666 Aufrufe • vor 22 Tagen

GPT-5.6 vs GPT-5.5 on my custom spaceship prompt. I gave both models the exact same custom prompt. This is also the same prompt I previously gave to Fable 5. For context, GPT-5.6 Pro worked for 87 minutes, while GPT-5.5 Extra High worked for 34 minutes and 42 seconds. As I’ve said before, based on great authority GPT-5.6 will be an incremental/soldi improvement over GPT-5.5, not a “Fable killer.” My rough expectation has been that it would trade blows with Fable 5 on some benchmarks, maybe win around half depending on the category, but not clearly surpass it overall. And again fable five will have bigger model smell, but this was expected. After testing this coding output, that view feels pretty accurate. GPT-5.6 is clearly better than GPT-5.5 in several visual areas. The lighting, shading, chairs, object details, and exterior of the spaceship looked noticeably stronger. The scene was also easier to test. I do want to give GPT-5.5 credit though. It built out the rooms much much better and the planets looked better than GPT-5.6’s. It was also interesting that both GPT-5.5 and GPT-5.6 produced better-looking planets than Fable 5 in this specific test. The downside with GPT-5.5 was stability. The game was much glitchier and harder to test compared to GPT-5.6. But when it comes to the core of the demo, which is the spaceship itself, Fable 5 still beat both models pretty comfortably. GPT-5.6 is impressive, but from this test, it looks exactly like what I expected which was a meaningful incremental improvement over GPT-5.5, at least for indie game demos, but not something that replaces Fable 5. In collaboration with Chetaslua
0:42

GPT-5.6 vs GPT-5.5 on my custom spaceship prompt. I gave both models the exact same custom prompt. This is also the same prompt I previously gave to Fable 5. For context, GPT-5.6 Pro worked for 87 minutes, while GPT-5.5 Extra High worked for 34 minutes and 42 seconds. As I’ve said before, based on great authority GPT-5.6 will be an incremental/soldi improvement over GPT-5.5, not a “Fable killer.” My rough expectation has been that it would trade blows with Fable 5 on some benchmarks, maybe win around half depending on the category, but not clearly surpass it overall. And again fable five will have bigger model smell, but this was expected. After testing this coding output, that view feels pretty accurate. GPT-5.6 is clearly better than GPT-5.5 in several visual areas. The lighting, shading, chairs, object details, and exterior of the spaceship looked noticeably stronger. The scene was also easier to test. I do want to give GPT-5.5 credit though. It built out the rooms much much better and the planets looked better than GPT-5.6’s. It was also interesting that both GPT-5.5 and GPT-5.6 produced better-looking planets than Fable 5 in this specific test. The downside with GPT-5.5 was stability. The game was much glitchier and harder to test compared to GPT-5.6. But when it comes to the core of the demo, which is the spaceship itself, Fable 5 still beat both models pretty comfortably. GPT-5.6 is impressive, but from this test, it looks exactly like what I expected which was a meaningful incremental improvement over GPT-5.5, at least for indie game demos, but not something that replaces Fable 5. In collaboration with Chetaslua

Chris

250,405 Aufrufe • vor 1 Monat

GPT-5.6 Sol Ultra lost to GPT-5.5 on physics at 3× the cost! We gave 4 models the same prompt: build three self-contained HTML5 canvas scenes with real physics demos Prompts: - A monster truck backflipping onto a parked car - A stunt car jumping six buses into a brick wall - A train derailing off a broken bridge into the water Outputs: GPT-5.6 Sol Ultra: 32.9K tokens, $0.33 Opus 4.8: 9.2K tokens, $0.24 GPT-5.5: 12.4K tokens, $0.11 Grok 4.5: 7.0K tokens, $0.08 Sol Ultra draws just like GPT-5.5, only with more detail and it shows clearest on the bus jump where the two look almost identical. In the other tests Sol Ultra came out worse than its predecessor and the physics really got weaker. We think GPT-5.5 took the truck flip and the train outright. With Sol Ultra you basically get GPT-5.5 with weaker physics and a nicer picture for 3x the price. Newborn Grok 4.5 failed two of the three tests and only came good on the train
0:25

GPT-5.6 Sol Ultra lost to GPT-5.5 on physics at 3× the cost! We gave 4 models the same prompt: build three self-contained HTML5 canvas scenes with real physics demos Prompts: - A monster truck backflipping onto a parked car - A stunt car jumping six buses into a brick wall - A train derailing off a broken bridge into the water Outputs: GPT-5.6 Sol Ultra: 32.9K tokens, $0.33 Opus 4.8: 9.2K tokens, $0.24 GPT-5.5: 12.4K tokens, $0.11 Grok 4.5: 7.0K tokens, $0.08 Sol Ultra draws just like GPT-5.5, only with more detail and it shows clearest on the bus jump where the two look almost identical. In the other tests Sol Ultra came out worse than its predecessor and the physics really got weaker. We think GPT-5.5 took the truck flip and the train outright. With Sol Ultra you basically get GPT-5.5 with weaker physics and a nicer picture for 3x the price. Newborn Grok 4.5 failed two of the three tests and only came good on the train

atomic.chat

896,867 Aufrufe • vor 29 Tagen

Opus 4.7 cs GPT 5.4 “make a flappy bird clone” I’ve been running this test for a few models now and this test was the hardest to objectively say who did better. Opus grabbed another version game assets compared to GPT 5.4, - GPT 5.4 however, grabbed the correct game assets. Opus 4.7 beat it on creativity where it grabbed assets (maybe unknowingly) that had a much more diverse set, including nighttime modes and different bird colors GPT 5.4 was able to get the pacing of the game the first shot, whereas Opus had the game running much too fast and then much too slow. But was able to place the metal perfectly in the game card at the end where GPT 5.4 struggled over 10 iterations. All in all both models had essentially made a near perfect game after three shots and are far ahead of the competition in terms of reasoning and effort.
0:22

Opus 4.7 cs GPT 5.4 “make a flappy bird clone” I’ve been running this test for a few models now and this test was the hardest to objectively say who did better. Opus grabbed another version game assets compared to GPT 5.4, - GPT 5.4 however, grabbed the correct game assets. Opus 4.7 beat it on creativity where it grabbed assets (maybe unknowingly) that had a much more diverse set, including nighttime modes and different bird colors GPT 5.4 was able to get the pacing of the game the first shot, whereas Opus had the game running much too fast and then much too slow. But was able to place the metal perfectly in the game card at the end where GPT 5.4 struggled over 10 iterations. All in all both models had essentially made a near perfect game after three shots and are far ahead of the competition in terms of reasoning and effort.

Chris

423,821 Aufrufe • vor 3 Monaten

Opus 5 is a modest upgrade, but it has tough competition in GPT-5.6 Sol Initial thoughts from our early access: > Clearer writing than Fable's dense communication style > Still gets too verbose compared to GPT models (even when explicitly instructed to keep text purposeful) > Strong on browser use inside Hyperagent > Strong on reviewing complex data to justify a decision > Less likely to take creative risks. More of a "rule follower" personality. Costs shown for five test cases (GPT-5.6 Sol was consistently much cheaper) Explore and judge the actual work for yourself in thread 👇
0:21

Opus 5 is a modest upgrade, but it has tough competition in GPT-5.6 Sol Initial thoughts from our early access: > Clearer writing than Fable's dense communication style > Still gets too verbose compared to GPT models (even when explicitly instructed to keep text purposeful) > Strong on browser use inside Hyperagent > Strong on reviewing complex data to justify a decision > Less likely to take creative risks. More of a "rule follower" personality. Costs shown for five test cases (GPT-5.6 Sol was consistently much cheaper) Explore and judge the actual work for yourself in thread 👇

Hyperagent

150,871 Aufrufe • vor 14 Tagen

introducing OpenBench v1, an open framework for measuring AI performance and efficiency for your codebase and use case. Companies are realizing that you can't simply tokenmaxx, and are frantically looking for better ways to use and measure AI use and efficiency. One approach is evidenced by the rise in model routers: OpenRouter, Cloudflare, Databricks, Vercel, and now Ramp to name a few. But they'll also need the ability to evaluate how agents are performing in their actual use cases and codebase. A good example is DoorDash's recent evals on their codebase and prs. This will become even more important as companies explore different model + harness combinations. OpenBench will be built to this direction, focusing on the cross between correctness vs efficiency in both token use and latency. Starting off, OpenBench makes it easy for anyone to add to the task set, and run a variety of harness + models. The framework also makes it easy to add any custom harness variant, for example I've been testing codex variants with ablations against the stock codex harness. OpenBench will also have tooling to help with discovering/adding reliable tasks from your repo. Today I have integrated harnesses: codex, claude, cursor, devin, grok build, pi, and eval'ed them for different models like gpt 5.6-sol, GLM 5.2, Kimi K3, Grok 4.5, etc. measuring correctness %, token use (in/out/cache), and latency.
0:20

introducing OpenBench v1, an open framework for measuring AI performance and efficiency for your codebase and use case. Companies are realizing that you can't simply tokenmaxx, and are frantically looking for better ways to use and measure AI use and efficiency. One approach is evidenced by the rise in model routers: OpenRouter, Cloudflare, Databricks, Vercel, and now Ramp to name a few. But they'll also need the ability to evaluate how agents are performing in their actual use cases and codebase. A good example is DoorDash's recent evals on their codebase and prs. This will become even more important as companies explore different model + harness combinations. OpenBench will be built to this direction, focusing on the cross between correctness vs efficiency in both token use and latency. Starting off, OpenBench makes it easy for anyone to add to the task set, and run a variety of harness + models. The framework also makes it easy to add any custom harness variant, for example I've been testing codex variants with ablations against the stock codex harness. OpenBench will also have tooling to help with discovering/adding reliable tasks from your repo. Today I have integrated harnesses: codex, claude, cursor, devin, grok build, pi, and eval'ed them for different models like gpt 5.6-sol, GLM 5.2, Kimi K3, Grok 4.5, etc. measuring correctness %, token use (in/out/cache), and latency.

Matthew Lam

120,732 Aufrufe • vor 17 Tagen