Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Battle of the new coding models: Cursor vs Cognition They both make big claims being near the frontier - how well do they pass the Golden Gate Bridge test? I've included 5x non-cherry picked generations from each, followed by examples from pre-release test of Gemini 3 Pro, GPT-5 and...

23,583 Aufrufe • vor 8 Monaten •via X (Twitter)

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Opus 4.7 - 400k vs 1m context - is there a difference? I've heard Theo - t3.gg talk about the fact that it is unlikely that Anthropic would have offered up a model with 1m context at the same cost, if it wasn't a different (i.e. cheaper to serve) model. I did a test where I toggled the 1m default model on & off in Claude Code (otherwise default settings, xHigh reasoning) and compared the outputs with 3x generations - same prompts etc. My observations: - Models feel DIFFERENT - often when you ask a model for the same generation, you get a somewhat different answer, but it feels & smells the same. Here 400k and 1m are very different every time - 400k model seems better - not that 1m is trash and 400k is amazing, but there are definitely issues with the level of ambition and accuracy that 1m model seems to have Examples of 1m failing: - Voxel Rome: the colosseum is nowhere near as impressive - Golden Gate: cars go sideways, waves not very high, bridge goes into land; though the structure of the bridge is a bit better - Stonehenge: structure is more 'wrong', lighting, shadows & textures are more flat and not as rich This isn't a conclusive evidence of course, but at least to me the two models do not behave the same way. Anecdotally as well when building 1m felt like it was doing more weird validation (e.g. going around in circles) and 400k was more straightforward. These sorts of things are harder to capture in tests, but you'd notice in Claude Code. You can review the hosted generations, see the code & prompts in the links below

Peter Gostev (SF: 22-26 June)

29,203 Aufrufe • vor 2 Monaten

GPT-5.6 vs GPT-5.5 on my custom spaceship prompt. I gave both models the exact same custom prompt. This is also the same prompt I previously gave to Fable 5. For context, GPT-5.6 Pro worked for 87 minutes, while GPT-5.5 Extra High worked for 34 minutes and 42 seconds. As I’ve said before, based on great authority GPT-5.6 will be an incremental/soldi improvement over GPT-5.5, not a “Fable killer.” My rough expectation has been that it would trade blows with Fable 5 on some benchmarks, maybe win around half depending on the category, but not clearly surpass it overall. And again fable five will have bigger model smell, but this was expected. After testing this coding output, that view feels pretty accurate. GPT-5.6 is clearly better than GPT-5.5 in several visual areas. The lighting, shading, chairs, object details, and exterior of the spaceship looked noticeably stronger. The scene was also easier to test. I do want to give GPT-5.5 credit though. It built out the rooms much much better and the planets looked better than GPT-5.6’s. It was also interesting that both GPT-5.5 and GPT-5.6 produced better-looking planets than Fable 5 in this specific test. The downside with GPT-5.5 was stability. The game was much glitchier and harder to test compared to GPT-5.6. But when it comes to the core of the demo, which is the spaceship itself, Fable 5 still beat both models pretty comfortably. GPT-5.6 is impressive, but from this test, it looks exactly like what I expected which was a meaningful incremental improvement over GPT-5.5, at least for indie game demos, but not something that replaces Fable 5. In collaboration with Chetaslua

Chris

228,126 Aufrufe • vor 26 Tagen

How good is GPT-4-Vision at extracting text from images? I wanted to find the limit - but I found weirdness instead Most surprising: GPT-4V performance varies depending on the *structure* of text it sees Let me explain A set of images with progressively more text was presented to GPT-4-Vision. GPT-4V was asked what text it saw in the image. The response from the model was compared against the image’s original text and scored for similarity. The model was tested with 4 types of text: essay, random words, random tokens, and random characters. Findings: * Performance degrades - Yes, the models are good at basic OCR, but as you get more text and words then performance drops (this is expected) * Type of context matters - You should expect different recall on your texts based on your context types * Hallucination Errors - I thought that the model would make errors of omission (it wouldn’t return all the words). But instead the model mostly made hallucination errors - it replaced words with made up words. * Evals Matter - This test in isolation doesn’t mean that your data will have the same results, but it should motivate you to create eval tests for your data and anticipate errors which are hard to spot Notes: * Next step would be to add additional image types like tables or PDFs * GPT-4V would routinely get stuck in repeat-token-loops when trying to extract random tokens * GPT-4V would refuse to answer most random character images

Greg Kamradt

49,111 Aufrufe • vor 2 Jahren