Some food for thought when designing benchmarks... So, here's a little computer-use (visual) comparison between GPT-5.6 Astra and Qwen3.8 Max. The task here was to recreate the image in the center using the Paint UI. Super interesting how the two different LLMs+Harnesses approached this totally differently by default. I.e.,... Astra tried to approach this by drawing and layering geometric shapes. Qwen approached this pixel by pixel. (Of course, the pixel-by-pixel result looks closer to the original, it's essentially a low-res version of that by nature.) So, the Qwen-generated image would surely score higher in the sense that it's closer to the original. But I wouldn’t conclude from this example that one LLM generalizes better than the other on other tasks. Also, I wouldn't say Qwen has better compute-use capabilities or better visual understanding than Astra. But it highlights an interesting point about how slippery benchmarks are when they only compare final results.show more

one paints like an artist, the other like a printer, non wonder the printer scores higher on being a printer

Maybe test 3.8max on Codex? Let's see if the model variance is still that obvious under the same harness.

Yeah unfortunately the Codex/ChatGPT app doesn't support third party LLMs. And the Codex CLI doesn't support computer use (i.e., UI use).

I see.. and is this a typo? Did you mean GPT-6 Astra rather than GPT-5.6 Astra?

Oh yeah that’s a typo lol

I really love the insight you reached from the example (the part about generalization). The geometric approach clearly shows parsimony, which is more “beautiful” than the pixel-by-pixel approach.

Qwen is better than I thought

this measures the harness way more than the model. swap the click layer and your ranking flips.

Yes, that's probably true

Astra wins this for sure because it is abstractly trying to represent the reference

Similar thing happens on our slides creation benchmark. Two slides can be very similar in looks but very different in further editability. And designing judges that take those params in account rather than just slide images is very challenging our design of judges:

If that’s for real, that’s a huge knowledge gap.

The scoring rule is doing the work here. A pixel similarity metric rewards the pixel by pixel strategy by construction, so it measures the harness default more than the model. Would reporting action count and wall clock next to the score separate those two?

wall clock was actually quite similar, both took around ~20 min

@MLaurusevicius If wall clock is tied at ~20 min, what metric actually separates them in production?

@MLaurusevicius the trajectory

the geometric-shapes approach is closer to what you'd want for real GUI work though. actually using layers, fills and selection tools generalizes to a different image, a raster pixel dump only proves it can reproduce this exact one.

Qwen behave as you’d expect from a robot and astra more like a human

perfect example. the benchmark says "qwen wins" because pixel-level copy scores higher. but astra's approach – decomposing the image into geometric primitives – is arguably stronger generalization. the score rewards reproduction, not understanding. two different strategies, one leaderboard.

Holy smokes 🔥😳 Qwen plays on a different level

one paint task mostly tells you about the harness. astra grinding vs qwen finishing is the interesting bit, not who won

the 'LLMs+Harnesses' callout is the real thing here if those differ, you're mostly benchmarking infrastructure choices, not models. my own prompt-format experiments moved accuracy more than model swaps did

Benchmarks that reward the final pixels can hide a lot about the path taken to get there.

Deeper issue: a task with one correct image is the wrong thing to benchmark generalization on. A crisp answer key rewards copying by design. Most real agent work has no target to diff against, so the thing you can score cleanly is, by that cleanliness, unlike production.

pixel-by-pixel is basically converting a visual reasoning task into mechanical transcription. wins the similarity score while completely sidestepping the capability the benchmark thinks it's testing. textbook Goodhart.

The shapes-versus-pixels detail is fascinating. I’d completely miss that difference if I only saw the finished pictures.

What is the prompt? I am curious what would happen if we ask Astra to do it pixel by pixel?

Output-only checks fool production pipelines too. I had generated images pass a file-type check and a dimensions check while one held only about a fifth of the picture, because the download was cut off. Only a check that walked the file's structure caught it.

a follow-up edit could be revealing here. 'move the window to the other side' asks whether the approach left anything reusable, and gives you a different comparison from the initial screenshot

visual computer-use benchmarks always expose how models handle feedback loops in real time. code is easy to verify, but UI actions show the real gap in spatial reasoning and agentic planning.

the benchmark should score recovery from UI mistakes and end-state reliability too, not just visual similarity

one drew the rocket with shapes, the other brought a pixel microscope

As per the use. Which one do you think is a better choice to use. In terms of quality and in terms of pricing.

A single screenshot hides the real failure mode. Pixel score tells you where it landed. The interaction trace tells you whether it had a plan.

What as the prompt?

Fairs

This is the same blind spot that shows up when AI decisions get evaluated by outcome alone. A result that happens to land close to the target isn't proof the process behind it was sound. If you can't see how a system got there, the score is the only thing you're checking.

the difference is being realistic and trying to be realistic, astra could have drawn the same thing , is it trying to be human? hence not drawing it near perfect?

Benchmarks often reveal more about the prompt engineering or tool-use strategy than the core model itself.

the difference in this case is mostly the aesthetic of the process. drawing large shapes mimick human painting so it is more beautiful. compare it to chess game, the AI could win either way but winning by beautiful moves are often more desirable by humans.