This harness situation makes ZERO sense 😭 GPT-5.6 Luna performs best through Codex. GPT-5.6 Sol performs WORST through Codex. Same model family. Completely opposite result. For Sol, OpenCode currently looks best to me—but the differences are extremely close.

Guys, how does this even happen? GPT-5.6 Luna produced its strongest result through the Codex harness. But when I tested GPT-5.6 Sol Medium, Codex produced the weakest result—and it was also significantly slower. Approximate runtimes: Pi: 14 minutes Claude Code: 14 minutes Codex: 26 minutes OpenCode: I unfortunately didn’t record it So Codex took almost twice as long and still delivered the weakest Sol generation. And this doesn’t appear to be a one-time accident. I ran a similar test around a month ago and saw basically the same pattern: Luna + Codex works extremely well. Sol + Codex does not. Looking more closely at the other Sol generations, the differences are much smaller. Pi created a more convincing fight because the characters were at least standing correctly on the ground. Claude Code built a slightly better arena and entrance, but parts of the jury were visibly bugged. OpenCode had a small and strangely positioned jury, yet from a distance, its arena looked the most polished to me—the surrounding lights and overall presentation were simply nicer. My current tendency is: OpenCode slightly ahead, with Claude Code and Pi extremely close behind. Codex is clearly last for Sol. But honestly, this benchmark may not be difficult enough to separate the top three properly. I’ll run a harder test next, because the differences here are almost too small. The main lesson remains the same: There is no best harness. There is only the best harness for each specific model.

Bro models are probabilistic, you need bigger samples to have a clear view on what is good or bad, those 1 shot tests doesnt prove anything, if you really want to test the models, make your own benchmark with a ton of test, light or heavy tasks depending on what you need to prove

You need to run the same tests multiple times. A single run in each harness is not proving anything here really. Because AI is non deterministic, you could have done all 4 of those tests in the same harness and one of them would have been better.

Same family, opposite results, that harness pairing really does matter.

Not using omp when comparing harnesses is criminal

Makes you realize the harness is part of the product, not just a wrapper

Are you fucking stupid or what are u Doing

honestly harness optimization feeling this arbitrary is wild. execution context routing for GPT models is turning into its own science at this point

If you just ran the prompt once… try doing it 9 more times at least. LLMs are not deterministic.

I might have to give open code a try no. Thanks! One for Hermes?

in my anecdotal experience, tasks in Codex seem to take significantly longer than they should. been wondering if it's the model or the harness.