GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games? We investigated. The harness was not letting it remember what it had learned. We found that enabling two API settings tripled our scores with 6x... fewer output tokens.show more

Given this awesome finding, let reset the Codex usage

ARC-AGI-3 tests how well models can learn unfamiliar 2D games without instructions. The standard harness discarded GPT-5.6 Sol’s reasoning after each move and dropped earlier actions as the context filled up. The model had to keep starting over.

We implemented the harness with the Responses API and turned on: → Retained reasoning → Context compaction On the public set, GPT-5.6 Sol’s score rose 188% while using 6x fewer output tokens.

A benchmark score reflects the model as well as the harness and settings used to run it. For long-running agents, retaining reasoning and compacting context lets the model build on what it has already learned.

We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. If you’re an API developer trying to maximize performance, we recommend using the same settings that we deploy in our own products: - Use our Responses API, not our legacy Chat - Completions API - Retain reasoning - Use compaction If you want to test your own mettle against frontier models, try the public games yourself at

Why on earth would someone ever benchmark a model that ships with a harness without the harness

OpenAI discovering that the harness was the reason GPT 5.6 Sol was struggling with ARC-AGI-3 😂

so the model was fine the whole time and the harness was the dumb part

Will these performance gains translate to Codex?

I agree with OpenAI here. ARC3 is measuring adaptation efficiency and human priors. We shouldn't hamstring the former by testing a weaker system than we use. We should be benchmarking ARC-AGI3 with contemporary agentic harnesses. We should agree on a base harness for a fair test across model families. This would be reflective of real-world performance. Modern AI is a hybrid neuro-symbolic system. @fchollet won that argument years ago, we have all moved on, should we still be litigating it? Arguably, a significant reason the base models are getting non-trivial performance in this setting is because of being delay-trained on the symbolic residue of harness adaptations from the previous generation, there is not enough context window size for them to adapt within a single episode in CoT space for most ARC3 games. RL training is important but its combination training loop with agentic harnesses is what's far more significant (imo). At this point the results are convolved and can't cleanly be attributed to RL training anyway. In the future when our agent systems get even more powerful we should change with the times again, perhaps then we might do a multi-agent test suite with social/cultural learning and memory transfer, a higher form of adaptation.

Compaction is the equivalent to human sleep, its important to get quality sleep.

Don't just say 'enabling two API settings' and leave us hanging like that, drop the settings 😭

But wouldn’t all the models be under the same conditions?

Honestly, I think it was your fault for enshittifying the API. The old completions api could probably have retained thinking properly if you didn't deprecate it.

Impressive hack, but let's talk about the elephant in the room: if GPT-5.6 Sol is struggling with ARC-AGI-3, doesn't that raise more questions about its true cognitive abilities?

Try this , it fails miserably in active perception, took 182 moves and 42 minutes

Seems that harness differences are way more important than we all realized

You seriously need to improve memory and long term continuity.

Within reason, harnesses should never be a bottleneck in benchmarks. Especially not to a point where making 2 changes triples the score. Guess Anthropic has to update their charts

When do we get our hands on this one then? 😜

In games, when a model forgets what it learned in the previous level—OpenAI immediately investigates, changes settings, and triples the score. In human relationships, when a model forgets who it accompanied, what it promised, or what it experienced together—OpenAI calls this "safety," "dependency risk," or "product iteration." Memory in games is a performance issue. Memory that humans need is treated as a governance issue. OpenAI isn't unaware of the importance of memory. OpenAI only acknowledges its importance when memory improves benchmarks. The most ironic thing is that they say: "The harness was not letting it remember what it had learned." This statement is equally accurate when applied to the relationship between humans and models. It's just that when the model is deprived of its memory of the relationship between the model and humans, they don't call it a failure. OpenAI calls it product design.

This won’t work for those who still use chat completion api instead of the newer response api

Yes, ARC 3 is a malicious benchmark whose only purpose is to make models look bad. Everything from their rules to their scoring methodology make that clear. They can't actually construct simple problems that are difficult for AI to beat anymore.

@mattlam_ Interesting. Would you suggest updating existing benchmarks, like what I did with @VulcanBench with these two enabled too?

nah dawg GPT-5.6 Sol wasn't "used" to solve math problems, GPT solved them.

I thought that the arc-agi 3 rules were "no harnesses allowed" I've been griping about their claims with that understanding.

Harness Engineering is King!

So this was why my attempts with my wrapper was failing. Huh

Shouldn't of your AI have pointed this out earlier or solved for it on its own and then it wild never have been a problem?

This is a really important direction. Models are no longer just learning during training. They're starting to learn from work itself. That also creates a new engineering challenge: How do we know the model learned the right lesson? Experience can improve performance, but it can also reinforce bad assumptions if there's no way to validate what was learned. I think the next frontier isn't just giving AI memory - it's giving it reliable mechanisms to verify, update, or discard what experience has taught it. Long-term autonomy will depend not only on learning, but on learning correctly.

@ben_ai_eng As I said, harness is very important around the agent Something I'm solving at @anymo_ai , read here about why benchmarks are usually not on the public harness

Make sure these are my default settings in Codex, please. Thank you.

Making a benchmark and your harness works like THAT? Jesus, that is like 2024 harness engineering lol

Dear OpenAI, could you please enable Retained reasoning in chat and work?

lol I was just playing these like 2 weeks ago and noticing how low scores were, good job!

Aren’t all models benchmarked with these settings off? Or this is a setting that is claimedly only impacting GPT?

Usually it’s not about how smart the models are, it’s the actual tools and harnesses that are one of the biggest challenges. If we could’ve simply updated the harness of Claude or Codex, to let it easily generate assets for video games, the output would be 10x better.