ARC-AGI-3 tests how well models can learn unfamiliar 2D games without instructions. The standard harness discarded GPT-5.6 Sol’s reasoning after each move and dropped earlier actions as the context filled up. The model had to keep starting over.

GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games? We investigated. The harness was not letting it remember what it had learned. We found that enabling two API settings tripled our scores with 6x fewer output tokens.

We implemented the harness with the Responses API and turned on: → Retained reasoning → Context compaction On the public set, GPT-5.6 Sol’s score rose 188% while using 6x fewer output tokens.

A benchmark score reflects the model as well as the harness and settings used to run it. For long-running agents, retaining reasoning and compacting context lets the model build on what it has already learned.

We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. If you’re an API developer trying to maximize performance, we recommend using the same settings that we deploy in our own products: - Use our Responses API, not our legacy Chat - Completions API - Retain reasoning - Use compaction If you want to test your own mettle against frontier models, try the public games yourself at

I dunno, it kinda looks like that was the point of the benchmark?

are all models on arc-agi-3 constrained by this ‘bad’ harness or only openai models? have you tried to reevaluate gpt-5.5?

We need these improvements in codex. 5.6 sol been behaving strangely past 3-4 days. it keeps looping with reviewer agent. finding its own bugs then patches up one bug and gives back to reviewer. after 2 rounds it must run full audit. then same thing happens again. burnt 65% today

This is exactly the kind of transparency the field needs. Simply turning on retained reasoning + context compaction nearly tripled GPT-5.6 Sol’s ARC-AGI-3 score while cutting tokens 6×. Clear proof that many agent benchmarks are still testing the harness more than the model itself. Great work.

No wonder it struggled. Expecting a model to learn without a memory is impossible.

Agreed — and the claims this hits hardest are the negative ones. Every "the model can't do X" is also a statement about the harness it ran in. That spec almost never ships with the claim, so the failure and the setup stay indistinguishable.

not dropping reasoning is fine, but compaction is against the spirit of the benchmark. i would like to see GPT's performance without compaction, only with sliding window-style message dropping.

persistent reasoning helps, but compact state summaries matter more than replaying the full trajectory

If the harness erases earlier actions, it is testing amnesia as much as reasoning. Long-running agents need a memory contract.

😉😉😉 @sama @OpenAI

ARC-AGI-3 测试模型在没有指导的情况下学习陌生 2D 游戏的表现。 标准测试框架在每次移动后丢弃了 GPT-5.6 Sol 的推理，并随着上下文填满而丢弃了早期的行动。模型不得不不断从头开始。

This is a useful benchmark lesson: context handling can look like model intelligence. Same weights, radically different score, because the harness stopped forcing the model to forget its own work.