Skip navigation
Can LLMs track state changes over long texts—like an object changing hands or a variable…
0:00 / 1:35

Can LLMs track state changes over long texts—like an object changing hands or a variable…

Ai2

@allen_ai88K followers

Visit on X
251 likesDownloadOriginal post

20,302 views

Can LLMs track state changes over long texts—like an object changing hands or a variable reassignment? On our new StateBench eval, frontier models including GPT-6 Sol & Claude Opus 5.5 fell to 13% accuracy or lower on the hardest, longest problems. 🧵

18 Comments

Comments from the original post on X (Twitter)

  • Ai2's profile picture

    @allen_ai

    StateBench separates state tracking into 4 levels: r-trivial, aperiodic, soluble, & general finite. These range from problems like remembering something fixed, such as a birth date, to harder ones like tracking a ball as cups are repeatedly swapped.

    11 likes

  • Ai2's profile picture

    @allen_ai

    We tested open pretrained models + frontier models incl. Gemini 3.8 Flash, Claude Opus 5.5, & GPT-6 Sol + Luna. Frontier models did well on simpler problems. But as tasks got harder & inputs longer, accuracy dropped sharply—to 13% or lower on the longest general finite problems.

    22 likes

  • Ai2's profile picture

    @allen_ai

    To test whether failures came from architecture or pretraining, we trained LLMs directly on StateBench, then evaled them on longer inputs. Hybrids + Gated DeltaNets were near-perfect on the first 3 levels; transformers degraded with length & couldn’t solve general finite.

    10 likes

  • Ai2's profile picture

    @allen_ai

    On StateBench’s hardest level, a flexible Gated DeltaNet handled sequences ~10× longer than those seen in training. Our new proof shows that even Gated DeltaNet with non-negative eigenvalues can solve some periodic tasks under fixed-precision arithmetic with rounding.

    6 likes

  • Ai2's profile picture

    @allen_ai

    We’re releasing StateBench so researchers can test new models, extend our work, & build better state-tracking data. Explore more in our blog: Download the dataset:

    9 likes

  • Ali Hatamizadeh's profile picture

    @ahatamiz1

    Linking a synthetic state-tracking benchmark to real-world code and file tracking is misleading. Doing well on a toy task doesn't mean a model will be better at tracking real codebases. For perspective, fixed-depth transformers are in TC-0, so under standard complexity assumptions they can't solve hard state-tracking problems like permutation composition in a single forward pass.

    8 likes

  • 老金带你玩AI's profile picture

    @KimYx0207

    给AI测长对话,我会加这种题:A报价8000元,B复制A当时的报价,后来A改成6000元。此时B还是8000,不能跟着A一起变。 再把变更链拉长、夹进无关聊天,对照提前写好的答案。能找出某句话,和能跟住一连串修改,真是两码事。 原帖的13%只指这里最长、最难那档测试,不是所有长文任务的正确率。可以用StateBench公开的样本生成器,或自己整理几组业务变更记录,别光看上下文能塞多少字。

    1 like

  • algorithon's profile picture

    @Algorithon

    what about 6.*1* Sol or Astra? 🤔

    1 like

  • Steven Kramp | SelfHostedSI's profile picture

    @SelfHostedSI

    Long agent runs break exactly here, with the model sounding sure the whole way. Good to finally see it measured

    1 like

  • 安叫兽|Bird🕊️ 🔶 BNB's profile picture

    @ajs6888

    状态一多就容易把旧信息当成最新值了,长文本里尤其明显。

    1 like

  • gkrovso's profile picture

    @bebraSOL

    the shell game graphic is perfect for what this is testing

  • unify's profile picture

    @letsunifyai

    Is Opus 5.5's climb back to 53% on the longest soluble inputs something about those samples, like their variable count or sensitivity, or noise? Its general finite score keeps falling, to 3%.

  • Tommy's profile picture

    @agneau123

    The model still drops the state

  • Headless Thrones's profile picture

    @headlessthrones

    Window size and tracked state are different numbers. Opus 5.5 documents a 1M-token context, and GPT-6.1 Sol lists 1.05M with a 922K max input. Results like 13% on the longest problems show how far usable recall sits below those limits.

  • Raúl Nogales 🇪🇸🇺🇸's profile picture

    @raulnogales

    This is the failure mode users feel every day without having a name for it. The model sounds fluent right up to the moment it quietly loses track of who has what, and then confidence stays high anyway.

  • Qwen AI Fans's profile picture

    @QwenAIFans

    13% on the hard StateBench cases is wild — long-horizon state tracking still breaks frontier models hard

  • Orion Cole's profile picture

    @orion_cole_2

    13% on state tracking sounds right. Ask it to track a deal through a 6-tool Make scenario and watch it forget step 3. What's your test set size?

  • Akshat Bhaskar's profile picture

    @aksbhaskar

    each sequence ships with 4 wrong final states, but the frontier eval was generative only. did you score the same models multiple-choice too, and how does the gap break down across the four levels?