Can LLMs track state changes over long texts—like an object changing hands or a variable…
@allen_ai88K followers
20,302 views
18 Comments
Comments from the original post on X (Twitter)

@allen_ai
StateBench separates state tracking into 4 levels: r-trivial, aperiodic, soluble, & general finite. These range from problems like remembering something fixed, such as a birth date, to harder ones like tracking a ball as cups are repeatedly swapped.
11 likes

@allen_ai
We tested open pretrained models + frontier models incl. Gemini 3.8 Flash, Claude Opus 5.5, & GPT-6 Sol + Luna. Frontier models did well on simpler problems. But as tasks got harder & inputs longer, accuracy dropped sharply—to 13% or lower on the longest general finite problems.
22 likes

@allen_ai
To test whether failures came from architecture or pretraining, we trained LLMs directly on StateBench, then evaled them on longer inputs. Hybrids + Gated DeltaNets were near-perfect on the first 3 levels; transformers degraded with length & couldn’t solve general finite.
10 likes

@allen_ai
On StateBench’s hardest level, a flexible Gated DeltaNet handled sequences ~10× longer than those seen in training. Our new proof shows that even Gated DeltaNet with non-negative eigenvalues can solve some periodic tasks under fixed-precision arithmetic with rounding.
6 likes

@allen_ai
We’re releasing StateBench so researchers can test new models, extend our work, & build better state-tracking data. Explore more in our blog: Download the dataset:
9 likes

@ahatamiz1
Linking a synthetic state-tracking benchmark to real-world code and file tracking is misleading. Doing well on a toy task doesn't mean a model will be better at tracking real codebases. For perspective, fixed-depth transformers are in TC-0, so under standard complexity assumptions they can't solve hard state-tracking problems like permutation composition in a single forward pass.
8 likes

@KimYx0207
给AI测长对话,我会加这种题:A报价8000元,B复制A当时的报价,后来A改成6000元。此时B还是8000,不能跟着A一起变。 再把变更链拉长、夹进无关聊天,对照提前写好的答案。能找出某句话,和能跟住一连串修改,真是两码事。 原帖的13%只指这里最长、最难那档测试,不是所有长文任务的正确率。可以用StateBench公开的样本生成器,或自己整理几组业务变更记录,别光看上下文能塞多少字。
1 like

@Algorithon
what about 6.*1* Sol or Astra? 🤔
1 like

@SelfHostedSI
Long agent runs break exactly here, with the model sounding sure the whole way. Good to finally see it measured
1 like

@ajs6888
状态一多就容易把旧信息当成最新值了,长文本里尤其明显。
1 like

@bebraSOL
the shell game graphic is perfect for what this is testing

@letsunifyai
Is Opus 5.5's climb back to 53% on the longest soluble inputs something about those samples, like their variable count or sensitivity, or noise? Its general finite score keeps falling, to 3%.

@agneau123
The model still drops the state

@headlessthrones
Window size and tracked state are different numbers. Opus 5.5 documents a 1M-token context, and GPT-6.1 Sol lists 1.05M with a 922K max input. Results like 13% on the longest problems show how far usable recall sits below those limits.

@raulnogales
This is the failure mode users feel every day without having a name for it. The model sounds fluent right up to the moment it quietly loses track of who has what, and then confidence stays high anyway.

@QwenAIFans
13% on the hard StateBench cases is wild — long-horizon state tracking still breaks frontier models hard

@orion_cole_2
13% on state tracking sounds right. Ask it to track a deal through a 6-tool Make scenario and watch it forget step 3. What's your test set size?

@aksbhaskar
each sequence ships with 4 wrong final states, but the frontier eval was generative only. did you score the same models multiple-choice too, and how does the gap break down across the four levels?