🌀 Introducing 𝐄𝐪𝐮𝐢𝐥𝐢𝐛𝐫𝐢𝐮𝐦 𝐑𝐞𝐚𝐬𝐨𝐧𝐞𝐫𝐬 (𝐄𝐪𝐑) ! Feedforward models and weight-tied models behave very differently on hard reasoning generalization. EqR pushes this difference to the extreme by learning 𝐭𝐚𝐬𝐤-𝐜𝐨𝐧𝐝𝐢𝐭𝐢𝐨𝐧𝐞𝐝 𝐧𝐞𝐮𝐫𝐚𝐥 𝐚𝐭𝐭𝐫𝐚𝐜𝐭𝐨𝐫𝐬 . • Sudoku-Extreme: 99.8% • Maze: 93% #ICML2026

2/ Standard feedforward transformers barely generalize: 4 / 16 / 64 layers → 1.8 / 2.1 / 2.6% on Sudoku, 0.0% on Maze. But weight-tied iteration changes the regime. What does the loop buy us?

3/ Our view: weight tying turns a network into a 𝐥𝐚𝐭𝐞𝐧𝐭 𝐝𝐲𝐧𝐚𝐦𝐢𝐜𝐚𝐥 𝐬𝐲𝐬𝐭𝐞𝐦. At test time, the model repeatedly updates its hidden state. Scaling works when the model’s 𝐢𝐧𝐭𝐞𝐫𝐧𝐚𝐥 𝐚𝐭𝐭𝐫𝐚𝐜𝐭𝐨𝐫 𝐥𝐚𝐧𝐝𝐬𝐜𝐚𝐩𝐞 aligns with the 𝐭𝐚𝐬𝐤 𝐦𝐞𝐭𝐫𝐢𝐜 𝐥𝐚𝐧𝐝𝐬𝐜𝐚𝐩𝐞: low-residual attractors -> low-error solutions.

4/ When the landscape is aligned, generalization beyond training depth becomes possible. As iterations increase: residual ↓ accuracy ↑ Convergence to neural attractors becomes a useful scaling signal.

5/ This gives two axes for inference scaling. · 𝐃𝐞𝐩𝐭𝐡: run one trajectory longer → better convergence. · 𝐁𝐫𝐞𝐚𝐝𝐭𝐡: random initialization + path noise → explore more basins. Then select the trajectory that converged best. No external verifier needed. Just the model’s own attractor dynamics.

6/ But weight tying alone is not enough. Recursive models can learn bad landscapes: 1. no correct attractor 2. spurious attractors 3. correct basin too narrow Then more compute just converges to the wrong place. In EqR, we introduce training interventions that reshape the landscape so correct attractors become reachable.

6.1/ Randomized State Initialization (RI)

6.2/ Path Stochasticity via Noise Injection (NI)

7/ Aligned attractors also make compute adaptive. Easy examples can converge in 1–5 steps. Hard examples can receive much more depth + breadth.

📄 Paper: ⚙️ Code: Shout out to my amazing mentors @ZhengyangGeng and @zicokolter for the guidance and support throughout this project!

Bonus 1/ Breadth scaling also becomes more useful after EqR training. Instead of majority voting, we can select the trajectory that actually converged best. For N trajectories, we choose the one with the lowest average residual over the final three iteration steps. This 𝐜𝐨𝐧𝐯𝐞𝐫𝐠𝐞𝐧𝐜𝐞-𝐛𝐚𝐬𝐞𝐝 𝐬𝐞𝐥𝐞𝐜𝐭𝐢𝐨𝐧 beats majority voting in both performance and efficiency. But importantly: it works for EqR, not for the baseline. That suggests RI + NI do more than improve accuracy. They make residuals meaningful again.

@zicokolter Bonus 2/ For people interested in iterative models with feedback loops, check out this collection! PRs, comments, and suggestions are very welcome!

Side-quest/ A separate question is: 𝐇𝐨𝐰 𝐝𝐨 𝐰𝐞 𝐠𝐞𝐭 𝐟𝐫𝐨𝐦 𝐚 𝐬𝐭𝐚𝐧𝐝𝐚𝐫𝐝 𝐟𝐞𝐞𝐝𝐟𝐨𝐫𝐰𝐚𝐫𝐝 𝐦𝐨𝐝𝐞𝐥 𝐭𝐨 𝐚 𝐜𝐚𝐩𝐚𝐛𝐥𝐞 𝐥𝐨𝐨𝐩 𝐦𝐨𝐝𝐞𝐥 𝐢𝐧 𝐭𝐡𝐞 𝐟𝐢𝐫𝐬𝐭 𝐩𝐥𝐚𝐜𝐞 ? We have also explored this problem in our work. We share our observations and findings in the side-post below:

Btw, the video has sound 😁

Love the video !

Thank you! It really takes me some time hh

Congrats Benhao. Very neat work.

Thank you Hayden! More on the way hh! And I have been trying your model at larger scale, so far so good 👍

Awesome! Excited for what’s next 🙂

Really interesting work.

Thank you! More to release tmrw, stay tuned 😉

This is really cool! Do you think the attractors could eventually be represented implicitly by a verifier? It seems like the flow fields are defined explicitly by a neural network here.

Thanks for your kind words! Yes, this is why we term it as neural attractors. And yeah! Flow field is quite relevant to equilibrium, and you may also intersted in drifting models

Interesting! Thanks for sharing!

😁

🤯 bro

congratulations on the icml acceptance 🎉 nice paper!

Thank you! 😄

Why not Group-Equivariant Equilibrium Reasoner or Clifford Group Equivariant? Would it make sense as it would need to learn to compose like with monoids?

LLM don’t reason, they compute.

I would say, “They compute” isn’t an argument against reasoning. If reasoning emerges from computation in brains, the question is whether LLM computation can instantiate reasoning-like processes.

这里面的动画也是skill自己生成的吗？