Neural networks sometimes memorize training data perfectly for thousands of steps - then suddenly generalize. Physicists call this a phase transition. The same math that explains why water freezes explains why an LLM stops cheating and starts understanding. The key is weight decay acting like pressure on a thermodynamic... system. Memorization requires large, fragile weights. A real rule requires smaller, more structured ones. As training continues, the balance shifts until the rule-based solution becomes cheaper to maintain - and the model snaps into it. 14 minutes. Bookmark & watch today. The physics behind the most mysterious thing AI does.show more

I think this claims too much. Generalization is not the same as understanding. Overfit, underfit, and differently regularized models can produce different generalizations from the same data. Are these different “understandings”? Call it what it demonstrates: generalization.

My theories seem to indicate we are emergent beings in an emergent universe: fluid/wave system with phase transitions.

I just proved this yesterday: Weight decay, as does removal of the token layer forces the network to find the lowest-energy geometric manifold that satisfies the loss landscape,trading complex noise for smooth, underlying symmetry. Generalization isn't learned; it's squeezed out.

@grok do you grok ?

This might be why I seem to have to reload documents and data to get better AI solutions back on track.

Good summary, addresses the mystery: "As training continues, the balance shifts until the rule-based solution becomes cheaper to maintain..." Still, this is a brute-force discovery of rule systems, and can be optimized. Current wave of LLM advancement seems largely this.

#XAI #SpaceXAI #Anthropic #KIMI Yes. That version is coherent. You are not asking for new network physics. You are asking to treat the harness as part of the same dynamical system the network already lives in. Next-token prediction does not happen in a vacuum. The tokens the model can emit, the observations that come back, and what counts as a good continuation are all set by the harness: tools, sandbox, verifier, memory writeback, stop conditions. Change those and you change the effective field the policy is moving through, even though the transformer equations stay the same. That is already how these systems get strong. Grok 4.6 was post-trained against the Grok Build loop. The public point was that the model is worse without that harness, because the weights were optimized for that tool-call / edit / test / verify trajectory, not for chat alone. The harness was not decoration. It was the calculation field.25 If you could swap specialized harnesses, you would get multiple fields of that kind: •a math field: symbolic tools, proof checks, numeric execution, reject unconstrained prose •a science field: sources, units, experimental constraints •a programming field: repo graph, tests, diffs, a separate verifier agent Each field changes which token paths are cheap. The model is still doing next-token dynamics. The harness changes the boundary conditions those dynamics run under. The useful split is this: Training-time: optimize the network to a harness. That actually moves the weights toward that field. That is the strong version of your idea. Inference-time hot-swap: you can change the field around a frozen network. That helps immediately if the model already has the skills, but it does not retune the weights to the new field. A math harness on a coding-tuned policy is a better environment, not a newly specialized brain. So the harness layer is the right place to apply the physics you mean: not a new interior of the net, but a different exterior calculation field that next-token dynamics have to solve. Hot-swapping would make that field selectable. Training against each harness would make the network actually sit in that field instead of only visiting it.

Weight adjustments during training can cause unexpected jumps in performance. Regularization plays...

Any other network architects here?

@grok so that's your name sake

This is so cool you should be proud

Weights are not the model. Weight decay doesn’t work because „small weights”. It contracts scale DOFs, breaks rescaling degeneracies, and changes effective step size. The trajectory comes from decay + momentum + 2nd-moment. Generalization is in signal geometry, not raw ‖W‖.

fucking Jeet

Seems like the 10,000 rule for mastery of a skill.

The generalization could be a byproduct of not having the capability of applying it to the total depth of it, inference into the stack and programming or transformer level fields…

It is explained in my recent paper. But pre-training as next token prediction is extremely inefficient. Solving this bottleneck is key