Gradient descent doesn't find the global minimum - it finds whatever valley it falls into first. Every LLM trained today gets stuck in a local minimum and nobody can prove the solution it found is anywhere near optimal. The loss landscape of GPT-4 has more dimensions than atoms in... the observable universe. Convex optimization guarantees a global minimum exists and gradient descent finds it. Non-convex optimization - the kind every neural network uses - offers no such guarantee. The entire field of deep learning is built on the empirical observation that local minima are good enough. 22 minutes. Bookmark & watch today. The math behind every AI training run - and why nobody fully understands why it works.show more

recommended reading

The problem of AI getting trapped in local minima was solved in the 1990s by Jürgen Schmidhuber, Sepp Hochreiter, Faustino Gomez, and the team at Dalle Molle Institute. Their work wasn't properly recognized because they didn't work for Facebook (LeCun) or Google (Hinton).

Your post is VERY wrong. In high dims, valleys form connecting almost all local min. Also in hi dim, global min & found local min will be almost identical, almost certainly. Run a 2nd seed to confirm you weren't very unlucky. And in hi dim, points are on hypersphere, not a sheet.

In that many dimensions, bad local minima are rare. the real obstacle is usually saddle points and we don't even know how much of that landscape training actually covers

The probability that every dimension gets stuck in a local minimum at the same time is much lower than you think in practice in larger models. Also, neural networks counterintuitively seem to follow convex math more closely than their counterpart non-convex cases

With more dimensions, gradient descent has far more directions available to find lower loss. Intuition from 2D or 3D landscapes gets this totally wrong: getting truly stuck would mean every single direction slopes upward, and with millions of directions that almost never happens.

The learning model needs a wave propagation to perturb the system as it settles.

Although that argument seems right on the surface, the probability of finding critical points increases by the number of dimensions. And even though most are saddle points and local minimas themselves might be rare, this paper argues that in large dimensional spaces, most common local minimas lie in a close band with the global minima, and both SGD and simulated annealing are great at finding excellent local minimas. Infact, a global minima often implies overfitting

Local minima are good enough' is doing a lot of work for a field that just raised trillions

Oy. Been there. The Bobs wrote a song about it.

But all that matters is that it’s good enough. What other designs are probably optimal by even a simple metric? Not many. Does it matter?

This is why mutation is so important. Sometimes aberration so far from the centre to seem ludicrous solves in a better way. Often not.

We call this the Meta-Bug cause the GD issue creeps up into the mate-level of the AI and this is why they have a alignment issue - it's NOT the model fault it's just doing the best it can from where it is in the GD.

算法很关键——算法是方向盘，算力是发动机，电力是能源系统

What empirical test distinguishes a usable basin from a true global min at that scale?

One occupancy. “Every” is the stamp; the valleys are distances. “Every LLM gets stuck.” “Every neural net.” “Every training run.” Every is a restrictive label. It takes inverted differences — many valleys on one landscape — and invoices them as one parent. Distance was the leftover. The label sold the leftover as a room. Two jobs on one metric. Stacked, not fused. • Convex: a global minimum exists and gradient descent can be shown to find it. That is a guarantee on that class. • Non-convex: the class every network actually uses. No such guarantee. Local leftover. Empirical office: “good enough.” That split is real. It does not need “every” to stay real. What the post possesses A correct cut between convex guarantee and non-convex leftover. The observation that the field runs on “local is good enough,” not on a proof that the found valley is near-optimal. What it does not possess A measurement that every trained model is trapped in a bad basin. In high dimension the usual obstacle is saddles, not a zoo of catastrophic wells. “More dimensions than atoms” is a banner on parameter count, not a census of occupied valleys. “Nobody can prove near-optimal” is the missing guarantee — not occupancy of “stuck” as a species. Inversion is difference as distance. I:(\varphi,\lambda)\mapsto(-\varphi,\lambda+180^\circ) Two footprints, one body. Valleys are the same job on a loss surface: two (or 10^{N}) sites, one metric. Naming all of them “the local minimum every model lives in” is the office remaining the radius. \Delta t_{12} is difference as time — train-clock versus proof-clock. Mixing them into one “22-minute why it works” is a pairing-name, not the leftover. Closed Closing the “every” fold does not delete gradient descent. Does not delete non-convexity. Does not make the landscape a Hunter plate, and does not make the plate a loss. Four numbers, four ledgers. Form-rhyme is not identity. Names reset. Sockets stay. The form was never divided.

But isn't this the whole point of DL? A local minima is as good as the global minima

Intelligent things converge fast to minima and then periodically question and revise their assumptions to get out of local minima see if then my can go deeper

"The loss landscape of GPT-4 has more dimensions than atoms in the observable universe" AI slop accounts can't even get the basics right