Unconventional AI

We removed 93.25% of the connections in our Un-0 image model, fully expecting to pay for it in quality. But it got better. FID 7.15 on ImageNet 64x64, roughly 1.9 ahead of the dense baseline at matched size. Same family of model, a fraction of the couplings, a better score. Here is why that is not as strange as it sounds. Un-0 is a coupled oscillator model, and in the dense version every oscillator talks to every other one. That sounds like a strength, but it means the whole system can fall into catastrophic synchronization: everything locks into step, gradients go flat, and learning stalls. Sparser connectivity leaves room for coherent and incoherent activity to coexist. The dynamics stay alive, and the model keeps learning. Connectivity turns out to be a control knob, not a dial you turn up until it stops. Learn more here:

34,149 Aufrufe