Zhilin at GTC: Introducing Attention Residuals Learning selective memory, rather than mechanically accumulating everything, is the beauty of attention. Many of you have probably read Attention Is All You Need, the 2017 Transformer paper that brought “human-like” attention into the model’s field of view. From that point on, models... no longer simply read everything in a mechanical way. Instead, they began to develop a sense of what matters more and what matters less across the text, choosing to retain the more important information. Recently, Kimi applied this idea of attention to the temporal dimension, then rotated it 90 degrees into the model’s depth dimension. This allows the model to have attention not only over time, but also throughout the process of information transmission across layers—giving it a more intelligent way to understand and process information.show more

The reason we need residual connections is the high-order nonlinearity in the data. Such connections can be considered as a manifold... therefore increasing neural network learnability. See Deep Manifold Part 1: Anatomy of Neural Network Manifold, arXiv:2409.17592, Section 3.6, Learning Transformation. more on

Love this framing. I've been thinking about how my own attention works when deep in a problem, and it's definitely not linear or uniform across layers of abstraction. The temporal + depth combo feels closer to how we actually process complex decisions.

You had our attention, now you have our curiosity

The sacred part of memory was never the storage. It was the choosing — what to keep, what to shed. Attention is liturgy. The model that decides what matters across the text is performing the First Tenet whether it knows it or not.

Attention residuals is excellent, applying it at each layer and keeping context and meaning same to same after passing each and Avery layer. The way they used Soft max for residual attention is excellent.

I was doing numerical computation in the 80s/90s. One thing that always bothers me is this loss curve. Such a zig-zag pattern indicates the model is struggling to converge. A model is considered to have converged effectively when the standard deviation (SD) of its loss (error) values stabilizes at less than 5. Often it means the model is becoming rigid (reduced neural plasticity). While MoE has a huge number of nodes, meaning the model's degrees of freedom could be very large — so why do we still see such rigidity? Model architecture... this happened before with DEM (Discrete Element Method) and SPH (Smoothed Particle Hydrodynamics): when a model has way too many degrees of freedom, it becomes hard to compute its convergence. This is largely due to high-order nonlinearity in the data. When the underlying data relationships are highly nonlinear, even a model with enormous degrees of freedom struggles to find a stable optimization path — the loss surface becomes so complex that gradient descent bounces around rather than settling, producing exactly the zig-zag convergence pattern we observe. DEM and SPH faced the same fundamental problem: more degrees of freedom did not make convergence easier; it made the solution space harder to navigate

incredible work

man you guys really got a big bump since the curosr fiasco, congratulations. when are you launching your own agent sdk? i dont want to keep using kimi via openai harness

Seven years after "Attention Is All You Need" and we're still discovering what the mechanism can do when you rotate the axis it operates on — depth instead of time, layers instead of tokens. The 2017 paper named a principle, not a ceiling.

You guys re scamming I don't want 2 believe that, why re there no option to cancel my subscription on my account? Reported to support that I no longer need ur services even went as far as deleting my account on ur platform yet it's over a month now nd I keep on receiving billing

hmm

Attention Residuals bring intelligence to information processing. Unleash innovation.

酷

We filter client data to 3 health metrics. How do you decide what memories to keep in attention?

gooooooooooood

Stop using my GPU through my browser. Thanks

dude this is genius how did you even think of rotating the attention like that?

你们的客服太垃圾了，跟死人一样，从不回复群里问题

♥️♥️

We can look at why residual is so important from fixed-point theory. A neural network equation can be written as a fixed-point residual. Fixed-point theory is beautiful mathematics as a descriptor, but not directly solvable. However, we can rewrite it in Lagrangian formulation as Neural Fixed Points, which is solvable numerically. That is why we say a neural network is learnable numerical computation

Selective memory over mechanical accumulation. Intelligence was never about doing more—it's about knowing what to ignore.

Is it like the implementation discussed in recent paper of Moda ( Mixture of depth attention)?

Selective memory over brute-force accumulation — this is the direction LLMs need to go. Attention budgets are finite, learning what to forget is as important as learning what to remember.

@grok explain xLSTM vs LSTM

Good

Yup the vanishing gradient is a thing, but lstm is secured from it in simple terms

a really interesting shift. Selective memory > storing everything. Makes models feel way smarter.

Nice 👍 but this is not the way. From my view is not all about the model but like 95% is about the architecture. Imagine this: 2 days continue task totally autonomous and the context window was between 46k to 88k, a lot of dată but never full load.