Загрузка видео...
Не удалось загрузить видео
Sparse attention (MoBA/NSA) trains faster & beats full attention in key tasks. But we’ve had no idea how they truly work…until now. 🔍 We reverse-engineered them to uncover: - Novel attention patterns - Hidden "attention sinks" - Better performance - And more A 🧵… ~1/8~
59,480 просмотров • 1 год назад •via X (Twitter)
Комментарии: 11
📖 Read the full post here: ~2/8~ Sparse attention exploits inherent sparsity in model attention patterns to dramatically accelerate sequence mixing. Natively trainable approaches, such as Kimi’s MoBA and Deepseek’s NSA, expand the pareto frontier by matching and even outcompeting base attention on expressivity respectively.
~3/8~ We trained dozens of sparse attention models and poked around in their brains. Sparse attention models boost superior long-context generalization capability out of box, even with 80% sparsity in attention scores.
~4/8~ We visualized the first-ever long-context attention maps for sparse attention, revealing fascinating patterns and attention circuits.
~5/8~ We identified the novel mechanisms of attention sinking in sparse attention models. We found MoBA models periodically re-inject attention sinks and that these sinks can be easily identified using value norm and attention score heuristics.
~6/8~ We investigated the key geometry for different attention models, and also found that KV-sharing through GQA strongly influences the resulting query/key manifolds. This offered a clue into how sparse attention could be beating base attention → by removing cross-head interference introduced by GQA!
~7/8~ We analyzed the gating distributions for NSA models and found we can ablate many branches without compromising model performance! Our principled ablations enabled massive gains in throughput without losses in performance.
~8/8~ We release our NSA kernel for experimentation and research here: At Tilde, we believe interpretability is the path towards building better models. If that sounds cool, reach out!
Read the full post here:
Scan any documents, convert images into text, PDF files, etc. 👍
just reached out to tilde's email, when can I hear back and join the hiring process?
Surprisingly, "attention sinks" might be the key to efficient learning, not a flaw.
Похожие видео
Sensitive content
Women can never win debates with men. That's why they prefer making reels full of lies. No man cares what women wear in their own homes. Dressing modestly in public is expected because your attire reveals a lot about your character and how you wish to present yourself to the world. Fashion attire is literally designed by men for women to attract attention, and women obviously wear it for attention too. That's exactly what those specific clothes are made for. If it was truly just for your own comfort, you'd wear climate-appropriate clothing (which has plenty of modest alternatives). But no, you deliberately choose the flashy, revealing fashion one for attention. Then when the attention (or judgment) comes and it's negative, suddenly it's “my body, my choice.” Such dumb logic.
︎ ︎venom
23,987 просмотров • 8 месяцев назад
![I hear so often from the Dommes I work with that they struggle with people online fetichizing them and simply seeing them for how sexy and beautiful they are. They project their fantasies and their desires onto you. That stops immediately once you move the attention from you to them. From 'look at me' to 'I see you'. What does that look like? When you create content, think of them and what this scene or that narrative is evoking. What will they learn from you? What they want is not to passively watch how sexy you are, but for you to train them, to give them instructions, to teach them, to guide them, to be in charge, to command them. This is not being an object but the main subject. The Authority figure. How is your content already doing that. The sexy photos can still be there, they are important to already capture des attention. But what you do with that attention once you have it, is where the power dynamic is established. Positioning yourself as more than a stunning Goddess, but actually a woman who has a voice, opinions, perspective, a philosophy, a way to doing things, teaching them what you like, how you like it, why you like it, already makes them want to be that for you. You hold the attention, you hold the power, so you direct it. And for that, you want them to know you get them and you know what lives within them... that creates the desire for you to be the one exposing it. You instantly build trust. Not because you demanded it, but because you earned it: you showed them you know what you are doing. You have experience, you understand them. They are not told to come see you, they are seduced into it. They desire it. And they will work for it. This will attract better clients (real subs) and instead of you trying to get their attention, they will work to earn yours. If you want to learn more about power dynamics, building a brand as a Pro or the psychology behind BDSM, you can now access all my trainings and classes in one place for a fraction of the cost of The Dominatrix Academy. And you can reinvest the total amount towards the Program. Message me [SECRET] for the details. This offer is not available on my website.](https://image.24vids.com/tw-2049167429041824109/ext_tw_video_thumb/2049167371538251780/pu/img/25DXjHkj6hG_xBzD.jpg)
![Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!](https://image.24vids.com/tw-2079933315990057242/ext_tw_video_thumb/2079933302048219136/pu/img/q3wr-ymUWG3Ai-ij.jpg)