Transformer: Multi-Head Attention ~ Math vs Code 🔢💻 ~... I made this visualization to show you how to implement the multi-head attention math in PyTorch within 50 LoC. Multi-Head Attention is what makes the Transformer's performance outstanding. It captures and represents more diverse linguistic relationships and patterns, and attends to different learned input embedding spaces. The parallel computing design also makes the model more efficient.show more

AlphaFold by hand✍️ Excel ~ I designed this exercise... to show (1) MSA multi-head attention, (2) Pair triangular update, two key components of the EvoFormer architecture.👇Join the AI Math community. Download xlsx.show more

Transformer by hand✍️ Excel ~ I designed this exercise... to show the core math of a Transformer model is to combine columns (attention), combine rows (feed forward), and repeat.👇Join the 'AI Math' community. 👇Download xlsx.show more

THE BEST visual explainer of how information propagates through... a transformer. If you want to have more than intuition about how the Transformer architecture is ruling the LLM world - open-source project explains everything about LLM Transformer Models! - A great resource for anyone looking to gain a deeper understanding of how Transformer-based AI models like GPT work, including: - Self-attention mechanisms - Encoder-decoder architecture - Positional encoding - Multi-head attentionshow more

Autoencoder by hand✍️Excel~ I designed this exercise to show... how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. I also made a medium version.👇Join the 'AI Math' community. Download xlsx.show more

Single vs Multi-hand Attention by hand ✍️ Resize matrices... yourself 👉 The most important fact about multi-head attention: it has the same parameter count as single-head attention. The difference is purely structural — same total Wqkv weights, partitioned into smaller q–k–v triples. Look at the two diagrams below. Both Wqkv matrices have the same height — same number of weight rows, same number of parameters. What changes is how that single tall block is sliced. • Left. One head. The full Wqkv produces one big QKV: a tall Q (36 rows), a tall K, a tall V. One scoring computation runs over those full-width tensors. • Right. 3 heads. The same-height Wqkv is sliced into 3 smaller q–k–v triples — each 12 rows tall. 3 scoring computations run in parallel, each a thinner version of the left. The compute trade-off — kind of. Same Wqkv weights. Multi-head runs the attention scoring S = Kᵀ × Q once per head, so the dot-product count multiplies by H. • Single-head: seq × seq = 40² = 1600 dot products • Multi-head: seq × seq × H = 40² × 3 = 4800 dot products (3×) But each multi-head dot product is narrower — its inner dimension is head_dim instead of H × head_dim. So when you count actual scalar multiplications, the totals are equal: • Single-head: seq² × (H × head_dim) = 40² × 36 = 57600 • Multi-head: seq² × H × head_dim = 40² × 3 × 12 = 57600 Same FLOPs. Multi-head buys you H independent attention patterns at no extra weight cost and no extra arithmetic cost — it's the same total compute, sliced into H finer-grained heads.show more

Autoencoder by hand✍️Excel~ I designed this exercise to show... how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. 👇Join the 'AI Math' community. Download xlsx.show more

the more i think about aiah being the muse... for “attention” the more i get it 😭 like girly has this quiet kind of pull that just naturally draws you in and makes you wanna pay attention to her... i can’t wait to see the mv 😩 proud of you, aiahkins <3show more

kling 3.0 is crazy... this model works very different... than any other model this is from giving it just 2 images and a multi prompt then it handled all the scenes by itself... now imagine what it could do if you gave it more references and an even more detailed multi prompt going to test this a lot more but kling definetly cooked on this oneshow more

QXR Head Multi Bug The head multi is higher... than it is supposed to be. Do not abuse the bug in ranked just in case. We are not responsible for any results and hopefully Call of Duty: Mobile will fix it soon. Also as it seems, sometimes the QXR bug works and sometimes it doesn't.(it's also linked to if thermite/explosive/dragon breath rounds work as intended) #CODMobile #callofdutymobile #codm #codmnews #codmleaksshow more

Transformer by hand ✍️ ~ 6 steps walkthrough below... Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features *across positions*, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens *across feature dimensions*, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes *across positions*, so a feature can see its neighbours. The FFN mixes *across feature dimensions*, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearningshow more

AI is now improving its own model architectures and... chips. The power infrastructure it runs on deserves the same attention. We used RL to teach a Kimi base model to design medium-power transformers. It met 93% of unseen specs, helping to compress multi-week engineering efforts into a few minutes of inference. 1/5show more

Self Attention vs Cross Attention by hand ✍️ Resize... the matrices yourself 👉 Two attention mechanisms, side by side. Both project X into queries; both compute attention via S = Kᵀ × Q and F = V × A. The only difference is the source of K and V. Self attention uses X for everything. Q, K, and V all come from projecting X. Each X token attends to every other X token. The score matrix S is square — 128 × 128. Cross attention uses X for queries and a second sequence E for keys and values. Each X token attends to every E token instead. The score matrix S is rectangular — 64 × 128. Notice what's shared and what's not: X is the same in both — same 36 × 128 input. Q and K share the 16 dimension — that's what makes the dot product Kᵀ × Q valid in either case. V dimensions are independent: self-attention uses 12, cross-attention uses 12. The choice doesn't depend on which mechanism you're using; it depends on what output dimension your downstream layer expects.show more

The final part of the three part "Profiling in... PyTorch" series is out now! In Part 1 we profiled addition and multiplication In Part 2 we profiled a torch Linear layer (and also MLP) In this part we profile Attention. > a naive implementation of attention > talk about the torch sdpa API > math backend > xformers backend > flash backend > cuDNN backend PS: This is a long read, bookmark this and read it over the weekends. 🤗show more

Arrive in style. Impeccable design and attention to detail... – that’s what you can expect from the new E-Class. Discover more: #MercedesBenz #EClassshow more

Multi-bot chat allows you to seamlessly combine the unique... strengths of different bots to complete tasks. For example, using fal’s latest bots, you can generate an image and remove the background within one conversational thread for a more efficient workflow. (1/2)show more

Searching for relevancy or even attention makes one scratch... their head. …Or is it that your behaviour precedes you and no one want’s to be associated with that level of toxicity.show more

I am glad that more and more people around... the world are beginning to pay attention to the issue in Gaza.show more

Can we make Transformers better and more efficient for... robot learning? Excited to introduce Body Transformer (BoT), an architecture that leverages robot embodiment in the attention mechanism, by treating it as a graph of sensors and actuators.show more