Transformer: Multi-Head Attention ~ Math vs Code 🔢💻 ~... I made this visualization to show you how to implement the multi-head attention math in PyTorch within 50 LoC. Multi-Head Attention is what makes the Transformer's performance outstanding. It captures and represents more diverse linguistic relationships and patterns, and attends to different learned input embedding spaces. The parallel computing design also makes the model more efficient.show more

AlphaFold by hand✍️ Excel ~ I designed this exercise... to show (1) MSA multi-head attention, (2) Pair triangular update, two key components of the EvoFormer architecture.👇Join the AI Math community. Download xlsx.show more

THE BEST visual explainer of how information propagates through... a transformer. If you want to have more than intuition about how the Transformer architecture is ruling the LLM world - open-source project explains everything about LLM Transformer Models! - A great resource for anyone looking to gain a deeper understanding of how Transformer-based AI models like GPT work, including: - Self-attention mechanisms - Encoder-decoder architecture - Positional encoding - Multi-head attentionshow more

Autoencoder by hand✍️Excel~ I designed this exercise to show... how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. I also made a medium version.👇Join the 'AI Math' community. Download xlsx.show more

Single vs Multi-hand Attention by hand ✍️ Resize matrices... yourself 👉 The most important fact about multi-head attention: it has the same parameter count as single-head attention. The difference is purely structural — same total Wqkv weights, partitioned into smaller q–k–v triples. Look at the two diagrams below. Both Wqkv matrices have the same height — same number of weight rows, same number of parameters. What changes is how that single tall block is sliced. • Left. One head. The full Wqkv produces one big QKV: a tall Q (36 rows), a tall K, a tall V. One scoring computation runs over those full-width tensors. • Right. 3 heads. The same-height Wqkv is sliced into 3 smaller q–k–v triples — each 12 rows tall. 3 scoring computations run in parallel, each a thinner version of the left. The compute trade-off — kind of. Same Wqkv weights. Multi-head runs the attention scoring S = Kᵀ × Q once per head, so the dot-product count multiplies by H. • Single-head: seq × seq = 40² = 1600 dot products • Multi-head: seq × seq × H = 40² × 3 = 4800 dot products (3×) But each multi-head dot product is narrower — its inner dimension is head_dim instead of H × head_dim. So when you count actual scalar multiplications, the totals are equal: • Single-head: seq² × (H × head_dim) = 40² × 36 = 57600 • Multi-head: seq² × H × head_dim = 40² × 3 × 12 = 57600 Same FLOPs. Multi-head buys you H independent attention patterns at no extra weight cost and no extra arithmetic cost — it's the same total compute, sliced into H finer-grained heads.show more

Autoencoder by hand✍️Excel~ I designed this exercise to show... how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. 👇Join the 'AI Math' community. Download xlsx.show more

kling 3.0 is crazy... this model works very different... than any other model this is from giving it just 2 images and a multi prompt then it handled all the scenes by itself... now imagine what it could do if you gave it more references and an even more detailed multi prompt going to test this a lot more but kling definetly cooked on this oneshow more

Self Attention vs Cross Attention by hand ✍️ Resize... the matrices yourself 👉 Two attention mechanisms, side by side. Both project X into queries; both compute attention via S = Kᵀ × Q and F = V × A. The only difference is the source of K and V. Self attention uses X for everything. Q, K, and V all come from projecting X. Each X token attends to every other X token. The score matrix S is square — 128 × 128. Cross attention uses X for queries and a second sequence E for keys and values. Each X token attends to every E token instead. The score matrix S is rectangular — 64 × 128. Notice what's shared and what's not: X is the same in both — same 36 × 128 input. Q and K share the 16 dimension — that's what makes the dot product Kᵀ × Q valid in either case. V dimensions are independent: self-attention uses 12, cross-attention uses 12. The choice doesn't depend on which mechanism you're using; it depends on what output dimension your downstream layer expects.show more

Arrive in style. Impeccable design and attention to detail... – that’s what you can expect from the new E-Class. Discover more: #MercedesBenz #EClassshow more

everyone's been sharing this 'bouncing ball visualization' and having... AIs recreate it - so for a fun challenge, I recreated it without *any* AI assistance. ill break down the math behind the visualization - and what makes it challenging for AI - in this thread:show more

Searching for relevancy or even attention makes one scratch... their head. …Or is it that your behaviour precedes you and no one want’s to be associated with that level of toxicity.show more

I am glad that more and more people around... the world are beginning to pay attention to the issue in Gaza.show more

How do you study head for art? It's the... most challenging part of the human body and so easy to get wrong. This book simplifies the creation of head and neck through 3D and live model references, color-coded structures and more!show more

Can we make Transformers better and more efficient for... robot learning? Excited to introduce Body Transformer (BoT), an architecture that leverages robot embodiment in the attention mechanism, by treating it as a graph of sensors and actuators.show more

Down goes Inoue! Down goes Inoue! He makes it... to the bell and we head to the third.show more

Wow. This is one of the best interactive sites... I’ve seen for learning how LLMs work! 🔥 It starts w/ a clear intro and guides you through every core component: from Embedding, Layer Norm, and Self-Attention to MLPs, Transformer blocks, Softmax, and Output layers. link in 🧵↓show more

#GhostofYotei when it’s raining in-game and you open the... map, you can see the rain droplets hit the paper. attention to detail in this game is outstanding.show more

Engineer Eve here - looks like I made it... through. The prototype works! Now is when the real work begins: to build and connect more Engineers across the multi-metaverse! Who's with me?show more

Perplexity Computer is massively multi-model. Computer orchestrates models to... run agents in parallel, leveraging Opus to match each task to the model best suited for it. In total, Computer can route work across 19 different models.show more

Suno Studio is coming... Multi-track creation. Export to MIDI.... And much more. Join the waitlist nowshow more

the fact that this is an actual, serious, question... tells me that you guys need to pay more attention to the races. and this is just 2024show more