Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Transformer: Multi-Head Attention ~ Math vs Code 🔢💻 ~ I made this visualization to show you how to implement the multi-head attention math in PyTorch within 50 LoC. Multi-Head Attention is what makes the Transformer's performance outstanding. It captures and represents more diverse linguistic relationships and patterns, and attends... show more

Yan Chen

1,866 subscribers

33,255 views • 1 year ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

AlphaFold by hand✍️ Excel ~ I designed this exercise to show (1) MSA multi-head attention, (2) Pair triangular update, two key components of the EvoFormer architecture.👇Join the AI Math community. Download xlsx.

AlphaFold by hand✍️ Excel ~ I designed this exercise to show (1) MSA multi-head attention, (2) Pair triangular update, two key components of the EvoFormer architecture.👇Join the AI Math community. Download xlsx.

Tom Yeh

104,965 views • 1 year ago

THE BEST visual explainer of how information propagates through a transformer. If you want to have more than intuition about how the Transformer architecture is ruling the LLM world - open-source project explains everything about LLM Transformer Models! - A great resource for anyone looking to gain a deeper understanding of how Transformer-based AI models like GPT work, including: - Self-attention mechanisms - Encoder-decoder architecture - Positional encoding - Multi-head attention

THE BEST visual explainer of how information propagates through a transformer. If you want to have more than intuition about how the Transformer architecture is ruling the LLM world - open-source project explains everything about LLM Transformer Models! - A great resource for anyone looking to gain a deeper understanding of how Transformer-based AI models like GPT work, including: - Self-attention mechanisms - Encoder-decoder architecture - Positional encoding - Multi-head attention

Rohan Paul

106,880 views • 1 year ago

Autoencoder by hand✍️Excel~ I designed this exercise to show how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. I also made a medium version.👇Join the 'AI Math' community. Download xlsx.

Autoencoder by hand✍️Excel~ I designed this exercise to show how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. I also made a medium version.👇Join the 'AI Math' community. Download xlsx.

Tom Yeh

54,473 views • 1 year ago

Single vs Multi-hand Attention by hand ✍️ Resize matrices yourself 👉 The most important fact about multi-head attention: it has the same parameter count as single-head attention. The difference is purely structural — same total Wqkv weights, partitioned into smaller q–k–v triples. Look at the two diagrams below. Both Wqkv matrices have the same height — same number of weight rows, same number of parameters. What changes is how that single tall block is sliced. • Left. One head. The full Wqkv produces one big QKV: a tall Q (36 rows), a tall K, a tall V. One scoring computation runs over those full-width tensors. • Right. 3 heads. The same-height Wqkv is sliced into 3 smaller q–k–v triples — each 12 rows tall. 3 scoring computations run in parallel, each a thinner version of the left. The compute trade-off — kind of. Same Wqkv weights. Multi-head runs the attention scoring S = Kᵀ × Q once per head, so the dot-product count multiplies by H. • Single-head: seq × seq = 40² = 1600 dot products • Multi-head: seq × seq × H = 40² × 3 = 4800 dot products (3×) But each multi-head dot product is narrower — its inner dimension is head_dim instead of H × head_dim. So when you count actual scalar multiplications, the totals are equal: • Single-head: seq² × (H × head_dim) = 40² × 36 = 57600 • Multi-head: seq² × H × head_dim = 40² × 3 × 12 = 57600 Same FLOPs. Multi-head buys you H independent attention patterns at no extra weight cost and no extra arithmetic cost — it's the same total compute, sliced into H finer-grained heads.

Single vs Multi-hand Attention by hand ✍️ Resize matrices yourself 👉 The most important fact about multi-head attention: it has the same parameter count as single-head attention. The difference is purely structural — same total Wqkv weights, partitioned into smaller q–k–v triples. Look at the two diagrams below. Both Wqkv matrices have the same height — same number of weight rows, same number of parameters. What changes is how that single tall block is sliced. • Left. One head. The full Wqkv produces one big QKV: a tall Q (36 rows), a tall K, a tall V. One scoring computation runs over those full-width tensors. • Right. 3 heads. The same-height Wqkv is sliced into 3 smaller q–k–v triples — each 12 rows tall. 3 scoring computations run in parallel, each a thinner version of the left. The compute trade-off — kind of. Same Wqkv weights. Multi-head runs the attention scoring S = Kᵀ × Q once per head, so the dot-product count multiplies by H. • Single-head: seq × seq = 40² = 1600 dot products • Multi-head: seq × seq × H = 40² × 3 = 4800 dot products (3×) But each multi-head dot product is narrower — its inner dimension is head_dim instead of H × head_dim. So when you count actual scalar multiplications, the totals are equal: • Single-head: seq² × (H × head_dim) = 40² × 36 = 57600 • Multi-head: seq² × H × head_dim = 40² × 3 × 12 = 57600 Same FLOPs. Multi-head buys you H independent attention patterns at no extra weight cost and no extra arithmetic cost — it's the same total compute, sliced into H finer-grained heads.

Tom Yeh

35,046 views • 1 month ago

Autoencoder by hand✍️Excel~ I designed this exercise to show how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. 👇Join the 'AI Math' community. Download xlsx.

Autoencoder by hand✍️Excel~ I designed this exercise to show how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. 👇Join the 'AI Math' community. Download xlsx.

Tom Yeh

101,555 views • 1 year ago

kling 3.0 is crazy... this model works very different than any other model this is from giving it just 2 images and a multi prompt then it handled all the scenes by itself... now imagine what it could do if you gave it more references and an even more detailed multi prompt going to test this a lot more but kling definetly cooked on this one

kling 3.0 is crazy... this model works very different than any other model this is from giving it just 2 images and a multi prompt then it handled all the scenes by itself... now imagine what it could do if you gave it more references and an even more detailed multi prompt going to test this a lot more but kling definetly cooked on this one

Miko

95,568 views • 4 months ago

Self Attention vs Cross Attention by hand ✍️ Resize the matrices yourself 👉 Two attention mechanisms, side by side. Both project X into queries; both compute attention via S = Kᵀ × Q and F = V × A. The only difference is the source of K and V. Self attention uses X for everything. Q, K, and V all come from projecting X. Each X token attends to every other X token. The score matrix S is square — 128 × 128. Cross attention uses X for queries and a second sequence E for keys and values. Each X token attends to every E token instead. The score matrix S is rectangular — 64 × 128. Notice what's shared and what's not: X is the same in both — same 36 × 128 input. Q and K share the 16 dimension — that's what makes the dot product Kᵀ × Q valid in either case. V dimensions are independent: self-attention uses 12, cross-attention uses 12. The choice doesn't depend on which mechanism you're using; it depends on what output dimension your downstream layer expects.

Self Attention vs Cross Attention by hand ✍️ Resize the matrices yourself 👉 Two attention mechanisms, side by side. Both project X into queries; both compute attention via S = Kᵀ × Q and F = V × A. The only difference is the source of K and V. Self attention uses X for everything. Q, K, and V all come from projecting X. Each X token attends to every other X token. The score matrix S is square — 128 × 128. Cross attention uses X for queries and a second sequence E for keys and values. Each X token attends to every E token instead. The score matrix S is rectangular — 64 × 128. Notice what's shared and what's not: X is the same in both — same 36 × 128 input. Q and K share the 16 dimension — that's what makes the dot product Kᵀ × Q valid in either case. V dimensions are independent: self-attention uses 12, cross-attention uses 12. The choice doesn't depend on which mechanism you're using; it depends on what output dimension your downstream layer expects.

Tom Yeh

61,300 views • 1 month ago

Arrive in style. Impeccable design and attention to detail – that’s what you can expect from the new E-Class. Discover more: #MercedesBenz #EClass

Arrive in style. Impeccable design and attention to detail – that’s what you can expect from the new E-Class. Discover more: #MercedesBenz #EClass

Mercedes-Benz

147,637 views • 3 years ago

everyone's been sharing this 'bouncing ball visualization' and having AIs recreate it - so for a fun challenge, I recreated it without *any* AI assistance. ill break down the math behind the visualization - and what makes it challenging for AI - in this thread:

everyone's been sharing this 'bouncing ball visualization' and having AIs recreate it - so for a fun challenge, I recreated it without any AI assistance. ill break down the math behind the visualization - and what makes it challenging for AI - in this thread:

N8 Programs

26,695 views • 1 year ago

Searching for relevancy or even attention makes one scratch their head. …Or is it that your behaviour precedes you and no one want’s to be associated with that level of toxicity.

Searching for relevancy or even attention makes one scratch their head. …Or is it that your behaviour precedes you and no one want’s to be associated with that level of toxicity.

Julius . D

89,716 views • 1 year ago

I am glad that more and more people around the world are beginning to pay attention to the issue in Gaza.

I am glad that more and more people around the world are beginning to pay attention to the issue in Gaza.

Roman Akramovich

21,609 views • 8 months ago

How do you study head for art? It's the most challenging part of the human body and so easy to get wrong. This book simplifies the creation of head and neck through 3D and live model references, color-coded structures and more!

How do you study head for art? It's the most challenging part of the human body and so easy to get wrong. This book simplifies the creation of head and neck through 3D and live model references, color-coded structures and more!

Anatomy For Sculptors

466,107 views • 2 years ago

Can we make Transformers better and more efficient for robot learning? Excited to introduce Body Transformer (BoT), an architecture that leverages robot embodiment in the attention mechanism, by treating it as a graph of sensors and actuators.

Can we make Transformers better and more efficient for robot learning? Excited to introduce Body Transformer (BoT), an architecture that leverages robot embodiment in the attention mechanism, by treating it as a graph of sensors and actuators.

Carlo Sferrazza

60,663 views • 1 year ago

Wow. This is one of the best interactive sites I’ve seen for learning how LLMs work! 🔥 It starts w/ a clear intro and guides you through every core component: from Embedding, Layer Norm, and Self-Attention to MLPs, Transformer blocks, Softmax, and Output layers. link in 🧵↓

Wow. This is one of the best interactive sites I’ve seen for learning how LLMs work! 🔥 It starts w/ a clear intro and guides you through every core component: from Embedding, Layer Norm, and Self-Attention to MLPs, Transformer blocks, Softmax, and Output layers. link in 🧵↓

Charly Wargnier

52,092 views • 1 year ago

Down goes Inoue! Down goes Inoue! He makes it to the bell and we head to the third.

Down goes Inoue! Down goes Inoue! He makes it to the bell and we head to the third.

Spinnin Backfist

192,610 views • 1 year ago

#GhostofYotei when it’s raining in-game and you open the map, you can see the rain droplets hit the paper. attention to detail in this game is outstanding.

#GhostofYotei when it’s raining in-game and you open the map, you can see the rain droplets hit the paper. attention to detail in this game is outstanding.

that 🥷🏾 you love to hate

49,586 views • 8 months ago

Engineer Eve here - looks like I made it through. The prototype works! Now is when the real work begins: to build and connect more Engineers across the multi-metaverse! Who's with me?

Engineer Eve here - looks like I made it through. The prototype works! Now is when the real work begins: to build and connect more Engineers across the multi-metaverse! Who's with me?

The Engineers NFT

13,519 views • 3 years ago

Perplexity Computer is massively multi-model. Computer orchestrates models to run agents in parallel, leveraging Opus to match each task to the model best suited for it. In total, Computer can route work across 19 different models.

Perplexity Computer is massively multi-model. Computer orchestrates models to run agents in parallel, leveraging Opus to match each task to the model best suited for it. In total, Computer can route work across 19 different models.

Perplexity

531,610 views • 3 months ago

Suno Studio is coming... Multi-track creation. Export to MIDI. And much more. Join the waitlist now

Suno Studio is coming... Multi-track creation. Export to MIDI. And much more. Join the waitlist now

Suno

105,135 views • 10 months ago

the fact that this is an actual, serious, question tells me that you guys need to pay more attention to the races. and this is just 2024

the fact that this is an actual, serious, question tells me that you guys need to pay more attention to the races. and this is just 2024

ela

68,324 views • 8 months ago