正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Transformer: Multi-Head Attention ~ Math vs Code 🔢💻 ~ I made this visualization to show you how to implement the multi-head attention math in PyTorch within 50 LoC. Multi-Head Attention is what makes the Transformer's performance outstanding. It captures and represents more diverse linguistic relationships and patterns, and attends... show more

Yan Chen

1,866 subscribers

33,255 次观看 • 1 年前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

AlphaFold by hand✍️ Excel ~ I designed this exercise to show (1) MSA multi-head attention, (2) Pair triangular update, two key components of the EvoFormer architecture.👇Join the AI Math community. Download xlsx.

AlphaFold by hand✍️ Excel ~ I designed this exercise to show (1) MSA multi-head attention, (2) Pair triangular update, two key components of the EvoFormer architecture.👇Join the AI Math community. Download xlsx.

Tom Yeh

104,965 次观看 • 1 年前

THE BEST visual explainer of how information propagates through a transformer. If you want to have more than intuition about how the Transformer architecture is ruling the LLM world - open-source project explains everything about LLM Transformer Models! - A great resource for anyone looking to gain a deeper understanding of how Transformer-based AI models like GPT work, including: - Self-attention mechanisms - Encoder-decoder architecture - Positional encoding - Multi-head attention

THE BEST visual explainer of how information propagates through a transformer. If you want to have more than intuition about how the Transformer architecture is ruling the LLM world - open-source project explains everything about LLM Transformer Models! - A great resource for anyone looking to gain a deeper understanding of how Transformer-based AI models like GPT work, including: - Self-attention mechanisms - Encoder-decoder architecture - Positional encoding - Multi-head attention

Rohan Paul

106,880 次观看 • 1 年前

Autoencoder by hand✍️Excel~ I designed this exercise to show how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. I also made a medium version.👇Join the 'AI Math' community. Download xlsx.

Autoencoder by hand✍️Excel~ I designed this exercise to show how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. I also made a medium version.👇Join the 'AI Math' community. Download xlsx.

Tom Yeh

54,473 次观看 • 1 年前

Single vs Multi-hand Attention by hand ✍️ Resize matrices yourself 👉 The most important fact about multi-head attention: it has the same parameter count as single-head attention. The difference is purely structural — same total Wqkv weights, partitioned into smaller q–k–v triples. Look at the two diagrams below. Both Wqkv matrices have the same height — same number of weight rows, same number of parameters. What changes is how that single tall block is sliced. • Left. One head. The full Wqkv produces one big QKV: a tall Q (36 rows), a tall K, a tall V. One scoring computation runs over those full-width tensors. • Right. 3 heads. The same-height Wqkv is sliced into 3 smaller q–k–v triples — each 12 rows tall. 3 scoring computations run in parallel, each a thinner version of the left. The compute trade-off — kind of. Same Wqkv weights. Multi-head runs the attention scoring S = Kᵀ × Q once per head, so the dot-product count multiplies by H. • Single-head: seq × seq = 40² = 1600 dot products • Multi-head: seq × seq × H = 40² × 3 = 4800 dot products (3×) But each multi-head dot product is narrower — its inner dimension is head_dim instead of H × head_dim. So when you count actual scalar multiplications, the totals are equal: • Single-head: seq² × (H × head_dim) = 40² × 36 = 57600 • Multi-head: seq² × H × head_dim = 40² × 3 × 12 = 57600 Same FLOPs. Multi-head buys you H independent attention patterns at no extra weight cost and no extra arithmetic cost — it's the same total compute, sliced into H finer-grained heads.

Single vs Multi-hand Attention by hand ✍️ Resize matrices yourself 👉 The most important fact about multi-head attention: it has the same parameter count as single-head attention. The difference is purely structural — same total Wqkv weights, partitioned into smaller q–k–v triples. Look at the two diagrams below. Both Wqkv matrices have the same height — same number of weight rows, same number of parameters. What changes is how that single tall block is sliced. • Left. One head. The full Wqkv produces one big QKV: a tall Q (36 rows), a tall K, a tall V. One scoring computation runs over those full-width tensors. • Right. 3 heads. The same-height Wqkv is sliced into 3 smaller q–k–v triples — each 12 rows tall. 3 scoring computations run in parallel, each a thinner version of the left. The compute trade-off — kind of. Same Wqkv weights. Multi-head runs the attention scoring S = Kᵀ × Q once per head, so the dot-product count multiplies by H. • Single-head: seq × seq = 40² = 1600 dot products • Multi-head: seq × seq × H = 40² × 3 = 4800 dot products (3×) But each multi-head dot product is narrower — its inner dimension is head_dim instead of H × head_dim. So when you count actual scalar multiplications, the totals are equal: • Single-head: seq² × (H × head_dim) = 40² × 36 = 57600 • Multi-head: seq² × H × head_dim = 40² × 3 × 12 = 57600 Same FLOPs. Multi-head buys you H independent attention patterns at no extra weight cost and no extra arithmetic cost — it's the same total compute, sliced into H finer-grained heads.

Tom Yeh

35,046 次观看 • 1 个月前

Autoencoder by hand✍️Excel~ I designed this exercise to show how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. 👇Join the 'AI Math' community. Download xlsx.

Autoencoder by hand✍️Excel~ I designed this exercise to show how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. 👇Join the 'AI Math' community. Download xlsx.

Tom Yeh

101,555 次观看 • 1 年前

kling 3.0 is crazy... this model works very different than any other model this is from giving it just 2 images and a multi prompt then it handled all the scenes by itself... now imagine what it could do if you gave it more references and an even more detailed multi prompt going to test this a lot more but kling definetly cooked on this one

kling 3.0 is crazy... this model works very different than any other model this is from giving it just 2 images and a multi prompt then it handled all the scenes by itself... now imagine what it could do if you gave it more references and an even more detailed multi prompt going to test this a lot more but kling definetly cooked on this one

Miko

95,568 次观看 • 4 个月前

Self Attention vs Cross Attention by hand ✍️ Resize the matrices yourself 👉 Two attention mechanisms, side by side. Both project X into queries; both compute attention via S = Kᵀ × Q and F = V × A. The only difference is the source of K and V. Self attention uses X for everything. Q, K, and V all come from projecting X. Each X token attends to every other X token. The score matrix S is square — 128 × 128. Cross attention uses X for queries and a second sequence E for keys and values. Each X token attends to every E token instead. The score matrix S is rectangular — 64 × 128. Notice what's shared and what's not: X is the same in both — same 36 × 128 input. Q and K share the 16 dimension — that's what makes the dot product Kᵀ × Q valid in either case. V dimensions are independent: self-attention uses 12, cross-attention uses 12. The choice doesn't depend on which mechanism you're using; it depends on what output dimension your downstream layer expects.

Self Attention vs Cross Attention by hand ✍️ Resize the matrices yourself 👉 Two attention mechanisms, side by side. Both project X into queries; both compute attention via S = Kᵀ × Q and F = V × A. The only difference is the source of K and V. Self attention uses X for everything. Q, K, and V all come from projecting X. Each X token attends to every other X token. The score matrix S is square — 128 × 128. Cross attention uses X for queries and a second sequence E for keys and values. Each X token attends to every E token instead. The score matrix S is rectangular — 64 × 128. Notice what's shared and what's not: X is the same in both — same 36 × 128 input. Q and K share the 16 dimension — that's what makes the dot product Kᵀ × Q valid in either case. V dimensions are independent: self-attention uses 12, cross-attention uses 12. The choice doesn't depend on which mechanism you're using; it depends on what output dimension your downstream layer expects.

Tom Yeh

61,300 次观看 • 1 个月前

Arrive in style. Impeccable design and attention to detail – that’s what you can expect from the new E-Class. Discover more: #MercedesBenz #EClass

Arrive in style. Impeccable design and attention to detail – that’s what you can expect from the new E-Class. Discover more: #MercedesBenz #EClass

Mercedes-Benz

147,637 次观看 • 3 年前

everyone's been sharing this 'bouncing ball visualization' and having AIs recreate it - so for a fun challenge, I recreated it without *any* AI assistance. ill break down the math behind the visualization - and what makes it challenging for AI - in this thread:

everyone's been sharing this 'bouncing ball visualization' and having AIs recreate it - so for a fun challenge, I recreated it without any AI assistance. ill break down the math behind the visualization - and what makes it challenging for AI - in this thread:

N8 Programs

26,695 次观看 • 1 年前

Searching for relevancy or even attention makes one scratch their head. …Or is it that your behaviour precedes you and no one want’s to be associated with that level of toxicity.

Searching for relevancy or even attention makes one scratch their head. …Or is it that your behaviour precedes you and no one want’s to be associated with that level of toxicity.

Julius . D

89,716 次观看 • 1 年前

I am glad that more and more people around the world are beginning to pay attention to the issue in Gaza.

I am glad that more and more people around the world are beginning to pay attention to the issue in Gaza.

Roman Akramovich

21,609 次观看 • 8 个月前

How do you study head for art? It's the most challenging part of the human body and so easy to get wrong. This book simplifies the creation of head and neck through 3D and live model references, color-coded structures and more!

How do you study head for art? It's the most challenging part of the human body and so easy to get wrong. This book simplifies the creation of head and neck through 3D and live model references, color-coded structures and more!

Anatomy For Sculptors

466,107 次观看 • 2 年前

Can we make Transformers better and more efficient for robot learning? Excited to introduce Body Transformer (BoT), an architecture that leverages robot embodiment in the attention mechanism, by treating it as a graph of sensors and actuators.

Can we make Transformers better and more efficient for robot learning? Excited to introduce Body Transformer (BoT), an architecture that leverages robot embodiment in the attention mechanism, by treating it as a graph of sensors and actuators.

Carlo Sferrazza

60,663 次观看 • 1 年前

Down goes Inoue! Down goes Inoue! He makes it to the bell and we head to the third.

Down goes Inoue! Down goes Inoue! He makes it to the bell and we head to the third.

Spinnin Backfist

192,610 次观看 • 1 年前

Wow. This is one of the best interactive sites I’ve seen for learning how LLMs work! 🔥 It starts w/ a clear intro and guides you through every core component: from Embedding, Layer Norm, and Self-Attention to MLPs, Transformer blocks, Softmax, and Output layers. link in 🧵↓

Wow. This is one of the best interactive sites I’ve seen for learning how LLMs work! 🔥 It starts w/ a clear intro and guides you through every core component: from Embedding, Layer Norm, and Self-Attention to MLPs, Transformer blocks, Softmax, and Output layers. link in 🧵↓

Charly Wargnier

52,092 次观看 • 1 年前

#GhostofYotei when it’s raining in-game and you open the map, you can see the rain droplets hit the paper. attention to detail in this game is outstanding.

#GhostofYotei when it’s raining in-game and you open the map, you can see the rain droplets hit the paper. attention to detail in this game is outstanding.

that 🥷🏾 you love to hate

49,586 次观看 • 8 个月前

Engineer Eve here - looks like I made it through. The prototype works! Now is when the real work begins: to build and connect more Engineers across the multi-metaverse! Who's with me?

Engineer Eve here - looks like I made it through. The prototype works! Now is when the real work begins: to build and connect more Engineers across the multi-metaverse! Who's with me?

The Engineers NFT

13,519 次观看 • 3 年前

Perplexity Computer is massively multi-model. Computer orchestrates models to run agents in parallel, leveraging Opus to match each task to the model best suited for it. In total, Computer can route work across 19 different models.

Perplexity Computer is massively multi-model. Computer orchestrates models to run agents in parallel, leveraging Opus to match each task to the model best suited for it. In total, Computer can route work across 19 different models.

Perplexity

531,610 次观看 • 3 个月前

Suno Studio is coming... Multi-track creation. Export to MIDI. And much more. Join the waitlist now

Suno Studio is coming... Multi-track creation. Export to MIDI. And much more. Join the waitlist now

Suno

105,135 次观看 • 10 个月前

the fact that this is an actual, serious, question tells me that you guys need to pay more attention to the races. and this is just 2024

the fact that this is an actual, serious, question tells me that you guys need to pay more attention to the races. and this is just 2024

ela

68,324 次观看 • 8 个月前