Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Self-Attention vs Cross-Attention interactive diagram. Open

Tom Yeh

65,671 subscribers

17,417 views • 24 days ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

1 Comments

Ahmad Mujtaba23 days ago

This is great ❤️

Related Videos

Last week, we launched "Attention in Transformers: Concepts and Code in PyTorch" instructed by Joshua Starmer! In this course, you'll: ✅ Learn how the attention mechanism in LLMs helps convert base token embeddings into rich context-aware embeddings. ✅ Understand the Query, Key, and Value matrices, what they are for, how to produce them, and how to use them in attention. ✅ Learn the difference between self-attention, masked self-attention, and cross-attention, and how multi-head attention scales the algorithm. 🔗 Enroll for free:

Last week, we launched "Attention in Transformers: Concepts and Code in PyTorch" instructed by Joshua Starmer! In this course, you'll: ✅ Learn how the attention mechanism in LLMs helps convert base token embeddings into rich context-aware embeddings. ✅ Understand the Query, Key, and Value matrices, what they are for, how to produce them, and how to use them in attention. ✅ Learn the difference between self-attention, masked self-attention, and cross-attention, and how multi-head attention scales the algorithm. 🔗 Enroll for free:

DeepLearning.AI

36,832 views • 1 year ago

Seedance 2.0 is impressive. But it's closed-source! Introducing our daVinci-MagiHuman — a single-stream 15B Transformer trained from scratch that jointly generates video + audio. No cross-attention. No multi-stream branches. Just self-attention. ⚡ 5s 1080p video in 38s on a single H100 🏆 80% win rate vs Ovi 1.1 | 60.9% vs LTX 2.3 (2,000 human comparisons) 🌍 6 languages 📦 Fully open-source Speed by simplicity. By @SII_GAIR × Sand.ai 📄 💻 🤗

Seedance 2.0 is impressive. But it's closed-source! Introducing our daVinci-MagiHuman — a single-stream 15B Transformer trained from scratch that jointly generates video + audio. No cross-attention. No multi-stream branches. Just self-attention. ⚡ 5s 1080p video in 38s on a single H100 🏆 80% win rate vs Ovi 1.1 | 60.9% vs LTX 2.3 (2,000 human comparisons) 🌍 6 languages 📦 Fully open-source Speed by simplicity. By @SII_GAIR × Sand.ai 📄 💻 🤗

Pengfei Liu

301,178 views • 6 months ago

self-attention explained by 3blue1brown's new video

self-attention explained by 3blue1brown's new video

ℏεsam

48,349 views • 1 year ago

🚨 ATTENTION 🚨 GAVIN CROSS HAS NOW HOMERED IN FIVE STRAIGHT GAMES!

🚨 ATTENTION 🚨 GAVIN CROSS HAS NOW HOMERED IN FIVE STRAIGHT GAMES!

Omaha Storm Chasers

83,236 views • 2 months ago

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,544 views • 1 year ago

Motions that grab attention. 📩 open for work!

Motions that grab attention. 📩 open for work!

ARIA

12,439 views • 9 months ago

Motions that grab attention. 📩 open for work!

Motions that grab attention. 📩 open for work!

ARIA

16,450 views • 10 months ago

Pretending to self-harm yourself in prison for attention. The lowest is what?

Sensitive content

Pretending to self-harm yourself in prison for attention. The lowest is what?

Boochie Myers🔪

94,245 views • 1 year ago

Pay deeper attention to the people around you, stop being so self absorbed.

Pay deeper attention to the people around you, stop being so self absorbed.

Robert Greene

223,238 views • 3 years ago

Nick Aldis (c) vs Cody Rhodes NWA World Heavyweight Championship All In 2018 🎶 Attention Attention - Shinedown 🎶 Cody Rhodes Nick Aldis

Nick Aldis (c) vs Cody Rhodes NWA World Heavyweight Championship All In 2018 🎶 Attention Attention - Shinedown 🎶 Cody Rhodes Nick Aldis

CV Highlights

104,958 views • 2 months ago

Pay attention when people are offended by the cross. It’s really all you need to know.

Pay attention when people are offended by the cross. It’s really all you need to know.

TheLizVariant

29,775 views • 8 months ago

A lady intentionally exposed herself while dancing. Attention-seeking or self-expression? What's your take?

A lady intentionally exposed herself while dancing. Attention-seeking or self-expression? What's your take?

Ella Nnabugo

16,494 views • 3 months ago

Easily create image maps on WordPress with the Draw Attention plugin. Immerse your audience with interactive content! Visit the demo below.

Easily create image maps on WordPress with the Draw Attention plugin. Immerse your audience with interactive content! Visit the demo below.

WP Draw Attention

8,384,147 views • 1 year ago

Attention Innovators💡 Applications for National Startup Awards 2023 are open. 👉

Attention Innovators💡 Applications for National Startup Awards 2023 are open. 👉

Piyush Goyal

586,545 views • 3 years ago

Turn any GitHub repository into a visual treat! Simply replace "hub" with "diagram" in a GitHub URL and instantly view the entire codebase as interactive diagrams for easier understanding. 100% open-source.

Turn any GitHub repository into a visual treat! Simply replace "hub" with "diagram" in a GitHub URL and instantly view the entire codebase as interactive diagrams for easier understanding. 100% open-source.

Akshay 🚀

125,479 views • 1 year ago

Found my usual spot at Kings Cross… Only this time… I gave the table leg some attention 😏

Found my usual spot at Kings Cross… Only this time… I gave the table leg some attention 😏

Chloe ✨

11,738 views • 1 month ago