Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Self-Attention vs Cross-Attention interactive diagram. Open

ProfTomYeh's profile picture

Tom Yeh

65,702 subscribers

21,228 Aufrufe • vor 2 Monaten •via X (Twitter)

Wissenschaft & TechnologieBildung
LIVE

Anya Rossi• Live Now

Private livecam show

1 Kommentare

Profilbild von Rajsuthan Gopinath
Rajsuthan Gopinathvor 2 Monaten

Thanks for sharing…

Ähnliche Videos

Last week, we launched "Attention in Transformers: Concepts and Code in PyTorch" instructed by Joshua Starmer! In this course, you'll: ✅ Learn how the attention mechanism in LLMs helps convert base token embeddings into rich context-aware embeddings. ✅ Understand the Query, Key, and Value matrices, what they are for, how to produce them, and how to use them in attention. ✅ Learn the difference between self-attention, masked self-attention, and cross-attention, and how multi-head attention scales the algorithm. 🔗 Enroll for free:
1:09

Last week, we launched "Attention in Transformers: Concepts and Code in PyTorch" instructed by Joshua Starmer! In this course, you'll: ✅ Learn how the attention mechanism in LLMs helps convert base token embeddings into rich context-aware embeddings. ✅ Understand the Query, Key, and Value matrices, what they are for, how to produce them, and how to use them in attention. ✅ Learn the difference between self-attention, masked self-attention, and cross-attention, and how multi-head attention scales the algorithm. 🔗 Enroll for free:

DeepLearning.AI

36,832 Aufrufe • vor 1 Jahr

Seedance 2.0 is impressive. But it's closed-source! Introducing our daVinci-MagiHuman — a single-stream 15B Transformer trained from scratch that jointly generates video + audio. No cross-attention. No multi-stream branches. Just self-attention. ⚡ 5s 1080p video in 38s on a single H100 🏆 80% win rate vs Ovi 1.1 | 60.9% vs LTX 2.3 (2,000 human comparisons) 🌍 6 languages 📦 Fully open-source Speed by simplicity. By @SII_GAIR × Sand.ai 📄 💻 🤗
2:03

Seedance 2.0 is impressive. But it's closed-source! Introducing our daVinci-MagiHuman — a single-stream 15B Transformer trained from scratch that jointly generates video + audio. No cross-attention. No multi-stream branches. Just self-attention. ⚡ 5s 1080p video in 38s on a single H100 🏆 80% win rate vs Ovi 1.1 | 60.9% vs LTX 2.3 (2,000 human comparisons) 🌍 6 languages 📦 Fully open-source Speed by simplicity. By @SII_GAIR × Sand.ai 📄 💻 🤗

Pengfei Liu

301,372 Aufrufe • vor 7 Monaten

self-attention explained by 3blue1brown's new video
1:51

self-attention explained by 3blue1brown's new video

ℏεsam

48,349 Aufrufe • vor 2 Jahren

🚨 ATTENTION 🚨 GAVIN CROSS HAS NOW HOMERED IN FIVE STRAIGHT GAMES!
0:22

🚨 ATTENTION 🚨 GAVIN CROSS HAS NOW HOMERED IN FIVE STRAIGHT GAMES!

Omaha Storm Chasers

83,236 Aufrufe • vor 3 Monaten

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:
2:41

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,792 Aufrufe • vor 1 Jahr

Motions that grab attention. 📩 open for work!
0:21

Motions that grab attention. 📩 open for work!

ARIA

12,439 Aufrufe • vor 10 Monaten

Motions that grab attention. 📩 open for work!
0:25

Motions that grab attention. 📩 open for work!

ARIA

16,450 Aufrufe • vor 11 Monaten

Pretending to self-harm yourself in prison for attention. The lowest is what?
0:37

Sensitive content

Pretending to self-harm yourself in prison for attention. The lowest is what?

Boochie Myers🔪

94,245 Aufrufe • vor 1 Jahr

Pay deeper attention to the people around you, stop being so self absorbed.
0:56

Pay deeper attention to the people around you, stop being so self absorbed.

Robert Greene

223,238 Aufrufe • vor 3 Jahren

Nick Aldis (c) vs Cody Rhodes NWA World Heavyweight Championship All In 2018 🎶 Attention Attention - Shinedown 🎶 Cody Rhodes Nick Aldis
3:06

Nick Aldis (c) vs Cody Rhodes NWA World Heavyweight Championship All In 2018 🎶 Attention Attention - Shinedown 🎶 Cody Rhodes Nick Aldis

CV Highlights

104,958 Aufrufe • vor 3 Monaten

Pay attention when people are offended by the cross. It’s really all you need to know.
2:13

Pay attention when people are offended by the cross. It’s really all you need to know.

TheLizVariant

29,775 Aufrufe • vor 9 Monaten

A lady intentionally exposed herself while dancing. Attention-seeking or self-expression? What's your take?
1:21

Sensitive content

A lady intentionally exposed herself while dancing. Attention-seeking or self-expression? What's your take?

Ella Nnabugo

16,494 Aufrufe • vor 4 Monaten

Easily create image maps on WordPress with the Draw Attention plugin. Immerse your audience with interactive content! Visit the demo below.
0:29

Easily create image maps on WordPress with the Draw Attention plugin. Immerse your audience with interactive content! Visit the demo below.

WP Draw Attention

9,283,360 Aufrufe • vor 1 Jahr

The song John would play to get attention at open mics😝
6:30

The song John would play to get attention at open mics😝

Priscilla | I LOVE TOM AND JERRY BY JOHN MAYER

23,253 Aufrufe • vor 13 Tagen

Attention Innovators💡 Applications for National Startup Awards 2023 are open. 👉
0:32

Attention Innovators💡 Applications for National Startup Awards 2023 are open. 👉

Piyush Goyal

586,545 Aufrufe • vor 3 Jahren

Feels like your attention is running away from you too often? Learn how to call it back and take charge of it with The Self-Command Reset! Cart is now open.
1:22

Feels like your attention is running away from you too often? Learn how to call it back and take charge of it with The Self-Command Reset! Cart is now open.

Command Within | Build Self-Command

15,724 Aufrufe • vor 1 Monat

Turn any GitHub repository into a visual treat! Simply replace "hub" with "diagram" in a GitHub URL and instantly view the entire codebase as interactive diagrams for easier understanding. 100% open-source.
0:21

Turn any GitHub repository into a visual treat! Simply replace "hub" with "diagram" in a GitHub URL and instantly view the entire codebase as interactive diagrams for easier understanding. 100% open-source.

Akshay 🚀

125,479 Aufrufe • vor 1 Jahr