Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Self-Attention vs Cross-Attention interactive diagram. Open

ProfTomYeh's profile picture

Tom Yeh

65,344 subscribers

20,878 Aufrufe • vor 3 Tagen •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Last week, we launched "Attention in Transformers: Concepts and Code in PyTorch" instructed by Joshua Starmer! In this course, you'll: ✅ Learn how the attention mechanism in LLMs helps convert base token embeddings into rich context-aware embeddings. ✅ Understand the Query, Key, and Value matrices, what they are for, how to produce them, and how to use them in attention. ✅ Learn the difference between self-attention, masked self-attention, and cross-attention, and how multi-head attention scales the algorithm. 🔗 Enroll for free:
1:09

Last week, we launched "Attention in Transformers: Concepts and Code in PyTorch" instructed by Joshua Starmer! In this course, you'll: ✅ Learn how the attention mechanism in LLMs helps convert base token embeddings into rich context-aware embeddings. ✅ Understand the Query, Key, and Value matrices, what they are for, how to produce them, and how to use them in attention. ✅ Learn the difference between self-attention, masked self-attention, and cross-attention, and how multi-head attention scales the algorithm. 🔗 Enroll for free:

DeepLearning.AI

36,832 Aufrufe • vor 1 Jahr

Seedance 2.0 is impressive. But it's closed-source! Introducing our daVinci-MagiHuman — a single-stream 15B Transformer trained from scratch that jointly generates video + audio. No cross-attention. No multi-stream branches. Just self-attention. ⚡ 5s 1080p video in 38s on a single H100 🏆 80% win rate vs Ovi 1.1 | 60.9% vs LTX 2.3 (2,000 human comparisons) 🌍 6 languages 📦 Fully open-source Speed by simplicity. By @SII_GAIR × Sand.ai 📄 💻 🤗
2:03

Seedance 2.0 is impressive. But it's closed-source! Introducing our daVinci-MagiHuman — a single-stream 15B Transformer trained from scratch that jointly generates video + audio. No cross-attention. No multi-stream branches. Just self-attention. ⚡ 5s 1080p video in 38s on a single H100 🏆 80% win rate vs Ovi 1.1 | 60.9% vs LTX 2.3 (2,000 human comparisons) 🌍 6 languages 📦 Fully open-source Speed by simplicity. By @SII_GAIR × Sand.ai 📄 💻 🤗

Pengfei Liu

300,272 Aufrufe • vor 5 Monaten

self-attention explained by 3blue1brown's new video
1:51

self-attention explained by 3blue1brown's new video

ℏεsam

48,349 Aufrufe • vor 1 Jahr

🚨 ATTENTION 🚨 GAVIN CROSS HAS NOW HOMERED IN FIVE STRAIGHT GAMES!
0:22

🚨 ATTENTION 🚨 GAVIN CROSS HAS NOW HOMERED IN FIVE STRAIGHT GAMES!

Omaha Storm Chasers

83,236 Aufrufe • vor 1 Monat

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:
2:41

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,400 Aufrufe • vor 1 Jahr

Motions that grab attention. 📩 open for work!
0:21

Motions that grab attention. 📩 open for work!

ARIA

12,439 Aufrufe • vor 8 Monaten

Motions that grab attention. 📩 open for work!
0:25

Motions that grab attention. 📩 open for work!

ARIA

16,382 Aufrufe • vor 8 Monaten

Pay attention to how Andrew Tate responds to this question… SELF HYPNOSIS…⚡️
0:40

Pay attention to how Andrew Tate responds to this question… SELF HYPNOSIS…⚡️

Tyson

41,673 Aufrufe • vor 9 Monaten

Pay deeper attention to the people around you, stop being so self absorbed.
0:56

Pay deeper attention to the people around you, stop being so self absorbed.

Robert Greene

223,238 Aufrufe • vor 3 Jahren

Pretending to self-harm yourself in prison for attention. The lowest is what?
0:37

Sensitive content

Pretending to self-harm yourself in prison for attention. The lowest is what?

Boochie is the Name

94,205 Aufrufe • vor 1 Jahr

Nick Aldis (c) vs Cody Rhodes NWA World Heavyweight Championship All In 2018 🎶 Attention Attention - Shinedown 🎶 Cody Rhodes Nick Aldis
3:06

Nick Aldis (c) vs Cody Rhodes NWA World Heavyweight Championship All In 2018 🎶 Attention Attention - Shinedown 🎶 Cody Rhodes Nick Aldis

CV Highlights

104,958 Aufrufe • vor 1 Monat

Pay attention when people are offended by the cross. It’s really all you need to know.
2:13

Pay attention when people are offended by the cross. It’s really all you need to know.

TheLizVariant

29,775 Aufrufe • vor 7 Monaten

A lady intentionally exposed herself while dancing. Attention-seeking or self-expression? What's your take?
1:21

A lady intentionally exposed herself while dancing. Attention-seeking or self-expression? What's your take?

Ella Nnabugo

16,459 Aufrufe • vor 1 Monat

Easily create image maps on WordPress with the Draw Attention plugin. Immerse your audience with interactive content! Visit the demo below.
0:29

Easily create image maps on WordPress with the Draw Attention plugin. Immerse your audience with interactive content! Visit the demo below.

WP Draw Attention

8,384,147 Aufrufe • vor 11 Monaten

Drake vs UMG is bigger then Kendrick vs Drake . Yall dot fans relax & pay attention
1:27

Drake vs UMG is bigger then Kendrick vs Drake . Yall dot fans relax & pay attention

lifestyle

20,324 Aufrufe • vor 10 Monaten

Attention Innovators💡 Applications for National Startup Awards 2023 are open. 👉
0:32

Attention Innovators💡 Applications for National Startup Awards 2023 are open. 👉

Piyush Goyal

586,545 Aufrufe • vor 3 Jahren

Turn any GitHub repository into a visual treat! Simply replace "hub" with "diagram" in a GitHub URL and instantly view the entire codebase as interactive diagrams for easier understanding. 100% open-source.
0:21

Turn any GitHub repository into a visual treat! Simply replace "hub" with "diagram" in a GitHub URL and instantly view the entire codebase as interactive diagrams for easier understanding. 100% open-source.

Akshay 🚀

125,479 Aufrufe • vor 1 Jahr

Attention Attention
0:40

Attention Attention

Zzizinga Dickson ™

253,482 Aufrufe • vor 3 Jahren

Attention! Attention!
1:24

Sensitive content

Attention! Attention!

Marceline Cameron

1,070,616 Aufrufe • vor 1 Monat