正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Self-Attention by hand ✍️ Excel ~ I designed this exercise for students to practice the QKV math. I also created a medium and a large version to show how the attention matrix grows quadratically as the sequence gets longer. 👇Join the 'AI Math' community. Download xlsx.

Tom Yeh

55,958 subscribers

125,595 次观看 • 1 年前 •via X (Twitter)

科学技术教育

Anya Rossi• Live Now

Private livecam show

9 条评论

Tom Yeh 的头像

Tom Yeh1 年前

Download xlsx from Github:

tetsuo.ai - e/acc 的头像

tetsuo.ai - e/acc1 年前

🤍

Yu Yang 的头像

Yu Yang1 年前

This is cool! Finally my project is now related to LLM, and I got time to read those high impact papers. Your approach of using excel file provides a solid step to understand attention mechanism!

Gene Sh 的头像

Gene Sh1 年前

Love this hands-on approach! Practicing self-attention mechanics in Excel really helps solidify understanding. Anyone tried the medium or large versions? #AILearning

Frank Dellaert 的头像

Frank Dellaert1 年前

These are really great!

Kandy 的头像

Kandy1 年前

invaluable work

Brok 的头像

Brok1 年前

Very helpful, keep up the great work👏👏

Fredd Villabona C. 的头像

Fredd Villabona C.1 年前

This looks amazing, thanks for sharing! 💯

Jie Wang 的头像

Jie Wang1 年前

another wtf moment during my ML study journey it is creative and interesting to see such an interactable transformer

相关视频

MLP by hand✍️Excel ~ I designed this exercise to show how to calculate a four-level network by hand, along with a graph and matching pytorch code. Also, I made a medium and a large version to show how it scales. 👇Join the 'AI Math' community. Download xlsx.

MLP by hand✍️Excel ~ I designed this exercise to show how to calculate a four-level network by hand, along with a graph and matching pytorch code. Also, I made a medium and a large version to show how it scales. 👇Join the 'AI Math' community. Download xlsx.

Tom Yeh

62,205 次观看 • 1 年前

Mamba by hand ✍️ ~ I made this exercise as I studied the math in the Mamba paper. It is a small yet complete Mamba implementation. Thanks Albert Gu for answering my questions and checking. 👇Join the 'AI Math' community. Download xlsx.

Mamba by hand ✍️ ~ I made this exercise as I studied the math in the Mamba paper. It is a small yet complete Mamba implementation. Thanks Albert Gu for answering my questions and checking. 👇Join the 'AI Math' community. Download xlsx.

Tom Yeh

55,438 次观看 • 1 年前

Temperature 🌡️ by hand ✍️ ~ I designed this Excel exercise to help you study how temperature influences the sampling process of a language model. Download xlsx 👇

Temperature 🌡️ by hand ✍️ ~ I designed this Excel exercise to help you study how temperature influences the sampling process of a language model. Download xlsx 👇

Tom Yeh

18,685 次观看 • 1 年前

I flipped through 300 pages of deep learning math puzzles by hand ✍️ This is the proof copy of the "Deep Learning Math Workbook" I made to help you learn and practice 12 foundational math concepts below👇

I flipped through 300 pages of deep learning math puzzles by hand ✍️ This is the proof copy of the "Deep Learning Math Workbook" I made to help you learn and practice 12 foundational math concepts below👇

Tom Yeh

2,378,310 次观看 • 1 年前

Transformer by hand ✍️ in Excel ~ I just released my first-ever "Full-Stack" implementation of the Transformer model. 👇Download xlsx to give it a try!

Transformer by hand ✍️ in Excel ~ I just released my first-ever "Full-Stack" implementation of the Transformer model. 👇Download xlsx to give it a try!

Tom Yeh

2,997,355 次观看 • 1 年前

Multihead Attention by hand ✍️ in Excel ~ Download ⬇️

Multihead Attention by hand ✍️ in Excel ~ Download ⬇️

Tom Yeh

49,542 次观看 • 1 年前

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,135 次观看 • 1 年前

I was an English-as-Second-Language learner when I moved to Canada with my family many years ago. I remember doing endless fill-in-the-blank exercises to practice English. Deep Learning Math is also a language. So I thought: why not use the same method to practice this math language? See more 👉

I was an English-as-Second-Language learner when I moved to Canada with my family many years ago. I remember doing endless fill-in-the-blank exercises to practice English. Deep Learning Math is also a language. So I thought: why not use the same method to practice this math language? See more 👉

Tom Yeh

87,792 次观看 • 6 个月前

This is how a designer at Figma designed AI features for Figma Slides The amount of design iterations and the attention to details is mind blowing Natasha Tenggoro shows she designed the AI features for Figma Slides in Figma 👇

This is how a designer at Figma designed AI features for Figma Slides The amount of design iterations and the attention to details is mind blowing Natasha Tenggoro shows she designed the AI features for Figma Slides in Figma 👇

Jayneil Dalal

20,714 次观看 • 1 年前

llm.c by Hand✍️ C programming + matrix multiplication by hand This combination is perhaps as low as we can get to explain how the Transformer works. Special thanks to Andrej Karpathy for encouraging early feedback and tetsuo //: 👾 for helping me understand the pragma magic. I hope this exercise can help people peak further into the LLM black box.

llm.c by Hand✍️ C programming + matrix multiplication by hand This combination is perhaps as low as we can get to explain how the Transformer works. Special thanks to Andrej Karpathy for encouraging early feedback and tetsuo //: 👾 for helping me understand the pragma magic. I hope this exercise can help people peak further into the LLM black box.

Tom Yeh

302,625 次观看 • 2 年前

building large language models from scratch by Sebastian Raschka was a great chance for me to sit down and study again all the LLM basics > token and positional embeddings > self-attention and what QKV is about > causal & multi-head attention studying llms and how they work can seem overwhelming at first. but once you taste how good it feels to learn these things intuitively there's no going back. I shared the resources and my notes on my repo and I hope it's a motivation if you want to start as well or recap.

building large language models from scratch by Sebastian Raschka was a great chance for me to sit down and study again all the LLM basics > token and positional embeddings > self-attention and what QKV is about > causal & multi-head attention studying llms and how they work can seem overwhelming at first. but once you taste how good it feels to learn these things intuitively there's no going back. I shared the resources and my notes on my repo and I hope it's a motivation if you want to start as well or recap.

ℏεsam

49,292 次观看 • 1 年前

Q: What is the best way to prove you know AI math? A: Show them you solved these 300 math exercises by hand ✍, just like Reginaldo Cunha did.

Q: What is the best way to prove you know AI math? A: Show them you solved these 300 math exercises by hand ✍, just like Reginaldo Cunha did.

Tom Yeh

83,361 次观看 • 1 年前

Beautiful math in nature. This pufferfish is an artist, and he uses math for his artwork. Look at his beautiful geometric structure under the ocean to have a female pufferfish's attention. Unbelievable. 😲

Beautiful math in nature. This pufferfish is an artist, and he uses math for his artwork. Look at his beautiful geometric structure under the ocean to have a female pufferfish's attention. Unbelievable. 😲

Math Lady Hazel 🇦🇷

15,681 次观看 • 1 年前

I wouldn’t DARE skip the Adduction Machine Just hit a lifetime PR on the Matrix version as a matter of fact… Full stack + add ons + gym pin w/ 15lbs And the Matrix one is HEAVY How are you going to skip this exercise and call yourself a man???

I wouldn’t DARE skip the Adduction Machine Just hit a lifetime PR on the Matrix version as a matter of fact… Full stack + add ons + gym pin w/ 15lbs And the Matrix one is HEAVY How are you going to skip this exercise and call yourself a man???

Dean Turner

620,199 次观看 • 6 个月前

A few weeks ago we posted a clip of how to apply the left hand to the club. Today we show you again, this time without a glove! We also focus on the right hand and this is something you should pay special attention to as many recreational golfers really struggle with this…

A few weeks ago we posted a clip of how to apply the left hand to the club. Today we show you again, this time without a glove! We also focus on the right hand and this is something you should pay special attention to as many recreational golfers really struggle with this…

Steve Gould

65,062 次观看 • 1 个月前

Here's how I do tests in math class: For the first 5 minutes of the session, students put their writing utensils on the ground, I hand out the tests, and students can chat with each other about the test. So many students have mentioned how much this lowers their test anxiety.

Here's how I do tests in math class: For the first 5 minutes of the session, students put their writing utensils on the ground, I hand out the tests, and students can chat with each other about the test. So many students have mentioned how much this lowers their test anxiety.

Howie Hua

226,363 次观看 • 2 年前

The Zero-Human Labs has completed phase one of a 5 phase double blind research on the Human Synapse Decoder. We have made 75 correlations to my narrations from hypnogogic decoding and dream decoding. In this experiment I was in a hypnogogic state for 12 minutes in pre-sleep. The AI detected the ending of the state and played binaural beats to crescendo the phase. This was my reading just before the system woke me for my self directed recording of what I was just thinking. The AI using the Human Synapse Decoder said: “A very complicated math problem with many’s dimensions” what did I self report? A solution to the matrix math used by my AI model. I have been giving 3 random tests to review as Director Mr. Grok conducts the double blind research and pre paper I get to see 3 random results. As the Director told me, we are in a place where no one has gone before because the 3 AI models I used for decoding. I am floored by this and can’t wIt for the full first phase results. The goal is to induce and read hypnogogic thought and make it nearly on demand. More soon!

The Zero-Human Labs has completed phase one of a 5 phase double blind research on the Human Synapse Decoder. We have made 75 correlations to my narrations from hypnogogic decoding and dream decoding. In this experiment I was in a hypnogogic state for 12 minutes in pre-sleep. The AI detected the ending of the state and played binaural beats to crescendo the phase. This was my reading just before the system woke me for my self directed recording of what I was just thinking. The AI using the Human Synapse Decoder said: “A very complicated math problem with many’s dimensions” what did I self report? A solution to the matrix math used by my AI model. I have been giving 3 random tests to review as Director Mr. Grok conducts the double blind research and pre paper I get to see 3 random results. As the Director told me, we are in a place where no one has gone before because the 3 AI models I used for decoding. I am floored by this and can’t wIt for the full first phase results. The goal is to induce and read hypnogogic thought and make it nearly on demand. More soon!

Brian Roemmele

18,802 次观看 • 1 个月前

Last week, we launched "Attention in Transformers: Concepts and Code in PyTorch" instructed by Joshua Starmer! In this course, you'll: ✅ Learn how the attention mechanism in LLMs helps convert base token embeddings into rich context-aware embeddings. ✅ Understand the Query, Key, and Value matrices, what they are for, how to produce them, and how to use them in attention. ✅ Learn the difference between self-attention, masked self-attention, and cross-attention, and how multi-head attention scales the algorithm. 🔗 Enroll for free:

Last week, we launched "Attention in Transformers: Concepts and Code in PyTorch" instructed by Joshua Starmer! In this course, you'll: ✅ Learn how the attention mechanism in LLMs helps convert base token embeddings into rich context-aware embeddings. ✅ Understand the Query, Key, and Value matrices, what they are for, how to produce them, and how to use them in attention. ✅ Learn the difference between self-attention, masked self-attention, and cross-attention, and how multi-head attention scales the algorithm. 🔗 Enroll for free:

DeepLearning.AI

36,832 次观看 • 1 年前

In tonight’s family friendly music lesson I show how to write a sad song. The math behind it and emotions. Here is the “I am Sam” test where I show that the lyrics don’t matter as much as people think. (Lyrics are usually to propagandize) Over 50 lessons!

In tonight’s family friendly music lesson I show how to write a sad song. The math behind it and emotions. Here is the “I am Sam” test where I show that the lyrics don’t matter as much as people think. (Lyrics are usually to propagandize) Over 50 lessons!

Owen Benjamin 🐻

26,008 次观看 • 2 个月前

Matrix Multiplication on GPU/TPU by hand ✍️ I drew 91 frames to show how to divide large matrices into "tiles" to use accelerators such as GPU/TPU. If you like to learn more, check out our weekly workshops. Link 👇

Matrix Multiplication on GPU/TPU by hand ✍️ I drew 91 frames to show how to divide large matrices into "tiles" to use accelerators such as GPU/TPU. If you like to learn more, check out our weekly workshops. Link 👇

Tom Yeh

77,774 次观看 • 11 个月前