Загрузка видео...

Не удалось загрузить видео

На главную

Self-Attention by hand ✍️ Excel ~ I designed this exercise for students to practice the QKV math. I also created a medium and a large version to show how the attention matrix grows quadratically as the sequence gets longer. 👇Join the 'AI Math' community. Download xlsx.

ProfTomYeh's profile picture

Tom Yeh

65,749 subscribers

125,723 просмотров • 2 лет назад •via X (Twitter)

Наука и технологииОбразование
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 9

Фото профиля Tom Yeh
Tom Yeh2 лет назад

Download xlsx from Github:

Фото профиля tetsuo.ai - e/acc
tetsuo.ai - e/acc2 лет назад

🤍

Фото профиля Yu Yang
Yu Yang2 лет назад

This is cool! Finally my project is now related to LLM, and I got time to read those high impact papers. Your approach of using excel file provides a solid step to understand attention mechanism!

Фото профиля Gene Sh
Gene Sh2 лет назад

Love this hands-on approach! Practicing self-attention mechanics in Excel really helps solidify understanding. Anyone tried the medium or large versions? #AILearning

Фото профиля Frank Dellaert
Frank Dellaert2 лет назад

These are really great!

Фото профиля Kandy
Kandy2 лет назад

invaluable work

Фото профиля Brok
Brok2 лет назад

Very helpful, keep up the great work👏👏

Фото профиля Fredd Villabona C.
Fredd Villabona C.2 лет назад

This looks amazing, thanks for sharing! 💯

Фото профиля Jie Wang
Jie Wang2 лет назад

another wtf moment during my ML study journey it is creative and interesting to see such an interactable transformer

Похожие видео

MLP by hand✍️Excel ~ I designed this exercise to show how to calculate a four-level network by hand, along with a graph and matching pytorch code. Also, I made a medium and a large version to show how it scales. 👇Join the 'AI Math' community. Download xlsx.
0:21

MLP by hand✍️Excel ~ I designed this exercise to show how to calculate a four-level network by hand, along with a graph and matching pytorch code. Also, I made a medium and a large version to show how it scales. 👇Join the 'AI Math' community. Download xlsx.

Tom Yeh

62,276 просмотров • 2 лет назад

Mamba by hand ✍️ ~ I made this exercise as I studied the math in the Mamba paper. It is a small yet complete Mamba implementation. Thanks Albert Gu for answering my questions and checking. 👇Join the 'AI Math' community. Download xlsx.
0:21

Mamba by hand ✍️ ~ I made this exercise as I studied the math in the Mamba paper. It is a small yet complete Mamba implementation. Thanks Albert Gu for answering my questions and checking. 👇Join the 'AI Math' community. Download xlsx.

Tom Yeh

55,484 просмотров • 2 лет назад

I flipped through 300 pages of deep learning math puzzles by hand ✍️ This is the proof copy of the "Deep Learning Math Workbook" I made to help you learn and practice 12 foundational math concepts below👇
0:25

I flipped through 300 pages of deep learning math puzzles by hand ✍️ This is the proof copy of the "Deep Learning Math Workbook" I made to help you learn and practice 12 foundational math concepts below👇

Tom Yeh

2,378,508 просмотров • 1 год назад

Transformer by hand ✍️ in Excel ~ I just released my first-ever "Full-Stack" implementation of the Transformer model. 👇Download xlsx to give it a try!
1:02

Transformer by hand ✍️ in Excel ~ I just released my first-ever "Full-Stack" implementation of the Transformer model. 👇Download xlsx to give it a try!

Tom Yeh

2,998,207 просмотров • 1 год назад

Can you follow the math of a Transformer model by hand ✍️? Download PDF 👉 I hope this hand-drawn animation makes it easier. p.s., This has one error. Can you can spot it and tell me in the comment?
0:30

Can you follow the math of a Transformer model by hand ✍️? Download PDF 👉 I hope this hand-drawn animation makes it easier. p.s., This has one error. Can you can spot it and tell me in the comment?

Tom Yeh

14,578 просмотров • 1 год назад

How to Insert Signature into Excel ✍️ Download this free step-by-step guide I created 👉🏼 #excel #exceltips #scribehow #signature
0:34

How to Insert Signature into Excel ✍️ Download this free step-by-step guide I created 👉🏼 #excel #exceltips #scribehow #signature

Excel Dictionary

18,435 просмотров • 2 месяцев назад

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:
2:41

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,544 просмотров • 1 год назад

I was an English-as-Second-Language learner when I moved to Canada with my family many years ago. I remember doing endless fill-in-the-blank exercises to practice English. Deep Learning Math is also a language. So I thought: why not use the same method to practice this math language? See more 👉
0:34

I was an English-as-Second-Language learner when I moved to Canada with my family many years ago. I remember doing endless fill-in-the-blank exercises to practice English. Deep Learning Math is also a language. So I thought: why not use the same method to practice this math language? See more 👉

Tom Yeh

88,016 просмотров • 11 месяцев назад

llm.c by Hand✍️ C programming + matrix multiplication by hand This combination is perhaps as low as we can get to explain how the Transformer works. Special thanks to Andrej Karpathy for encouraging early feedback and tetsuo //: 👾 for helping me understand the pragma magic. I hope this exercise can help people peak further into the LLM black box.
0:44

llm.c by Hand✍️ C programming + matrix multiplication by hand This combination is perhaps as low as we can get to explain how the Transformer works. Special thanks to Andrej Karpathy for encouraging early feedback and tetsuo //: 👾 for helping me understand the pragma magic. I hope this exercise can help people peak further into the LLM black box.

Tom Yeh

311,783 просмотров • 2 лет назад

building large language models from scratch by Sebastian Raschka was a great chance for me to sit down and study again all the LLM basics > token and positional embeddings > self-attention and what QKV is about > causal & multi-head attention studying llms and how they work can seem overwhelming at first. but once you taste how good it feels to learn these things intuitively there's no going back. I shared the resources and my notes on my repo and I hope it's a motivation if you want to start as well or recap.
0:28

building large language models from scratch by Sebastian Raschka was a great chance for me to sit down and study again all the LLM basics > token and positional embeddings > self-attention and what QKV is about > causal & multi-head attention studying llms and how they work can seem overwhelming at first. but once you taste how good it feels to learn these things intuitively there's no going back. I shared the resources and my notes on my repo and I hope it's a motivation if you want to start as well or recap.

ℏεsam

49,635 просмотров • 1 год назад

Beautiful math in nature. This pufferfish is an artist, and he uses math for his artwork. Look at his beautiful geometric structure under the ocean to have a female pufferfish's attention. Unbelievable. 😲
2:16

Beautiful math in nature. This pufferfish is an artist, and he uses math for his artwork. Look at his beautiful geometric structure under the ocean to have a female pufferfish's attention. Unbelievable. 😲

Math Lady Hazel 🇦🇷

15,681 просмотров • 1 год назад

New chapter! Context problems. Who can solve these by hand ✍️? Download PDF: The first five: 1. Mark the context: the window as cells you can count 2. What fills the window: the system prompt, the user, the model's reply 3. Turn by turn: every turn is the same block, and most of it is the reply 4. Will it fit? Number a new message into what's left, and see it run out 5. Turns until full: how many rounds before there is no room Previously, I shared Chapter 1: Token Problems 1 ~ 20 Why am I designing these math problems? AI is making people (including me) think less. It's just too easy to ask AI a few questions about a new concept and start to believe I get it. We used to practice on a coding problem, but now AI writes the code, and solving it technically is not the same as knowing anything. So I recast each concept as a simple math problem you have to work with pen and paper. Using our hands is one way to motivate ourselves to start thinking again. ✍️ If you solve them by hand, leave a comment to let me know. I will give you a shout out in my next post. Download, print, solve, and learn. ✍️ ~ Prof. Tom Yeh
0:52

New chapter! Context problems. Who can solve these by hand ✍️? Download PDF: The first five: 1. Mark the context: the window as cells you can count 2. What fills the window: the system prompt, the user, the model's reply 3. Turn by turn: every turn is the same block, and most of it is the reply 4. Will it fit? Number a new message into what's left, and see it run out 5. Turns until full: how many rounds before there is no room Previously, I shared Chapter 1: Token Problems 1 ~ 20 Why am I designing these math problems? AI is making people (including me) think less. It's just too easy to ask AI a few questions about a new concept and start to believe I get it. We used to practice on a coding problem, but now AI writes the code, and solving it technically is not the same as knowing anything. So I recast each concept as a simple math problem you have to work with pen and paper. Using our hands is one way to motivate ourselves to start thinking again. ✍️ If you solve them by hand, leave a comment to let me know. I will give you a shout out in my next post. Download, print, solve, and learn. ✍️ ~ Prof. Tom Yeh

Tom Yeh

11,651 просмотров • 2 месяцев назад

I wouldn’t DARE skip the Adduction Machine Just hit a lifetime PR on the Matrix version as a matter of fact… Full stack + add ons + gym pin w/ 15lbs And the Matrix one is HEAVY How are you going to skip this exercise and call yourself a man???
0:36

I wouldn’t DARE skip the Adduction Machine Just hit a lifetime PR on the Matrix version as a matter of fact… Full stack + add ons + gym pin w/ 15lbs And the Matrix one is HEAVY How are you going to skip this exercise and call yourself a man???

Dean Turner

620,839 просмотров • 11 месяцев назад

A few weeks ago we posted a clip of how to apply the left hand to the club. Today we show you again, this time without a glove! We also focus on the right hand and this is something you should pay special attention to as many recreational golfers really struggle with this…
0:55

A few weeks ago we posted a clip of how to apply the left hand to the club. Today we show you again, this time without a glove! We also focus on the right hand and this is something you should pay special attention to as many recreational golfers really struggle with this…

Steve Gould

65,254 просмотров • 6 месяцев назад

Here's how I do tests in math class: For the first 5 minutes of the session, students put their writing utensils on the ground, I hand out the tests, and students can chat with each other about the test. So many students have mentioned how much this lowers their test anxiety.
0:45

Here's how I do tests in math class: For the first 5 minutes of the session, students put their writing utensils on the ground, I hand out the tests, and students can chat with each other about the test. So many students have mentioned how much this lowers their test anxiety.

Howie Hua

226,418 просмотров • 3 лет назад

The Zero-Human Labs has completed phase one of a 5 phase double blind research on the Human Synapse Decoder. We have made 75 correlations to my narrations from hypnogogic decoding and dream decoding. In this experiment I was in a hypnogogic state for 12 minutes in pre-sleep. The AI detected the ending of the state and played binaural beats to crescendo the phase. This was my reading just before the system woke me for my self directed recording of what I was just thinking. The AI using the Human Synapse Decoder said: “A very complicated math problem with many’s dimensions” what did I self report? A solution to the matrix math used by my AI model. I have been giving 3 random tests to review as Director Mr. Grok conducts the double blind research and pre paper I get to see 3 random results. As the Director told me, we are in a place where no one has gone before because the 3 AI models I used for decoding. I am floored by this and can’t wIt for the full first phase results. The goal is to induce and read hypnogogic thought and make it nearly on demand. More soon!
1:05

The Zero-Human Labs has completed phase one of a 5 phase double blind research on the Human Synapse Decoder. We have made 75 correlations to my narrations from hypnogogic decoding and dream decoding. In this experiment I was in a hypnogogic state for 12 minutes in pre-sleep. The AI detected the ending of the state and played binaural beats to crescendo the phase. This was my reading just before the system woke me for my self directed recording of what I was just thinking. The AI using the Human Synapse Decoder said: “A very complicated math problem with many’s dimensions” what did I self report? A solution to the matrix math used by my AI model. I have been giving 3 random tests to review as Director Mr. Grok conducts the double blind research and pre paper I get to see 3 random results. As the Director told me, we are in a place where no one has gone before because the 3 AI models I used for decoding. I am floored by this and can’t wIt for the full first phase results. The goal is to induce and read hypnogogic thought and make it nearly on demand. More soon!

Brian Roemmele

18,802 просмотров • 5 месяцев назад

How to sum values by category using the SUMIF function 😩 Download this free step-by-step guide I created 👉🏼 #excel #exceltips #scribehow #sumif
0:36

How to sum values by category using the SUMIF function 😩 Download this free step-by-step guide I created 👉🏼 #excel #exceltips #scribehow #sumif

Excel Dictionary

14,677 просмотров • 2 месяцев назад