Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

THE BEST visual explainer of how information propagates through a transformer. If you want to have more than intuition about how the Transformer architecture is ruling the LLM world - open-source project explains everything about LLM Transformer Models! - A great resource for anyone looking to gain a deeper... show more

Rohan Paul

160,578 subscribers

107,001 görüntüleme • 2 yıl önce •via X (Twitter)

Bilim & Teknoloji Haberler & Politika Eğitim

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

How does a transformer work ?

How does a transformer work ?

Science Simplified

26,475 görüntüleme • 3 ay önce

Wow. This is one of the best interactive sites I’ve seen for learning how LLMs work! 🔥 It starts w/ a clear intro and guides you through every core component: from Embedding, Layer Norm, and Self-Attention to MLPs, Transformer blocks, Softmax, and Output layers. link in 🧵↓

Wow. This is one of the best interactive sites I’ve seen for learning how LLMs work! 🔥 It starts w/ a clear intro and guides you through every core component: from Embedding, Layer Norm, and Self-Attention to MLPs, Transformer blocks, Softmax, and Output layers. link in 🧵↓

Charly Wargnier

52,092 görüntüleme • 1 yıl önce

Transformer by hand✍️ Excel ~ I designed this exercise to show the core math of a Transformer model is to combine columns (attention), combine rows (feed forward), and repeat.👇Join the 'AI Math' community. 👇Download xlsx.

Transformer by hand✍️ Excel ~ I designed this exercise to show the core math of a Transformer model is to combine columns (attention), combine rows (feed forward), and repeat.👇Join the 'AI Math' community. 👇Download xlsx.

Tom Yeh

66,816 görüntüleme • 2 yıl önce

Can we make Transformers better and more efficient for robot learning? Excited to introduce Body Transformer (BoT), an architecture that leverages robot embodiment in the attention mechanism, by treating it as a graph of sensors and actuators.

Can we make Transformers better and more efficient for robot learning? Excited to introduce Body Transformer (BoT), an architecture that leverages robot embodiment in the attention mechanism, by treating it as a graph of sensors and actuators.

Carlo Sferrazza

60,714 görüntüleme • 2 yıl önce

Transformer: Multi-Head Attention ~ Math vs Code 🔢💻 ~ I made this visualization to show you how to implement the multi-head attention math in PyTorch within 50 LoC. Multi-Head Attention is what makes the Transformer's performance outstanding. It captures and represents more diverse linguistic relationships and patterns, and attends to different learned input embedding spaces. The parallel computing design also makes the model more efficient.

Transformer: Multi-Head Attention ~ Math vs Code 🔢💻 ~ I made this visualization to show you how to implement the multi-head attention math in PyTorch within 50 LoC. Multi-Head Attention is what makes the Transformer's performance outstanding. It captures and represents more diverse linguistic relationships and patterns, and attends to different learned input embedding spaces. The parallel computing design also makes the model more efficient.

Yan Chen

33,343 görüntüleme • 2 yıl önce

This is how transformer a boring wall

This is how transformer a boring wall

Interior design ideas

133,964 görüntüleme • 1 ay önce

How do you like my transformer?

How do you like my transformer?

Ilya Dreamy

36,219 görüntüleme • 8 ay önce

🚨 This is the best way to learn how LLMs work. Interactive. 3D. Step-by-step. Covers: → Embedding → Layer Norm → Self-Attention → MLP → Transformer layers → Softmax → Output Stop reading papers. Start seeing. Link in comments. Save this immediately.

🚨 This is the best way to learn how LLMs work. Interactive. 3D. Step-by-step. Covers: → Embedding → Layer Norm → Self-Attention → MLP → Transformer layers → Softmax → Output Stop reading papers. Start seeing. Link in comments. Save this immediately.

Tech with Mak

82,419 görüntüleme • 8 ay önce

The rise and fall of popular transformer architectures 📈 One pro of having a lot of transformer architectures implemented in transformers & hosted on the Hugging Face Hub is that one can see the evolution of popularity of different models 👀

The rise and fall of popular transformer architectures 📈 One pro of having a lot of transformer architectures implemented in transformers & hosted on the Hugging Face Hub is that one can see the evolution of popularity of different models 👀

Lysandre

31,744 görüntüleme • 3 yıl önce

We’re releasing model weights for our 8B- parameter Dynamic Byte Latent Transformer, an alternative to traditional tokenization methods with the potential to redefine the standards for language model efficiency and reliability. Learn more about how Dynamic Byte Latent Transformer is paving the way for groundbreaking developments in the field of language modeling. Read the research paper, and download the model and code.

We’re releasing model weights for our 8B- parameter Dynamic Byte Latent Transformer, an alternative to traditional tokenization methods with the potential to redefine the standards for language model efficiency and reliability. Learn more about how Dynamic Byte Latent Transformer is paving the way for groundbreaking developments in the field of language modeling. Read the research paper, and download the model and code.

AI at Meta

195,705 görüntüleme • 1 yıl önce

I feel like people don't talk about how America still knows how to build stuff. Hitachi Energy just announced a $528M transformer factory in Mississippi. They say that'll lead to creating more than 700 jobs and more than double its transformer production capacity in the area. And this isn't just some random product. Transformers are a critical piece of the electrical grid. That's the kind of "Made in America" story people can actually point to. A real factory, with real equipment and real jobs. Build in America.

I feel like people don't talk about how America still knows how to build stuff. Hitachi Energy just announced a $528M transformer factory in Mississippi. They say that'll lead to creating more than 700 jobs and more than double its transformer production capacity in the area. And this isn't just some random product. Transformers are a critical piece of the electrical grid. That's the kind of "Made in America" story people can actually point to. A real factory, with real equipment and real jobs. Build in America.

Defiant L’s

43,330 görüntüleme • 1 ay önce

Transformer by hand ✍️ ~ 6 steps walkthrough below Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features *across positions*, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens *across feature dimensions*, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes *across positions*, so a feature can see its neighbours. The FFN mixes *across feature dimensions*, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearning

Transformer by hand ✍️ ~ 6 steps walkthrough below Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features across positions, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens across feature dimensions, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes across positions, so a feature can see its neighbours. The FFN mixes across feature dimensions, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearning

Tom Yeh

26,279 görüntüleme • 3 ay önce

Check this!! A 100% open-source toolkit to work with LLMs. Transformer Lab is an app to experiment with LLMs: - Train, fine-tune, or chat. - One-click LLM download (DeepSeek, Gemma, etc.) - Drag-n-drop UI for RAG. - Built-in logging, and more. 100% local!

Check this!! A 100% open-source toolkit to work with LLMs. Transformer Lab is an app to experiment with LLMs: - Train, fine-tune, or chat. - One-click LLM download (DeepSeek, Gemma, etc.) - Drag-n-drop UI for RAG. - Built-in logging, and more. 100% local!

Avi Chawla

74,961 görüntüleme • 1 yıl önce

Part 2/30 of the LLM Series: RoPE (Rotary Position Embedding) How does a transformer know the difference between - "the dog bit the man" and "the man bit the dog"? The words are almost identical, but the meaning changes completely. RoPE encodes position as rotation, allowing transformers to understand relative order through geometry. Read more:

Part 2/30 of the LLM Series: RoPE (Rotary Position Embedding) How does a transformer know the difference between - "the dog bit the man" and "the man bit the dog"? The words are almost identical, but the meaning changes completely. RoPE encodes position as rotation, allowing transformers to understand relative order through geometry. Read more:

TensorTonic

11,766 görüntüleme • 5 ay önce

Day 15 of learning AI bs from scratch > looked into seq2seq models > understood the encoder-decoder architecture > implemented a basic Seq2Seq model using RNNs > how the decoder generates outputs one step at a time > read couple more chapters of > understood the fixed-context bottleneck I updated the resource list blog, link below, it has more resources organized now. Fell asleep on the desk towards the morning, rweeting rn after waking up.

Day 15 of learning AI bs from scratch > looked into seq2seq models > understood the encoder-decoder architecture > implemented a basic Seq2Seq model using RNNs > how the decoder generates outputs one step at a time > read couple more chapters of > understood the fixed-context bottleneck I updated the resource list blog, link below, it has more resources organized now. Fell asleep on the desk towards the morning, rweeting rn after waking up.

Arham Amin

31,853 görüntüleme • 3 ay önce

Get ready for the PDE-Transformer: our new NN architecture tailored to scientific tasks. It combines hierarchical processing (UDiT), scalability (SWin) and flexible conditioning mechanisms. The paper shows it outperforming existing SOTA architectures 😁

Get ready for the PDE-Transformer: our new NN architecture tailored to scientific tasks. It combines hierarchical processing (UDiT), scalability (SWin) and flexible conditioning mechanisms. The paper shows it outperforming existing SOTA architectures 😁

Thuerey Group at TUM

13,422 görüntüleme • 1 yıl önce

The best Tesla costume. Transformer.

The best Tesla costume. Transformer.

Tesla Owners Silicon Valley

57,978 görüntüleme • 1 yıl önce

$Transformer and Mixture of Experts, explained visually! Mixture of Experts (MoE) is a popular architecture that uses different experts to improve Transformer models. Transformer and MoE differ in the decoder block: - Transformer uses a feed-forward network. - MoE uses experts, which are feed-forward networks but smaller compared to those Transformer. During inference, a subset of experts are selected. This makes inference faster in MoE. Also, since the network has multiple decoder layers: - The text passes through different experts across layers. - The chosen experts also differ between tokens. But how does the model decide which experts should be ideal? The router does that. It is a multi-class classifier that produces softmax scores over experts to select the top K experts. The router is trained with the network, and it learns to select the best experts. But it isn't straightforward. There are challenges! Challenge 1) Notice this pattern at the start of training: - Say, the model selects "Expert 2" - This expert gets a bit better - It may get selected again since it's the "best" - It learns more - It gets selected again in the next iteration - It learns more, and so on! This means many experts can go under-trained due to the overselection of a few experts! We solve this in two steps: - Add noise to the feed-forward output of the router so that other experts can get higher logits. - Set all but the top K logits to -infinity. After softmax, these scores become zero. This way, other experts also get the opportunity to train. Challenge 2) Some experts may get exposed to more tokens than others, leading to under-trained experts. We prevent this by limiting the number of tokens an expert can process. If an expert reaches the limit, the token is passed to the next best expert. Overall, MoEs have more parameters to load. But a fraction of them are activated during inference. This leads to faster inference. Mixtral 8x7B and Llama 4 are two popular MoE-based LLMs. Have you used MoEs in production yet? To dive deeper into how MoE inference works in production, we wrote a full article covering token dispatch, grouped expert computation, model-weight memory, multi-GPU communication, expert placement, load imbalance, and performance diagnosis. Read it below.$

Transformer and Mixture of Experts, explained visually! Mixture of Experts (MoE) is a popular architecture that uses different experts to improve Transformer models. Transformer and MoE differ in the decoder block: - Transformer uses a feed-forward network. - MoE uses experts, which are feed-forward networks but smaller compared to those Transformer. During inference, a subset of experts are selected. This makes inference faster in MoE. Also, since the network has multiple decoder layers: - The text passes through different experts across layers. - The chosen experts also differ between tokens. But how does the model decide which experts should be ideal? The router does that. It is a multi-class classifier that produces softmax scores over experts to select the top K experts. The router is trained with the network, and it learns to select the best experts. But it isn't straightforward. There are challenges! Challenge 1) Notice this pattern at the start of training: - Say, the model selects "Expert 2" - This expert gets a bit better - It may get selected again since it's the "best" - It learns more - It gets selected again in the next iteration - It learns more, and so on! This means many experts can go under-trained due to the overselection of a few experts! We solve this in two steps: - Add noise to the feed-forward output of the router so that other experts can get higher logits. - Set all but the top K logits to -infinity. After softmax, these scores become zero. This way, other experts also get the opportunity to train. Challenge 2) Some experts may get exposed to more tokens than others, leading to under-trained experts. We prevent this by limiting the number of tokens an expert can process. If an expert reaches the limit, the token is passed to the next best expert. Overall, MoEs have more parameters to load. But a fraction of them are activated during inference. This leads to faster inference. Mixtral 8x7B and Llama 4 are two popular MoE-based LLMs. Have you used MoEs in production yet? To dive deeper into how MoE inference works in production, we wrote a full article covering token dispatch, grouped expert computation, model-weight memory, multi-GPU communication, expert placement, load imbalance, and performance diagnosis. Read it below.

Daily Dose of Data Science

55,651 görüntüleme • 1 ay önce

Transformers & LLMs cheatsheets for Stanford's CME-295! Covering tokenization, self-attention, prompting, fine-tuning, LLM-as-a-judge, RAG, AI Agents, and reasoning models. 100% free and open-source.

Transformers & LLMs cheatsheets for Stanford's CME-295! Covering tokenization, self-attention, prompting, fine-tuning, LLM-as-a-judge, RAG, AI Agents, and reasoning models. 100% free and open-source.

Akshay 🚀

101,701 görüntüleme • 1 yıl önce