Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Wow. This is one of the best interactive sites I’ve seen for learning how LLMs work! 🔥 It starts w/ a clear intro and guides you through every core component: from Embedding, Layer Norm, and Self-Attention to MLPs, Transformer blocks, Softmax, and Output layers. link in 🧵↓

Charly Wargnier

153,980 subscribers

52,092 views • 1 year ago •via X (Twitter)

Science & Technology News & Politics Education

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

🚨 This is the best way to learn how LLMs work. Interactive. 3D. Step-by-step. Covers: → Embedding → Layer Norm → Self-Attention → MLP → Transformer layers → Softmax → Output Stop reading papers. Start seeing. Link in comments. Save this immediately.

🚨 This is the best way to learn how LLMs work. Interactive. 3D. Step-by-step. Covers: → Embedding → Layer Norm → Self-Attention → MLP → Transformer layers → Softmax → Output Stop reading papers. Start seeing. Link in comments. Save this immediately.

Tech with Mak

82,419 views • 7 months ago

THE BEST visual explainer of how information propagates through a transformer. If you want to have more than intuition about how the Transformer architecture is ruling the LLM world - open-source project explains everything about LLM Transformer Models! - A great resource for anyone looking to gain a deeper understanding of how Transformer-based AI models like GPT work, including: - Self-attention mechanisms - Encoder-decoder architecture - Positional encoding - Multi-head attention

THE BEST visual explainer of how information propagates through a transformer. If you want to have more than intuition about how the Transformer architecture is ruling the LLM world - open-source project explains everything about LLM Transformer Models! - A great resource for anyone looking to gain a deeper understanding of how Transformer-based AI models like GPT work, including: - Self-attention mechanisms - Encoder-decoder architecture - Positional encoding - Multi-head attention

Rohan Paul

106,897 views • 2 years ago

Transformer by hand ✍️ ~ 6 steps walkthrough below Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features *across positions*, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens *across feature dimensions*, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes *across positions*, so a feature can see its neighbours. The FFN mixes *across feature dimensions*, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearning

Transformer by hand ✍️ ~ 6 steps walkthrough below Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features across positions, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens across feature dimensions, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes across positions, so a feature can see its neighbours. The FFN mixes across feature dimensions, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearning

Tom Yeh

26,211 views • 2 months ago

Bit of a BOOKISH NEWSFLASH for you… For the last five years I’ve been researching and writing a book I’ve poured everything I’ve got in to it, and then some It is about this insanely beautiful place… and the people who live and work in it

Bit of a BOOKISH NEWSFLASH for you… For the last five years I’ve been researching and writing a book I’ve poured everything I’ve got in to it, and then some It is about this insanely beautiful place… and the people who live and work in it

James Rebanks

148,500 views • 2 years ago

Transformer: Multi-Head Attention ~ Math vs Code 🔢💻 ~ I made this visualization to show you how to implement the multi-head attention math in PyTorch within 50 LoC. Multi-Head Attention is what makes the Transformer's performance outstanding. It captures and represents more diverse linguistic relationships and patterns, and attends to different learned input embedding spaces. The parallel computing design also makes the model more efficient.

Transformer: Multi-Head Attention ~ Math vs Code 🔢💻 ~ I made this visualization to show you how to implement the multi-head attention math in PyTorch within 50 LoC. Multi-Head Attention is what makes the Transformer's performance outstanding. It captures and represents more diverse linguistic relationships and patterns, and attends to different learned input embedding spaces. The parallel computing design also makes the model more efficient.

Yan Chen

33,326 views • 2 years ago

- WUTHERING WAVES NEWS - I’ve officially done it! Guides for every single Wuthering Waves character are done! It took a lot of work but excited to jump in and play tomorrow! I hope you guys enjoy them all and thank you for all the support 🥰❤️ #WutheringWaves

- WUTHERING WAVES NEWS - I’ve officially done it! Guides for every single Wuthering Waves character are done! It took a lot of work but excited to jump in and play tomorrow! I hope you guys enjoy them all and thank you for all the support 🥰❤️ #WutheringWaves

SipSipStefen

51,623 views • 2 years ago

FIVE LAYERS OF AGENT ENGINEERING, EACH ONE WRAPS THE ONE BELOW IT. IF YOU SKIP LAYER 2, YOUR LAYER 5 WILL LOOK BROKEN WHEN IT IS ACTUALLY JUST STANDING ON NOTHING. for weeks i debated harness vs loop vs graph like they were competing choices. then a stack diagram made the shape obvious. they are not choices. they are floors. 01 | prompt engineering. the message. unit of work: one input. inputs are role, instructions, examples, format. output is a single raw response. 02 | context engineering. the memory. unit of work: what stays in the window. a curator selects, compresses, and drops from query, docs, memory, prior turns, and tool outputs before the prompt runs. 03 | harness engineering. the machine. unit of work: the machine itself. gather (context + prompt) → LLM → tools or sub-agents → verifier → final response. the article calls this the operating environment. 04 | loop engineering. the system. unit of work: the run. goal + success criteria + max iterations + budget + completion check wrap around one harness pass. failed pass appends results to context and retries. 05 | graph engineering. the topology. unit of work: the graph run. goal + nodes + edges + state schema. graph routes to agent nodes, tool nodes, or human approval. a reviewer node with a different model and fresh context checks the final answer. the wrapping is the whole point. layer 5 assumes layer 4 works. layer 4 assumes layer 3 works. skip layer 2 and layer 3's verifier keeps failing without a clear reason. this is why swapping the model is a one-day project and swapping the stack is a quarter. the model is the commodity. the five layers around it are the engineering. full three-layer breakdown of the top of the stack (harness, loop, graph) in the post below.

FIVE LAYERS OF AGENT ENGINEERING, EACH ONE WRAPS THE ONE BELOW IT. IF YOU SKIP LAYER 2, YOUR LAYER 5 WILL LOOK BROKEN WHEN IT IS ACTUALLY JUST STANDING ON NOTHING. for weeks i debated harness vs loop vs graph like they were competing choices. then a stack diagram made the shape obvious. they are not choices. they are floors. 01 | prompt engineering. the message. unit of work: one input. inputs are role, instructions, examples, format. output is a single raw response. 02 | context engineering. the memory. unit of work: what stays in the window. a curator selects, compresses, and drops from query, docs, memory, prior turns, and tool outputs before the prompt runs. 03 | harness engineering. the machine. unit of work: the machine itself. gather (context + prompt) → LLM → tools or sub-agents → verifier → final response. the article calls this the operating environment. 04 | loop engineering. the system. unit of work: the run. goal + success criteria + max iterations + budget + completion check wrap around one harness pass. failed pass appends results to context and retries. 05 | graph engineering. the topology. unit of work: the graph run. goal + nodes + edges + state schema. graph routes to agent nodes, tool nodes, or human approval. a reviewer node with a different model and fresh context checks the final answer. the wrapping is the whole point. layer 5 assumes layer 4 works. layer 4 assumes layer 3 works. skip layer 2 and layer 3's verifier keeps failing without a clear reason. this is why swapping the model is a one-day project and swapping the stack is a quarter. the model is the commodity. the five layers around it are the engineering. full three-layer breakdown of the top of the stack (harness, loop, graph) in the post below.

kocer

31,198 views • 1 month ago

I can't believe how good this turned out. Recently, I put out my first physics paper proposing that life and the universe might be the result of a law governing how oscillators sync up and couple together. I find it beautiful, awe-inspiring, and deeply meaningful. It aligns with how I have always seen the world. It suggests that there is an arrow of negative entropy, a law governing how life forms in all of its complexity. Learning about it is intimidating. I am self-taught, and have struggled for months to understand and articulate the core concepts and framework. After I put out the paper, I was challenged by Danielle Fong 🔆 to make an 'explorable explanation'. A guide you can walk through, and which will hold your hand through each concept, equation, and principle. It was amazing to make. In fact, I plan to make many more covering the wide range of topics I have been researching and exploring. Both to help me, and because I believe that this is something the world needs right now. Link in comments

I can't believe how good this turned out. Recently, I put out my first physics paper proposing that life and the universe might be the result of a law governing how oscillators sync up and couple together. I find it beautiful, awe-inspiring, and deeply meaningful. It aligns with how I have always seen the world. It suggests that there is an arrow of negative entropy, a law governing how life forms in all of its complexity. Learning about it is intimidating. I am self-taught, and have struggled for months to understand and articulate the core concepts and framework. After I put out the paper, I was challenged by Danielle Fong 🔆 to make an 'explorable explanation'. A guide you can walk through, and which will hold your hand through each concept, equation, and principle. It was amazing to make. In fact, I plan to make many more covering the wide range of topics I have been researching and exploring. Both to help me, and because I believe that this is something the world needs right now. Link in comments

Parzival - ∞/89

23,155 views • 5 months ago

One of the greatest keys to power and consistency in the golf swing is learning how to properly rotate through the ball. Initiating the downswing is one of the most important tools in consistent golf. It is something I have worked on virtually every day of my life, from hitting balls as a young boy in South Africa to competing against the best players in the world. GP #PXGTroops

One of the greatest keys to power and consistency in the golf swing is learning how to properly rotate through the ball. Initiating the downswing is one of the most important tools in consistent golf. It is something I have worked on virtually every day of my life, from hitting balls as a young boy in South Africa to competing against the best players in the world. GP #PXGTroops

GARY PLAYER

51,687 views • 4 months ago

Full Fine-tuning vs. Freezing Layers. Interact 👉 and == Full Fine-tuning == A real network has many — three layers in this example, billions of parameters in a production model. What does fine-tuning look like when you update all of them? That’s full fine-tuning: continue training every weight in the pretrained network on your new task. Every layer’s W gets its own ΔW. Nothing is frozen — every parameter is in play. Think of an MLP as a chain of prerequisites leading to an advanced course. Layer 1 might be Linear Algebra, layer 2 Probability, layer 3 Advanced Machine Learning — each one building on what came before. Fine-tuning is what happens during graduate study: the foundations are already there from undergrad, so you’re not re-learning. Full fine-tuning is reviewing every prerequisite to see what new topics have appeared and what discoveries the field has made since the last time you sat through them. Effective — but exhausting. This diagram shows the same three-layer MLP twice, side by side. On the left, the pretrained network runs on input X: three weight matrices W₁, W₂, W₃, each followed by a ReLU activation. Full fine-tuning gives the model the most freedom to specialize. Every parameter can move — and every parameter that can move must be stored. But not every prerequisite needs revisiting. The further you go back in the chain, the less the material has changed since pretraining — the linear-algebra basics under your computer-vision course are largely the same as they ever were. The next page does exactly that: freeze the prerequisites that haven’t moved, and only refresh the advanced one closest to your specialization. == Freezing Layers == Full fine-tuning reviewed every prerequisite — Linear Algebra, Probability, Advanced ML — to refresh each subject with the latest topics. Effective, but exhausting. Then you realize something. The prerequisites haven’t actually changed that much. Linear Algebra is still Linear Algebra; the matrix decompositions you learned still hold. Probability is still Probability; the distributions and Bayes’ rule haven’t moved. Almost all the new material — the new ideas, the recent discoveries — lives in the advanced layer at the top. That’s freezing layers: keep the prerequisite layers fixed at their pretrained state, and only update the advanced one. In the diagram below, W1 and W2 — the foundational prerequisites — stay frozen. Only W3 — the layer closest to your task-specific output — gets a ΔW.

Full Fine-tuning vs. Freezing Layers. Interact 👉 and == Full Fine-tuning == A real network has many — three layers in this example, billions of parameters in a production model. What does fine-tuning look like when you update all of them? That’s full fine-tuning: continue training every weight in the pretrained network on your new task. Every layer’s W gets its own ΔW. Nothing is frozen — every parameter is in play. Think of an MLP as a chain of prerequisites leading to an advanced course. Layer 1 might be Linear Algebra, layer 2 Probability, layer 3 Advanced Machine Learning — each one building on what came before. Fine-tuning is what happens during graduate study: the foundations are already there from undergrad, so you’re not re-learning. Full fine-tuning is reviewing every prerequisite to see what new topics have appeared and what discoveries the field has made since the last time you sat through them. Effective — but exhausting. This diagram shows the same three-layer MLP twice, side by side. On the left, the pretrained network runs on input X: three weight matrices W₁, W₂, W₃, each followed by a ReLU activation. Full fine-tuning gives the model the most freedom to specialize. Every parameter can move — and every parameter that can move must be stored. But not every prerequisite needs revisiting. The further you go back in the chain, the less the material has changed since pretraining — the linear-algebra basics under your computer-vision course are largely the same as they ever were. The next page does exactly that: freeze the prerequisites that haven’t moved, and only refresh the advanced one closest to your specialization. == Freezing Layers == Full fine-tuning reviewed every prerequisite — Linear Algebra, Probability, Advanced ML — to refresh each subject with the latest topics. Effective, but exhausting. Then you realize something. The prerequisites haven’t actually changed that much. Linear Algebra is still Linear Algebra; the matrix decompositions you learned still hold. Probability is still Probability; the distributions and Bayes’ rule haven’t moved. Almost all the new material — the new ideas, the recent discoveries — lives in the advanced layer at the top. That’s freezing layers: keep the prerequisite layers fixed at their pretrained state, and only update the advanced one. In the diagram below, W1 and W2 — the foundational prerequisites — stay frozen. Only W3 — the layer closest to your task-specific output — gets a ΔW.

Tom Yeh

27,740 views • 5 months ago

Someone sent this in from Elrow. I’ve been to a few of these myself in Ibiza. One in the one of the warehouses along the docks too. Probably the best club night in the world. All the stage starts coming alive and it all starts coming towards you. People on stilts. It’s mad. Crazy to watch when you are twisted. I’d highly recommend it

Someone sent this in from Elrow. I’ve been to a few of these myself in Ibiza. One in the one of the warehouses along the docks too. Probably the best club night in the world. All the stage starts coming alive and it all starts coming towards you. People on stilts. It’s mad. Crazy to watch when you are twisted. I’d highly recommend it

Around Liverpool

32,677 views • 4 months ago

Self Attention vs Cross Attention by hand ✍️ Resize the matrices yourself 👉 Two attention mechanisms, side by side. Both project X into queries; both compute attention via S = Kᵀ × Q and F = V × A. The only difference is the source of K and V. Self attention uses X for everything. Q, K, and V all come from projecting X. Each X token attends to every other X token. The score matrix S is square — 128 × 128. Cross attention uses X for queries and a second sequence E for keys and values. Each X token attends to every E token instead. The score matrix S is rectangular — 64 × 128. Notice what's shared and what's not: X is the same in both — same 36 × 128 input. Q and K share the 16 dimension — that's what makes the dot product Kᵀ × Q valid in either case. V dimensions are independent: self-attention uses 12, cross-attention uses 12. The choice doesn't depend on which mechanism you're using; it depends on what output dimension your downstream layer expects.

Self Attention vs Cross Attention by hand ✍️ Resize the matrices yourself 👉 Two attention mechanisms, side by side. Both project X into queries; both compute attention via S = Kᵀ × Q and F = V × A. The only difference is the source of K and V. Self attention uses X for everything. Q, K, and V all come from projecting X. Each X token attends to every other X token. The score matrix S is square — 128 × 128. Cross attention uses X for queries and a second sequence E for keys and values. Each X token attends to every E token instead. The score matrix S is rectangular — 64 × 128. Notice what's shared and what's not: X is the same in both — same 36 × 128 input. Q and K share the 16 dimension — that's what makes the dot product Kᵀ × Q valid in either case. V dimensions are independent: self-attention uses 12, cross-attention uses 12. The choice doesn't depend on which mechanism you're using; it depends on what output dimension your downstream layer expects.

Tom Yeh

62,508 views • 5 months ago

March 18, 2025 marked the public launch of OptimAI. In one year, it has evolved from a lightweight node layer into a decentralized intelligence infrastructure powering real-time data, compute, and reinforcement for agentic systems. Not just nodes. Not just data. A continuously learning, network-driven intelligence layer. This is infrastructure for a new class of software: autonomous agents that persist, adapt, and operate across environments. Year one established the network. Year two is where it compounds into coordination and value flow. Personal agents. Reinforcement at network scale. Emerging primitives for AgentFi. New layers coming online. 2026 won’t just be about scale, it’s where the network starts to operate. Keep building!

March 18, 2025 marked the public launch of OptimAI. In one year, it has evolved from a lightweight node layer into a decentralized intelligence infrastructure powering real-time data, compute, and reinforcement for agentic systems. Not just nodes. Not just data. A continuously learning, network-driven intelligence layer. This is infrastructure for a new class of software: autonomous agents that persist, adapt, and operate across environments. Year one established the network. Year two is where it compounds into coordination and value flow. Personal agents. Reinforcement at network scale. Emerging primitives for AgentFi. New layers coming online. 2026 won’t just be about scale, it’s where the network starts to operate. Keep building!

OptimAI Network

34,462 views • 6 months ago

This has to be one of the best workout videos I’ve seen - and yeah, I’m officially claiming it for the gays 🏳️‍🌈🔥 Who’s coming with me for the next workout routine?👇🏿😁

This has to be one of the best workout videos I’ve seen - and yeah, I’m officially claiming it for the gays 🏳️‍🌈🔥 Who’s coming with me for the next workout routine?👇🏿😁

Sl!m

24,444 views • 4 months ago

In this Korty’s interview with Asake, this part stood out for me. Asake, while prompting his car’s voice assistant to play him one of his songs, first spoke with a slight accent. But the voice assistant couldn’t understand him and he quickly switched to his original, African self. This, coupled with the usage of the Yoruba language in the interview, shows that to cement your place in the world and get the admiration of the entire world, you must be your original self. This is one of the best interviews I’ve seen and I encourage everyone to go see it as well. God bless all the creatives behind this.

In this Korty’s interview with Asake, this part stood out for me. Asake, while prompting his car’s voice assistant to play him one of his songs, first spoke with a slight accent. But the voice assistant couldn’t understand him and he quickly switched to his original, African self. This, coupled with the usage of the Yoruba language in the interview, shows that to cement your place in the world and get the admiration of the entire world, you must be your original self. This is one of the best interviews I’ve seen and I encourage everyone to go see it as well. God bless all the creatives behind this.

Ayo-Elesho

162,519 views • 6 months ago

This was us haha. Sreejith PP Arpit Saxena and I built Andrej Karpathy-fy! It takes a paperswithcode link, your current proficiency and gives you a tailored 5 lesson plan to go from where you are to the meaty parts of the paper implementation - like the karpathy video series / labml style. This is a task none of the previous models could do, and O1 was really good at. The best part? The output is an executable ipynb notebook (which you can use minusxai to work through :) ) The below video has apple's recent monocular depth paper. I could actually go through and understand it, with loss functions from the paper and everything!

This was us haha. Sreejith PP Arpit Saxena and I built Andrej Karpathy-fy! It takes a paperswithcode link, your current proficiency and gives you a tailored 5 lesson plan to go from where you are to the meaty parts of the paper implementation - like the karpathy video series / labml style. This is a task none of the previous models could do, and O1 was really good at. The best part? The output is an executable ipynb notebook (which you can use minusxai to work through :) ) The below video has apple's recent monocular depth paper. I could actually go through and understand it, with loss functions from the paper and everything!

Vivek Aithal

200,848 views • 2 years ago

gpt-image-2 + Seedance just took AI realism to another level. I've been testing this combo for a few days. The output is genuinely hard to tell apart from real footage. And the best part? I automated the entire workflow. One click. Same quality every time. rt + Comment "workflow" and I'll DM you the link.

gpt-image-2 + Seedance just took AI realism to another level. I've been testing this combo for a few days. The output is genuinely hard to tell apart from real footage. And the best part? I automated the entire workflow. One click. Same quality every time. rt + Comment "workflow" and I'll DM you the link.

Ahmad

43,992 views • 5 months ago