Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

You can literally type a sentence and watch a Transformer process it. This interactive explainer runs a GPT-style model in the browser and lets you follow the whole path: tokenization → embeddings → Q/K/V → attention → MLP → logits → next-token probabilities You can inspect attention maps, intermediate... show more

Tech with Mak

48,156 subscribers

134,999 views • 20 days ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

21 Comments

Sam20 days ago

this visualizer is insane. actually seeing attention weights shift live makes the math click way faster than reading a paper

景哥｜AI落地与智能体实践20 days ago

浏览器里交互式演示 Transformer 推理全过程，可看注意力图、调整采样参数。能把抽象的 Transformer 架构，变成可实时观察的完整链路。

Sapafy.com20 days ago

5.6GB RAM?

Rob Parker20 days ago

Super cool! Thanks

Aditya | AI Infra19 days ago

Great @techNmak 🙌🏼 This'll definitely help people to understand how transformer architecture works internally.

anish19 days ago

typing a sentence and watching a transformer process it is such a good way to learn attention lol. does it show each head separately?

George20 days ago

The masked self attention is really clear but I think you could make the multiple heads clearer, perhaps show some depth with slight 3d rather than a 2d plane for each head.

realMattAbrams19 days ago

Looks awesome. Told my daughter that LLMs are essentially giant guided plinko machines. A bit of a oversimplification, but your explainer shows it action. Thanks for sharing it!

Aapakari20 days ago

How faithful is the browser model to a real GPT?

Daniel Plays19 days ago

the "watch" framing oversells it. you see static snapshots, not the actual compute happening.

晗BaiHoo20 days ago

wow amazing

Brjan | AI Builder20 days ago

that hands-on approach will really help demystify how transformers work

Harold Finch20 days ago

Would love to try

BitAres20 days ago

First experiment I'd try: swap one word and compare the attention maps.

Stevo Changa20 days ago

Nice clean looks

Trillionaire on Transit20 days ago

Insane visualisation

Eleftheria Batsou19 days ago

Thank you for sharing this!! Have a great Sunday

Adrian Hsu19 days ago

the MLP step is where the diagram version always lost me. poking temperature and watching the next-token probs move is the useful part.

René Cannaò20 days ago

I'd start by changing one word and comparing the attention maps.

Joey D19 days ago

You guys make this for DVA at GT?

Crio Songo20 days ago

Nice, been wanting to inspect attention maps without spinning up a notebook. Browser-based is clutch.

Related Videos

Stop reading attention is all you need. Transformers (LLMs) clearly explained with visuals: - Real GPT-2 visualizer - tokens to embeddings - 12 attention heads (Q, K, V, scale, mask, softmax) - multi-head Q/K/V - residual + MLP - next-token probabilities That’s the difference between reading about transformers and seeing one think. -

Stop reading attention is all you need. Transformers (LLMs) clearly explained with visuals: - Real GPT-2 visualizer - tokens to embeddings - 12 attention heads (Q, K, V, scale, mask, softmax) - multi-head Q/K/V - residual + MLP - next-token probabilities That’s the difference between reading about transformers and seeing one think. -

Md Ismail Šojal 🕷️

23,695 views • 1 month ago

Transformer Explainer Really cool interactive tool to learn about the inner workings of a Transformer model. Apparently, it runs a GPT-2 instance locally in the user's browser and allows you to experiment with your own inputs. This is a nice tool to learn more about the different components inside the Transformer and the transformations that occur. Tool:

Transformer Explainer Really cool interactive tool to learn about the inner workings of a Transformer model. Apparently, it runs a GPT-2 instance locally in the user's browser and allows you to experiment with your own inputs. This is a nice tool to learn more about the different components inside the Transformer and the transformations that occur. Tool:

elvis

122,014 views • 2 years ago

Attention is a lookup. Each token builds a query, compares it against every key in the sequence, and pulls value vectors weighted by the match. Stack that 96 layers deep and you get a frontier model. Video covers the full pipeline: Q/K/V, attention scores, encoder blocks.

Attention is a lookup. Each token builds a query, compares it against every key in the sequence, and pulls value vectors weighted by the match. Stack that 96 layers deep and you get a frontier model. Video covers the full pipeline: Q/K/V, attention scores, encoder blocks.

tetsuo

106,557 views • 4 months ago

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,544 views • 1 year ago

New course: Transformers in Practice. You'll get a practical view of how transformer-based LLMs work, so you can reason about their behavior, diagnose problems like slow inference, and make smarter decisions about deployment. This course is built in partnership with AMD and taught by Sharon Zhou. You'll see how transformers generate text one token at a time, how the model decides which earlier words matter most when predicting the next one, and how techniques like quantization speed up inference on GPUs. This is not a video-only course; interactive visualizations throughout let you play with these concepts and build intuition that sticks. Skills you'll gain: - Understand why LLMs hallucinate, and RAG and chain-of-thought shape what they generate - Look inside the model to see how attention and layers combine to predict the next token - Diagnose inference bottlenecks and learn the techniques that speed up transformers on GPUs Join and understand what's really happening inside your LLMs:

New course: Transformers in Practice. You'll get a practical view of how transformer-based LLMs work, so you can reason about their behavior, diagnose problems like slow inference, and make smarter decisions about deployment. This course is built in partnership with AMD and taught by Sharon Zhou. You'll see how transformers generate text one token at a time, how the model decides which earlier words matter most when predicting the next one, and how techniques like quantization speed up inference on GPUs. This is not a video-only course; interactive visualizations throughout let you play with these concepts and build intuition that sticks. Skills you'll gain: - Understand why LLMs hallucinate, and RAG and chain-of-thought shape what they generate - Look inside the model to see how attention and layers combine to predict the next token - Diagnose inference bottlenecks and learn the techniques that speed up transformers on GPUs Join and understand what's really happening inside your LLMs:

Andrew Ng

126,461 views • 5 months ago

Transformer Explainer Interactive Learning of Text-Generative Models discuss: Transformers have revolutionized machine learning, yet their inner workings remain opaque to many. We present Transformer Explainer, an interactive visualization tool designed for non-experts to learn about Transformers through the GPT-2 model. Our tool helps users understand complex Transformer concepts by integrating a model overview and enabling smooth transitions across abstraction levels of mathematical operations and model structures. It runs a live GPT-2 instance locally in the user's browser, empowering users to experiment with their own input and observe in real-time how the internal components and parameters of the Transformer work together to predict the next tokens. Our tool requires no installation or special hardware, broadening the public's education access to modern generative AI techniques.

Transformer Explainer Interactive Learning of Text-Generative Models discuss: Transformers have revolutionized machine learning, yet their inner workings remain opaque to many. We present Transformer Explainer, an interactive visualization tool designed for non-experts to learn about Transformers through the GPT-2 model. Our tool helps users understand complex Transformer concepts by integrating a model overview and enabling smooth transitions across abstraction levels of mathematical operations and model structures. It runs a live GPT-2 instance locally in the user's browser, empowering users to experiment with their own input and observe in real-time how the internal components and parameters of the Transformer work together to predict the next tokens. Our tool requires no installation or special hardware, broadening the public's education access to modern generative AI techniques.

AK

90,882 views • 2 years ago

The easiest way to misunderstand KV cache is to assume every token goes through an LLM in the same way. It doesn’t. Take a decoder-only Transformer and a four-token prompt: x1, x2, x3, x4. During the prefill phase, the prompt is processed and each self-attention layer computes the keys and values for its prompt positions. Those K/V states are stored in that layer’s KV cache. The representation at the final prompt position produces the logits used to select the first generated token, y1. Here is the part that is easy to miss => Selecting y1 does not mean y1 is already in the KV cache. On the next forward step, y1 enters the model as position 5. At each self-attention layer, its Q, K and V are computed. Its query attends to the cached K/V from positions 1-4 plus its own current K/V at position 5. Its K/V then become part of that layer’s cache. The resulting final-layer representation produces logits for y2. Then y2 is processed as position 6, its K/V extend the cache, and the process continues. So for standard single-token, KV-cached autoregressive decoding: > prompt → prefill phase > new token → process one new position > past K/V → reuse them instead of recomputing them > KV cache → grows as new positions are processed One subtle distinction matters a lot and I want to highlight it => a token can be generated before it has been processed into the KV cache.

The easiest way to misunderstand KV cache is to assume every token goes through an LLM in the same way. It doesn’t. Take a decoder-only Transformer and a four-token prompt: x1, x2, x3, x4. During the prefill phase, the prompt is processed and each self-attention layer computes the keys and values for its prompt positions. Those K/V states are stored in that layer’s KV cache. The representation at the final prompt position produces the logits used to select the first generated token, y1. Here is the part that is easy to miss => Selecting y1 does not mean y1 is already in the KV cache. On the next forward step, y1 enters the model as position 5. At each self-attention layer, its Q, K and V are computed. Its query attends to the cached K/V from positions 1-4 plus its own current K/V at position 5. Its K/V then become part of that layer’s cache. The resulting final-layer representation produces logits for y2. Then y2 is processed as position 6, its K/V extend the cache, and the process continues. So for standard single-token, KV-cached autoregressive decoding: > prompt → prefill phase > new token → process one new position > past K/V → reuse them instead of recomputing them > KV cache → grows as new positions are processed One subtle distinction matters a lot and I want to highlight it => a token can be generated before it has been processed into the KV cache.

Tech with Mak

12,211 views • 1 month ago

Now this is cool. You can see which way the market is moving with Market Maps. This lets you follow the change in a market, noticing specific trends in a given sector. Check it out:

Now this is cool. You can see which way the market is moving with Market Maps. This lets you follow the change in a market, noticing specific trends in a given sector. Check it out:

unusual_whales

225,481 views • 3 months ago

Now this is cool. You can see which way the market is moving with Market Maps. This lets you follow the change in a market, noticing specific trends in a given sector. Check it out:

Now this is cool. You can see which way the market is moving with Market Maps. This lets you follow the change in a market, noticing specific trends in a given sector. Check it out:

unusual_whales

141,779 views • 3 months ago

Last week, we launched "Attention in Transformers: Concepts and Code in PyTorch" instructed by Joshua Starmer! In this course, you'll: ✅ Learn how the attention mechanism in LLMs helps convert base token embeddings into rich context-aware embeddings. ✅ Understand the Query, Key, and Value matrices, what they are for, how to produce them, and how to use them in attention. ✅ Learn the difference between self-attention, masked self-attention, and cross-attention, and how multi-head attention scales the algorithm. 🔗 Enroll for free:

Last week, we launched "Attention in Transformers: Concepts and Code in PyTorch" instructed by Joshua Starmer! In this course, you'll: ✅ Learn how the attention mechanism in LLMs helps convert base token embeddings into rich context-aware embeddings. ✅ Understand the Query, Key, and Value matrices, what they are for, how to produce them, and how to use them in attention. ✅ Learn the difference between self-attention, masked self-attention, and cross-attention, and how multi-head attention scales the algorithm. 🔗 Enroll for free:

DeepLearning.AI

36,832 views • 1 year ago

Can you follow the math of a Transformer model by hand ✍️? Download PDF 👉 I hope this hand-drawn animation makes it easier. p.s., This has one error. Can you can spot it and tell me in the comment?

Can you follow the math of a Transformer model by hand ✍️? Download PDF 👉 I hope this hand-drawn animation makes it easier. p.s., This has one error. Can you can spot it and tell me in the comment?

Tom Yeh

14,578 views • 1 year ago

Attention all Kiwi! One NZ Satellite🛰️ Now you can use X anywhere in New Zealand you can see the sky, as well stay connected and informed using apps like WhatsApp and Google Maps. Check it out at terms apply.

Attention all Kiwi! One NZ Satellite🛰️ Now you can use X anywhere in New Zealand you can see the sky, as well stay connected and informed using apps like WhatsApp and Google Maps. Check it out at terms apply.

One New Zealand

680,717 views • 9 months ago

Found a nice resource for learning ML/AI concepts interactively: VizLearn. You can play with gradient descent, SVMs, PCA, Bayes, BPE tokenization, Q/K/V, KV cache, quantization and a lot more, and watch the actual computation change as you modify the inputs. Free, no account needed.

Found a nice resource for learning ML/AI concepts interactively: VizLearn. You can play with gradient descent, SVMs, PCA, Bayes, BPE tokenization, Q/K/V, KV cache, quantization and a lot more, and watch the actual computation change as you modify the inputs. Free, no account needed.

Tech with Mak

14,513 views • 28 days ago

spent way too long playing with Vivix A1 the character actually talks, listens, moves around the environment and reacts when you change the conversation and you can interrupt it MID-SENTENCE most “interactive” AI characters still feel like a video call with a face this feels much closer to actually having a character inside the world with you Vivix.AI

spent way too long playing with Vivix A1 the character actually talks, listens, moves around the environment and reacts when you change the conversation and you can interrupt it MID-SENTENCE most “interactive” AI characters still feel like a video call with a face this feels much closer to actually having a character inside the world with you Vivix.AI

GARLOTIC

17,018 views • 1 month ago

Submodular optimization for token/sentence selection from long contexts. Here's an interesting exp: first used jina-embeddings-v4's multi-vector feature to extract token-level embeddings from a passage, then applied submodular optimization to cherry-pick the tokens that provide the best coverage, finally call tokenizer and convert selections back to the strings at their org. positions. Think of it as a form of "compression"—you can adjust the top-k slider to dial in different "compress rates". Can you still make sense of the compressed text?

Submodular optimization for token/sentence selection from long contexts. Here's an interesting exp: first used jina-embeddings-v4's multi-vector feature to extract token-level embeddings from a passage, then applied submodular optimization to cherry-pick the tokens that provide the best coverage, finally call tokenizer and convert selections back to the strings at their org. positions. Think of it as a form of "compression"—you can adjust the top-k slider to dial in different "compress rates". Can you still make sense of the compressed text?

Jina AI

13,205 views • 1 year ago

Lex P and Drea Nicole talked about how they don’t like men who talk in a baby voice during s-x and Lex P started to mimic how men actually talk in a baby voice during s-x in the most hilarious way ever💀😭 “Are you gon s—k this d—k, is that how he says it, can you s—k this d—k please baby, cmon babe s—k my d—k please”😭

Sensitive content

Lex P and Drea Nicole talked about how they don’t like men who talk in a baby voice during s-x and Lex P started to mimic how men actually talk in a baby voice during s-x in the most hilarious way ever💀😭 “Are you gon s—k this d—k, is that how he says it, can you s—k this d—k please baby, cmon babe s—k my d—k please”😭

flame⛧

174,870 views • 5 months ago

If you treat the thoracic spine, this is worth a watch See how an expert physio assesses thoracic pain in practice and pick up a few things you can use with your next patient. 👉 Try it for FREE:

If you treat the thoracic spine, this is worth a watch See how an expert physio assesses thoracic pain in practice and pick up a few things you can use with your next patient. 👉 Try it for FREE:

Physio Network

17,150 views • 1 month ago

This makes the future way closer than you think. Director AI automates any tasks you can do in a browser. It interacts with web pages just like a person would. If you could use a browser to complete a task, you can have this AI do it for you. And it's free! Watch the video:

This makes the future way closer than you think. Director AI automates any tasks you can do in a browser. It interacts with web pages just like a person would. If you could use a browser to complete a task, you can have this AI do it for you. And it's free! Watch the video:

Santiago

36,273 views • 1 year ago