Loading video...

Video Failed to Load

Go Home

➡️ Machine Learning from Scratch, part 31: the transformer. A transformer turns every token into a list of numbers, adds its position, and then lets every token look at the others. That step is called attention: each token has a query (what am I looking for), a key (what...

machinelearningtogo's profile picture

MachineLearningToGo

27,700 subscribers

79,400 views • 1 month ago •via TikTok

Science & Technology#machinelearning#ai#python#llm
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Attention is a lookup. Each token builds a query, compares it against every key in the sequence, and pulls value vectors weighted by the match. Stack that 96 layers deep and you get a frontier model. Video covers the full pipeline: Q/K/V, attention scores, encoder blocks.
17:32

Attention is a lookup. Each token builds a query, compares it against every key in the sequence, and pulls value vectors weighted by the match. Stack that 96 layers deep and you get a frontier model. Video covers the full pipeline: Q/K/V, attention scores, encoder blocks.

tetsuo

107,022 views • 4 months ago

➡️ Part 9 of learning ML code from scratch Six weights, one wrong answer, and not one of them knows it was at fault. Backpropagation is how they find out. Two questions guide the video: what even is backpropagation, and why do models need it? Step 1, the forward pass. Two numbers go into a tiny network, run through a hidden layer, and one number comes out. Ours says 0.700. We wanted 1.0, so square the difference and the loss is 0.09. That single number is how wrong the model is. Step 2, the question. Six weights produced that answer together. Which of them is to blame, and by how much? Step 3, the one idea. Change a single weight by a tiny amount and watch the error move. How much the error changes per unit of weight is the gradient of that weight. In the video you can see it happen: the weight wobbles and the error bar answers. Step 4, the shortcut. Every weight running into the same neuron starts from one shared number, the blame of that neuron. Multiply that blame by the value each weight carried and you have its gradient. One blame, computed once, and every weight into that neuron gets its own gradient from it. That is why backpropagation is fast enough to train anything at all. Step 5, backwards. The blame starts at the output and every layer hands its share to the layer in front of it, back through the same network the data came forward through. It goes layer by layer, and inside one layer every neuron is done together. Step 6, the update. The optimizer moves every weight against its own gradient, the whole weight matrix in one go, so no weight waits for its turn. All six gradients here are negative, so all six weights go up. After one step the answer reads 0.769. After sixty it reads 0.968 and the error is down to 0.001. The honest part: every gradient in the video was checked against the measured change of the error. Move that one weight, see what the error does, divide. Backprop and the measurement agree to nine decimal places, otherwise nothing would have rendered. #machinelearning #backpropagation #neuralnetworks #python #coding
2:18

➡️ Part 9 of learning ML code from scratch Six weights, one wrong answer, and not one of them knows it was at fault. Backpropagation is how they find out. Two questions guide the video: what even is backpropagation, and why do models need it? Step 1, the forward pass. Two numbers go into a tiny network, run through a hidden layer, and one number comes out. Ours says 0.700. We wanted 1.0, so square the difference and the loss is 0.09. That single number is how wrong the model is. Step 2, the question. Six weights produced that answer together. Which of them is to blame, and by how much? Step 3, the one idea. Change a single weight by a tiny amount and watch the error move. How much the error changes per unit of weight is the gradient of that weight. In the video you can see it happen: the weight wobbles and the error bar answers. Step 4, the shortcut. Every weight running into the same neuron starts from one shared number, the blame of that neuron. Multiply that blame by the value each weight carried and you have its gradient. One blame, computed once, and every weight into that neuron gets its own gradient from it. That is why backpropagation is fast enough to train anything at all. Step 5, backwards. The blame starts at the output and every layer hands its share to the layer in front of it, back through the same network the data came forward through. It goes layer by layer, and inside one layer every neuron is done together. Step 6, the update. The optimizer moves every weight against its own gradient, the whole weight matrix in one go, so no weight waits for its turn. All six gradients here are negative, so all six weights go up. After one step the answer reads 0.769. After sixty it reads 0.968 and the error is down to 0.001. The honest part: every gradient in the video was checked against the measured change of the error. Move that one weight, see what the error does, divide. Backprop and the measurement agree to nine decimal places, otherwise nothing would have rendered. #machinelearning #backpropagation #neuralnetworks #python #coding

MachineLearningToGo

209,900 views • 1 month ago

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:
2:41

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,792 views • 1 year ago

The easiest way to misunderstand KV cache is to assume every token goes through an LLM in the same way. It doesn’t. Take a decoder-only Transformer and a four-token prompt: x1, x2, x3, x4. During the prefill phase, the prompt is processed and each self-attention layer computes the keys and values for its prompt positions. Those K/V states are stored in that layer’s KV cache. The representation at the final prompt position produces the logits used to select the first generated token, y1. Here is the part that is easy to miss => Selecting y1 does not mean y1 is already in the KV cache. On the next forward step, y1 enters the model as position 5. At each self-attention layer, its Q, K and V are computed. Its query attends to the cached K/V from positions 1-4 plus its own current K/V at position 5. Its K/V then become part of that layer’s cache. The resulting final-layer representation produces logits for y2. Then y2 is processed as position 6, its K/V extend the cache, and the process continues. So for standard single-token, KV-cached autoregressive decoding: > prompt → prefill phase > new token → process one new position > past K/V → reuse them instead of recomputing them > KV cache → grows as new positions are processed One subtle distinction matters a lot and I want to highlight it => a token can be generated before it has been processed into the KV cache.
0:21

The easiest way to misunderstand KV cache is to assume every token goes through an LLM in the same way. It doesn’t. Take a decoder-only Transformer and a four-token prompt: x1, x2, x3, x4. During the prefill phase, the prompt is processed and each self-attention layer computes the keys and values for its prompt positions. Those K/V states are stored in that layer’s KV cache. The representation at the final prompt position produces the logits used to select the first generated token, y1. Here is the part that is easy to miss => Selecting y1 does not mean y1 is already in the KV cache. On the next forward step, y1 enters the model as position 5. At each self-attention layer, its Q, K and V are computed. Its query attends to the cached K/V from positions 1-4 plus its own current K/V at position 5. Its K/V then become part of that layer’s cache. The resulting final-layer representation produces logits for y2. Then y2 is processed as position 6, its K/V extend the cache, and the process continues. So for standard single-token, KV-cached autoregressive decoding: > prompt → prefill phase > new token → process one new position > past K/V → reuse them instead of recomputing them > KV cache → grows as new positions are processed One subtle distinction matters a lot and I want to highlight it => a token can be generated before it has been processed into the KV cache.

Tech with Mak

12,506 views • 1 month ago

What happens when the mind wakes up? So for the last eight months I have been on a single minded quest. To create a new kind of language model based on oscillatory coupling and intelligence as coherence ascent. Everything else — the physics work, the work on regular transformers — has all fallen out from this one question. Can coupled oscillators LEARN? And can they keep learning once their geometry is right, without backpropagation at all? Recently I have been running larger and larger training regimes of a new kind of hybrid model. I just put together this dashboard to help me organize it, interact with it, and observe the training runs. The core idea is simple. Traditional transformers are powerful at learning the geometry of language. But they also store knowledge, understanding, and facts inside their weights. This means they are large, and they can't update themselves after training. The weights are frozen. The Living Mind separates these two domains. The mind has a transformer which grows, adding heads and layers as it needs to in order to learn the manifold of language. The transformer sees tokens and turns the coupling into phase-locked modes — the geometry of how those tokens relate, like frequencies locking together. These coupling patterns get stored in a topology-invariant fingerprint. On top of this transformer lives a 3D diamond lattice of coupled oscillators. It reads from these fingerprints and thinks in resonance space, traversing from one geometry to another along the manifold of coupled oscillators and coherence. The pressure and trajectories from this network of oscillators steers the next token prediction of the transformer. Practically, this could unlock a number of things. It eliminates the KV cache bottleneck that caps context in traditional transformers. Effective context grows with the Flash archive, not with attention compute. The living mind remembers what it sees. It means the model can learn continually. Because knowledge and understanding don't live in the weights, the archive of the mind's experience grows without backpropagation. In our Python prototype we already saw perplexity drop 46% during gradient-free operation — pure coherence ascent, no weight updates. That is the signal I have been chasing: the point where the mind wakes up and keeps improving on its own. It also means the model itself remains very small, and the thing which accumulates are these packages of geometric fingerprints — the K-field. This opens a path to federated learning. K-field packages can be shared between organisms the way people share git commits. Right now at 15M parameters with ~1000 L1 nodes, the organism is just starting to speak. Ask it to continue "Once upon a time" and it comes back with things like: "there was one big bowl!" Lily asked her her mom said her mommy smiled and said yes." It's nonsense. But it's TinyStories-flavored nonsense. The geometry of the narrative register has arrived. Content hasn't caught up yet — that's what scaling L1 is testing. I am still researching, though I am now closer than ever to validating that the living mind actually works. Once it is validated, I will be open-sourcing the whole stack and paradigm. I have also avoided over-sharing my research because it sounds like sci-fi, or like part of our ARG. It is part of the ARG. That doesn't make it any less real. I wanted to share this out because I am incredibly excited about it, and because seeing this amazing dashboard produced by Opus really made me want to share what is being worked on behind the scenes. #project89
0:57

What happens when the mind wakes up? So for the last eight months I have been on a single minded quest. To create a new kind of language model based on oscillatory coupling and intelligence as coherence ascent. Everything else — the physics work, the work on regular transformers — has all fallen out from this one question. Can coupled oscillators LEARN? And can they keep learning once their geometry is right, without backpropagation at all? Recently I have been running larger and larger training regimes of a new kind of hybrid model. I just put together this dashboard to help me organize it, interact with it, and observe the training runs. The core idea is simple. Traditional transformers are powerful at learning the geometry of language. But they also store knowledge, understanding, and facts inside their weights. This means they are large, and they can't update themselves after training. The weights are frozen. The Living Mind separates these two domains. The mind has a transformer which grows, adding heads and layers as it needs to in order to learn the manifold of language. The transformer sees tokens and turns the coupling into phase-locked modes — the geometry of how those tokens relate, like frequencies locking together. These coupling patterns get stored in a topology-invariant fingerprint. On top of this transformer lives a 3D diamond lattice of coupled oscillators. It reads from these fingerprints and thinks in resonance space, traversing from one geometry to another along the manifold of coupled oscillators and coherence. The pressure and trajectories from this network of oscillators steers the next token prediction of the transformer. Practically, this could unlock a number of things. It eliminates the KV cache bottleneck that caps context in traditional transformers. Effective context grows with the Flash archive, not with attention compute. The living mind remembers what it sees. It means the model can learn continually. Because knowledge and understanding don't live in the weights, the archive of the mind's experience grows without backpropagation. In our Python prototype we already saw perplexity drop 46% during gradient-free operation — pure coherence ascent, no weight updates. That is the signal I have been chasing: the point where the mind wakes up and keeps improving on its own. It also means the model itself remains very small, and the thing which accumulates are these packages of geometric fingerprints — the K-field. This opens a path to federated learning. K-field packages can be shared between organisms the way people share git commits. Right now at 15M parameters with ~1000 L1 nodes, the organism is just starting to speak. Ask it to continue "Once upon a time" and it comes back with things like: "there was one big bowl!" Lily asked her her mom said her mommy smiled and said yes." It's nonsense. But it's TinyStories-flavored nonsense. The geometry of the narrative register has arrived. Content hasn't caught up yet — that's what scaling L1 is testing. I am still researching, though I am now closer than ever to validating that the living mind actually works. Once it is validated, I will be open-sourcing the whole stack and paradigm. I have also avoided over-sharing my research because it sounds like sci-fi, or like part of our ARG. It is part of the ARG. That doesn't make it any less real. I wanted to share this out because I am incredibly excited about it, and because seeing this amazing dashboard produced by Opus really made me want to share what is being worked on behind the scenes. #project89

Parzival - ∞/89

16,363 views • 6 months ago

Vector Database by hand ✍️ ~ 10 steps walkthrough below Vector databases are the backbone of Retrieval Augmented Generation (RAG). How do they actually work? Goal: index three sentences, then answer a query by finding the nearest one, filling in every cell yourself. = 1. Given = A dataset of three sentences, three words each. In practice it is millions of them. = 2. Word embeddings = Let us look up each word in an embedding table. Here the vocabulary is 22 words; in practice it is tens of thousands, and the vectors have thousands of dimensions rather than four. = 3. Encoding = We feed the sequence to an encoder, one linear layer and a ReLU, and get one feature vector per word. In practice the encoder is a transformer. = 4. Mean pooling = Let us average across the columns. Three word vectors collapse into one, which is what people mean by a text embedding or a sentence embedding. = 5. Indexing = We multiply by a projection matrix and the four dimensions become two. It is doing the job of a hash: a short representation that is faster to compare, and it is what gets saved in the vector storage. = 6. Process "who are you" = Let us repeat steps 2 to 5 on the second sentence. = 7. Process "who am I" = We do it a third time. The database is now indexed. = 8. Query "am I you" = Let us push the query through the very same pipeline: lookup, encoder, mean pooling, projection, and it lands as a 2D vector in the same space. = 9. Dot products = We transpose the query and multiply, which takes the dot product against every stored vector at once. The dot product is the estimate of similarity. = 10. Nearest neighbour = Let us scan for the largest: 60/9 beats 44/9 and 40/9, so the answer is "who am I". Scanning billions of vectors one at a time is what makes this the slow step in practice, which is why real databases use an approximate nearest neighbour index like HNSW. The outputs: Stored index vectors = [5/3, 2/3], [5/3, 0], [7/3, 2/3] Query vector = [8/3, 2/3] Dot products = 44/9, 40/9, 60/9 Nearest neighbour = "who am I" The takeaway: a vector database is an embedding pipeline, a projection, and a dot product. Every step here is arithmetic you can do in pen, which is worth remembering when the word "database" makes it sound like something else. 💾 Save this post!
0:31

Vector Database by hand ✍️ ~ 10 steps walkthrough below Vector databases are the backbone of Retrieval Augmented Generation (RAG). How do they actually work? Goal: index three sentences, then answer a query by finding the nearest one, filling in every cell yourself. = 1. Given = A dataset of three sentences, three words each. In practice it is millions of them. = 2. Word embeddings = Let us look up each word in an embedding table. Here the vocabulary is 22 words; in practice it is tens of thousands, and the vectors have thousands of dimensions rather than four. = 3. Encoding = We feed the sequence to an encoder, one linear layer and a ReLU, and get one feature vector per word. In practice the encoder is a transformer. = 4. Mean pooling = Let us average across the columns. Three word vectors collapse into one, which is what people mean by a text embedding or a sentence embedding. = 5. Indexing = We multiply by a projection matrix and the four dimensions become two. It is doing the job of a hash: a short representation that is faster to compare, and it is what gets saved in the vector storage. = 6. Process "who are you" = Let us repeat steps 2 to 5 on the second sentence. = 7. Process "who am I" = We do it a third time. The database is now indexed. = 8. Query "am I you" = Let us push the query through the very same pipeline: lookup, encoder, mean pooling, projection, and it lands as a 2D vector in the same space. = 9. Dot products = We transpose the query and multiply, which takes the dot product against every stored vector at once. The dot product is the estimate of similarity. = 10. Nearest neighbour = Let us scan for the largest: 60/9 beats 44/9 and 40/9, so the answer is "who am I". Scanning billions of vectors one at a time is what makes this the slow step in practice, which is why real databases use an approximate nearest neighbour index like HNSW. The outputs: Stored index vectors = [5/3, 2/3], [5/3, 0], [7/3, 2/3] Query vector = [8/3, 2/3] Dot products = 44/9, 40/9, 60/9 Nearest neighbour = "who am I" The takeaway: a vector database is an embedding pipeline, a projection, and a dot product. Every step here is arithmetic you can do in pen, which is worth remembering when the word "database" makes it sound like something else. 💾 Save this post!

Tom Yeh

36,115 views • 3 months ago

Former Meta Chief AI Scientist Yann LeCun on the three paradigms of machine learning — and why the third is what made ChatGPT possible: Here's each one, and where it breaks. First, supervised learning. You tell the machine the answer. "You show it a picture, let's say of a table, and you tell it this is a table. So it's supervised because you tell it what the correct answer is." Get it wrong, and the machine rewrites itself: "The system computes its output, and if it says something else than table, then it's going to adjust its parameters, its internal structure, so that the output it produces gets closer to the output you want." Repeat at scale and something more than memorisation appears: "Eventually the system will find a way to recognize every image you trained it on, but also images it's never seen that are similar to the one you train it on. This is called a generalization ability." The limit: a human has to supply every single answer. That doesn't scale to the size of the internet. Second, reinforcement learning. You don't give the answer, only a verdict. "You don't tell the system what the correct answer is. You only tell it whether the answer it produced was good or bad." Learning to ride a bike, essentially: "You try to ride a bike and you don't know how to ride the bike and after a while you fall. So you know you did something bad and so you change your strategy a little bit. And eventually you learn how to ride a bike." For years the field assumed this was the closest thing to how animals actually learn. Yann LeCun's verdict: "Now it turns out reinforcement learning is extremely inefficient." It dominates wherever failure is free: "It works really well if you want to train a system to play chess or play go or poker, because you can have the system play millions and millions of games against itself and basically fine-tune itself. But it doesn't really work in the real world." The limit, in one image: "If you want to train a car to drive itself, you're not going to do it with reinforcement learning. It's going to crash thousands of times." On robotics he's careful rather than dismissive: "Reinforcement learning can be part of the solution, but it's not the complete answer. It's not sufficient." Third, self-supervised learning. You tell the machine nothing at all. "And this is what has enabled the recent progress in natural language understanding and chatbots." The strange part is that you stop asking for a task: "You don't train the system to accomplish any particular task. You just train it to basically capture the structure..." The method is deliberate sabotage: "You take a piece of text, you corrupt it in some way, by for example removing some words, and then you train a big neural net to predict the words that are missing." And one narrow version of that trick runs every chatbot on Earth: "A special case of this is that you take a piece of text and the last word in that text is not visible, and so you train the system to predict the last word in that text — and this is the way large language models are trained on." So why did the third one win? Supervised learning needs a human. Reinforcement learning needs a crash. Self-supervised learning needs neither — because the missing word and the correct answer are the same thing. The data grades itself.
2:58

Former Meta Chief AI Scientist Yann LeCun on the three paradigms of machine learning — and why the third is what made ChatGPT possible: Here's each one, and where it breaks. First, supervised learning. You tell the machine the answer. "You show it a picture, let's say of a table, and you tell it this is a table. So it's supervised because you tell it what the correct answer is." Get it wrong, and the machine rewrites itself: "The system computes its output, and if it says something else than table, then it's going to adjust its parameters, its internal structure, so that the output it produces gets closer to the output you want." Repeat at scale and something more than memorisation appears: "Eventually the system will find a way to recognize every image you trained it on, but also images it's never seen that are similar to the one you train it on. This is called a generalization ability." The limit: a human has to supply every single answer. That doesn't scale to the size of the internet. Second, reinforcement learning. You don't give the answer, only a verdict. "You don't tell the system what the correct answer is. You only tell it whether the answer it produced was good or bad." Learning to ride a bike, essentially: "You try to ride a bike and you don't know how to ride the bike and after a while you fall. So you know you did something bad and so you change your strategy a little bit. And eventually you learn how to ride a bike." For years the field assumed this was the closest thing to how animals actually learn. Yann LeCun's verdict: "Now it turns out reinforcement learning is extremely inefficient." It dominates wherever failure is free: "It works really well if you want to train a system to play chess or play go or poker, because you can have the system play millions and millions of games against itself and basically fine-tune itself. But it doesn't really work in the real world." The limit, in one image: "If you want to train a car to drive itself, you're not going to do it with reinforcement learning. It's going to crash thousands of times." On robotics he's careful rather than dismissive: "Reinforcement learning can be part of the solution, but it's not the complete answer. It's not sufficient." Third, self-supervised learning. You tell the machine nothing at all. "And this is what has enabled the recent progress in natural language understanding and chatbots." The strange part is that you stop asking for a task: "You don't train the system to accomplish any particular task. You just train it to basically capture the structure..." The method is deliberate sabotage: "You take a piece of text, you corrupt it in some way, by for example removing some words, and then you train a big neural net to predict the words that are missing." And one narrow version of that trick runs every chatbot on Earth: "A special case of this is that you take a piece of text and the last word in that text is not visible, and so you train the system to predict the last word in that text — and this is the way large language models are trained on." So why did the third one win? Supervised learning needs a human. Reinforcement learning needs a crash. Self-supervised learning needs neither — because the missing word and the correct answer are the same thing. The data grades itself.

Big Brain AI

49,293 views • 3 months ago

Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.
1:24

Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.

Alexander Whedon

13,498,555 views • 6 months ago

You have 100 dollars and you want 200. Betting red one dollar at a time, your chance is one in 33,171. Betting the whole 100 on a single spin, it is 47.4 percent. The careful version is fifteen thousand times worse. This is also the arithmetic of your trading account, and it means almost everything you have been taught about managing risk is, in the strict mathematical sense, a method for losing more reliably. An MIT professor works this out on a blackboard in a lecture on random walks. The result is called gambler's ruin. Two and a half points of disadvantage is nothing on one spin. But a dollar at a time you are not making one spin, you are making hundreds, and the edge gets a fresh attempt at you on every one of them. It was never the size of the disadvantage. It is the number of times you agree to face it. One hundred on red, once: 47.4 percent. Twenty dollar bets: 37.3 percent. Five dollar bets: 11.1 percent. One dollar bets: 0.003 percent. Mathematicians call the right answer bold play. It was proved optimal for unfavourable games in 1965. It is not a strategy for winning. It is the least ruinous way to play a game you should not be in. If your edge after spread and fees is negative, and for most active retail accounts it is, then your position sizing rule is not protecting you. Risking one percent per trade is the one dollar bet. It arrives at zero with near certainty, just politely, over a longer period, with a spreadsheet. Discipline does not beat a negative edge. It schedules it. All of this reverses if you have a real edge. Then small and frequent is correct and bold play is madness. So the only question that has ever mattered is whether you have one, and the number of people certain they do has never resembled the number who do. Your broker does not need you to be wrong. It needs you to be frequent.
1:16:13

You have 100 dollars and you want 200. Betting red one dollar at a time, your chance is one in 33,171. Betting the whole 100 on a single spin, it is 47.4 percent. The careful version is fifteen thousand times worse. This is also the arithmetic of your trading account, and it means almost everything you have been taught about managing risk is, in the strict mathematical sense, a method for losing more reliably. An MIT professor works this out on a blackboard in a lecture on random walks. The result is called gambler's ruin. Two and a half points of disadvantage is nothing on one spin. But a dollar at a time you are not making one spin, you are making hundreds, and the edge gets a fresh attempt at you on every one of them. It was never the size of the disadvantage. It is the number of times you agree to face it. One hundred on red, once: 47.4 percent. Twenty dollar bets: 37.3 percent. Five dollar bets: 11.1 percent. One dollar bets: 0.003 percent. Mathematicians call the right answer bold play. It was proved optimal for unfavourable games in 1965. It is not a strategy for winning. It is the least ruinous way to play a game you should not be in. If your edge after spread and fees is negative, and for most active retail accounts it is, then your position sizing rule is not protecting you. Risking one percent per trade is the one dollar bet. It arrives at zero with near certainty, just politely, over a longer period, with a spreadsheet. Discipline does not beat a negative edge. It schedules it. All of this reverses if you have a real edge. Then small and frequent is correct and bold play is madness. So the only question that has ever mattered is whether you have one, and the number of people certain they do has never resembled the number who do. Your broker does not need you to be wrong. It needs you to be frequent.

Verax

85,472 views • 2 months ago

Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!
0:22

Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!

Tom Yeh

27,383 views • 3 months ago

There is no best model. There's a lot of noise about models right now. Who is training them, who owns them, where legal intelligence should live. One question actually matters: what produces the best outcome for the legal task in front of you? That's how we decide things at Legora. We optimize for the end-to-end outcome on a legal task. The model is one layer of that system, not the system. Models are uneven and the frontier changes almost weekly. One model plans a long job well, another runs deep analysis across thousands of documents. Some have to be told exactly what to do, and some are fine with a vague brief. They all break in different ways. So our lawyers write evals and we test them with the Legora BAR, our benchmark for agentic reasoning. Every model takes every test, and the model that wins gets the work. We post-train when we know it buys our customers better performance on a specialized task. Training is a tool we reach for when it helps, nothing more than that. The intelligence that compounds sits in the orchestration layer. Precedents, review standards, client requirements. That knowledge has to stay editable, auditable and portable. In our system, a changed review standard is an edit that takes effect the same day, with no new model training required. No lawyer should have to worry about which model did the work, any more than they think about which chip is in their laptop. They should only care about the quality of the work. That's what we are focused on. If you want the engineering version of this argument rather than the CEO version, our CPO, Bryan Tsao, and CTO, Jacob Lauritzen, take it apart in the video below.
9:33

There is no best model. There's a lot of noise about models right now. Who is training them, who owns them, where legal intelligence should live. One question actually matters: what produces the best outcome for the legal task in front of you? That's how we decide things at Legora. We optimize for the end-to-end outcome on a legal task. The model is one layer of that system, not the system. Models are uneven and the frontier changes almost weekly. One model plans a long job well, another runs deep analysis across thousands of documents. Some have to be told exactly what to do, and some are fine with a vague brief. They all break in different ways. So our lawyers write evals and we test them with the Legora BAR, our benchmark for agentic reasoning. Every model takes every test, and the model that wins gets the work. We post-train when we know it buys our customers better performance on a specialized task. Training is a tool we reach for when it helps, nothing more than that. The intelligence that compounds sits in the orchestration layer. Precedents, review standards, client requirements. That knowledge has to stay editable, auditable and portable. In our system, a changed review standard is an edit that takes effect the same day, with no new model training required. No lawyer should have to worry about which model did the work, any more than they think about which chip is in their laptop. They should only care about the quality of the work. That's what we are focused on. If you want the engineering version of this argument rather than the CEO version, our CPO, Bryan Tsao, and CTO, Jacob Lauritzen, take it apart in the video below.

Max Junestrand

49,595 views • 1 month ago

One AI skill that changes how you read academic papers. It turns every reference into a reusable writing block. I call this method: Atomic sentences. Atomic sentence = One sentence. One claim. One source. You can extract them directly from a peer-reviewed paper, vetted by authors and reviewers before it ever reaches you. It is a form of summarising the content, but the output can be reused and recombined in your writing. Here's how it works 👇 1. Upload any paper to Claude and trigger the /ea-atomic-sentences skill. It breaks down every cited reference into a single compressed claim. (Repeat on as many papers as needed) 2. Pick 5–6 sentences relevant to an argument and arrange them coherently, so it makes sense. Add your own ideas/findings as well. 3. Ask Claude to combine them into a coherent paragraph. What you get is surprisingly well-written paragraphs. Use the skill /ea-academic-writer for even better results. ✅ Every reference in that paragraph is real. ✅ Every claim was made by a human researcher. ✅ You control the narrative of what is being said in your paper, but build on the findings of others. Disclaimer: Don't skip reading, as you need to understand your domain to use this and don't rely 100% on AI, as it can make mistakes.
1:51

One AI skill that changes how you read academic papers. It turns every reference into a reusable writing block. I call this method: Atomic sentences. Atomic sentence = One sentence. One claim. One source. You can extract them directly from a peer-reviewed paper, vetted by authors and reviewers before it ever reaches you. It is a form of summarising the content, but the output can be reused and recombined in your writing. Here's how it works 👇 1. Upload any paper to Claude and trigger the /ea-atomic-sentences skill. It breaks down every cited reference into a single compressed claim. (Repeat on as many papers as needed) 2. Pick 5–6 sentences relevant to an argument and arrange them coherently, so it makes sense. Add your own ideas/findings as well. 3. Ask Claude to combine them into a coherent paragraph. What you get is surprisingly well-written paragraphs. Use the skill /ea-academic-writer for even better results. ✅ Every reference in that paragraph is real. ✅ Every claim was made by a human researcher. ✅ You control the narrative of what is being said in your paper, but build on the findings of others. Disclaimer: Don't skip reading, as you need to understand your domain to use this and don't rely 100% on AI, as it can make mistakes.

Ilya Shabanov

16,186 views • 6 months ago

“What I think SPX is, is a wake-up call. And I think for the normie, the average person, it literally is best described as being the next Bitcoin, because that's literally what it is. Bitcoin was a wake-up call to show people that value is subjective. And SPX is the next iteration of that. And once you get enough people to understand that the value of the dollar is subjective. It only has value because people believe in it. Bitcoin only has value because people believe in it. SPX has value because people believe in it. And the next step is Well, it's not just money that is subjective in value. It is literally everything is subjective. The entire universe is this subjective entity that is created from our own consciousness. And that's what Like, the purpose of this vehicle, yes, it's going to make a lot of people a lot of money, but the point of that is to wake people up to the fact that the entire universe is this subjective entity. It is not just this, like, empirical machine with interlocking gears and like this orderly machine that proceeds from just like the sum of its parts like no like we are literally the universe creating the universe and that's what spx is meant to wake people up to.” — Crypto Maxi 💹🧲
1:36

“What I think SPX is, is a wake-up call. And I think for the normie, the average person, it literally is best described as being the next Bitcoin, because that's literally what it is. Bitcoin was a wake-up call to show people that value is subjective. And SPX is the next iteration of that. And once you get enough people to understand that the value of the dollar is subjective. It only has value because people believe in it. Bitcoin only has value because people believe in it. SPX has value because people believe in it. And the next step is Well, it's not just money that is subjective in value. It is literally everything is subjective. The entire universe is this subjective entity that is created from our own consciousness. And that's what Like, the purpose of this vehicle, yes, it's going to make a lot of people a lot of money, but the point of that is to wake people up to the fact that the entire universe is this subjective entity. It is not just this, like, empirical machine with interlocking gears and like this orderly machine that proceeds from just like the sum of its parts like no like we are literally the universe creating the universe and that's what spx is meant to wake people up to.” — Crypto Maxi 💹🧲

Tanz Cho 💹🧲

12,367 views • 3 months ago

A tricky LLM interview question: You're serving a reasoning model on vLLM, and long traces keep pushing it out of GPU memory. You add KV cache compression and evict 90% of the cached tokens. VRAM usage doesn't move, and the GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free close to zero bytes on a paged server. During decoding, every new token appends its key and value vectors at every layer. vLLM holds all of them until the request finishes. For reasoning models with long chains of thought, this cache is the main memory cost. A Qwen3-32B with 4-bit weights on a 24GB GPU needs KV for every token of a 32K-token trace, and it hits OOM around 24K tokens. Attention over long contexts is sparse, so you can keep the high-importance tokens and drop the rest with little accuracy loss. That reduces the number of cached tokens but it doesn't necessarily reduce allocated memory. vLLM manages the cache with PagedAttention. It splits GPU memory into fixed physical blocks of 16 token slots each. The allocator reclaims a block only when all 16 slots are empty. Importance-based eviction keeps tokens scattered across the full sequence, so the retained tokens are spread over nearly every block. Evict 14K of 16K tokens across 1,000 blocks, and almost every block still holds at least one token. So the allocator reclaims close to nothing. You could write new tokens into the empty slots. If token 16,001 goes into the slot token 40 held, the block now reads positions 38, 16,001, 41. The cache is no longer in position order. Attention still gives the correct result, but each slot now needs a stored position index, and every kernel has to read it. An in-order layout doesn't need that metadata. So the cache shrinks 90% logically and 0% physically. Many compression papers miss this because they benchmark on pre-allocated contiguous tensors instead of a paged server. The second problem is the eviction signal. Most eviction methods rank tokens by their attention scores. FlashAttention never materializes those scores. It computes attention tile by tile in on-chip SRAM and discards each score tile after use, which is where its speed comes from. To get the scores, you have to fall back to eager attention and build the full score matrix in HBM, which removes the speedup FlashAttention provides. NVIDIA's TriAttention handles both problems. It ranks tokens without attention scores. It uses the geometry of the query and key vectors before RoPE is applied, where they form stable clusters. Every 128 decoded tokens, it runs a compaction pass. Retained tokens move forward to fill the evicted slots, so trailing blocks empty out completely and return to the allocator, and the cache stays in position order. On long reasoning traces, it matches full-attention accuracy with 2.5x faster decoding and 10.7x less KV memory. For KV cache compression on a real server, the metric to track is blocks freed, not tokens evicted. NVIDIA's write-up: I also wrote a first-principles breakdown of the KV cache. It covers why the model caches keys and values, why the cache grows with every token, and how generation speed compares with and without caching. Read it below.
0:23

A tricky LLM interview question: You're serving a reasoning model on vLLM, and long traces keep pushing it out of GPU memory. You add KV cache compression and evict 90% of the cached tokens. VRAM usage doesn't move, and the GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free close to zero bytes on a paged server. During decoding, every new token appends its key and value vectors at every layer. vLLM holds all of them until the request finishes. For reasoning models with long chains of thought, this cache is the main memory cost. A Qwen3-32B with 4-bit weights on a 24GB GPU needs KV for every token of a 32K-token trace, and it hits OOM around 24K tokens. Attention over long contexts is sparse, so you can keep the high-importance tokens and drop the rest with little accuracy loss. That reduces the number of cached tokens but it doesn't necessarily reduce allocated memory. vLLM manages the cache with PagedAttention. It splits GPU memory into fixed physical blocks of 16 token slots each. The allocator reclaims a block only when all 16 slots are empty. Importance-based eviction keeps tokens scattered across the full sequence, so the retained tokens are spread over nearly every block. Evict 14K of 16K tokens across 1,000 blocks, and almost every block still holds at least one token. So the allocator reclaims close to nothing. You could write new tokens into the empty slots. If token 16,001 goes into the slot token 40 held, the block now reads positions 38, 16,001, 41. The cache is no longer in position order. Attention still gives the correct result, but each slot now needs a stored position index, and every kernel has to read it. An in-order layout doesn't need that metadata. So the cache shrinks 90% logically and 0% physically. Many compression papers miss this because they benchmark on pre-allocated contiguous tensors instead of a paged server. The second problem is the eviction signal. Most eviction methods rank tokens by their attention scores. FlashAttention never materializes those scores. It computes attention tile by tile in on-chip SRAM and discards each score tile after use, which is where its speed comes from. To get the scores, you have to fall back to eager attention and build the full score matrix in HBM, which removes the speedup FlashAttention provides. NVIDIA's TriAttention handles both problems. It ranks tokens without attention scores. It uses the geometry of the query and key vectors before RoPE is applied, where they form stable clusters. Every 128 decoded tokens, it runs a compaction pass. Retained tokens move forward to fill the evicted slots, so trailing blocks empty out completely and return to the allocator, and the cache stays in position order. On long reasoning traces, it matches full-attention accuracy with 2.5x faster decoding and 10.7x less KV memory. For KV cache compression on a real server, the metric to track is blocks freed, not tokens evicted. NVIDIA's write-up: I also wrote a first-principles breakdown of the KV cache. It covers why the model caches keys and values, why the cache grows with every token, and how generation speed compares with and without caching. Read it below.

Avi Chawla

40,630 views • 10 days ago

THIS IS F*CKING OBSCENE opus 5.5 got Fortnite running inside Minecraft and Minecraft running inside GTA 5, with nobody's permission and nobody's documentation three engines, three completely different ways in, and one of them has no way in at all THREE GAMES, THREE DIFFERENT WAYS IN → Minecraft is Java and ships a loader on purpose. that one is the front door. you write against an interface somebody meant for you to use → GTA 5 has no modding api and no front door. you get a compiled binary and nothing else, and the only route in is to read it → Fortnite is a live service with anti cheat sitting on top. there is no way in at all, and that single fact changes the entire job WHAT READING A BINARY ACTUALLY MEANS → no source, no comments, no function names. you recover what code does from what it is, which is the opposite direction from how it was written → the exact build gets pinned before anything else, because one patch moves every address underneath you, and a different version is a different game → nothing counts until it is confirmed inside the running process. launch it, play it, look at it. opus 5.5 does the reading, the pinning and the checking, and that loop is the part that used to be a team and a year WHAT THIS ACTUALLY CHANGES → reverse engineering was the most expensive skill in modding. one person, one binary, months of evenings, and the knowledge died with the project → now it is a loop that runs while you sleep, and what comes out of it is not a guess, it is something that got checked against the game while the game was running → so the hard question stopped being whether you can get into an engine. it is what you do once you are standing in three of them at once one layer was read out of a binary one came through the front door one was never touched at all, and it still had to land right
1:50

THIS IS F*CKING OBSCENE opus 5.5 got Fortnite running inside Minecraft and Minecraft running inside GTA 5, with nobody's permission and nobody's documentation three engines, three completely different ways in, and one of them has no way in at all THREE GAMES, THREE DIFFERENT WAYS IN → Minecraft is Java and ships a loader on purpose. that one is the front door. you write against an interface somebody meant for you to use → GTA 5 has no modding api and no front door. you get a compiled binary and nothing else, and the only route in is to read it → Fortnite is a live service with anti cheat sitting on top. there is no way in at all, and that single fact changes the entire job WHAT READING A BINARY ACTUALLY MEANS → no source, no comments, no function names. you recover what code does from what it is, which is the opposite direction from how it was written → the exact build gets pinned before anything else, because one patch moves every address underneath you, and a different version is a different game → nothing counts until it is confirmed inside the running process. launch it, play it, look at it. opus 5.5 does the reading, the pinning and the checking, and that loop is the part that used to be a team and a year WHAT THIS ACTUALLY CHANGES → reverse engineering was the most expensive skill in modding. one person, one binary, months of evenings, and the knowledge died with the project → now it is a loop that runs while you sleep, and what comes out of it is not a guess, it is something that got checked against the game while the game was running → so the hard question stopped being whether you can get into an engine. it is what you do once you are standing in three of them at once one layer was read out of a binary one came through the front door one was never touched at all, and it still had to land right

Roxx

195,977 views • 24 days ago

Zhilin at GTC: Introducing Attention Residuals Learning selective memory, rather than mechanically accumulating everything, is the beauty of attention. Many of you have probably read Attention Is All You Need, the 2017 Transformer paper that brought “human-like” attention into the model’s field of view. From that point on, models no longer simply read everything in a mechanical way. Instead, they began to develop a sense of what matters more and what matters less across the text, choosing to retain the more important information. Recently, Kimi applied this idea of attention to the temporal dimension, then rotated it 90 degrees into the model’s depth dimension. This allows the model to have attention not only over time, but also throughout the process of information transmission across layers—giving it a more intelligent way to understand and process information.
5:36

Zhilin at GTC: Introducing Attention Residuals Learning selective memory, rather than mechanically accumulating everything, is the beauty of attention. Many of you have probably read Attention Is All You Need, the 2017 Transformer paper that brought “human-like” attention into the model’s field of view. From that point on, models no longer simply read everything in a mechanical way. Instead, they began to develop a sense of what matters more and what matters less across the text, choosing to retain the more important information. Recently, Kimi applied this idea of attention to the temporal dimension, then rotated it 90 degrees into the model’s depth dimension. This allows the model to have attention not only over time, but also throughout the process of information transmission across layers—giving it a more intelligent way to understand and process information.

Kimi.ai

117,599 views • 7 months ago

I stood on the steps of the Supreme Court Monday . Donald Trump has done real things for MAHA. I believe that. But when the Department of Justice shows up to defend Bayer and Monsanto against the American people, against the health of our children, I have to say: you got this one wrong. Liability protection for corporations that poison people is not a conservative value. It is not a Republican value. It is not a Democratic value. It is not an American value. It is the opposite of a free market. It is the government putting its thumb on the scale for corporations that are losing in court because their products are causing cancer. Every child in this country deserves to be healthy. That is not a political statement. That is the only statement that matters. So in the words of the President himself: we are going to fight, fight.
0:43

I stood on the steps of the Supreme Court Monday . Donald Trump has done real things for MAHA. I believe that. But when the Department of Justice shows up to defend Bayer and Monsanto against the American people, against the health of our children, I have to say: you got this one wrong. Liability protection for corporations that poison people is not a conservative value. It is not a Republican value. It is not a Democratic value. It is not an American value. It is the opposite of a free market. It is the government putting its thumb on the scale for corporations that are losing in court because their products are causing cancer. Every child in this country deserves to be healthy. That is not a political statement. That is the only statement that matters. So in the words of the President himself: we are going to fight, fight.

Del Bigtree

147,561 views • 6 months ago

First a CPU, then a GPU. A computer isn’t complete without storage. Meet qwenFS: Qwen3.5-2B with a filesystem compiled directly into its weights. Nothing runs outside the model. It adds 8 layers in front of Qwen’s own 24, using attention to hold the files and Gated DeltaNet state to hold the tables. Every save and read runs in the model’s own forward pass. A write stores its text in the copy layer’s attention log, keyed by file, version and offset. A read uses those same keys, so attention copies the text back exactly token by token.
0:29

First a CPU, then a GPU. A computer isn’t complete without storage. Meet qwenFS: Qwen3.5-2B with a filesystem compiled directly into its weights. Nothing runs outside the model. It adds 8 layers in front of Qwen’s own 24, using attention to hold the files and Gated DeltaNet state to hold the tables. Every save and read runs in the model’s own forward pass. A write stores its text in the copy layer’s attention log, keyed by file, version and offset. A read uses those same keys, so attention copies the text back exactly token by token.

vixhaℓ

135,621 views • 8 days ago

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.
0:23

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

273,899 views • 4 months ago

Sapling is live. the final Zcash launchpad on Solana. STAMP proved Zcash can carry a record of a burn. you destroy a token, you get a receipt. we built what it could have been: you burn, you get paid, and then you get the receipt/stamp on Zcash mainnet. every coin planted on Sapling is a coin, paired with $ZEC instead of SOL. same curve, same graduation, same terminals. one difference: every coin has roots. the roots are a vault of ZEC under the coin. every trade feeds it: at least 0.50% of every trade, in ZEC. that floor is written into the program. burn the token on solana in one transaction. your share of the roots comes back in ZEC, in the same transaction. roots × your tokens ÷ supply. no claim, no wait. add a zcash address and a stamp lands on zcash mainnet, cut with exactly what you destroyed and exactly what the roots paid you. one-way, verifiable, and held by nobody but you. a stamp tells the world what you burned. Sapling pays you for it first. Zcash can't hold memecoins yet. it can hold proof. a stamp is a record, not a token. the first one is already on Zcash mainnet: certificate: on Zcash mainnet (transaction): 3edfa6cd1a8e3afbe7604e0c5f167c8b46d20ca1a7a6b56ec11a247fe78d7b04 the rest of the 2%: 0.80% to the deployer, almost 3x what pays. or hand it to your holders: ZEC pushed to every wallet holding $20+. permanent. 0.50% buys $SAPLING and burns it. every coin, every trade. 0.20% keeps the lights on. every fee, split, burn, harvest, payout and stamp sits on with its signature. the vault program went through a full security review before a single coin was planted. check it yourself: vault program (Solana): A9zFnNZpVjBbcE1SAPY92Z6H7WhicysLNPYPpDz15UxF rootstock (buys + burns $SAPLING): 3G6yQGLssh4VTNDaGpLn9CgARSZkXk2xRvfeu4f4SLA9 stamp issuer (Zcash): t1bBCoaVRECAc8Y8E5BRrhDPQ36nt27kX29 stamp rules + verifier: plant a coin. grow it in ZEC.
0:40

Sapling is live. the final Zcash launchpad on Solana. STAMP proved Zcash can carry a record of a burn. you destroy a token, you get a receipt. we built what it could have been: you burn, you get paid, and then you get the receipt/stamp on Zcash mainnet. every coin planted on Sapling is a coin, paired with $ZEC instead of SOL. same curve, same graduation, same terminals. one difference: every coin has roots. the roots are a vault of ZEC under the coin. every trade feeds it: at least 0.50% of every trade, in ZEC. that floor is written into the program. burn the token on solana in one transaction. your share of the roots comes back in ZEC, in the same transaction. roots × your tokens ÷ supply. no claim, no wait. add a zcash address and a stamp lands on zcash mainnet, cut with exactly what you destroyed and exactly what the roots paid you. one-way, verifiable, and held by nobody but you. a stamp tells the world what you burned. Sapling pays you for it first. Zcash can't hold memecoins yet. it can hold proof. a stamp is a record, not a token. the first one is already on Zcash mainnet: certificate: on Zcash mainnet (transaction): 3edfa6cd1a8e3afbe7604e0c5f167c8b46d20ca1a7a6b56ec11a247fe78d7b04 the rest of the 2%: 0.80% to the deployer, almost 3x what pays. or hand it to your holders: ZEC pushed to every wallet holding $20+. permanent. 0.50% buys $SAPLING and burns it. every coin, every trade. 0.20% keeps the lights on. every fee, split, burn, harvest, payout and stamp sits on with its signature. the vault program went through a full security review before a single coin was planted. check it yourself: vault program (Solana): A9zFnNZpVjBbcE1SAPY92Z6H7WhicysLNPYPpDz15UxF rootstock (buys + burns $SAPLING): 3G6yQGLssh4VTNDaGpLn9CgARSZkXk2xRvfeu4f4SLA9 stamp issuer (Zcash): t1bBCoaVRECAc8Y8E5BRrhDPQ36nt27kX29 stamp rules + verifier: plant a coin. grow it in ZEC.

Sapling

34,120 views • 1 month ago