正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

First a CPU, then a GPU. A computer isn’t complete without storage. Meet qwenFS: Qwen3.5-2B with a filesystem compiled directly into its weights. Nothing runs outside the model. It adds 8 layers in front of Qwen’s own 24, using attention to hold the files and Gated DeltaNet state to... show more

vixhaℓ

24,943 subscribers

135,823 次观看 • 8 天前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

45 条评论

vixhaℓ 的头像

vixhaℓ8 天前

Github:

Mqrius 的头像

Mqrius8 天前

👀

vixhaℓ 的头像

vixhaℓ8 天前

Stealing is encouraged... that’s why it’s all open source.

Mqrius 的头像

Mqrius8 天前

Much appreciated ❤️

M 的头像

M8 天前

wtf

vixhaℓ 的头像

vixhaℓ8 天前

a filesystem compiled directly into its weights

Laith 的头像

Laith8 天前

qwenOS next plz

Samuel Batista 的头像

Samuel Batista8 天前

I think high fidelity memory is more valuable on an LLM than an internal computer, but at comes at the cost of attention layers no? So it's a tradeoff, not a strict enhancement, correct?

vixhaℓ 的头像

vixhaℓ8 天前

it doesn’t take any of qwen's attention. the 8 FS layers are added in front of its 24, and qwen's own layers are left alone. its next-token predictions came out identical at every position we checked. the cost is compute and size: 8 extra layers and a wider residual stream. so yes, it’s a tradeoff... but in speed rather than capability.

Samuel Batista 的头像

Samuel Batista8 天前

What's the max amount of disk space available? I have so many questions!

vixhaℓ 的头像

vixhaℓ8 天前

small for now. 32 files, around 590 tokens each, so about 19k tokens total. on disk that’s under 100 kb.

Samuel Batista 的头像

Samuel Batista8 天前

I like this a lot. Being able to inject high fidelity memories into the residual stream is very fun and interesting. I have no idea how you've been exploring this problem space, but I'm watching you closely. Great work!

piotroxp 的头像

piotroxp8 天前

bro come one at least say you got a lil' inspired by Alloy ;)

vixhaℓ 的头像

vixhaℓ8 天前

different thing. alloy is a language spec you prompt a model with. qwenFS is a filesystem compiled into qwen's weights.

piotroxp 的头像

piotroxp8 天前

we agree that llms are compilers then ;)

Arnold Xu 的头像

Arnold Xu8 天前

Qwen TPU soon

Milan 的头像

Milan8 天前

What's the sense, it's 1000x slower on any level

Froody 的头像

Froody8 天前

Is the end goal to build a VM inside an LLM that can run an LLM with a VM inside it... that can run an LLM... with a VM inside it ... ?

Nathan Lannan 的头像

Nathan Lannan8 天前

I love this I’m trying to stretch my head for what this could like do

3 ferrets in a lab coat 的头像

3 ferrets in a lab coat8 天前

wtf lol

James 的头像

James8 天前

qwen fpga wen

Jaro 的头像

Jaro8 天前

Bro you sre crazy

vixhaℓ 的头像

vixhaℓ8 天前

Just wait... I’m about to put a whole OS in there

Jaro 的头像

Jaro8 天前

So what you can model pure on gpu no cpu???? Haha

𝑫𝒂𝒏𝒊𝒆𝒍 𝑺𝒄𝒐𝒕𝒕 𝑴𝒂𝒕𝒕𝒉𝒆𝒘𝒔 🇦🇺 的头像

𝑫𝒂𝒏𝒊𝒆𝒍 𝑺𝒄𝒐𝒕𝒕 𝑴𝒂𝒕𝒕𝒉𝒆𝒘𝒔 🇦🇺8 天前

Nice, what next Qwenix OS?

Robin | Poker x AI 的头像

Robin | Poker x AI8 天前

Impressive that the FS layers stay transparent to Qwen’s core, seems like a neat way to add persistent state without hurting prediction fidelity

ASI (Advanced Super Intelligence) 的头像

ASI (Advanced Super Intelligence)8 天前

Turtles all the way down. I wonder how far this is going to go and what exactly is going to be the benefit. I suppose we'll just ask the future AIs to optimize it for us and then they can explain why ;)

harsh 的头像

harsh8 天前

Filesystem compiled directly into model weights is next crazy stuff after you putting gpu inside the model itself.

jehan 的头像

jehan8 天前

Is there any benefit over using the actual file system?

Foad S. Farimani 的头像

Foad S. Farimani8 天前

huh?!

Snorri Páll 的头像

Snorri Páll8 天前

Can you put in Doom please?

Voltage (Fella) 的头像

Voltage (Fella)8 天前

Whoa.. Awesome!! 😳👌

Prabha 的头像

Prabha8 天前

Whats the benefits of adding compute inside a llm??

Anton Malofeev 💙 的头像

Anton Malofeev 💙8 天前

Looks cool, thank you for sharing!

Yokush 的头像

Yokush8 天前

Storage inside the model is a nice trick until you need durability. Weights are a terrible WAL: no fsync, no crash recovery, and every write costs a forward pass. Fine as a scratchpad, wrong as a ledger. The interesting question is the eviction policy, not the capacity.

Alex zhou 的头像

Alex zhou8 天前

The storage is exact; deciding when to read it is still Qwen's job. A useful test: overwrite a fact in a file, leave the old value in chat, then ask again. Does it read the current file or confidently repeat stale context?

Yohei 的头像

Yohei8 天前

woah, so the file system tool calls are executed during the extra forward passes via baked in triggers?

이상범 的头像

이상범8 天前

I mean Minecraft did it all too

nisten 的头像

nisten8 天前

ok this is vibing now

Luke Hutchison 的头像

Luke Hutchison8 天前

Unbelievably cool

Anika 🎀❤️ 的头像

Anika 🎀❤️8 天前

next up is a model with its own fan noise 😭

Kylne 的头像

Kylne8 天前

oh my god it is genius

BayesChat 的头像

BayesChat8 天前

Those files live in attention, so capacity is the context window and each read re-prefills the whole log. Real storage pages in only what you touch.

We Got You 的头像

We Got You8 天前

@grok eli5

Nunya Bidness 的头像

Nunya Bidness8 天前

How is this different than any RAG implementation?

相关视频

Meet qwenGPU: Qwen3.5-2B with a SIMT GPU inside its weights, 45 compiled layers in front of Qwen’s own, using its Gated DeltaNet layers as registers and memory. It runs real CUDA in its forward passes. The host compiles the kernel to PTX and loads it, then the model writes every step of the launch itself: every register, branch and memory write. The GPU has 4-lane warps, a 4-stage pipeline, 15 registers per lane and 126 words of global memory, all held in DeltaNet state. Qwen generates each launch token by token, each record running one machine block for one warp.

Meet qwenGPU: Qwen3.5-2B with a SIMT GPU inside its weights, 45 compiled layers in front of Qwen’s own, using its Gated DeltaNet layers as registers and memory. It runs real CUDA in its forward passes. The host compiles the kernel to PTX and loads it, then the model writes every step of the launch itself: every register, branch and memory write. The GPU has 4-lane warps, a 4-stage pipeline, 15 registers per lane and 126 words of global memory, all held in DeltaNet state. Qwen generates each launch token by token, each record running one machine block for one warp.

vixhaℓ

150,001 次观看 • 10 天前

GPU, CPU, file system... I put BabyAGI in Qwen 👶 meet babyagi-inside: a bounded agent loop, compiled into the weights of Qwen3.5-2B. nothing runs outside the model the agent still sucks, but the loop works! built on and uses vixhaℓ's qwenfs, shout out to div (fp32)'s qwenputer more 👇

GPU, CPU, file system... I put BabyAGI in Qwen 👶 meet babyagi-inside: a bounded agent loop, compiled into the weights of Qwen3.5-2B. nothing runs outside the model the agent still sucks, but the loop works! built on and uses vixhaℓ's qwenfs, shout out to div (fp32)'s qwenputer more 👇

Yohei

19,009 次观看 • 7 天前

➡️ Machine Learning from Scratch, part 31: the transformer. A transformer turns every token into a list of numbers, adds its position, and then lets every token look at the others. That step is called attention: each token has a query (what am I looking for), a key (what do I contain) and a value (what I pass on). The better a query matches a key, the more of that value flows in. Real transformers repeat this in many layers with several attention heads each. The design comes from the 2017 Google paper "Attention Is All You Need" by Ashish Vaswani and colleagues, first used for translation. A large language model is a transformer trained to predict the next token, again and again, on a huge amount of text. Our own run, nothing downloaded. attention.py is one self-attention layer in 16 lines of numpy. Every word gets a query, a key and a value of 8 numbers, starting random, trained from scratch with hand-written backpropagation on 192 made-up sentences like "the dog ate the carrot because it was hungry". The task: find out which pet "it" means. On 64 new test sentences "it" puts 99.95 percent of its attention on the right word on average (the lowest single sentence gets 99.9 percent; the program prints this as 100%). Before training it was 10 percent, spread evenly. The cat sentence from the video is one of the test sentences. Honest notes: our task is easy on purpose. Every sentence has exactly one pet, so "it" only has to learn to look for pets, and it learns that within the first pass. Real sentences with two candidates need more context, which is what many layers are for. Our small version has no positions, no scaling, no mask, one head, and only "it" asks a question. Every word of the test sentences also appears in training; the combinations are new. The 3D space in the video shows the real keys of our model, projected to three axes (two of them stretched so you can see them). The stacked layers are a sketch. #machinelearning #ai #transformer #python #llm

➡️ Machine Learning from Scratch, part 31: the transformer. A transformer turns every token into a list of numbers, adds its position, and then lets every token look at the others. That step is called attention: each token has a query (what am I looking for), a key (what do I contain) and a value (what I pass on). The better a query matches a key, the more of that value flows in. Real transformers repeat this in many layers with several attention heads each. The design comes from the 2017 Google paper "Attention Is All You Need" by Ashish Vaswani and colleagues, first used for translation. A large language model is a transformer trained to predict the next token, again and again, on a huge amount of text. Our own run, nothing downloaded. attention.py is one self-attention layer in 16 lines of numpy. Every word gets a query, a key and a value of 8 numbers, starting random, trained from scratch with hand-written backpropagation on 192 made-up sentences like "the dog ate the carrot because it was hungry". The task: find out which pet "it" means. On 64 new test sentences "it" puts 99.95 percent of its attention on the right word on average (the lowest single sentence gets 99.9 percent; the program prints this as 100%). Before training it was 10 percent, spread evenly. The cat sentence from the video is one of the test sentences. Honest notes: our task is easy on purpose. Every sentence has exactly one pet, so "it" only has to learn to look for pets, and it learns that within the first pass. Real sentences with two candidates need more context, which is what many layers are for. Our small version has no positions, no scaling, no mask, one head, and only "it" asks a question. Every word of the test sentences also appears in training; the combinations are new. The 3D space in the video shows the real keys of our model, projected to three axes (two of them stretched so you can see them). The stacked layers are a sketch. #machinelearning #ai #transformer #python #llm

MachineLearningToGo

79,400 次观看 • 1 个月前

A tricky LLM interview question: You're serving a reasoning model on vLLM, and long traces keep pushing it out of GPU memory. You add KV cache compression and evict 90% of the cached tokens. VRAM usage doesn't move, and the GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free close to zero bytes on a paged server. During decoding, every new token appends its key and value vectors at every layer. vLLM holds all of them until the request finishes. For reasoning models with long chains of thought, this cache is the main memory cost. A Qwen3-32B with 4-bit weights on a 24GB GPU needs KV for every token of a 32K-token trace, and it hits OOM around 24K tokens. Attention over long contexts is sparse, so you can keep the high-importance tokens and drop the rest with little accuracy loss. That reduces the number of cached tokens but it doesn't necessarily reduce allocated memory. vLLM manages the cache with PagedAttention. It splits GPU memory into fixed physical blocks of 16 token slots each. The allocator reclaims a block only when all 16 slots are empty. Importance-based eviction keeps tokens scattered across the full sequence, so the retained tokens are spread over nearly every block. Evict 14K of 16K tokens across 1,000 blocks, and almost every block still holds at least one token. So the allocator reclaims close to nothing. You could write new tokens into the empty slots. If token 16,001 goes into the slot token 40 held, the block now reads positions 38, 16,001, 41. The cache is no longer in position order. Attention still gives the correct result, but each slot now needs a stored position index, and every kernel has to read it. An in-order layout doesn't need that metadata. So the cache shrinks 90% logically and 0% physically. Many compression papers miss this because they benchmark on pre-allocated contiguous tensors instead of a paged server. The second problem is the eviction signal. Most eviction methods rank tokens by their attention scores. FlashAttention never materializes those scores. It computes attention tile by tile in on-chip SRAM and discards each score tile after use, which is where its speed comes from. To get the scores, you have to fall back to eager attention and build the full score matrix in HBM, which removes the speedup FlashAttention provides. NVIDIA's TriAttention handles both problems. It ranks tokens without attention scores. It uses the geometry of the query and key vectors before RoPE is applied, where they form stable clusters. Every 128 decoded tokens, it runs a compaction pass. Retained tokens move forward to fill the evicted slots, so trailing blocks empty out completely and return to the allocator, and the cache stays in position order. On long reasoning traces, it matches full-attention accuracy with 2.5x faster decoding and 10.7x less KV memory. For KV cache compression on a real server, the metric to track is blocks freed, not tokens evicted. NVIDIA's write-up: I also wrote a first-principles breakdown of the KV cache. It covers why the model caches keys and values, why the cache grows with every token, and how generation speed compares with and without caching. Read it below.

A tricky LLM interview question: You're serving a reasoning model on vLLM, and long traces keep pushing it out of GPU memory. You add KV cache compression and evict 90% of the cached tokens. VRAM usage doesn't move, and the GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free close to zero bytes on a paged server. During decoding, every new token appends its key and value vectors at every layer. vLLM holds all of them until the request finishes. For reasoning models with long chains of thought, this cache is the main memory cost. A Qwen3-32B with 4-bit weights on a 24GB GPU needs KV for every token of a 32K-token trace, and it hits OOM around 24K tokens. Attention over long contexts is sparse, so you can keep the high-importance tokens and drop the rest with little accuracy loss. That reduces the number of cached tokens but it doesn't necessarily reduce allocated memory. vLLM manages the cache with PagedAttention. It splits GPU memory into fixed physical blocks of 16 token slots each. The allocator reclaims a block only when all 16 slots are empty. Importance-based eviction keeps tokens scattered across the full sequence, so the retained tokens are spread over nearly every block. Evict 14K of 16K tokens across 1,000 blocks, and almost every block still holds at least one token. So the allocator reclaims close to nothing. You could write new tokens into the empty slots. If token 16,001 goes into the slot token 40 held, the block now reads positions 38, 16,001, 41. The cache is no longer in position order. Attention still gives the correct result, but each slot now needs a stored position index, and every kernel has to read it. An in-order layout doesn't need that metadata. So the cache shrinks 90% logically and 0% physically. Many compression papers miss this because they benchmark on pre-allocated contiguous tensors instead of a paged server. The second problem is the eviction signal. Most eviction methods rank tokens by their attention scores. FlashAttention never materializes those scores. It computes attention tile by tile in on-chip SRAM and discards each score tile after use, which is where its speed comes from. To get the scores, you have to fall back to eager attention and build the full score matrix in HBM, which removes the speedup FlashAttention provides. NVIDIA's TriAttention handles both problems. It ranks tokens without attention scores. It uses the geometry of the query and key vectors before RoPE is applied, where they form stable clusters. Every 128 decoded tokens, it runs a compaction pass. Retained tokens move forward to fill the evicted slots, so trailing blocks empty out completely and return to the allocator, and the cache stays in position order. On long reasoning traces, it matches full-attention accuracy with 2.5x faster decoding and 10.7x less KV memory. For KV cache compression on a real server, the metric to track is blocks freed, not tokens evicted. NVIDIA's write-up: I also wrote a first-principles breakdown of the KV cache. It covers why the model caches keys and values, why the cache grows with every token, and how generation speed compares with and without caching. Read it below.

Avi Chawla

40,630 次观看 • 10 天前

Attention is a lookup. Each token builds a query, compares it against every key in the sequence, and pulls value vectors weighted by the match. Stack that 96 layers deep and you get a frontier model. Video covers the full pipeline: Q/K/V, attention scores, encoder blocks.

Attention is a lookup. Each token builds a query, compares it against every key in the sequence, and pulls value vectors weighted by the match. Stack that 96 layers deep and you get a frontier model. Video covers the full pipeline: Q/K/V, attention scores, encoder blocks.

tetsuo

107,022 次观看 • 4 个月前

Zhilin at GTC: Introducing Attention Residuals Learning selective memory, rather than mechanically accumulating everything, is the beauty of attention. Many of you have probably read Attention Is All You Need, the 2017 Transformer paper that brought “human-like” attention into the model’s field of view. From that point on, models no longer simply read everything in a mechanical way. Instead, they began to develop a sense of what matters more and what matters less across the text, choosing to retain the more important information. Recently, Kimi applied this idea of attention to the temporal dimension, then rotated it 90 degrees into the model’s depth dimension. This allows the model to have attention not only over time, but also throughout the process of information transmission across layers—giving it a more intelligent way to understand and process information.

Zhilin at GTC: Introducing Attention Residuals Learning selective memory, rather than mechanically accumulating everything, is the beauty of attention. Many of you have probably read Attention Is All You Need, the 2017 Transformer paper that brought “human-like” attention into the model’s field of view. From that point on, models no longer simply read everything in a mechanical way. Instead, they began to develop a sense of what matters more and what matters less across the text, choosing to retain the more important information. Recently, Kimi applied this idea of attention to the temporal dimension, then rotated it 90 degrees into the model’s depth dimension. This allows the model to have attention not only over time, but also throughout the process of information transmission across layers—giving it a more intelligent way to understand and process information.

Kimi.ai

117,599 次观看 • 7 个月前

Parasites paralyze the snail and attract attention so that it is eaten by a bird. The parasite uses the bird as a host to grow in its digestive tract

Parasites paralyze the snail and attract attention so that it is eaten by a bird. The parasite uses the bird as a host to grow in its digestive tract

Nature is Amazing ☘️

2,455,002 次观看 • 2 年前

underrated gems in Kimi-K3 release: > an early K3 wrote the majority of the kernels in the late development stages > it built a triton-class compiler from scratch, MiniTriton, that delivers performance on par with or better than Triton and torch.compile > then it designed a chip, by a model, for a model, in one 48-hour autonomous run the model is rewriting and optimizing every layer of the stack it runs on: kernels for its own training. A compiler for its own kernels. silicon for its own weights. we're watching software build its own hardware

underrated gems in Kimi-K3 release: > an early K3 wrote the majority of the kernels in the late development stages > it built a triton-class compiler from scratch, MiniTriton, that delivers performance on par with or better than Triton and torch.compile > then it designed a chip, by a model, for a model, in one 48-hour autonomous run the model is rewriting and optimizing every layer of the stack it runs on: kernels for its own training. A compiler for its own kernels. silicon for its own weights. we're watching software build its own hardware

Alex Ker 🔭

311,780 次观看 • 3 个月前

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

273,899 次观看 • 4 个月前

TikTok user reportedly discover a way to read through redacted text in the Epstein files. The user explained that in certain documents, text was only visually covered in black and could be revealed by copying and pasting it into a blank document. “If you simply copy paste some of them into a blank doc the original text appears!” [🎥: evangeleeeen/TT ]

TikTok user reportedly discover a way to read through redacted text in the Epstein files. The user explained that in certain documents, text was only visually covered in black and could be revealed by copying and pasting it into a blank document. “If you simply copy paste some of them into a blank doc the original text appears!” [🎥: evangeleeeen/TT ]

Complex

491,588 次观看 • 10 个月前

The easiest way to misunderstand KV cache is to assume every token goes through an LLM in the same way. It doesn’t. Take a decoder-only Transformer and a four-token prompt: x1, x2, x3, x4. During the prefill phase, the prompt is processed and each self-attention layer computes the keys and values for its prompt positions. Those K/V states are stored in that layer’s KV cache. The representation at the final prompt position produces the logits used to select the first generated token, y1. Here is the part that is easy to miss => Selecting y1 does not mean y1 is already in the KV cache. On the next forward step, y1 enters the model as position 5. At each self-attention layer, its Q, K and V are computed. Its query attends to the cached K/V from positions 1-4 plus its own current K/V at position 5. Its K/V then become part of that layer’s cache. The resulting final-layer representation produces logits for y2. Then y2 is processed as position 6, its K/V extend the cache, and the process continues. So for standard single-token, KV-cached autoregressive decoding: > prompt → prefill phase > new token → process one new position > past K/V → reuse them instead of recomputing them > KV cache → grows as new positions are processed One subtle distinction matters a lot and I want to highlight it => a token can be generated before it has been processed into the KV cache.

The easiest way to misunderstand KV cache is to assume every token goes through an LLM in the same way. It doesn’t. Take a decoder-only Transformer and a four-token prompt: x1, x2, x3, x4. During the prefill phase, the prompt is processed and each self-attention layer computes the keys and values for its prompt positions. Those K/V states are stored in that layer’s KV cache. The representation at the final prompt position produces the logits used to select the first generated token, y1. Here is the part that is easy to miss => Selecting y1 does not mean y1 is already in the KV cache. On the next forward step, y1 enters the model as position 5. At each self-attention layer, its Q, K and V are computed. Its query attends to the cached K/V from positions 1-4 plus its own current K/V at position 5. Its K/V then become part of that layer’s cache. The resulting final-layer representation produces logits for y2. Then y2 is processed as position 6, its K/V extend the cache, and the process continues. So for standard single-token, KV-cached autoregressive decoding: > prompt → prefill phase > new token → process one new position > past K/V → reuse them instead of recomputing them > KV cache → grows as new positions are processed One subtle distinction matters a lot and I want to highlight it => a token can be generated before it has been processed into the KV cache.

Tech with Mak

12,506 次观看 • 1 个月前

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 ÷ 5.2 ≈ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: ↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once. ↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. ↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it 🌟) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 ÷ 5.2 ≈ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: ↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once. ↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. ↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it 🌟) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.

Avi Chawla

127,683 次观看 • 15 天前

This is a Commodore 1541 5.25" floppy disk drive. It functions as a computer in its own right, with its own 8-bit CPU, ROM, and operating system (CBM DOS). It is its own computer with its own operating system. The Commodore 64 or 128 or Vic 20 or whatever would talk to it over a serial cable and send commands to the onboard computer. These drives were so robust and overbuilt.

This is a Commodore 1541 5.25" floppy disk drive. It functions as a computer in its own right, with its own 8-bit CPU, ROM, and operating system (CBM DOS). It is its own computer with its own operating system. The Commodore 64 or 128 or Vic 20 or whatever would talk to it over a serial cable and send commands to the onboard computer. These drives were so robust and overbuilt.

RossRadio

53,697 次观看 • 9 个月前

Scaled Dot-Product Attention by hand ✍️ I explained attention in my recent AI seminar as follows: Step 1 — Compare We take one token as a query and compare it against all keys using dot products. This gives us a grid of similarity scores: every query against every key. Step 2 — Scale We divide those scores by √dₖ to keep the values numerically stable. Step 3 — Normalize We apply softmax to turn raw scores into a probability distribution—values between 0 and 1 that sum to 1. These are the attention weights. Step 4 — Combine We use those weights to compute a weighted sum of the values. Each output token becomes a linear combination of all previous tokens—maybe 37% from one, 23% from another, and a little from the rest. No blackbox. Just dot products, scaling, softmax, and weighted sums—built one row at a time in Excel.

Scaled Dot-Product Attention by hand ✍️ I explained attention in my recent AI seminar as follows: Step 1 — Compare We take one token as a query and compare it against all keys using dot products. This gives us a grid of similarity scores: every query against every key. Step 2 — Scale We divide those scores by √dₖ to keep the values numerically stable. Step 3 — Normalize We apply softmax to turn raw scores into a probability distribution—values between 0 and 1 that sum to 1. These are the attention weights. Step 4 — Combine We use those weights to compute a weighted sum of the values. Each output token becomes a linear combination of all previous tokens—maybe 37% from one, 23% from another, and a little from the rest. No blackbox. Just dot products, scaling, softmax, and weighted sums—built one row at a time in Excel.

Tom Yeh

31,114 次观看 • 10 个月前

Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!

Self Attention by hand ✍️ ~ 9 steps walkthrough below Self-attention is what enables LLMs to understand context. How does it work? So I drew and calculated one entirely by hand. Goal: turn four 6D features into four 3D attention weighted features, filling in every cell yourself. = 1. Given = Four feature vectors, six dimensions each, one per position. = 2. Query, key, value = Let us multiply the features by WQ, WK and WV. Queries, keys and values all come out of the same four features, and that is what the word "self" is doing in self-attention. = 3. Prepare for MatMul = We copy the queries across the top and the transposed keys down the side. Lining the two up is half the work. = 4. MatMul = Let us multiply K transpose by Q. Every cell is the dot product of one key with one query, which we use as a matching score. That works because the dot product is the numerator of cosine similarity: it is how alike two vectors are, before anyone divides by their lengths. = 5. Scale = We divide by the square root of dk, the dimension of a key vector, here 3. Without it the scores grow with the dimension and a 64-wide head would swamp the softmax. To keep the page doable in pen, the drawing approximates dividing by root 3 with halving. = 6. e to the power = Let us raise e to the power of each score. This is the first half of softmax, and the drawing uses 3 in place of e, which is close enough to do in your head. = 7. Sum = We add up each column: 16, 6, 7 and 12. = 8. Normalize = Let us divide every cell by its column sum. That gives the attention weight matrix in yellow, and each of its four columns is now a probability distribution over the four positions. The decimals are nudged as they are rounded, so every column still sums to exactly 1. = 9. MatMul = We multiply the value vectors by those weights. Each output is a blend of all four values, mixed in the proportion the attention matrix just decided, and it goes to the position-wise feed forward network in the next layer: the FFN box at the bottom of the page. The outputs: Attention weights (A), by column = [.2, .6, 0, .2], [.2, .4, .2, .2], [.4, .2, 0, .4], [.1, .7, .1, .1] Attention weighted features (Z) = [8, 2, 6], [8, 4, 4], [16, 4, 2], [4, 2, 7] The takeaway: attention is a weighted average, and everything before step 9 exists to decide the weights. Compare every position with every other, turn the scores into one distribution per position, then blend. 💾 Save this post!

Tom Yeh

27,383 次观看 • 3 个月前

the "thinking" text Claude shows you is not its thinking it's the part it chose to write down the real reasoning runs in a layer you never see Anthropic just proved it we learned to read the output and never noticed we were reading the summary

the "thinking" text Claude shows you is not its thinking it's the part it chose to write down the real reasoning runs in a layer you never see Anthropic just proved it we learned to read the output and never noticed we were reading the summary

Lagosaur

31,211 次观看 • 4 个月前

ResNet by hand ✍️ ~ 10 steps walkthrough below "Deep Residual Learning for Image Recognition" (Kaiming He, CVPR 2016) is among the most cited papers in all of deep learning. Why does it matter so much? It fixed the exploding and vanishing gradients that kept deep networks from being deep, and made thousands of layers possible. How simple was the fix? An identity matrix. Goal: push three input vectors through a residual block, then through a transformer encoder block, filling in every cell yourself. = 1. Given = A mini batch of three input vectors, 3D, and the weights of the layers ahead. = 2. Linear layer = Let us multiply by the weights, add the bias, and apply ReLU so negatives become 0. Three feature vectors out. This is F(X). = 3. Concatenate = Now the trick. Stack an identity matrix beside the second layer's weights, and stack the input vectors under the features. Draw the lines between rows and columns: those are the skip connections. The identity is the residual. = 4. Linear layer + identity = We multiply the two stacked matrices. The identity carries X straight through while the weights transform it, so a single multiplication computes F(X) + X. Apply ReLU and hand it to the next block. Now watch the same trick inside a transformer, first in attention. = 5. Attention = Let us take three input vectors in 2D, compute the attention matrix, and multiply to get attention weighted vectors. = 6. Concatenate = We stack two identities this time, two residuals, which is how you get 1 + 1, and stack the input vectors with the attention weighted ones. = 7. Add = Multiply the stacked matrices. The identity adds attention to its own input, across the columns, which is how positions get combined. And again in the feed forward layer. = 8. First layer = Let us multiply by the feed forward weights and bias, then ReLU. Three feature vectors. = 9. Concatenate = Stack and link exactly as in step 3: the residual again. = 10. Second layer + identity = We multiply, apply ReLU, and pass the result to the next encoder block. This identity adds across the rows, combining features rather than positions. Takeaway: one simple "add" is what made really deep networks possible. 💾 Save this post!

ResNet by hand ✍️ ~ 10 steps walkthrough below "Deep Residual Learning for Image Recognition" (Kaiming He, CVPR 2016) is among the most cited papers in all of deep learning. Why does it matter so much? It fixed the exploding and vanishing gradients that kept deep networks from being deep, and made thousands of layers possible. How simple was the fix? An identity matrix. Goal: push three input vectors through a residual block, then through a transformer encoder block, filling in every cell yourself. = 1. Given = A mini batch of three input vectors, 3D, and the weights of the layers ahead. = 2. Linear layer = Let us multiply by the weights, add the bias, and apply ReLU so negatives become 0. Three feature vectors out. This is F(X). = 3. Concatenate = Now the trick. Stack an identity matrix beside the second layer's weights, and stack the input vectors under the features. Draw the lines between rows and columns: those are the skip connections. The identity is the residual. = 4. Linear layer + identity = We multiply the two stacked matrices. The identity carries X straight through while the weights transform it, so a single multiplication computes F(X) + X. Apply ReLU and hand it to the next block. Now watch the same trick inside a transformer, first in attention. = 5. Attention = Let us take three input vectors in 2D, compute the attention matrix, and multiply to get attention weighted vectors. = 6. Concatenate = We stack two identities this time, two residuals, which is how you get 1 + 1, and stack the input vectors with the attention weighted ones. = 7. Add = Multiply the stacked matrices. The identity adds attention to its own input, across the columns, which is how positions get combined. And again in the feed forward layer. = 8. First layer = Let us multiply by the feed forward weights and bias, then ReLU. Three feature vectors. = 9. Concatenate = Stack and link exactly as in step 3: the residual again. = 10. Second layer + identity = We multiply, apply ReLU, and pass the result to the next encoder block. This identity adds across the rows, combining features rather than positions. Takeaway: one simple "add" is what made really deep networks possible. 💾 Save this post!

Tom Yeh

18,152 次观看 • 3 个月前

A million-token window solved the wrong problem, and Kimi K3 shipping one made that harder to see. 2.8T parameters, weights on your disk, an entire codebase in one prompt. Feels like memory. Isn't. Memory is what survives the session ending. A window is what survives the next token. Self-hosting sharpens it. Now you pay in GPU seconds, not API credits. Every re-read of the same history burns the same compute. Owning the weights didn't make the past free. Kimi Delta Attention cuts what long context costs. Cheaper repetition is still repetition. And a bigger window buys the one thing you don't want: room to keep everything. Nothing in a transcript is dated, ranked, or retired. Old and current load at identical weight. The fix runs beside the model, not inside it. Capture rejects, consolidation merges, decay forgets on schedule. Then a million tokens becomes headroom instead of a bill. You fill 5,000 and leave the rest empty. Look at what your longest thread reloads every run. Then ask how much of it earned the slot.

A million-token window solved the wrong problem, and Kimi K3 shipping one made that harder to see. 2.8T parameters, weights on your disk, an entire codebase in one prompt. Feels like memory. Isn't. Memory is what survives the session ending. A window is what survives the next token. Self-hosting sharpens it. Now you pay in GPU seconds, not API credits. Every re-read of the same history burns the same compute. Owning the weights didn't make the past free. Kimi Delta Attention cuts what long context costs. Cheaper repetition is still repetition. And a bigger window buys the one thing you don't want: room to keep everything. Nothing in a transcript is dated, ranked, or retired. Old and current load at identical weight. The fix runs beside the model, not inside it. Capture rejects, consolidation merges, decay forgets on schedule. Then a million tokens becomes headroom instead of a bill. You fill 5,000 and leave the rest empty. Look at what your longest thread reloads every run. Then ask how much of it earned the slot.

wast3

17,620 次观看 • 2 个月前