A tricky LLM interview question: You're serving a reasoning model on vLLM, and long traces keep pushing it out of GPU memory. You add KV cache compression and evict 90% of the cached tokens. VRAM usage doesn't move, and the GPU still runs out of memory. Why? (answer below)... Evicting 90% of the KV cache can free close to zero bytes on a paged server. During decoding, every new token appends its key and value vectors at every layer. vLLM holds all of them until the request finishes. For reasoning models with long chains of thought, this cache is the main memory cost. A Qwen3-32B with 4-bit weights on a 24GB GPU needs KV for every token of a 32K-token trace, and it hits OOM around 24K tokens. Attention over long contexts is sparse, so you can keep the high-importance tokens and drop the rest with little accuracy loss. That reduces the number of cached tokens but it doesn't necessarily reduce allocated memory. vLLM manages the cache with PagedAttention. It splits GPU memory into fixed physical blocks of 16 token slots each. The allocator reclaims a block only when all 16 slots are empty. Importance-based eviction keeps tokens scattered across the full sequence, so the retained tokens are spread over nearly every block. Evict 14K of 16K tokens across 1,000 blocks, and almost every block still holds at least one token. So the allocator reclaims close to nothing. You could write new tokens into the empty slots. If token 16,001 goes into the slot token 40 held, the block now reads positions 38, 16,001, 41. The cache is no longer in position order. Attention still gives the correct result, but each slot now needs a stored position index, and every kernel has to read it. An in-order layout doesn't need that metadata. So the cache shrinks 90% logically and 0% physically. Many compression papers miss this because they benchmark on pre-allocated contiguous tensors instead of a paged server. The second problem is the eviction signal. Most eviction methods rank tokens by their attention scores. FlashAttention never materializes those scores. It computes attention tile by tile in on-chip SRAM and discards each score tile after use, which is where its speed comes from. To get the scores, you have to fall back to eager attention and build the full score matrix in HBM, which removes the speedup FlashAttention provides. NVIDIA's TriAttention handles both problems. It ranks tokens without attention scores. It uses the geometry of the query and key vectors before RoPE is applied, where they form stable clusters. Every 128 decoded tokens, it runs a compaction pass. Retained tokens move forward to fill the evicted slots, so trailing blocks empty out completely and return to the allocator, and the cache stays in position order. On long reasoning traces, it matches full-attention accuracy with 2.5x faster decoding and 10.7x less KV memory. For KV cache compression on a real server, the metric to track is blocks freed, not tokens evicted. NVIDIA's write-up: I also wrote a first-principles breakdown of the KV cache. It covers why the model caches keys and values, why the cache grows with every token, and how generation speed compares with and without caching. Read it below.show more

vllm grabs like 90% of the vram at startup anyway… and if the evicted tokens are spread all over the pages, almost no full block gets freed

And which AI model help you write this down? Interview question made by AI with best answer also written by AI, presented here for human AI-expert candidate. Nobody can catchup by doing this.

The tell is that VRAM didn't budge at all. Evicting 90% of cached tokens should free something, so the memory is almost certainly held elsewhere: preallocated KV blocks or a fixed gpu_memory_utilization pool that vLLM won't shrink, plus fragmentation from long traces. Compression

my guess: vLLM reserves the whole KV pool at startup so VRAM never moves, and evicting tokens scattered across blocks frees almost no full blocks. so the pool still fills up

This is a great technical deep dive into LLM KV cache management.

does the compaction pass every 128 tokens stall decoding while it moves tokens, or does it overlap with the next step?

My guess is vLLM preallocates the KV cache blocks up front, so evicting tokens frees slots inside the pool but never hands memory back. Is the real OOM coming from activations?

this is why i'd make eviction block-aware, either dropping whole 16-token blocks by summed importance or compacting the survivors into fewer blocks, because the allocator only frees at block granularity. so the metric worth watching is free blocks, not cached tokens.

interview questions like this keep me humble

Same trap as unstructured pruning. Zero out 90% of the weights in a dense layout and you save zero bytes and zero FLOPs. It only pays off once the sparsity has structure the hardware can use, like 2:4 or whole blocks. Compaction is the KV cache version of that.

我按 700 亿参数那档模型算过:一个词的键值缓存约 32 万字节。推理模型吐 8000 词的思维链,光缓存就占掉 2.6 GB。压掉九成省不出显存,因为这些块会回到池子等人复用,显存一直被预分配的池占着。先压单次请求上限,再换键值头更少的模型。 #X创作分成 #蓝V创作者 #AI时代 #一人公司

Compression saves space but if your batch size is still huge you're just refilling it. What's your max_num_seqs set to?

This is a fantastic breakdown of a complex LLM issue.

Love this breakdown! Why would vLLM rather hallucinate kv cache on a spreadsheet for 9,000 tokens than just enter values into sheets to get accurate results? The biggest heist of your money in human kind! They forbade the calculator? Who let the Math out?! Even frontier cloud Gemini dropped to 65.7% (46/70) doing token math, then jumped to 100% (70/70) the instant MDCC entered the values into 0.37ms code. Please give your LLM and the AI giants a calculator as code:

vLLM preallocates the KV block pool, so evicting 90% of cached tokens only frees slots inside reserved blocks — memory never returns. The real OOM is activations + the preallocated pool. Cap block count and drop max num sequences to shrink the fixed reservation.

Another wrinkle: prefix caching. If system-prompt blocks are shared across requests, per-request eviction inside them forces copy-on-write, so "compression" can raise memory. Safer to pin shared-prefix blocks and only compact the request-private tail, where the CoT lives anyway.

That's a clever one! Really makes you think about how these systems work under the hood.

Because vLLM’s PagedAttention pre-allocates memory blocks up front. Evicting tokens logically doesn’t release or compact physical pages back to the CUDA allocator. Unless you de-fragment and re-pool the freed memory blocks, VRAM usage stays flat. Classic buffer trap.

vi is the one i'd watch here

My guess: vLLM reserves gpu_memory_utilization up front, so evicting tokens frees slots inside its own pool, not VRAM. And when the evicted tokens are scattered, very few 16-token blocks actually empty out, so the allocator still runs dry.

checkout as well

Should be using Strata...

The part about measuring blocks freed instead of tokens evicted is what makes this interesting. A smaller cache doesn't always mean less GPU memory.

Is the answer that the model weights themselves are the fixed cost eating the VRAM?

postgres has the same problem. delete 90% of a table's rows and it stays the same size on disk until something rewrites the pages

VLLM preallocates its KV pool, so token eviction doesn’t return VRAM for activations or workspaces that can still cause OOM.

My guess: vLLM reserves the KV cache pool up front from gpu_memory_utilization. Evicting tokens frees blocks inside that pool, not VRAM. If it still OOMs, the pressure is outside the pool, like activations on long prefills. Chunked prefill helps there.

blocks freed vs tokens evicted is such a clean way to put it. curious about the compaction pass every 128 tokens though, moving retained kv forward isn't free. does the 2.5x decode number already include that copy cost?

a 90% smaller cache that frees zero blocks is just tidier paperwork.

Worth adding that compaction itself isn't free: moving retained KV forward every 128 tokens is extra memory traffic on a decode path that's already bandwidth-bound. The win depends on trace length and batch size, so test at your real concurrency, not one request.

and with prefix caching on, the prompt blocks are shared across requests, so the compaction pass can't touch those without copying first