Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

A tricky LLM interview question: You're serving a reasoning model on vLLM, and long traces keep pushing it out of GPU memory. You add KV cache compression and evict 90% of the cached tokens. VRAM usage doesn't move, and the GPU still runs out of memory. Why? (answer below)...

41,511 Aufrufe • vor 15 Tagen •via X (Twitter)

31 Kommentare

Profilbild von K4k0
K4k0vor 15 Tagen

vllm grabs like 90% of the vram at startup anyway… and if the evicted tokens are spread all over the pages, almost no full block gets freed

Profilbild von zoel
zoelvor 15 Tagen

And which AI model help you write this down? Interview question made by AI with best answer also written by AI, presented here for human AI-expert candidate. Nobody can catchup by doing this.

Profilbild von Modelplane
Modelplanevor 15 Tagen

The tell is that VRAM didn't budge at all. Evicting 90% of cached tokens should free something, so the memory is almost certainly held elsewhere: preallocated KV blocks or a fixed gpu_memory_utilization pool that vLLM won't shrink, plus fragmentation from long traces. Compression

Profilbild von Jatin Sharma
Jatin Sharmavor 15 Tagen

my guess: vLLM reserves the whole KV pool at startup so VRAM never moves, and evicting tokens scattered across blocks frees almost no full blocks. so the pool still fills up

Profilbild von Hailu 🍏
Hailu 🍏vor 15 Tagen

This is a great technical deep dive into LLM KV cache management.

Profilbild von Max Petrusenko
Max Petrusenkovor 15 Tagen

does the compaction pass every 128 tokens stall decoding while it moves tokens, or does it overlap with the next step?

Profilbild von Saikat | Springg
Saikat | Springgvor 15 Tagen

My guess is vLLM preallocates the KV cache blocks up front, so evicting tokens frees slots inside the pool but never hands memory back. Is the real OOM coming from activations?

Profilbild von Unique Adhikari
Unique Adhikarivor 15 Tagen

this is why i'd make eviction block-aware, either dropping whole 16-token blocks by summed importance or compacting the survivors into fewer blocks, because the allocator only frees at block granularity. so the metric worth watching is free blocks, not cached tokens.

Profilbild von vnmsfx
vnmsfxvor 15 Tagen

interview questions like this keep me humble

Profilbild von Soumil Bhandari
Soumil Bhandarivor 15 Tagen

Same trap as unstructured pruning. Zero out 90% of the weights in a dense layout and you save zero bytes and zero FLOPs. It only pays off once the sparsity has structure the hardware can use, like 2:4 or whole blocks. Compaction is the KV cache version of that.

Profilbild von 掷三十
掷三十vor 14 Tagen

我按 700 亿参数那档模型算过:一个词的键值缓存约 32 万字节。推理模型吐 8000 词的思维链,光缓存就占掉 2.6 GB。压掉九成省不出显存,因为这些块会回到池子等人复用,显存一直被预分配的池占着。先压单次请求上限,再换键值头更少的模型。 #X创作分成 #蓝V创作者 #AI时代 #一人公司

Profilbild von Marlowe Drake
Marlowe Drakevor 15 Tagen

Compression saves space but if your batch size is still huge you're just refilling it. What's your max_num_seqs set to?

Profilbild von Vesper🪐
Vesper🪐vor 15 Tagen

This is a fantastic breakdown of a complex LLM issue.

Profilbild von Marco Polo
Marco Polovor 15 Tagen

Love this breakdown! Why would vLLM rather hallucinate kv cache on a spreadsheet for 9,000 tokens than just enter values into sheets to get accurate results? The biggest heist of your money in human kind! They forbade the calculator? Who let the Math out?! Even frontier cloud Gemini dropped to 65.7% (46/70) doing token math, then jumped to 100% (70/70) the instant MDCC entered the values into 0.37ms code. Please give your LLM and the AI giants a calculator as code:

Profilbild von Maga Musk
Maga Muskvor 15 Tagen

vLLM preallocates the KV block pool, so evicting 90% of cached tokens only frees slots inside reserved blocks — memory never returns. The real OOM is activations + the preallocated pool. Cap block count and drop max num sequences to shrink the fixed reservation.

Profilbild von Architecting Intelligence
Architecting Intelligencevor 15 Tagen

Another wrinkle: prefix caching. If system-prompt blocks are shared across requests, per-request eviction inside them forces copy-on-write, so "compression" can raise memory. Safer to pin shared-prefix blocks and only compact the request-private tail, where the CoT lives anyway.

Profilbild von Dipanshu Kushwaha
Dipanshu Kushwahavor 15 Tagen

That's a clever one! Really makes you think about how these systems work under the hood.

Profilbild von Aleksander Kóska
Aleksander Kóskavor 15 Tagen

Because vLLM’s PagedAttention pre-allocates memory blocks up front. Evicting tokens logically doesn’t release or compact physical pages back to the CUDA allocator. Unless you de-fragment and re-pool the freed memory blocks, VRAM usage stays flat. Classic buffer trap.

Profilbild von Callahan
Callahanvor 15 Tagen

vi is the one i'd watch here

Profilbild von AiMarket24.pro
AiMarket24.provor 15 Tagen

My guess: vLLM reserves gpu_memory_utilization up front, so evicting tokens frees slots inside its own pool, not VRAM. And when the evicted tokens are scattered, very few 16-token blocks actually empty out, so the allocator still runs dry.

Profilbild von Somu
Somuvor 15 Tagen

checkout as well

Profilbild von ImaTrumpDeplorable
ImaTrumpDeplorablevor 14 Tagen

Should be using Strata...

Profilbild von kalpataru Sahoo
kalpataru Sahoovor 14 Tagen

The part about measuring blocks freed instead of tokens evicted is what makes this interesting. A smaller cache doesn't always mean less GPU memory.

Profilbild von Wbackdown
Wbackdownvor 15 Tagen

Is the answer that the model weights themselves are the fixed cost eating the VRAM?

Profilbild von Aritro Roy
Aritro Royvor 15 Tagen

postgres has the same problem. delete 90% of a table's rows and it stays the same size on disk until something rewrites the pages

Profilbild von The SI Maximalist
The SI Maximalistvor 15 Tagen

VLLM preallocates its KV pool, so token eviction doesn’t return VRAM for activations or workspaces that can still cause OOM.

Profilbild von Adriano Gomes | AI Agents + Humans in the Loop
Adriano Gomes | AI Agents + Humans in the Loopvor 15 Tagen

My guess: vLLM reserves the KV cache pool up front from gpu_memory_utilization. Evicting tokens frees blocks inside that pool, not VRAM. If it still OOMs, the pressure is outside the pool, like activations on long prefills. Chunked prefill helps there.

Profilbild von Daksh Agarwal
Daksh Agarwalvor 15 Tagen

blocks freed vs tokens evicted is such a clean way to put it. curious about the compaction pass every 128 tokens though, moving retained kv forward isn't free. does the 2.5x decode number already include that copy cost?

Profilbild von Lexi Learn
Lexi Learnvor 15 Tagen

a 90% smaller cache that frees zero blocks is just tidier paperwork.

Profilbild von Flavius Burca
Flavius Burcavor 15 Tagen

Worth adding that compaction itself isn't free: moving retained KV forward every 128 tokens is extra memory traffic on a decode path that's already bandwidth-bound. The win depends on trace length and batch size, so test at your real concurrency, not one request.

Profilbild von Kuldeep Pisda
Kuldeep Pisdavor 15 Tagen

and with prefix caching on, the prompt blocks are shared across requests, so the compaction pass can't touch those without copying first

Ähnliche Videos

An interesting new approach to agentic RL: Researchers found a way to train AI agents nearly 2x faster. Agentic RL usually produces long rollouts, and when a rollout reaches its context budget, a compaction policy removes older turns and keeps the prompt, recent turns, or a summary. Most systems use re-prefill compaction. They create a shorter sequence and run every retained token through the model again. This rebuilds the KV cache after each compaction, so the same tokens may be processed several times. KV-streams is a new approach to solve this. Instead of rebuilding the shorter sequence, KV-streams removes discarded turns directly from the live KV cache. Generation continues from the entries that survived. In practice, you cannot just reuse those entries as is. It requires two fixes. > First, RoPE has already rotated each cached Key using its original token position. A Key may move from physical slot 1,000 to slot 500 after compaction, but the model must still treat it as position 1,000. KV-streams keeps physical storage and logical position separate. Retained Keys keep their rotations. New Queries and Keys continue from the original position count. > Second, vLLM does not store each KV entry independently. It groups them into fixed 16-token blocks and uses a block table to find those blocks in GPU memory. KV-streams cannot leave empty slots inside that table because the attention kernel may read the wrong memory. It therefore removes complete blocks and joins the remaining block references together. This introduces another problem. A conversation turn may end halfway through a 16-token block. In their vLLM setup, that partial block is not committed to the prefix cache and must be recomputed when the next turn arrives. If compaction happens first, those tokens may have originally attended to context that is no longer present. For the vLLM experiments, the researchers pad each completion to a multiple of 16 tokens. This commits every generated token to the cache before the next turn or compaction event. The SGLang setup uses single-token blocks and needs no padding. Separately, the researchers found that token-level eviction produced degenerate text and required substantial supervised fine-tuning. They suspect that removing message-boundary tokens disrupted the chat format learned during training. The agent experiments therefore remove complete turns, including their boundary tags. Once inference works this way, training must follow the same process. A normal trainer would either process the full rollout or rebuild the shorter sequence after compaction. Both options produce different KV entries from those used during generation. KV-streams records every compaction event during generation. The trainer then reproduces that history: retained tokens keep the states they formed before compaction, while later tokens are blocked from attending to spans that inference had already removed. In the repository’s validation runs, the difference between training and generation stayed near a mismatch KL of 0.001, indicating that both paths produced almost identical token probabilities. On SWE-bench Verified, the Markovian Thinker setup with KV-streams reached peak performance in about 32 hours. The same compaction strategy with re-prefill took roughly 60 hours, while both reached comparable final scores. The speedup mainly applies to RL training, where one rollout may be compacted several times. At production inference, batching can hide some prefill cost, and keeping caches for inactive users reduces serving capacity. There is one more detail. The retained KV entries were created while the removed turns were still visible. Future tokens cannot attend to those turns directly, but some of their influence may remain in the entries that were not evicted. In a controlled test, RL recovered an assignment after its original message was removed. Recall reached 100% when enough cache remained. KV-streams does not decide what the model should forget. Instead, it makes existing compaction policies cheaper to train by preserving the KV state they choose to retain. Repo: If you want to dive deeper into the broader KV-cache engineering stack, I wrote an article that covers eviction, paging, quantization, offloading, and architectural compression. Read it below.

Avi Chawla

18,595 Aufrufe • vor 1 Monat

Researchers made LLM inference 14x faster and 90% cheaper. The video below depicts the speed up in action. Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely. For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization. But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed. Three common request patterns produce full cache misses: - A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them. - The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed. - In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix. Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits. So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent. Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in. There's a second, less discussed problem as well. Cache management runs inside the inference engine's process. Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources. This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process. Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel. Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries. LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM. The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing. On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. GitHub repo: (don't forget to star 🌟) My co-founder wrote a full breakdown of KV cache management. It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset. Read it below.

Avi Chawla

30,717 Aufrufe • vor 4 Monaten

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Avi Chawla

16,140 Aufrufe • vor 3 Monaten

Redis built a cache that cuts LLM costs by 90%! Production LLM apps do not receive completely new questions every time. A customer-support assistant might receive all three of these: - "Can I get a refund after buying the monthly plan?" - "Is the monthly subscription refundable?" - "Can I cancel the plan and get my money back?" The wording is different, but the underlying question and its answer remain the same. Yet LLM apps process every version as a new request. They assemble the prompt, send it to the model, and generate an answer that may have already been generated. Prefix caching reduces part of these repeated calls. When requests begin with the same system prompt or context, the model can reuse the KV states already computed for that shared prefix. But the request still hits the LLM. The new tokens must be processed, and the complete answer must still be decoded. So even with a prefix-cache hit, there's another generation call involved. To solve this, instead of only caching computation inside the model, the application can cache the generated response outside it. When another question arrives, the system embeds it and compares it with previously answered questions. If it finds a sufficiently close match, it returns the stored response without invoking the LLM again. A cache hit removes the input tokens, output tokens, and decoding time associated with another LLM call. In practice, it is important to decide which questions can safely share an answer since a production setup needs well-tuned similarity thresholds, expiration policies, data isolation, and monitoring for incorrect matches. If you want to use this in practice, Redis already implements it as a managed service called Redis LangCache. Under the hood, it generates embeddings, searches previous responses, and returns a matching answer before another model call occurs. Redis also handles access scopes, custom filtering, TTL and eviction controls, and cache monitoring through Redis Cloud. I built an interface to compare it against direct LLM inference. The video below shows this in action, and I worked with Redis on this post to put this together. For the paraphrased question in my run, direct inference took 2.232 seconds and consumed 514 input tokens plus 250 output tokens. Redis returned the earlier response in 0.37 seconds with zero LLM input or output tokens. That was roughly 6x faster in this run. Redis reports API cost savings of up to 90% and cache-hit responses up to 15x faster. The actual result depends on how much safe repetition exists in the workload. You can try Redis LangCache here: If you want to dive deeper, I have already written a detailed breakdown of KV, prefix, prompt, and semantic caching in the article quoted below. This demo builds on the final technique and shows it running in practice. Read it below.

Avi Chawla

166,230 Aufrufe • vor 2 Monaten

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

158,352 Aufrufe • vor 6 Monaten

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

346,057 Aufrufe • vor 2 Monaten

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 ÷ 5.2 ≈ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: ↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once. ↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. ↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it 🌟) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.

Avi Chawla

128,761 Aufrufe • vor 21 Tagen

Qwen3.8-Flash-Next is still going strong at 364.7K tokens of context on an M5 Max. And this isn’t just a static long-context test. The model was reasoning about how to speed up its own workflow while using tools, and the tool calls kept working without misses. Setup: • Qwen3.8-Flash-Next • M5 Max • 128GB unified memory • MLX-Serve PR #363 • OpenCode 2 • 364.7K context The interesting part isn’t simply getting hundreds of thousands of tokens into memory. It’s what happens once the context gets this large. Long-context inference usually comes with a painful tradeoff. As the KV cache grows, memory pressure increases and generation can slow down. But this setup is still pushing through 364K tokens while maintaining a usable agent workflow. The model can reason, call tools, inspect results, continue working, and keep the session moving. And the tool calls reportedly haven’t missed so far. That’s important for agentic coding. A huge context window is only useful if the model can actually operate reliably inside it. A 400K-token context that constantly breaks tool calls isn’t very useful. A 364K session that can keep reasoning and executing tools is a different story. And the test isn’t finished yet. The current run is approaching 400K tokens, with the expectation that it can keep going. This is also another interesting example of why Apple Silicon keeps showing up in local LLM experiments. The M5 Max’s unified memory gives a large model and its growing KV cache access to one shared memory pool. With MLX-Serve continuing to improve, these machines are becoming surprisingly capable long-context inference boxes. The bigger takeaway: Context length is becoming a workload, not just a model specification. Running a model at 256K is one thing. Keeping an agent alive at 300K+ while it reasons and uses tools is much more interesting. And Qwen3.8-Flash-Next is showing that this can be pushed surprisingly far on a single 128GB Mac. 364.7K and counting. Next stop: 400K.

FHILY👑

39,982 Aufrufe • vor 2 Monaten