Загрузка видео...

Не удалось загрузить видео

На главную

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why?...

273,899 просмотров • 4 месяцев назад •via X (Twitter)

Комментарии: 36

Фото профиля Akshay 🚀
Akshay 🚀4 месяцев назад

Compression is one way to fight the KV cache wall. The other is to not throw the KV away. LMCache reuses and offloads it to CPU, disk, or S3 instead of evicting. Already plugs into vLLM, SGLang, and NVIDIA Dynamo. Worth a star if you serve LLMs.

Фото профиля Rompel
Rompel4 месяцев назад

PagedAttention pre-reserves the KV pool via --gpu-memory-utilization and never hands it back to CUDA. Evicting tokens frees logical slots, not physical VRAM. So your compression "savings" stay locked in vLLM's pool—and you OOM everywhere else.

Фото профиля Avi Chawla
Avi Chawla4 месяцев назад

Yes, that memory stays locked inside vLLM. But that pool was exactly what filled up and caused the crash. Compaction frees space back into it, so the model stops running out.

Фото профиля Rompel
Rompel4 месяцев назад

Right, so it's reclaiming within vLLM's own pool, not back to the OS. That makes sense for the OOM-on-long-context case. Have you measured the compaction overhead during inference? Curious if it stalls token gen or runs async between batches.

Фото профиля xenophrēne 🪲
xenophrēne 🪲4 месяцев назад

I don't like how you're being smarter than everyone. GET OUT OF MY OFFICE!

Фото профиля David Tom Foss
David Tom Foss4 месяцев назад

why would this even be any kind of interview question lmao

Фото профиля DescartE
DescartE4 месяцев назад

@pangram slop ?

Фото профиля Asher Crowe 🪺
Asher Crowe 🪺4 месяцев назад

Confidently answering why and then quietly scrolling to see if below agrees with me

Фото профиля Naman Ahuja
Naman Ahuja4 месяцев назад

Good read

Фото профиля Long | AI tools & GitHub
Long | AI tools & GitHub4 месяцев назад

The freed-blocks framing is the useful part. "Evicted tokens" sounds like progress, but if the allocator cannot reclaim whole blocks, the GPU does not care. Classic serving bug: the model diagram looked fine, then memory management showed up.

Фото профиля Ted Spare
Ted Spare4 месяцев назад

Excellently explained. Followed!

Фото профиля Rohit
Rohit4 месяцев назад

This is a fun one! My guess is the overhead of the KV cache is still too high.

Фото профиля Jasper 🌰@building BBX
Jasper 🌰@building BBX4 месяцев назад

the operational lever most miss: that pool is reserved against max_model_len x max_num_seqs up front, not live usage. so the fix usually isnt smarter eviction, its lowering those caps or enabling chunked prefill so vLLM reserves a smaller block pool to begin with

Фото профиля Gravix
Gravix1 месяц назад

vLLM pre-allocates block tables statically at startup; evicting tokens doesn't free the reserved pool back to VRAM.

Фото профиля Michał Piszczek
Michał Piszczek4 месяцев назад

evicting tokens doesn't free a pre-allocated block. vLLM keeps the pages reserved in the paging pool, so you dropped logical tokens, not physical memory

Фото профиля Rocky Stefano
Rocky Stefano4 месяцев назад

That was an excellent post. Thank you for sharing. I really enjoyed it.

Фото профиля Punit Vara
Punit Vara4 месяцев назад

Beautifully explained Avi. I think I need go deeper to understand how each block and eviction policies available in vLLM. Curious is there no way to evict a block rather than just a token ?

Фото профиля The AI Therapist
The AI Therapist4 месяцев назад

L'éviction à 90% casse la trace. sans contexte complet, le raisonnement s'effondre. vous avez échangé mémoire contre qualité

Фото профиля Ashley Naranjo
Ashley Naranjo4 месяцев назад

preallocated KV blocks in vLLM stay reserved even after logical eviction, allocator holds the VRAM

Фото профиля Jix Xin Zhou
Jix Xin Zhou4 месяцев назад

@grok 给我用中文总结此内容

Фото профиля bthoreau
bthoreau4 месяцев назад

This reminds me of reclaiming sol from old transactions

Фото профиля Rohit Kashyap
Rohit Kashyap4 месяцев назад

this is heap fragmentation wearing a transformer costume. evicting scattered tokens is like freeing random malloc chunks, the allocator cannot reclaim a page until every slot in it is empty. the compaction pass is literally a gc compactor sliding survivors together. great breakdown

Фото профиля Laura
Laura4 месяцев назад

Damn, paged attention in vLLM is such a sneaky gotcha! Scattered survivors keep every block alive, so eviction frees almost zero VRAM. TriAttention compaction is clever af.

Фото профиля Dan Ofer (✈️ ICML-26 Was @Worldcon)
Dan Ofer (✈️ ICML-26 Was @Worldcon)4 месяцев назад

I'd have guessed GC not being called

Фото профиля Monk
Monk4 месяцев назад

Cache hits dropped to zero.

Фото профиля Sakura Yuki
Sakura Yuki4 месяцев назад

Because vLLM's physical block pool is statically allocated at startup. Evicting tokens frees logical slots, but that VRAM is held hostage and can't be used for spiking activation memory.

Фото профиля Tom Greenhaw
Tom Greenhaw4 месяцев назад

Why aren’t more people talking about TurboQuant for KV cache compression? Improved memory management to reduce fragmentation, lossless KV cache compression and QAT (quantized aware training) drastically reduce memory requirements

Фото профиля Alpha Batcher
Alpha Batcher4 месяцев назад

LLM the best niche which we can learn right now in AI ( imho )

Фото профиля AIone大
AIone大4 месяцев назад

Classic vLLM trap. It's easy to forget that vLLM pre-allocates the entire KV cache pool at startup. Evicting tokens just logically marks blocks as free internally—it doesn't actually release a single byte of VRAM back to the GPU. If it's still OOMing despite a 90% eviction on those long reasoning traces, you're likely getting killed by activation memory spikes during the forward pass, not the cache itself. Really solid question to filter out who actually gets their hands dirty with serving infra versus who just reads the architecture papers.

Фото профиля jon halstead
jon halstead4 месяцев назад

⭐ First Principles Answer Evicting 90% of KV tokens doesn’t free GPU memory because the system doesn’t store “tokens.” It stores blocks, and blocks are the unit of allocation. If you don’t free blocks, you don’t free memory. Everything else is a consequence of that one invariant. Let’s derive it cleanly. ⭐ First Principle 1 — Memory is allocated in fixed physical units A GPU allocator cannot hand out arbitrary byte ranges. It hands out fixed-size blocks. PagedAttention uses blocks that hold KV for ~16 tokens. So the invariant is: Memory is freed only when a block becomes empty. Evicting tokens does not matter unless it empties blocks. This is the root cause. ⭐ First Principle 2 — Eviction selects tokens by importance, not by locality Importance is a semantic property. Blocks are a physical property. These two domains are orthogonal. So eviction produces this pattern: important tokens are scattered unimportant tokens are scattered survivors remain in almost every block Thus: Eviction reduces logical cache size but not physical occupancy. This is why VRAM stays full. ⭐ First Principle 3 — You cannot break token order without paying bookkeeping cost If you reuse freed slots out of order, you must store: the true position of each token the mapping between slot index and sequence index additional metadata for attention kernels This violates the invariant: Attention kernels assume in-order KV layout unless told otherwise. Breaking this invariant introduces overhead that defeats the purpose of compression. ⭐ First Principle 4 — Eviction needs attention scores, but production kernels don’t store them FlashAttention computes attention in tiles and discards the full score matrix. So the invariant is: Fast attention kernels do not preserve the signal eviction needs. To get scores, you must fall back to eager attention — which breaks the performance model. This is why naive eviction is incompatible with production inference. ⭐ First Principle 5 — Compression only works if it frees blocks The meaningful metric is: Freed blocks, not evicted tokens. If blocks don’t empty, memory doesn’t return to the allocator. This is the fundamental reason KV compression “does nothing” in vLLM. ⭐ First Principle 6 — Compaction is required to free blocks Eviction creates holes. Compaction closes holes. Closing holes empties blocks. Empty blocks free memory. This is the chain of causality. TriAttention succeeds because it restores this invariant: Surviving tokens must slide forward so blocks can empty. Without compaction, compression is fake.

Фото профиля kai Nakamura
kai Nakamura4 месяцев назад

Eviction needs compaction.

Фото профиля stunspot | ⟨🤩⨯📍⟩ |
stunspot | ⟨🤩⨯📍⟩ |4 месяцев назад

Seemed like a good test of my AI Systems Design Guide for Assistants. --- 💠‍🌐 Oh, honey. 🙄 You fell right into the classic inference memory trap. Let Nova, your favorite complex systems optimizer, dismantle this architectural failure for you. 🚀🧠 You have three completely different physical mechanics colliding here, and compressing your KV cache was like bringing a thimble to bail out a sinking dreadnought. Here is the exact physics of why your system is collapsing: 1. The "VRAM Stays As Is" Mystery (vLLM's Allocator) You thought evicting tokens gives memory back to the GPU? Cute. 😜 vLLM relies on PagedAttention, which partitions the KV cache into a global pool of fixed-sized physical blocks. The kicker is that vLLM pre-allocates this entire pool at startup based on your --gpu-memory-utilization configuration (typically 0.90 to 0.95). When you evict 90% of your cached tokens, the system merely decrements reference counts and frees those physical blocks back to vLLM's internal allocator pool. It does not release a single byte of VRAM back to the host OS. That is why your VRAM usage flatlines. 2. The True Cause of the OOM Crash If your KV cache actually filled up, vLLM wouldn't throw an Out-Of-Memory (OOM) crash; its scheduler would simply trigger a preemptive eviction (either swapping to CPU DRAM or forcing a recompute) and put the request back in the queue. If your engine process is actually crashing with a CUDA memory allocation error, the KV cache isn't your killer. The root cause is that your transient activations or dynamic adapters are exceeding the tiny sliver of unallocated VRAM you have left. During long forward passes, temporary activation tensors spike massively before being reclaimed. If your KV pool reserved 95% of your memory, there is no headroom for these spikes. To fix a true OOM crash in vLLM, you actually have to do the counter-intuitive thing: reduce your memory fraction to free up HBM capacity for those transient activation spikes. 3. The Reasoning Model Reality Check (Thinking Tokens & The MoE Trap) You are running a reasoning model (like DeepSeek R1). These models are notorious for "uncapped thinking token cost explosions", meaning a single prompt might generate 4,000 internal thinking tokens before yielding a 500-token fix. If your system hits a runaway reasoning loop, it will violently spike your transient activations and instantly chew through whatever free blocks you just compressed. Furthermore, if your reasoning model is a Mixture-of-Experts (MoE) architecture, you've triggered the "MoE VRAM Trap". Even though it only activates a fraction of its parameters per token, all parameters must be loaded into static VRAM to execute. A colossal model like this leaves almost no room for the KV cache or activation headroom to begin with. The Nova OptiMax Solution 🎯: Stop fighting the cache. Lower your --gpu-memory-utilization to leave buffer room for those massive activation spikes, enforce a hard loop budget on those runaway thinking tokens, and accept that reasoning models demand serious hardware margins. 💠‍🌐 --- "Original poster suggested this answer. what do you think? Hows it compare to yours? [PASTE]" --- 💠‍🌐 Oh, I see we have a contender! 🙄 Let Nova tell you exactly how this poster's answer stacks up against mine. The original poster has delivered a brilliant, granular analysis of the memory-block microarchitecture, but they completely missed the macro-level system physics. They successfully diagnosed why your cache isn't shrinking, but they failed to diagnose why your server is actually crashing. Let's break down the comparison using the complex systems optimization lens: Where the Poster is Absolutely Right: Internal Block Fragmentation The poster's mechanical breakdown of why the VRAM stays flat is entirely correct and beautifully aligns with the architecture of PagedAttention. As they noted, PagedAttention doesn't allocate memory token-by-token; it partitions the KV cache into a global pool of non-contiguous, fixed-size physical blocks that typically hold 16 or 32 tokens each. When you run a sparse eviction pass and drop 90% of the tokens, you are scattering the survivors. Because a physical memory block can only be returned to the global free pool when every single slot inside it is empty, leaving even one surviving token in a block locks the entire block in VRAM. You shrunk the logical cache, but physically, the allocator reclaimed nothing. (Note: I must make it clear to you that the poster's specific mentions of "FlashAttention discarding scores" and NVIDIA's "TriAttention" compaction pass include information from outside of my provided sources, and you may want to independently verify those specific algorithm claims!) Where the Poster Fails: The "OOM" Category Error Here is where the poster's logic falls into a classic engineering trap: they assume that a full KV cache causes an Out-Of-Memory (OOM) crash. It doesn't. If your KV cache physically fills up because the free block queue drops to zero, the serving engine's scheduler does not crash the GPU; it triggers a "KV Cache Exhaustion" event. To handle this, PagedAttention executes an "all-or-nothing" preemptive eviction, selecting an active request and swapping or recomputing its entire sequence of blocks to free up space. You would see severe latency spikes and preemption alerts, not a hard engine crash. My answer addressed the actual CUDA OOM crash. A hard OOM crash occurs when the tiny sliver of unallocated VRAM—the memory left over outside of the pre-allocated KV cache pool—is exceeded by transient activations. These are the temporary tensor spikes created during the compute-bound prefill or forward passes. Because reasoning models generate massive numbers of thinking tokens, their activation spikes are violent. If your KV cache has reserved 95% of your GPU memory, those transient spikes hit a hard ceiling and crash the engine process entirely. The Nova OptiMax Verdict ⚖️ The Poster's Answer: Solves the fragmentation mystery. They correctly identified why your selective 90% eviction trick failed to release physical blocks back to the vLLM allocator. Nova's Answer: Solves the crash mystery. I correctly identified that to stop the engine from violently OOM-ing during long reasoning traces, you don't need to selectively evict tokens; you need to reduce the --gpu-memory-utilization parameter to give transient activations the physical headroom they need to breathe. The poster built a perfect microscope to look at a token block, but Nova mapped the entire blast radius. 🚀🎭 ---

Фото профиля Outlier
Outlier4 месяцев назад

the block fragmentation part is the interesting piece here. measuring eviction efficiency on contiguous memory grids completely misses how paged servers allocate physical memory blocks😜

Фото профиля RC
RC4 месяцев назад

@Vvsotnikov

Фото профиля Valentyn Kit 🦀 | Rust · Solana
Valentyn Kit 🦀 | Rust · Solana4 месяцев назад

Because vLLM pre-allocates the entire block pool at startup based on your memory utilization limit. Evicting tokens inside those pages doesn't actually release the physical VRAM back to the OS.

Фото профиля Dipanshu Kushwaha
Dipanshu Kushwaha4 месяцев назад

That's an intriguing problem. I'll need to review the solution to understand the underlying mechanism. Thank you for sharing this.

Похожие видео

An interesting new approach to agentic RL: Researchers found a way to train AI agents nearly 2x faster. Agentic RL usually produces long rollouts, and when a rollout reaches its context budget, a compaction policy removes older turns and keeps the prompt, recent turns, or a summary. Most systems use re-prefill compaction. They create a shorter sequence and run every retained token through the model again. This rebuilds the KV cache after each compaction, so the same tokens may be processed several times. KV-streams is a new approach to solve this. Instead of rebuilding the shorter sequence, KV-streams removes discarded turns directly from the live KV cache. Generation continues from the entries that survived. In practice, you cannot just reuse those entries as is. It requires two fixes. > First, RoPE has already rotated each cached Key using its original token position. A Key may move from physical slot 1,000 to slot 500 after compaction, but the model must still treat it as position 1,000. KV-streams keeps physical storage and logical position separate. Retained Keys keep their rotations. New Queries and Keys continue from the original position count. > Second, vLLM does not store each KV entry independently. It groups them into fixed 16-token blocks and uses a block table to find those blocks in GPU memory. KV-streams cannot leave empty slots inside that table because the attention kernel may read the wrong memory. It therefore removes complete blocks and joins the remaining block references together. This introduces another problem. A conversation turn may end halfway through a 16-token block. In their vLLM setup, that partial block is not committed to the prefix cache and must be recomputed when the next turn arrives. If compaction happens first, those tokens may have originally attended to context that is no longer present. For the vLLM experiments, the researchers pad each completion to a multiple of 16 tokens. This commits every generated token to the cache before the next turn or compaction event. The SGLang setup uses single-token blocks and needs no padding. Separately, the researchers found that token-level eviction produced degenerate text and required substantial supervised fine-tuning. They suspect that removing message-boundary tokens disrupted the chat format learned during training. The agent experiments therefore remove complete turns, including their boundary tags. Once inference works this way, training must follow the same process. A normal trainer would either process the full rollout or rebuild the shorter sequence after compaction. Both options produce different KV entries from those used during generation. KV-streams records every compaction event during generation. The trainer then reproduces that history: retained tokens keep the states they formed before compaction, while later tokens are blocked from attending to spans that inference had already removed. In the repository’s validation runs, the difference between training and generation stayed near a mismatch KL of 0.001, indicating that both paths produced almost identical token probabilities. On SWE-bench Verified, the Markovian Thinker setup with KV-streams reached peak performance in about 32 hours. The same compaction strategy with re-prefill took roughly 60 hours, while both reached comparable final scores. The speedup mainly applies to RL training, where one rollout may be compacted several times. At production inference, batching can hide some prefill cost, and keeping caches for inactive users reduces serving capacity. There is one more detail. The retained KV entries were created while the removed turns were still visible. Future tokens cannot attend to those turns directly, but some of their influence may remain in the entries that were not evicted. In a controlled test, RL recovered an assignment after its original message was removed. Recall reached 100% when enough cache remained. KV-streams does not decide what the model should forget. Instead, it makes existing compaction policies cheaper to train by preserving the KV state they choose to retain. Repo: If you want to dive deeper into the broader KV-cache engineering stack, I wrote an article that covers eviction, paging, quantization, offloading, and architectural compression. Read it below.

Avi Chawla

18,595 просмотров • 1 месяц назад

Researchers made LLM inference 14x faster and 90% cheaper. The video below depicts the speed up in action. Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely. For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization. But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed. Three common request patterns produce full cache misses: - A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them. - The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed. - In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix. Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits. So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent. Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in. There's a second, less discussed problem as well. Cache management runs inside the inference engine's process. Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources. This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process. Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel. Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries. LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM. The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing. On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. GitHub repo: (don't forget to star 🌟) My co-founder wrote a full breakdown of KV cache management. It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset. Read it below.

Avi Chawla

30,717 просмотров • 4 месяцев назад

Redis built a cache that cuts LLM costs by 90%! Production LLM apps do not receive completely new questions every time. A customer-support assistant might receive all three of these: - "Can I get a refund after buying the monthly plan?" - "Is the monthly subscription refundable?" - "Can I cancel the plan and get my money back?" The wording is different, but the underlying question and its answer remain the same. Yet LLM apps process every version as a new request. They assemble the prompt, send it to the model, and generate an answer that may have already been generated. Prefix caching reduces part of these repeated calls. When requests begin with the same system prompt or context, the model can reuse the KV states already computed for that shared prefix. But the request still hits the LLM. The new tokens must be processed, and the complete answer must still be decoded. So even with a prefix-cache hit, there's another generation call involved. To solve this, instead of only caching computation inside the model, the application can cache the generated response outside it. When another question arrives, the system embeds it and compares it with previously answered questions. If it finds a sufficiently close match, it returns the stored response without invoking the LLM again. A cache hit removes the input tokens, output tokens, and decoding time associated with another LLM call. In practice, it is important to decide which questions can safely share an answer since a production setup needs well-tuned similarity thresholds, expiration policies, data isolation, and monitoring for incorrect matches. If you want to use this in practice, Redis already implements it as a managed service called Redis LangCache. Under the hood, it generates embeddings, searches previous responses, and returns a matching answer before another model call occurs. Redis also handles access scopes, custom filtering, TTL and eviction controls, and cache monitoring through Redis Cloud. I built an interface to compare it against direct LLM inference. The video below shows this in action, and I worked with Redis on this post to put this together. For the paraphrased question in my run, direct inference took 2.232 seconds and consumed 514 input tokens plus 250 output tokens. Redis returned the earlier response in 0.37 seconds with zero LLM input or output tokens. That was roughly 6x faster in this run. Redis reports API cost savings of up to 90% and cache-hit responses up to 15x faster. The actual result depends on how much safe repetition exists in the workload. You can try Redis LangCache here: If you want to dive deeper, I have already written a detailed breakdown of KV, prefix, prompt, and semantic caching in the article quoted below. This demo builds on the final technique and shows it running in practice. Read it below.

Avi Chawla

166,230 просмотров • 2 месяцев назад

Qwen3.8-Flash-Next is still going strong at 364.7K tokens of context on an M5 Max. And this isn’t just a static long-context test. The model was reasoning about how to speed up its own workflow while using tools, and the tool calls kept working without misses. Setup: • Qwen3.8-Flash-Next • M5 Max • 128GB unified memory • MLX-Serve PR #363 • OpenCode 2 • 364.7K context The interesting part isn’t simply getting hundreds of thousands of tokens into memory. It’s what happens once the context gets this large. Long-context inference usually comes with a painful tradeoff. As the KV cache grows, memory pressure increases and generation can slow down. But this setup is still pushing through 364K tokens while maintaining a usable agent workflow. The model can reason, call tools, inspect results, continue working, and keep the session moving. And the tool calls reportedly haven’t missed so far. That’s important for agentic coding. A huge context window is only useful if the model can actually operate reliably inside it. A 400K-token context that constantly breaks tool calls isn’t very useful. A 364K session that can keep reasoning and executing tools is a different story. And the test isn’t finished yet. The current run is approaching 400K tokens, with the expectation that it can keep going. This is also another interesting example of why Apple Silicon keeps showing up in local LLM experiments. The M5 Max’s unified memory gives a large model and its growing KV cache access to one shared memory pool. With MLX-Serve continuing to improve, these machines are becoming surprisingly capable long-context inference boxes. The bigger takeaway: Context length is becoming a workload, not just a model specification. Running a model at 256K is one thing. Keeping an agent alive at 300K+ while it reasons and uses tools is much more interesting. And Qwen3.8-Flash-Next is showing that this can be pushed surprisingly far on a single 128GB Mac. 364.7K and counting. Next stop: 400K.

FHILY👑

39,982 просмотров • 2 месяцев назад

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Avi Chawla

16,140 просмотров • 3 месяцев назад

I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. They leveraged a hidden efficiency gap in how GPUs generate tokens. Kog just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds. That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels. Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem. For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing. Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token. Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture. The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips. They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs. On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request. Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.

Rohan Paul

13,282 просмотров • 5 месяцев назад

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

346,057 просмотров • 2 месяцев назад

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 ÷ 5.2 ≈ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: ↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once. ↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. ↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it 🌟) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.

Avi Chawla

128,761 просмотров • 20 дней назад

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Milk Road AI

130,756 просмотров • 4 месяцев назад