Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

I built an interactive simulator that shows why LLM inference capacity is hard. Batching, queues, KV cache, GPU decode, priority tiers - all coupled. Change one knob and the bottleneck moves.

Khushboo Verma

164,918 subscribers

25,217 просмотров • 5 месяцев назад •via X (Twitter)

Наука и технологии

Anya Rossi• Live Now

Private livecam show

Комментарии: 23

Фото профиля Khushboo Verma

Khushboo Verma5 месяцев назад

The core problem: GPUs hate serving one request at a time. Decode is memory-bandwidth-bound, every forward pass streams the whole model's weights from memory to produce... one token. Serve requests sequentially and your $40k GPU idles between tokens.

Фото профиля Khushboo Verma

Khushboo Verma5 месяцев назад

The fix: batching. One forward pass = one next token for EVERY sequence in the batch. 16 requests, one pass, 16 tokens. The weight reads are shared, so step cost grows much slower than batch size. Throughput is almost free. Almost.

Фото профиля Khushboo Verma

Khushboo Verma5 месяцев назад

But requests finish at different lengths. Static batching strands empty slots until the longest sequence finishes, wasted bandwidth every step. Continuous batching backfills each slot the moment it frees, between decode steps. Utilization goes from ~60% to ~95%.

Фото профиля Khushboo Verma

Khushboo Verma5 месяцев назад

And priority tiers make it worse. Strict priority is great for enterprise traffic and starves free-tier users forever under load. Real schedulers bound starvation: weighted fair queueing, max-wait promotion, reserved slots.

Фото профиля Khushboo Verma

Khushboo Verma5 месяцев назад

The part nobody tells you: the bottleneck doesn't stay put. Long prompts? Prefill steals decode time → inter-token latency spikes Long outputs? Decode-bound → queues explode Small KV cache? Memory blocks admissions while GPUs sit idle Few CPU workers? GPUs starve

Фото профиля Khushboo Verma

Khushboo Verma5 месяцев назад

So I built a simulator where you can watch all of this happen. ⚙️ Tune request rate, token distributions, GPUs, batch size, wait deadline, tier mix, SLOs, see throughput, TTFT, p99 and a live "current bottleneck" banner respond instantly.

Фото профиля Khushboo Verma

Khushboo Verma5 месяцев назад

7 one-click failure scenarios included: - decode saturation - prefill storm - CPU starvation - batching tradeoff - tier starvation - KV cache crunch Each one teaches a different capacity lesson in ~10 seconds. Try now! 👀

Фото профиля PsudoMike 🇨🇦

PsudoMike 🇨🇦5 месяцев назад

KV cache fragmentation is what actually surprises engineers the first time. You've got headroom in theory, but contiguous memory requirements choke it. Spent two weeks tuning batch sizes before realizing the bottleneck was cache eviction, not GPU utilization.

Фото профиля Khushboo Verma

Khushboo Verma5 месяцев назад

Yeah, complex systems

Фото профиля Harnoor Singh

Harnoor Singh5 месяцев назад

That’s a great explanation!

Фото профиля Khushboo Verma

Khushboo Verma5 месяцев назад

glad you liked it :)

Фото профиля Shivay Lamba

Shivay Lamba5 месяцев назад

great visuals!

Фото профиля Khushboo Verma

Khushboo Verma5 месяцев назад

Thanks, so far, haven't had much luck in video generation using AI models, this one turned out nice

Фото профиля David Uche

David Uche5 месяцев назад

Nice stuff

Фото профиля Khushboo Verma

Khushboo Verma5 месяцев назад

Thanks 🙏

Фото профиля NoiselessTrading

NoiselessTrading5 месяцев назад

Insanely well done,! 🙌 The interactive simulator + failure scenarios make LLM serving concepts finally click. Super valuable tool!”

Фото профиля Khushboo Verma

Khushboo Verma5 месяцев назад

Thanks :)

Фото профиля Aniket Maurya

Aniket Maurya5 месяцев назад

great explaination!

Фото профиля Khushboo Verma

Khushboo Verma5 месяцев назад

Thank you :)

Фото профиля PsudoMike 🇨🇦

PsudoMike 🇨🇦5 месяцев назад

Seen this in payments. Optimize auth latency and now settlement throughput is the ceiling. Fix settlement, reconciliation piles up. Every component looks fine in isolation. You only see the coupling when one part starts to scale and the others don't follow.

Фото профиля PsudoMike 🇨🇦

PsudoMike 🇨🇦5 месяцев назад

The KV cache piece is what gets people. Adding more GPUs sounds like the fix but long sequences fill KV memory fast. You end up memory bound before you're ever compute bound. What latency targets are you modeling at different request rates?

Фото профиля Gaurav Nardia

Gaurav Nardia5 месяцев назад

Khushboo, I DMed you something. Can you please check? I think it's in the request tab.

Фото профиля Ishwar | Infrastructure Systems

Ishwar | Infrastructure Systems4 месяцев назад

The moving bottleneck aspect is underrated. A lot of engineering intuition assumes "find bottleneck -> optimize bottleneck." In serving systems, fixing one bottleneck often just reveals the next one.

Похожие видео

Batching strategies in LLM inference, clearly explained! (bookmark it) - Static - Dynamic - And continuous batching I wrote a detailed article explaining how each works and why serving LLMs is a different problem from traditional ML inference. The article is quoted below.

Batching strategies in LLM inference, clearly explained! (bookmark it) - Static - Dynamic - And continuous batching I wrote a detailed article explaining how each works and why serving LLMs is a different problem from traditional ML inference. The article is quoted below.

Akshay 🚀

30,234 просмотров • 2 месяцев назад

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 просмотров • 2 месяцев назад

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

h100envy

61,470 просмотров • 4 месяцев назад

LLM inference speed with vs. without KV caching: (learn how and why it works below)

LLM inference speed with vs. without KV caching: (learn how and why it works below)

Avi Chawla

396,012 просмотров • 7 месяцев назад

𝗞𝗩 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝗘𝘅𝗽𝗹𝗮𝗶𝗻𝗲𝗱 Ever wondered how LLMs remember thousands of tokens without processing everything from scratch every time? That's where KV Caching comes in. When an LLM generates a response, it doesn't need to recompute the Key and Value representations of all previous tokens at every step. Instead, it stores them in a cache and reuses them while generating the next token. Sounds simple, but this makes a huge difference in inference speed and computational efficiency. Here's what happens: - During prefill, the model processes the prompt and stores the Keys and Values. - During decoding, it computes Q, K and V only for the new token. - The new K and V are added to the existing cache. - Previous KV values are reused instead of being recomputed. But there's one important thing to understand: KV Caching doesn't eliminate attention over previous tokens. It eliminates the need to recompute their Keys and Values. As context grows, KV cache memory can become a major bottleneck. That's why techniques like: - MQA (Multi-Query Attention) - GQA (Grouped-Query Attention) - PagedAttention - Prefix Caching play an important role in making LLM inference more efficient. Understanding KV Caching is essential if you're learning how LLM inference actually works or building systems that serve LLMs at scale.

𝗞𝗩 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝗘𝘅𝗽𝗹𝗮𝗶𝗻𝗲𝗱 Ever wondered how LLMs remember thousands of tokens without processing everything from scratch every time? That's where KV Caching comes in. When an LLM generates a response, it doesn't need to recompute the Key and Value representations of all previous tokens at every step. Instead, it stores them in a cache and reuses them while generating the next token. Sounds simple, but this makes a huge difference in inference speed and computational efficiency. Here's what happens: - During prefill, the model processes the prompt and stores the Keys and Values. - During decoding, it computes Q, K and V only for the new token. - The new K and V are added to the existing cache. - Previous KV values are reused instead of being recomputed. But there's one important thing to understand: KV Caching doesn't eliminate attention over previous tokens. It eliminates the need to recompute their Keys and Values. As context grows, KV cache memory can become a major bottleneck. That's why techniques like: - MQA (Multi-Query Attention) - GQA (Grouped-Query Attention) - PagedAttention - Prefix Caching play an important role in making LLM inference more efficient. Understanding KV Caching is essential if you're learning how LLM inference actually works or building systems that serve LLMs at scale.

𝗿𝗮𝗺𝗮𝗸𝗿𝘂𝘀𝗵𝗻𝗮— 𝗲/𝗮𝗰𝗰

23,260 просмотров • 6 дней назад

LLM inference speed with vs. without KV caching: (learn how and why it works below)

LLM inference speed with vs. without KV caching: (learn how and why it works below)

Daily Dose of Data Science

59,218 просмотров • 7 месяцев назад

We’re thrilled to open-source TriAttention! 🚀 🦞 Deploy OpenClaw (32B LLM) on a single 24GB RTX 4090 locally 💻Full code open-source & vLLM-ready for one-click deployment ⚡️ 2.5× faster inference speed & 10.7× less KV cache memory usage TriAttention is a novel KV cache compression method built on rigorous trigonometric analysis in the Pre‑RoPE space for efficient LLM long reasoning. Github Repo: Paper Link: Homepage:

We’re thrilled to open-source TriAttention! 🚀 🦞 Deploy OpenClaw (32B LLM) on a single 24GB RTX 4090 locally 💻Full code open-source & vLLM-ready for one-click deployment ⚡️ 2.5× faster inference speed & 10.7× less KV cache memory usage TriAttention is a novel KV cache compression method built on rigorous trigonometric analysis in the Pre‑RoPE space for efficient LLM long reasoning. Github Repo: Paper Link: Homepage:

Yukang Chen

199,463 просмотров • 7 месяцев назад

Our SVP of Applied AI, Ryan Wang explains the importance of KV Cache management in enabling LLM Inferencing work efficiently at scale. For starters, When an autoregressive LLM generates new text, it looks at all the previous tokens to predict the next one. And to do that, it turns each token into a (K) key (what it’s about) and a (V) value (what it carries). Since the past keys and values do not change, the system stores them instead of recomputing them, And, That is KV cache.

Our SVP of Applied AI, Ryan Wang explains the importance of KV Cache management in enabling LLM Inferencing work efficiently at scale. For starters, When an autoregressive LLM generates new text, it looks at all the previous tokens to predict the next one. And to do that, it turns each token into a (K) key (what it’s about) and a (V) value (what it carries). Since the past keys and values do not change, the system stores them instead of recomputing them, And, That is KV cache.

Nebius

26,867 просмотров • 7 дней назад

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

Reese Chong

53,026 просмотров • 6 месяцев назад

The most important 40 seconds of the Dario pod…. Why do these models feel like magic all of a sudden (fintwit)!? “KV CACHE OPTIMIZATION”…which is still a major major bottleneck btw…

The most important 40 seconds of the Dario pod…. Why do these models feel like magic all of a sudden (fintwit)!? “KV CACHE OPTIMIZATION”…which is still a major major bottleneck btw…

Jared L Kubin

137,743 просмотров • 9 месяцев назад

Jim Cramer on Leopold Aschenbrenner’s $NBIS stake “When this man touches something, it turns to gold.” The next AI bottleneck is efficient inference at scale which is why Nebius Token Factory is built around optimization, orchestration and agentic deployment rather than basic GPU rental.

Jim Cramer on Leopold Aschenbrenner’s $NBIS stake “When this man touches something, it turns to gold.” The next AI bottleneck is efficient inference at scale which is why Nebius Token Factory is built around optimization, orchestration and agentic deployment rather than basic GPU rental.

Shay Boloor

289,153 просмотров • 5 месяцев назад

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

h100envy

32,605 просмотров • 4 месяцев назад

KV caching is the fundamental optimization underpinning autoregressive LLM inference. Transformer layers store keys and values from earlier tokens, then reuse them as new tokens are generated. This avoids recomputation, but at the cost of memory capacity and bandwidth.

KV caching is the fundamental optimization underpinning autoregressive LLM inference. Transformer layers store keys and values from earlier tokens, then reuse them as new tokens are generated. This avoids recomputation, but at the cost of memory capacity and bandwidth.

Alec Helbling

36,773 просмотров • 2 месяцев назад

I made an interactive enzyme simulator. Change the temperature, add or remove inhibitors, and tune the probability that enzymes bind their substrates. Watch as the reaction rate moves up or down in real-time. Maybe this will be a helpful teaching tool.

I made an interactive enzyme simulator. Change the temperature, add or remove inhibitors, and tune the probability that enzymes bind their substrates. Watch as the reaction rate moves up or down in real-time. Maybe this will be a helpful teaching tool.

Niko McCarty.

60,605 просмотров • 1 год назад

The reason I own $AAOI is because it builds the optical connectivity that lets AI chips move data across data centers. As AI scales toward 500K+ GPU factories, copper becomes the bottleneck & optics become plumbing that moves data fast enough for training & inference to work.

The reason I own $AAOI is because it builds the optical connectivity that lets AI chips move data across data centers. As AI scales toward 500K+ GPU factories, copper becomes the bottleneck & optics become plumbing that moves data fast enough for training & inference to work.

Shay Boloor

188,217 просмотров • 5 месяцев назад

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Avi Chawla

16,140 просмотров • 3 месяцев назад

Researchers made LLM inference 14x faster and 90% cheaper. The video below depicts the speed up in action. Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely. For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization. But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed. Three common request patterns produce full cache misses: - A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them. - The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed. - In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix. Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits. So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent. Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in. There's a second, less discussed problem as well. Cache management runs inside the inference engine's process. Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources. This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process. Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel. Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries. LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM. The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing. On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. GitHub repo: (don't forget to star 🌟) My co-founder wrote a full breakdown of KV cache management. It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset. Read it below.

Researchers made LLM inference 14x faster and 90% cheaper. The video below depicts the speed up in action. Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely. For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization. But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed. Three common request patterns produce full cache misses: - A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them. - The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed. - In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix. Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits. So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent. Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in. There's a second, less discussed problem as well. Cache management runs inside the inference engine's process. Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources. This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process. Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel. Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries. LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM. The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing. On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. GitHub repo: (don't forget to star 🌟) My co-founder wrote a full breakdown of KV cache management. It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset. Read it below.

Avi Chawla

30,717 просмотров • 4 месяцев назад

Introducing Universal Simulator (UniSim), an interactive simulator of the real world. Interactive website: Paper:

Introducing Universal Simulator (UniSim), an interactive simulator of the real world. Interactive website: Paper:

Sherry Yang

440,007 просмотров • 3 лет назад

SITUATION EXPLAINED: Cerebras raised $5.55 billion in their IPO and closing their first day of trading valued at $66 billion, making it the biggest US tech IPO since Snowflake in 2020. Cerebras makes Wafer-Scale Engine chips built for AI inference. We asked Sarah Fong the main difference between wafer-scale chips and traditional GPUs: - GPUs are great at parallel work (graphics, training) - AI inference is sequential, AKA one token at a time This causes the "memory wall" problem: - Every GPU core needs model weights, KV cache, and activations to do its math - On a GPU, that data lives in off-chip memory (HBM) - Cores constantly load and offload from off-chip memory, which is a huge bottleneck; hardware accounts for ~70% of inference latency Cerebras' chips: -Dinner-plate sized (vs. GPUs which are palm-sized) with tens of thousands of cores -Memory sits directly on top of the cores as distributed SRAM -Weights and KV cache can be accessed at on-chip speeds in the PB/s range, compared with off-chip speeds in the TB/s range achieved by GPUs with HBM.

SITUATION EXPLAINED: Cerebras raised $5.55 billion in their IPO and closing their first day of trading valued at $66 billion, making it the biggest US tech IPO since Snowflake in 2020. Cerebras makes Wafer-Scale Engine chips built for AI inference. We asked Sarah Fong the main difference between wafer-scale chips and traditional GPUs: - GPUs are great at parallel work (graphics, training) - AI inference is sequential, AKA one token at a time This causes the "memory wall" problem: - Every GPU core needs model weights, KV cache, and activations to do its math - On a GPU, that data lives in off-chip memory (HBM) - Cores constantly load and offload from off-chip memory, which is a huge bottleneck; hardware accounts for ~70% of inference latency Cerebras' chips: -Dinner-plate sized (vs. GPUs which are palm-sized) with tens of thousands of cores -Memory sits directly on top of the cores as distributed SRAM -Weights and KV cache can be accessed at on-chip speeds in the PB/s range, compared with off-chip speeds in the TB/s range achieved by GPUs with HBM.

MTS

44,057 просмотров • 6 месяцев назад

This guy built a real motion racing simulator that actually moves, spins and drifts with the game.

This guy built a real motion racing simulator that actually moves, spins and drifts with the game.

Science girl

41,677 просмотров • 7 месяцев назад