正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

LLM inference speed with vs. without KV caching: (learn KV cache management in LLMs below)

Avi Chawla

77,660 subscribers

41,032 次观看 • 22 天前 •via X (Twitter)

科学技术教育健康养生

Anya Rossi• Live Now

Private livecam show

8 条评论

Jonathan Sandhu 的头像

Jonathan Sandhu22 天前

Useful taxonomy. The real control is not “KV cache on or off.” It is the model’s cache geometry, the serving engine’s implementation, context and concurrency, cache dtype, paging and eviction policy, offload path, and actual memory topology. We just shipped Aperture Recipe Lab to translate published inference recipes into capability checks for the hardware someone actually owns. Which KV-cache recipe should we encode first?

Misha_Cripto 的头像

Misha_Cripto22 天前

Left side finished the story and went outside. Right side is still explaining that the little ones were playing. KV cache is just the model finally taking notes.

cordivai | Machine Learning & AI 的头像

cordivai | Machine Learning & AI22 天前

Good framing for LLM research work. The practical part is not just trying a stronger model, but logging baselines, data splits, task-specific metrics, and failure cases so the result is reproducible.

Sean Dong 的头像

Sean Dong20 天前

at the short term cost of everyone iPhone is $100-200 more expensive

Tang Vu 的头像

Tang Vu22 天前

Continuous batching changes KV cache overhead significantly, was this measured with static batches or dynamic token-level batching? The difference can be 3-5x on real workloads.

deezzex 的头像

deezzex22 天前

how much of the overhead is from memory access vs actual computation?

Avi Chawla 的头像

Avi Chawla22 天前

Good question. Actually, there is no fixed percentage. It depends on context length, batch size, attention type, precision, GPU, and the serving engine. But during token-by-token decoding, KV-cache attention is usually limited more by memory bandwidth than computation. Prefill is a different phase and is generally more compute-heavy (covered here: At shorter contexts, model-weight reads, projections, MLP layers, communication, and kernel overhead can matter more. As the context grows, KV traffic grows linearly and can dominate. This is why GQA, KV quantization, eviction, and sparse attention improve more than capacity. They reduce the number of bytes each decoding step has to move.

maguyva 的头像

maguyva21 天前

kv caching. you don't respect it til you've run inference without it once. night and day

相关视频

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

Reese Chong

53,026 次观看 • 5 个月前

We’re thrilled to open-source TriAttention! 🚀 🦞 Deploy OpenClaw (32B LLM) on a single 24GB RTX 4090 locally 💻Full code open-source & vLLM-ready for one-click deployment ⚡️ 2.5× faster inference speed & 10.7× less KV cache memory usage TriAttention is a novel KV cache compression method built on rigorous trigonometric analysis in the Pre‑RoPE space for efficient LLM long reasoning. Github Repo: Paper Link: Homepage:

We’re thrilled to open-source TriAttention! 🚀 🦞 Deploy OpenClaw (32B LLM) on a single 24GB RTX 4090 locally 💻Full code open-source & vLLM-ready for one-click deployment ⚡️ 2.5× faster inference speed & 10.7× less KV cache memory usage TriAttention is a novel KV cache compression method built on rigorous trigonometric analysis in the Pre‑RoPE space for efficient LLM long reasoning. Github Repo: Paper Link: Homepage:

Yukang Chen

199,246 次观看 • 5 个月前

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

h100envy

61,470 次观看 • 2 个月前

I built an interactive simulator that shows why LLM inference capacity is hard. Batching, queues, KV cache, GPU decode, priority tiers - all coupled. Change one knob and the bottleneck moves.

I built an interactive simulator that shows why LLM inference capacity is hard. Batching, queues, KV cache, GPU decode, priority tiers - all coupled. Change one knob and the bottleneck moves.

Khushboo Verma

25,217 次观看 • 3 个月前

KV caching is the fundamental optimization underpinning autoregressive LLM inference. Transformer layers store keys and values from earlier tokens, then reuse them as new tokens are generated. This avoids recomputation, but at the cost of memory capacity and bandwidth.

KV caching is the fundamental optimization underpinning autoregressive LLM inference. Transformer layers store keys and values from earlier tokens, then reuse them as new tokens are generated. This avoids recomputation, but at the cost of memory capacity and bandwidth.

Alec Helbling

36,484 次观看 • 24 天前

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Andrew Ng

104,727 次观看 • 2 年前

Woah... This is the fastest Transformers.js has ever been! 🤯 Run a 1.7B LLM 100% locally in your browser at over 130 tokens per second! 🚀 No server required. ⚡️ WebGPU-accelerated in-browser inference 📦 Optimized ONNX exports w/ GQA 💬 Multi-round conversations w/ KV caching

Woah... This is the fastest Transformers.js has ever been! 🤯 Run a 1.7B LLM 100% locally in your browser at over 130 tokens per second! 🚀 No server required. ⚡️ WebGPU-accelerated in-browser inference 📦 Optimized ONNX exports w/ GQA 💬 Multi-round conversations w/ KV caching

Xenova

54,099 次观看 • 1 年前

Researchers made LLM inference 14x faster and 90% cheaper. The video below depicts the speed up in action. Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely. For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization. But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed. Three common request patterns produce full cache misses: - A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them. - The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed. - In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix. Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits. So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent. Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in. There's a second, less discussed problem as well. Cache management runs inside the inference engine's process. Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources. This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process. Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel. Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries. LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM. The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing. On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. GitHub repo: (don't forget to star 🌟) My co-founder wrote a full breakdown of KV cache management. It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset. Read it below.

Researchers made LLM inference 14x faster and 90% cheaper. The video below depicts the speed up in action. Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely. For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization. But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed. Three common request patterns produce full cache misses: - A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them. - The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed. - In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix. Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits. So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent. Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in. There's a second, less discussed problem as well. Cache management runs inside the inference engine's process. Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources. This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process. Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel. Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries. LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM. The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing. On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. GitHub repo: (don't forget to star 🌟) My co-founder wrote a full breakdown of KV cache management. It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset. Read it below.

Avi Chawla

30,717 次观看 • 2 个月前

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 次观看 • 17 天前

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

Andrew Ng

102,747 次观看 • 5 个月前

Teaching Cloudflare Developers Workers AI-powered caching! Built this today: An AI-powered middleware that auto-decides cache durations for every page on my site using Deepseek + Workers KV. Watch it think in real-time ⚡️

Teaching Cloudflare Developers Workers AI-powered caching! Built this today: An AI-powered middleware that auto-decides cache durations for every page on my site using Deepseek + Workers KV. Watch it think in real-time ⚡️

Kristian Freeman

22,631 次观看 • 1 年前

Reasoning from scratch round 2: In this video, I cover the text generation process in LLMs and KV caching (to prepare the base model before adding reasoning techniques in the upcoming ones). 00:00 Introduction and reasoning model demo 01:55 How to work through the book 05:00 Chapter 2 overview 08:25 Checking PyTorch and hardware support 10:26 Apple silicon and MPS caveats 15:00 Cloud GPU options 16:08 Tokens and tokenization 18:20 Qwen3 and the Reasoning From Scratch package 23:05 Encoding and decoding text 26:24 Downloading weights and selecting a device 31:01 Loading the pretrained Qwen3 model 34:32 How LLMs generate text 36:47 Input tensors and batch dimensions 41:48 Running the model in inference mode 44:11 Logits and next-token predictions 49:21 Greedy decoding with argmax 52:28 Building a streaming text generator 01:01:28 Generating text and handling end-of-sequence tokens 01:06:00 Benchmarking text generation 01:14:34 How KV caching works 01:17:22 Adding KV caching and measuring the speedup 01:24:31 Model compilation with torch.compile 01:30:33 Combining compilation with KV caching 01:32:53 Comparing CPU and GPU performance 01:35:32 Recap and next steps

Reasoning from scratch round 2: In this video, I cover the text generation process in LLMs and KV caching (to prepare the base model before adding reasoning techniques in the upcoming ones). 00:00 Introduction and reasoning model demo 01:55 How to work through the book 05:00 Chapter 2 overview 08:25 Checking PyTorch and hardware support 10:26 Apple silicon and MPS caveats 15:00 Cloud GPU options 16:08 Tokens and tokenization 18:20 Qwen3 and the Reasoning From Scratch package 23:05 Encoding and decoding text 26:24 Downloading weights and selecting a device 31:01 Loading the pretrained Qwen3 model 34:32 How LLMs generate text 36:47 Input tensors and batch dimensions 41:48 Running the model in inference mode 44:11 Logits and next-token predictions 49:21 Greedy decoding with argmax 52:28 Building a streaming text generator 01:01:28 Generating text and handling end-of-sequence tokens 01:06:00 Benchmarking text generation 01:14:34 How KV caching works 01:17:22 Adding KV caching and measuring the speedup 01:24:31 Model compilation with torch.compile 01:30:33 Combining compilation with KV caching 01:32:53 Comparing CPU and GPU performance 01:35:32 Recap and next steps

Sebastian Raschka

66,239 次观看 • 22 天前

First local agent to beat Hermes on benchmarks! ✦ runs Qwen, Gemma, Llama via llama.cpp ✦ stable-prefix caching keeps sessions cheap ✦ TurboQuant cuts the KV-cache 6.4× smaller ✦ 37 tasks solved vs Hermes' 31 on GAIA Level 1 Open source on macOS, Windows & Linux 👇

First local agent to beat Hermes on benchmarks! ✦ runs Qwen, Gemma, Llama via llama.cpp ✦ stable-prefix caching keeps sessions cheap ✦ TurboQuant cuts the KV-cache 6.4× smaller ✦ 37 tasks solved vs Hermes' 31 on GAIA Level 1 Open source on macOS, Windows & Linux 👇

Atomic Agent

303,163 次观看 • 2 个月前

How do we cache prompts with LLMs? Interview with Tanishq Singh. This is a real interview question from a big tech company, asked to a candidate in their technical interview round. The video explains the answer in roughly 6 minutes. 00:00 Question - Prompt Caching with LLMs 00:39 Exact Query Caching 02:04 Semantic Caching 04:14 Tradeoffs 05:52 Blooper :p AI engineering program for software engineers: #LLMs #AI #InterviewReady

How do we cache prompts with LLMs? Interview with Tanishq Singh. This is a real interview question from a big tech company, asked to a candidate in their technical interview round. The video explains the answer in roughly 6 minutes. 00:00 Question - Prompt Caching with LLMs 00:39 Exact Query Caching 02:04 Semantic Caching 04:14 Tradeoffs 05:52 Blooper :p AI engineering program for software engineers: #LLMs #AI #InterviewReady

Gaurav Sen

94,937 次观看 • 1 个月前

Introducing DuoAttention: Our new framework slashes both memory and latency for long-context LLMs without sacrificing performance! By applying full KV cache only to critical heads, we achieve: ⚡ 2.55x memory reduction ⚡ 2.18x decoding speedup ⚡ 3.3M tokens on a single A100 GPU

Introducing DuoAttention: Our new framework slashes both memory and latency for long-context LLMs without sacrificing performance! By applying full KV cache only to critical heads, we achieve: ⚡ 2.55x memory reduction ⚡ 2.18x decoding speedup ⚡ 3.3M tokens on a single A100 GPU

Guangxuan Xiao

31,108 次观看 • 1 年前

Batching strategies in LLM inference, clearly explained! (bookmark it) - Static - Dynamic - And continuous batching I wrote a detailed article explaining how each works and why serving LLMs is a different problem from traditional ML inference. The article is quoted below.

Batching strategies in LLM inference, clearly explained! (bookmark it) - Static - Dynamic - And continuous batching I wrote a detailed article explaining how each works and why serving LLMs is a different problem from traditional ML inference. The article is quoted below.

Akshay 🚀

29,245 次观看 • 27 天前

Great visual of NVIDIA’s Inference Context Memory architecture. KV cache offload to NVMe via BlueField-4 DPU this is how Rubin handles massive context windows at scale. Similar to Amazon’s approach but with full-stack hardware integration. H/T 程工

Great visual of NVIDIA’s Inference Context Memory architecture. KV cache offload to NVMe via BlueField-4 DPU this is how Rubin handles massive context windows at scale. Similar to Amazon’s approach but with full-stack hardware integration. H/T 程工

Ben Pouladian

54,335 次观看 • 8 个月前

i spent 3 hours finding the sweet spot for hermes 4.3 36B on a single RTX 3090. saving you the trouble, anon. the model is 21.8GB at Q4_K_M. that leaves 2.2GB free on 24GB VRAM. not much room for KV cache. here's what actually happened: 4K: 35.3 tok/s 8K: 35.2 tok/s 16K: 34.8 tok/s 32K: 34.6 tok/s 64K: 6.4 tok/s 128K: 1.9 tok/s flat from 4K to 32K. then it falls off a cliff at 64K. the trick is quantized KV cache. without it you OOM at 16K. with quantized KV cache you get 32K at full speed. all 64 layers on GPU. at 64K something weird happens. ngl 99 (all layers on GPU) = 3.96 tok/s. the KV cache silently spills to CPU. drop to ngl 55 and speed jumps to 6.37. drop to ngl 48 and it gets worse again (3.46). there's an offload sweet spot where you free just enough VRAM for the cache without losing too much compute to PCIe transfers. 128K works at ngl 32 but you're at 1.95 tok/s. half the model on CPU. usable for batch work, not for interactive. the sweet spot command: llama-server -m hermes-4.3-36b-Q4_K_M.gguf -ngl 99 -c 32768 --cache-type-k q4_0 --cache-type-v q4_0 32K context. 34.6 tok/s. all on GPU. this is where dense 36B lives on 24GB. for comparison, qwen 3.5 (35B MoE, 3B active) holds 112 tok/s from 4K all the way to 262K on the same GPU. no speed drop. same total params, completely different architecture. hybrid linear attention means flat context scaling. dense pays for every token in the KV cache. code and quality comparison coming next. fast vs slow generation side by side in the videos below.

i spent 3 hours finding the sweet spot for hermes 4.3 36B on a single RTX 3090. saving you the trouble, anon. the model is 21.8GB at Q4_K_M. that leaves 2.2GB free on 24GB VRAM. not much room for KV cache. here's what actually happened: 4K: 35.3 tok/s 8K: 35.2 tok/s 16K: 34.8 tok/s 32K: 34.6 tok/s 64K: 6.4 tok/s 128K: 1.9 tok/s flat from 4K to 32K. then it falls off a cliff at 64K. the trick is quantized KV cache. without it you OOM at 16K. with quantized KV cache you get 32K at full speed. all 64 layers on GPU. at 64K something weird happens. ngl 99 (all layers on GPU) = 3.96 tok/s. the KV cache silently spills to CPU. drop to ngl 55 and speed jumps to 6.37. drop to ngl 48 and it gets worse again (3.46). there's an offload sweet spot where you free just enough VRAM for the cache without losing too much compute to PCIe transfers. 128K works at ngl 32 but you're at 1.95 tok/s. half the model on CPU. usable for batch work, not for interactive. the sweet spot command: llama-server -m hermes-4.3-36b-Q4_K_M.gguf -ngl 99 -c 32768 --cache-type-k q4_0 --cache-type-v q4_0 32K context. 34.6 tok/s. all on GPU. this is where dense 36B lives on 24GB. for comparison, qwen 3.5 (35B MoE, 3B active) holds 112 tok/s from 4K all the way to 262K on the same GPU. no speed drop. same total params, completely different architecture. hybrid linear attention means flat context scaling. dense pays for every token in the KV cache. code and quality comparison coming next. fast vs slow generation side by side in the videos below.

Sudo su

52,268 次观看 • 7 个月前

You can't buffer an LLM. Unlike video streaming, where you can render lower-resolution frames while data loads, an LLM requires 100% of its KV cache to generate the next token. This means your slowest chunk dictates your entire inference speed. If you are managing LLM infrastructure, optimizing for average latency isn't enough; your architecture is entirely bound by P99 performance. Samuel Shen breaks down how dropping variable-sized metadata and enforcing fixed-size chunking doubled LMCache's throughput from 2GB/s to 5GB/s. Watch the full technical breakdown here: #LMCache #KVCache #LLM #AIInfrastructure #OpenSource

You can't buffer an LLM. Unlike video streaming, where you can render lower-resolution frames while data loads, an LLM requires 100% of its KV cache to generate the next token. This means your slowest chunk dictates your entire inference speed. If you are managing LLM infrastructure, optimizing for average latency isn't enough; your architecture is entirely bound by P99 performance. Samuel Shen breaks down how dropping variable-sized metadata and enforcing fixed-size chunking doubled LMCache's throughput from 2GB/s to 5GB/s. Watch the full technical breakdown here: #LMCache #KVCache #LLM #AIInfrastructure #OpenSource

LMCache Lab

10,360 次观看 • 3 个月前