Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

LLM inference speed with vs. without KV caching: (learn how and why it works below)

Avi Chawla

79,092 subscribers

396,012 görüntüleme • 7 ay önce •via X (Twitter)

Bilim & Teknoloji

Anya Rossi• Live Now

Private livecam show

13 Yorum

0x_Vivek profil fotoğrafı

0x_Vivek7 ay önce

kv caching is basically the llm refusing to do the same math twice. saving memory so it can hallucinate faster.

Ivan Zhang profil fotoğrafı

Ivan Zhang7 ay önce

This is awesome, thanks Avi.

Finding Moat profil fotoğrafı

Finding Moat7 ay önce

What laptop/pc are you using?

Eric Henderson profil fotoğrafı

Eric Henderson7 ay önce

I have much to learn

Jayakrishnan R profil fotoğrafı

Jayakrishnan R7 ay önce

Entire MQA/GQA Attention is built over this

Ghulam profil fotoğrafı

Ghulam7 ay önce

Uses the same peak memory tho

0x001A profil fotoğrafı

0x001A7 ay önce

Bookmarked.

Açoine profil fotoğrafı

Açoine7 ay önce

V cool

Dwgs profil fotoğrafı

Dwgs7 ay önce

kv cache really makes the speed jump crazy fast

Imtiaz Nabi profil fotoğrafı

Imtiaz Nabi7 ay önce

@_avichawla great tut as usual Is there any better and effective way or KV caching is still SOTA?

Xiayi Sun profil fotoğrafı

Xiayi Sun7 ay önce

KV caching feels like the first optimization everyone learns, but the interesting question is where memory bandwidth becomes the real bottleneck.

Xiayi Sun profil fotoğrafı

Xiayi Sun7 ay önce

KV caching feels like the moment LLMs became real systems engineering problems, where memory bandwidth can matter as much as model quality.

Vijay Khambe profil fotoğrafı

Vijay Khambe7 ay önce

KV cache removes recomputation. It shifts decode from compute-bound → memory-bound. But at scale, KV itself becomes the bottleneck.

Benzer Videolar

𝗞𝗩 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝗘𝘅𝗽𝗹𝗮𝗶𝗻𝗲𝗱 Ever wondered how LLMs remember thousands of tokens without processing everything from scratch every time? That's where KV Caching comes in. When an LLM generates a response, it doesn't need to recompute the Key and Value representations of all previous tokens at every step. Instead, it stores them in a cache and reuses them while generating the next token. Sounds simple, but this makes a huge difference in inference speed and computational efficiency. Here's what happens: - During prefill, the model processes the prompt and stores the Keys and Values. - During decoding, it computes Q, K and V only for the new token. - The new K and V are added to the existing cache. - Previous KV values are reused instead of being recomputed. But there's one important thing to understand: KV Caching doesn't eliminate attention over previous tokens. It eliminates the need to recompute their Keys and Values. As context grows, KV cache memory can become a major bottleneck. That's why techniques like: - MQA (Multi-Query Attention) - GQA (Grouped-Query Attention) - PagedAttention - Prefix Caching play an important role in making LLM inference more efficient. Understanding KV Caching is essential if you're learning how LLM inference actually works or building systems that serve LLMs at scale.

𝗞𝗩 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝗘𝘅𝗽𝗹𝗮𝗶𝗻𝗲𝗱 Ever wondered how LLMs remember thousands of tokens without processing everything from scratch every time? That's where KV Caching comes in. When an LLM generates a response, it doesn't need to recompute the Key and Value representations of all previous tokens at every step. Instead, it stores them in a cache and reuses them while generating the next token. Sounds simple, but this makes a huge difference in inference speed and computational efficiency. Here's what happens: - During prefill, the model processes the prompt and stores the Keys and Values. - During decoding, it computes Q, K and V only for the new token. - The new K and V are added to the existing cache. - Previous KV values are reused instead of being recomputed. But there's one important thing to understand: KV Caching doesn't eliminate attention over previous tokens. It eliminates the need to recompute their Keys and Values. As context grows, KV cache memory can become a major bottleneck. That's why techniques like: - MQA (Multi-Query Attention) - GQA (Grouped-Query Attention) - PagedAttention - Prefix Caching play an important role in making LLM inference more efficient. Understanding KV Caching is essential if you're learning how LLM inference actually works or building systems that serve LLMs at scale.

𝗿𝗮𝗺𝗮𝗸𝗿𝘂𝘀𝗵𝗻𝗮— 𝗲/𝗮𝗰𝗰

22,663 görüntüleme • 4 gün önce

Batching strategies in LLM inference, clearly explained! (bookmark it) - Static - Dynamic - And continuous batching I wrote a detailed article explaining how each works and why serving LLMs is a different problem from traditional ML inference. The article is quoted below.

Batching strategies in LLM inference, clearly explained! (bookmark it) - Static - Dynamic - And continuous batching I wrote a detailed article explaining how each works and why serving LLMs is a different problem from traditional ML inference. The article is quoted below.

Akshay 🚀

30,234 görüntüleme • 2 ay önce

KV caching is the fundamental optimization underpinning autoregressive LLM inference. Transformer layers store keys and values from earlier tokens, then reuse them as new tokens are generated. This avoids recomputation, but at the cost of memory capacity and bandwidth.

KV caching is the fundamental optimization underpinning autoregressive LLM inference. Transformer layers store keys and values from earlier tokens, then reuse them as new tokens are generated. This avoids recomputation, but at the cost of memory capacity and bandwidth.

Alec Helbling

36,773 görüntüleme • 2 ay önce

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

Reese Chong

53,026 görüntüleme • 6 ay önce

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Andrew Ng

104,727 görüntüleme • 2 yıl önce

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

h100envy

61,470 görüntüleme • 4 ay önce

ROTATIONAL ARTHERECTOMY Learn how it works below 👇🏼

ROTATIONAL ARTHERECTOMY Learn how it works below 👇🏼

Pharm. Billy-young

1,517,631 görüntüleme • 3 ay önce

I built an interactive simulator that shows why LLM inference capacity is hard. Batching, queues, KV cache, GPU decode, priority tiers - all coupled. Change one knob and the bottleneck moves.

I built an interactive simulator that shows why LLM inference capacity is hard. Batching, queues, KV cache, GPU decode, priority tiers - all coupled. Change one knob and the bottleneck moves.

Khushboo Verma

25,217 görüntüleme • 5 ay önce

Watch Luba Kravchenko's Ship 2025 session to learn how Vercel's framework-defined infrastructure simplifies caching and content delivery. See how it works, the difference between CDN and runtime caches, and a deep dive into revalidation with Incremental Static Regeneration (ISR).

Watch Luba Kravchenko's Ship 2025 session to learn how Vercel's framework-defined infrastructure simplifies caching and content delivery. See how it works, the difference between CDN and runtime caches, and a deep dive into revalidation with Incremental Static Regeneration (ISR).

Vercel

29,830 görüntüleme • 1 yıl önce

Have you heard of v-tone before? Learn how it works below:

Have you heard of v-tone before? Learn how it works below:

Pharm. Billy-young

4,606,109 görüntüleme • 6 ay önce

Woah... This is the fastest Transformers.js has ever been! 🤯 Run a 1.7B LLM 100% locally in your browser at over 130 tokens per second! 🚀 No server required. ⚡️ WebGPU-accelerated in-browser inference 📦 Optimized ONNX exports w/ GQA 💬 Multi-round conversations w/ KV caching

Woah... This is the fastest Transformers.js has ever been! 🤯 Run a 1.7B LLM 100% locally in your browser at over 130 tokens per second! 🚀 No server required. ⚡️ WebGPU-accelerated in-browser inference 📦 Optimized ONNX exports w/ GQA 💬 Multi-round conversations w/ KV caching

Xenova

54,099 görüntüleme • 1 yıl önce

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

Andrew Ng

103,664 görüntüleme • 7 ay önce

Soly has tried TheStack. It works. We brought Sasho MacKenzie on the pod to explain how and why it works, and how you can add speed to your golf swing. Spotify: YouTube:

Soly has tried TheStack. It works. We brought Sasho MacKenzie on the pod to explain how and why it works, and how you can add speed to your golf swing. Spotify: YouTube:

No Laying Up

45,949 görüntüleme • 2 yıl önce

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 görüntüleme • 2 ay önce

Reasoning from scratch round 2: In this video, I cover the text generation process in LLMs and KV caching (to prepare the base model before adding reasoning techniques in the upcoming ones). 00:00 Introduction and reasoning model demo 01:55 How to work through the book 05:00 Chapter 2 overview 08:25 Checking PyTorch and hardware support 10:26 Apple silicon and MPS caveats 15:00 Cloud GPU options 16:08 Tokens and tokenization 18:20 Qwen3 and the Reasoning From Scratch package 23:05 Encoding and decoding text 26:24 Downloading weights and selecting a device 31:01 Loading the pretrained Qwen3 model 34:32 How LLMs generate text 36:47 Input tensors and batch dimensions 41:48 Running the model in inference mode 44:11 Logits and next-token predictions 49:21 Greedy decoding with argmax 52:28 Building a streaming text generator 01:01:28 Generating text and handling end-of-sequence tokens 01:06:00 Benchmarking text generation 01:14:34 How KV caching works 01:17:22 Adding KV caching and measuring the speedup 01:24:31 Model compilation with torch.compile 01:30:33 Combining compilation with KV caching 01:32:53 Comparing CPU and GPU performance 01:35:32 Recap and next steps

Reasoning from scratch round 2: In this video, I cover the text generation process in LLMs and KV caching (to prepare the base model before adding reasoning techniques in the upcoming ones). 00:00 Introduction and reasoning model demo 01:55 How to work through the book 05:00 Chapter 2 overview 08:25 Checking PyTorch and hardware support 10:26 Apple silicon and MPS caveats 15:00 Cloud GPU options 16:08 Tokens and tokenization 18:20 Qwen3 and the Reasoning From Scratch package 23:05 Encoding and decoding text 26:24 Downloading weights and selecting a device 31:01 Loading the pretrained Qwen3 model 34:32 How LLMs generate text 36:47 Input tensors and batch dimensions 41:48 Running the model in inference mode 44:11 Logits and next-token predictions 49:21 Greedy decoding with argmax 52:28 Building a streaming text generator 01:01:28 Generating text and handling end-of-sequence tokens 01:06:00 Benchmarking text generation 01:14:34 How KV caching works 01:17:22 Adding KV caching and measuring the speedup 01:24:31 Model compilation with torch.compile 01:30:33 Combining compilation with KV caching 01:32:53 Comparing CPU and GPU performance 01:35:32 Recap and next steps

Sebastian Raschka

68,300 görüntüleme • 2 ay önce

New course announcement: Semantic Caching for AI Agents, taught by Tyler Hutcherson and Iliya Zhechev from Redis. Semantic caching can significantly reduce your AI application's inference costs and latency. If someone asks "How do I get a refund?" and another later asks "I want my money back," semantic caching recognizes these mean the same thing so it can use a cached response instead of making another model call. This short course takes you from building your first semantic cache from scratch to implementing production-ready systems using Redis' open-source tools. Skills you'll gain: - Build semantic caches from scratch, then implement them using Redis' SDK with production features - Measure cache performance using hit rate, precision, recall, and latency - Enhance accuracy with threshold tuning, cross-encoders, LLM validation, and fuzzy matching Join and learn to reduce your agentic AI's costs and improve speed!

New course announcement: Semantic Caching for AI Agents, taught by Tyler Hutcherson and Iliya Zhechev from Redis. Semantic caching can significantly reduce your AI application's inference costs and latency. If someone asks "How do I get a refund?" and another later asks "I want my money back," semantic caching recognizes these mean the same thing so it can use a cached response instead of making another model call. This short course takes you from building your first semantic cache from scratch to implementing production-ready systems using Redis' open-source tools. Skills you'll gain: - Build semantic caches from scratch, then implement them using Redis' SDK with production features - Measure cache performance using hit rate, precision, recall, and latency - Enhance accuracy with threshold tuning, cross-encoders, LLM validation, and fuzzy matching Join and learn to reduce your agentic AI's costs and improve speed!

Andrew Ng

62,497 görüntüleme • 11 ay önce

Learn how it works

Learn how it works

Savanii

1,722,400 görüntüleme • 1 ay önce

#M5StackNew 🎊 The LLM630 Compute Kit is an #AI large language model (#LLM) inference development kit, powered by the #Axera #AX630C SoC with a 3.2 TOPs NPU, it delivers efficient AI inference for tasks like computer vision (CV) and LLM processing.

#M5StackNew 🎊 The LLM630 Compute Kit is an #AI large language model (#LLM) inference development kit, powered by the #Axera #AX630C SoC with a 3.2 TOPs NPU, it delivers efficient AI inference for tasks like computer vision (CV) and LLM processing.

M5Stack

16,859 görüntüleme • 1 yıl önce

We’re thrilled to open-source TriAttention! 🚀 🦞 Deploy OpenClaw (32B LLM) on a single 24GB RTX 4090 locally 💻Full code open-source & vLLM-ready for one-click deployment ⚡️ 2.5× faster inference speed & 10.7× less KV cache memory usage TriAttention is a novel KV cache compression method built on rigorous trigonometric analysis in the Pre‑RoPE space for efficient LLM long reasoning. Github Repo: Paper Link: Homepage:

We’re thrilled to open-source TriAttention! 🚀 🦞 Deploy OpenClaw (32B LLM) on a single 24GB RTX 4090 locally 💻Full code open-source & vLLM-ready for one-click deployment ⚡️ 2.5× faster inference speed & 10.7× less KV cache memory usage TriAttention is a novel KV cache compression method built on rigorous trigonometric analysis in the Pre‑RoPE space for efficient LLM long reasoning. Github Repo: Paper Link: Homepage:

Yukang Chen

199,463 görüntüleme • 7 ay önce