Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

LLM inference speed with vs. without KV caching: (learn how and why it works below)

Daily Dose of Data Science

48,955 subscribers

59,218 views • 2 months ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

Reese Chong

52,588 views • 2 months ago

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Andrew Ng

104,727 views • 2 years ago

Watch Luba Kravchenko's Ship 2025 session to learn how Vercel's framework-defined infrastructure simplifies caching and content delivery. See how it works, the difference between CDN and runtime caches, and a deep dive into revalidation with Incremental Static Regeneration (ISR).

Watch Luba Kravchenko's Ship 2025 session to learn how Vercel's framework-defined infrastructure simplifies caching and content delivery. See how it works, the difference between CDN and runtime caches, and a deep dive into revalidation with Incremental Static Regeneration (ISR).

Vercel

29,652 views • 11 months ago

Have you heard of v-tone before? Learn how it works below:

Have you heard of v-tone before? Learn how it works below:

Pharm. Billy-young

4,586,844 views • 2 months ago

Woah... This is the fastest Transformers.js has ever been! 🤯 Run a 1.7B LLM 100% locally in your browser at over 130 tokens per second! 🚀 No server required. ⚡️ WebGPU-accelerated in-browser inference 📦 Optimized ONNX exports w/ GQA 💬 Multi-round conversations w/ KV caching

Woah... This is the fastest Transformers.js has ever been! 🤯 Run a 1.7B LLM 100% locally in your browser at over 130 tokens per second! 🚀 No server required. ⚡️ WebGPU-accelerated in-browser inference 📦 Optimized ONNX exports w/ GQA 💬 Multi-round conversations w/ KV caching

Xenova

54,099 views • 1 year ago

Did you know this? It’s called a ring pessary. Learn how it works below.

Did you know this? It’s called a ring pessary. Learn how it works below.

Pharm. Billy-young

2,801,647 views • 2 months ago

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

Andrew Ng

99,019 views • 3 months ago

Soly has tried TheStack. It works. We brought Sasho MacKenzie on the pod to explain how and why it works, and how you can add speed to your golf swing. Spotify: YouTube:

Soly has tried TheStack. It works. We brought Sasho MacKenzie on the pod to explain how and why it works, and how you can add speed to your golf swing. Spotify: YouTube:

No Laying Up

45,914 views • 1 year ago

New course announcement: Semantic Caching for AI Agents, taught by Tyler Hutcherson and Iliya Zhechev from Redis. Semantic caching can significantly reduce your AI application's inference costs and latency. If someone asks "How do I get a refund?" and another later asks "I want my money back," semantic caching recognizes these mean the same thing so it can use a cached response instead of making another model call. This short course takes you from building your first semantic cache from scratch to implementing production-ready systems using Redis' open-source tools. Skills you'll gain: - Build semantic caches from scratch, then implement them using Redis' SDK with production features - Measure cache performance using hit rate, precision, recall, and latency - Enhance accuracy with threshold tuning, cross-encoders, LLM validation, and fuzzy matching Join and learn to reduce your agentic AI's costs and improve speed!

New course announcement: Semantic Caching for AI Agents, taught by Tyler Hutcherson and Iliya Zhechev from Redis. Semantic caching can significantly reduce your AI application's inference costs and latency. If someone asks "How do I get a refund?" and another later asks "I want my money back," semantic caching recognizes these mean the same thing so it can use a cached response instead of making another model call. This short course takes you from building your first semantic cache from scratch to implementing production-ready systems using Redis' open-source tools. Skills you'll gain: - Build semantic caches from scratch, then implement them using Redis' SDK with production features - Measure cache performance using hit rate, precision, recall, and latency - Enhance accuracy with threshold tuning, cross-encoders, LLM validation, and fuzzy matching Join and learn to reduce your agentic AI's costs and improve speed!

Andrew Ng

61,939 views • 7 months ago

We’re thrilled to open-source TriAttention! 🚀 🦞 Deploy OpenClaw (32B LLM) on a single 24GB RTX 4090 locally 💻Full code open-source & vLLM-ready for one-click deployment ⚡️ 2.5× faster inference speed & 10.7× less KV cache memory usage TriAttention is a novel KV cache compression method built on rigorous trigonometric analysis in the Pre‑RoPE space for efficient LLM long reasoning. Github Repo: Paper Link: Homepage:

We’re thrilled to open-source TriAttention! 🚀 🦞 Deploy OpenClaw (32B LLM) on a single 24GB RTX 4090 locally 💻Full code open-source & vLLM-ready for one-click deployment ⚡️ 2.5× faster inference speed & 10.7× less KV cache memory usage TriAttention is a novel KV cache compression method built on rigorous trigonometric analysis in the Pre‑RoPE space for efficient LLM long reasoning. Github Repo: Paper Link: Homepage:

Yukang Chen

197,334 views • 3 months ago

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

Madison Kanna

119,268 views • 3 months ago

in the lectures below, i hold your hand through low-level LLM systems engineering. it includes everything up to TODAY! 1) pytorch tensors 2) large matmul on cpu vs gpu 3) JAX (and why xAI uses it instead of pytorch) 4) raw cuda kernels and global threading indexing 5) triton design philosophy and softmax example 6) HIP kernels 7) mapping out the ENTIRE ecosystem + differences between CUDA and ROCm/HIP (BLAS, FFT, DNN) 8) cutlass and cute-dsl 9) pretraining, finetuning, rl, unsloth, axolotl, megatron-lm, deepspeed, nanogpt, nanochat 10) training vs inference, inference serving problems, throughput vs latency vs concurrency scaling, vllm, sglang, tensorrt-llm, tensorrt, llama.cpp, exllamav2, exllamav3, benchmark comparisons 11) projects/companies using llms to generate SOTA cuda/triton kernels 12) luminal inference 13) mojo/modular/max

in the lectures below, i hold your hand through low-level LLM systems engineering. it includes everything up to TODAY! 1) pytorch tensors 2) large matmul on cpu vs gpu 3) JAX (and why xAI uses it instead of pytorch) 4) raw cuda kernels and global threading indexing 5) triton design philosophy and softmax example 6) HIP kernels 7) mapping out the ENTIRE ecosystem + differences between CUDA and ROCm/HIP (BLAS, FFT, DNN) 8) cutlass and cute-dsl 9) pretraining, finetuning, rl, unsloth, axolotl, megatron-lm, deepspeed, nanogpt, nanochat 10) training vs inference, inference serving problems, throughput vs latency vs concurrency scaling, vllm, sglang, tensorrt-llm, tensorrt, llama.cpp, exllamav2, exllamav3, benchmark comparisons 11) projects/companies using llms to generate SOTA cuda/triton kernels 12) luminal inference 13) mojo/modular/max

Elliot Arledge

57,855 views • 8 months ago

Have you seen lattice multiplication? Here's how it works and why it works!

Have you seen lattice multiplication? Here's how it works and why it works!

Howie Hua

43,530 views • 2 years ago

🚨Today we’re rolling out Prompt Caching on GroqCloud. Keep hot prompts in memory, cut cached token costs by 50% and slash latency. Faster response, smarter inference. Learn more 👇

🚨Today we’re rolling out Prompt Caching on GroqCloud. Keep hot prompts in memory, cut cached token costs by 50% and slash latency. Faster response, smarter inference. Learn more 👇

Groq Inc

20,383 views • 10 months ago

Now live: Auto Top-Ups on Hyperbolic Nothing’s more frustrating than having your instance shut down mid-job—or having your inference request fail—just because your balance ran out. Now, your account will automatically recharge when your credits drop below your set threshold, so your workloads keep running without interruption. Learn how it works:

Now live: Auto Top-Ups on Hyperbolic Nothing’s more frustrating than having your instance shut down mid-job—or having your inference request fail—just because your balance ran out. Now, your account will automatically recharge when your credits drop below your set threshold, so your workloads keep running without interruption. Learn how it works:

Hyperbolic

13,892 views • 1 year ago

can you chat privately with a cloud llm—*without* sacrificing speed? excited to release minions secure chat: an open-source protocol for end-to-end encrypted llm chat with <1% latency overhead (even @ 30B+ params!). cloud providers can’t peek—messages decrypt only inside a secure gpu enclave, where inference stays fully confidential 🤯 links + code in comments👇

can you chat privately with a cloud llm—without sacrificing speed? excited to release minions secure chat: an open-source protocol for end-to-end encrypted llm chat with <1% latency overhead (even @ 30B+ params!). cloud providers can’t peek—messages decrypt only inside a secure gpu enclave, where inference stays fully confidential 🤯 links + code in comments👇

Avanika Narayan

79,190 views • 1 year ago

Great football starts with great passing. In GOALS, charge isn't just about power, it's about intent. Learn how our passing system works and why we built it to be fast, predictable, and responsive ↓

Great football starts with great passing. In GOALS, charge isn't just about power, it's about intent. Learn how our passing system works and why we built it to be fast, predictable, and responsive ↓

GOALS

24,408 views • 1 month ago