正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

LLM inference speed with vs. without KV caching: (learn how and why it works below)

Daily Dose of Data Science

48,955 subscribers

59,218 次观看 • 2 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at

Reese Chong

52,588 次观看 • 2 个月前

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Andrew Ng

104,727 次观看 • 2 年前

Watch Luba Kravchenko's Ship 2025 session to learn how Vercel's framework-defined infrastructure simplifies caching and content delivery. See how it works, the difference between CDN and runtime caches, and a deep dive into revalidation with Incremental Static Regeneration (ISR).

Watch Luba Kravchenko's Ship 2025 session to learn how Vercel's framework-defined infrastructure simplifies caching and content delivery. See how it works, the difference between CDN and runtime caches, and a deep dive into revalidation with Incremental Static Regeneration (ISR).

Vercel

29,652 次观看 • 11 个月前

Have you heard of v-tone before? Learn how it works below:

Have you heard of v-tone before? Learn how it works below:

Pharm. Billy-young

4,586,844 次观看 • 2 个月前

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

Andrew Ng

99,125 次观看 • 3 个月前

Woah... This is the fastest Transformers.js has ever been! 🤯 Run a 1.7B LLM 100% locally in your browser at over 130 tokens per second! 🚀 No server required. ⚡️ WebGPU-accelerated in-browser inference 📦 Optimized ONNX exports w/ GQA 💬 Multi-round conversations w/ KV caching

Woah... This is the fastest Transformers.js has ever been! 🤯 Run a 1.7B LLM 100% locally in your browser at over 130 tokens per second! 🚀 No server required. ⚡️ WebGPU-accelerated in-browser inference 📦 Optimized ONNX exports w/ GQA 💬 Multi-round conversations w/ KV caching

Xenova

54,099 次观看 • 1 年前

Did you know this? It’s called a ring pessary. Learn how it works below.

Did you know this? It’s called a ring pessary. Learn how it works below.

Pharm. Billy-young

2,801,647 次观看 • 2 个月前

Soly has tried TheStack. It works. We brought Sasho MacKenzie on the pod to explain how and why it works, and how you can add speed to your golf swing. Spotify: YouTube:

Soly has tried TheStack. It works. We brought Sasho MacKenzie on the pod to explain how and why it works, and how you can add speed to your golf swing. Spotify: YouTube:

No Laying Up

45,914 次观看 • 1 年前

New course announcement: Semantic Caching for AI Agents, taught by Tyler Hutcherson and Iliya Zhechev from Redis. Semantic caching can significantly reduce your AI application's inference costs and latency. If someone asks "How do I get a refund?" and another later asks "I want my money back," semantic caching recognizes these mean the same thing so it can use a cached response instead of making another model call. This short course takes you from building your first semantic cache from scratch to implementing production-ready systems using Redis' open-source tools. Skills you'll gain: - Build semantic caches from scratch, then implement them using Redis' SDK with production features - Measure cache performance using hit rate, precision, recall, and latency - Enhance accuracy with threshold tuning, cross-encoders, LLM validation, and fuzzy matching Join and learn to reduce your agentic AI's costs and improve speed!

New course announcement: Semantic Caching for AI Agents, taught by Tyler Hutcherson and Iliya Zhechev from Redis. Semantic caching can significantly reduce your AI application's inference costs and latency. If someone asks "How do I get a refund?" and another later asks "I want my money back," semantic caching recognizes these mean the same thing so it can use a cached response instead of making another model call. This short course takes you from building your first semantic cache from scratch to implementing production-ready systems using Redis' open-source tools. Skills you'll gain: - Build semantic caches from scratch, then implement them using Redis' SDK with production features - Measure cache performance using hit rate, precision, recall, and latency - Enhance accuracy with threshold tuning, cross-encoders, LLM validation, and fuzzy matching Join and learn to reduce your agentic AI's costs and improve speed!

Andrew Ng

61,939 次观看 • 7 个月前

We’re thrilled to open-source TriAttention! 🚀 🦞 Deploy OpenClaw (32B LLM) on a single 24GB RTX 4090 locally 💻Full code open-source & vLLM-ready for one-click deployment ⚡️ 2.5× faster inference speed & 10.7× less KV cache memory usage TriAttention is a novel KV cache compression method built on rigorous trigonometric analysis in the Pre‑RoPE space for efficient LLM long reasoning. Github Repo: Paper Link: Homepage:

We’re thrilled to open-source TriAttention! 🚀 🦞 Deploy OpenClaw (32B LLM) on a single 24GB RTX 4090 locally 💻Full code open-source & vLLM-ready for one-click deployment ⚡️ 2.5× faster inference speed & 10.7× less KV cache memory usage TriAttention is a novel KV cache compression method built on rigorous trigonometric analysis in the Pre‑RoPE space for efficient LLM long reasoning. Github Repo: Paper Link: Homepage:

Yukang Chen

197,334 次观看 • 3 个月前

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

Madison Kanna

119,268 次观看 • 3 个月前

in the lectures below, i hold your hand through low-level LLM systems engineering. it includes everything up to TODAY! 1) pytorch tensors 2) large matmul on cpu vs gpu 3) JAX (and why xAI uses it instead of pytorch) 4) raw cuda kernels and global threading indexing 5) triton design philosophy and softmax example 6) HIP kernels 7) mapping out the ENTIRE ecosystem + differences between CUDA and ROCm/HIP (BLAS, FFT, DNN) 8) cutlass and cute-dsl 9) pretraining, finetuning, rl, unsloth, axolotl, megatron-lm, deepspeed, nanogpt, nanochat 10) training vs inference, inference serving problems, throughput vs latency vs concurrency scaling, vllm, sglang, tensorrt-llm, tensorrt, llama.cpp, exllamav2, exllamav3, benchmark comparisons 11) projects/companies using llms to generate SOTA cuda/triton kernels 12) luminal inference 13) mojo/modular/max

in the lectures below, i hold your hand through low-level LLM systems engineering. it includes everything up to TODAY! 1) pytorch tensors 2) large matmul on cpu vs gpu 3) JAX (and why xAI uses it instead of pytorch) 4) raw cuda kernels and global threading indexing 5) triton design philosophy and softmax example 6) HIP kernels 7) mapping out the ENTIRE ecosystem + differences between CUDA and ROCm/HIP (BLAS, FFT, DNN) 8) cutlass and cute-dsl 9) pretraining, finetuning, rl, unsloth, axolotl, megatron-lm, deepspeed, nanogpt, nanochat 10) training vs inference, inference serving problems, throughput vs latency vs concurrency scaling, vllm, sglang, tensorrt-llm, tensorrt, llama.cpp, exllamav2, exllamav3, benchmark comparisons 11) projects/companies using llms to generate SOTA cuda/triton kernels 12) luminal inference 13) mojo/modular/max

Elliot Arledge

57,855 次观看 • 8 个月前

Have you seen lattice multiplication? Here's how it works and why it works!

Have you seen lattice multiplication? Here's how it works and why it works!

Howie Hua

43,530 次观看 • 2 年前

🚨Today we’re rolling out Prompt Caching on GroqCloud. Keep hot prompts in memory, cut cached token costs by 50% and slash latency. Faster response, smarter inference. Learn more 👇

🚨Today we’re rolling out Prompt Caching on GroqCloud. Keep hot prompts in memory, cut cached token costs by 50% and slash latency. Faster response, smarter inference. Learn more 👇

Groq Inc

20,383 次观看 • 10 个月前

Now live: Auto Top-Ups on Hyperbolic Nothing’s more frustrating than having your instance shut down mid-job—or having your inference request fail—just because your balance ran out. Now, your account will automatically recharge when your credits drop below your set threshold, so your workloads keep running without interruption. Learn how it works:

Now live: Auto Top-Ups on Hyperbolic Nothing’s more frustrating than having your instance shut down mid-job—or having your inference request fail—just because your balance ran out. Now, your account will automatically recharge when your credits drop below your set threshold, so your workloads keep running without interruption. Learn how it works:

Hyperbolic

13,892 次观看 • 1 年前

can you chat privately with a cloud llm—*without* sacrificing speed? excited to release minions secure chat: an open-source protocol for end-to-end encrypted llm chat with <1% latency overhead (even @ 30B+ params!). cloud providers can’t peek—messages decrypt only inside a secure gpu enclave, where inference stays fully confidential 🤯 links + code in comments👇

can you chat privately with a cloud llm—without sacrificing speed? excited to release minions secure chat: an open-source protocol for end-to-end encrypted llm chat with <1% latency overhead (even @ 30B+ params!). cloud providers can’t peek—messages decrypt only inside a secure gpu enclave, where inference stays fully confidential 🤯 links + code in comments👇

Avanika Narayan

79,190 次观看 • 1 年前

Great football starts with great passing. In GOALS, charge isn't just about power, it's about intent. Learn how our passing system works and why we built it to be fast, predictable, and responsive ↓

Great football starts with great passing. In GOALS, charge isn't just about power, it's about intent. Learn how our passing system works and why we built it to be fast, predictable, and responsive ↓

GOALS

24,408 次观看 • 1 个月前