Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Batching strategies in LLM inference, clearly explained! (bookmark it) - Static - Dynamic - And continuous batching I wrote a detailed article explaining how each works and why serving LLMs is a different problem from traditional ML inference. The article is quoted below.

Akshay 🚀

290,555 subscribers

30,234 views • 1 month ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

10 Comments

Hussain Hashim | Building SundayBack1 month ago

@akshay_pachaar always wondered how batching affects LLMs specifically. gonna dive into your article, thanks for breaking it down!

Vikrant Guleria1 month ago

Continuous batching solved the head-of-line blocking problem but introduced its own headache with KV cache fragmentation once you have dozens of sequences finishing at different times. PagedAttention style memory management ends up being just as important as the batching strategy itself.

Gregor1 month ago

hit this last month self-hosting a small model. one long completion was holding the whole batch hostage and my p99 looked terrible while p50 was fine. took embarrassingly long to trace it. does the article explain why most frameworks still default to static?

Workspai1 month ago

LLM inference really is its own kind of systems problem.

Anoy1 month ago

continuous batching is the one that changed everything for me.

DISCOVER Robotics1 month ago

LLM inference isn't just about generating tokens—it's about deciding which tokens to generate next across hundreds of concurrent requests.

Grace Green1 month ago

Great breakdown! Dynamic batching seems like a real game-changer for LLM inference. Looking forward to diving into the article to learn more!

AI Mastery Guide1 month ago

Great breakdown, saving this one

Lee Garcia1 month ago

Nice

Taiko TOEDA1 month ago

One dimension these comparisons almost never carry: batching changes the text, not only the schedule. Someone published a clean measurement of it today, on llama.cpp with greedy decoding at temperature 0. Sequential runs were byte identical. At 64 slots, eight distinct prompts produced 27 distinct answers, and the same prompt returned up to five completions depending on which slot it landed in. The part he did not expect: for two of the three prompts he checked, none of the eight batched slots matched the sequential greedy output. Batching was not selecting among answers the model gives, it was producing ones it does not give at parallel 1. My own numbers agree from a different angle, repeats inside one configuration byte identical at 200 of 200 and agreement dropping to 112 of 200 when only the speculative path changed. The practical consequence for someone using your article to pick a strategy is that you evaluate at parallel 1 and serve at 64, so the thing you validated is not the thing you ship. Worth a line beside the throughput charts.

Related Videos

I built an interactive simulator that shows why LLM inference capacity is hard. Batching, queues, KV cache, GPU decode, priority tiers - all coupled. Change one knob and the bottleneck moves.

I built an interactive simulator that shows why LLM inference capacity is hard. Batching, queues, KV cache, GPU decode, priority tiers - all coupled. Change one knob and the bottleneck moves.

Khushboo Verma

25,217 views • 4 months ago

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Avi Chawla

16,140 views • 2 months ago

Your Mac is about to run inference like a datacenter. Coming soon to MLX-Swift: Continuous batching: the fastest way to handle multiple inference streams locally. It starts with regular inference and seamlessly upgrades to batched mode when new requests arrive. The best of both worlds. Based on the work of and Awni Hannun

Your Mac is about to run inference like a datacenter. Coming soon to MLX-Swift: Continuous batching: the fastest way to handle multiple inference streams locally. It starts with regular inference and seamlessly upgrades to batched mode when new requests arrive. The best of both worlds. Based on the work of and Awni Hannun

Ronald Mannak

60,570 views • 11 months ago

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 views • 1 month ago

Met a guy making $1.6 million/year as an LLM engineer. I asked him how he learned LLMs from scratch. He sent me the exact video that got him in. A 1 hour course on how LLMs actually work. He shows how transformers inside LLMs like ChatGPT & Claude are actually built. I watched it last night. Halfway through, I realized LLM architecture is way simpler than they make it look. Bookmark this and read the article below. • 00:00 - LLM foundations • 04:21 - LLM tokenization • 05:43 - LLMs vector embeddings • 22:16 - attention mechanism of LLM • 43:42 - LLM multi head attention

Met a guy making $1.6 million/year as an LLM engineer. I asked him how he learned LLMs from scratch. He sent me the exact video that got him in. A 1 hour course on how LLMs actually work. He shows how transformers inside LLMs like ChatGPT & Claude are actually built. I watched it last night. Halfway through, I realized LLM architecture is way simpler than they make it look. Bookmark this and read the article below. • 00:00 - LLM foundations • 04:21 - LLM tokenization • 05:43 - LLMs vector embeddings • 22:16 - attention mechanism of LLM • 43:42 - LLM multi head attention

Roan

81,846 views • 2 months ago

Every AI product running in production depends on an inference system someone had to engineer, optimize and more. Scheduling, batching, routing, cost per token. This is a craft. The Inference Frontier Program spotlights the builders behind that work. 💡 Watch the video and nominate a team:

Every AI product running in production depends on an inference system someone had to engineer, optimize and more. Scheduling, batching, routing, cost per token. This is a craft. The Inference Frontier Program spotlights the builders behind that work. 💡 Watch the video and nominate a team:

Nebius

54,742 views • 7 months ago

LLM Routing, clearly explained! (routing can cost more than not routing at all) The basic idea sounds simple: Send easy requests to cheaper models and reserve frontier models for difficult tasks. But production routing is rarely that straightforward. Every classification adds latency. General-purpose models can misread intent. Routing rules become outdated. And switching models during an agent session can destroy prefix-cache savings, increase costs, and create inconsistent behavior. In this video, I cover: → Why DIY routing often breaks in production → How prefix caching changes the economics → Why agents need session pinning and model affinity → How Plano and Arch-Router handle intent resolution → How to build, test, and evaluate a router with DigitalOcean Routing works, but only when the infrastructure accounts for the full session, not just the latest request. The real question is not whether routing can reduce costs. It is how much of your AI traffic has been overpaying this whole time. Get started with Digital Ocean inference router: I recently wrote a detailed article on the same topic. The article is quoted below.

LLM Routing, clearly explained! (routing can cost more than not routing at all) The basic idea sounds simple: Send easy requests to cheaper models and reserve frontier models for difficult tasks. But production routing is rarely that straightforward. Every classification adds latency. General-purpose models can misread intent. Routing rules become outdated. And switching models during an agent session can destroy prefix-cache savings, increase costs, and create inconsistent behavior. In this video, I cover: → Why DIY routing often breaks in production → How prefix caching changes the economics → Why agents need session pinning and model affinity → How Plano and Arch-Router handle intent resolution → How to build, test, and evaluate a router with DigitalOcean Routing works, but only when the infrastructure accounts for the full session, not just the latest request. The real question is not whether routing can reduce costs. It is how much of your AI traffic has been overpaying this whole time. Get started with Digital Ocean inference router: I recently wrote a detailed article on the same topic. The article is quoted below.

Akshay 🚀

16,223 views • 13 days ago

SGLang-Diffusion is a high-performance serving framework for diffusion models, designed for both large-scale offline generation and latency-sensitive real-time inference. At PyTorch Conference North America 2026, Yihao Wang and Kevin Mi of RadixArk will present the design of SGLang-Diffusion and the PyTorch-based inference optimizations behind it, including efficient request scheduling, memory management, batching strategies, and execution-path optimizations for diffusion pipelines. Join us in San Jose, October 20-21: #PyTorchCon

SGLang-Diffusion is a high-performance serving framework for diffusion models, designed for both large-scale offline generation and latency-sensitive real-time inference. At PyTorch Conference North America 2026, Yihao Wang and Kevin Mi of RadixArk will present the design of SGLang-Diffusion and the PyTorch-based inference optimizations behind it, including efficient request scheduling, memory management, batching strategies, and execution-path optimizations for diffusion pipelines. Join us in San Jose, October 20-21: #PyTorchCon

PyTorch

17,055 views • 10 days ago

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Andrew Ng

104,727 views • 2 years ago

Went deeper on the “how I built it” with a solid article breaking down each step, how it works under the hood, and the code behind explained with SVG animations.

Went deeper on the “how I built it” with a solid article breaking down each step, how it works under the hood, and the code behind explained with SVG animations.

R

13,487 views • 17 days ago

I wrote a technical demo article explaining how you can build a seamless interactive web experience using the GSAP animation library alongside other techs like WebGL and page transitions. The article is live on Codrops ↓

I wrote a technical demo article explaining how you can build a seamless interactive web experience using the GSAP animation library alongside other techs like WebGL and page transitions. The article is live on Codrops ↓

chakib

36,885 views • 8 months ago

Andrej Karpathy(Anthropic Engineer) dropped a full 6-hour course on how to build LLMs from scratch and use it: • 00:00 - Full dive into LLMs • 03:31:23 - Building LLM from scratch • 05:27:43 - How to use LLMs (Karpathy method) This course can replace a $150K Stanford LLM senior degree and Bootkamp Start watching today, then read article below

Andrej Karpathy(Anthropic Engineer) dropped a full 6-hour course on how to build LLMs from scratch and use it: • 00:00 - Full dive into LLMs • 03:31:23 - Building LLM from scratch • 05:27:43 - How to use LLMs (Karpathy method) This course can replace a $150K Stanford LLM senior degree and Bootkamp Start watching today, then read article below

Morty

15,840 views • 1 month ago

My friend applied to 200 tech jobs in two years. No PhD. No Stanford. Last month Anthropic offered him $750,000. I asked him how he broke in from zero. He sent me a course that was never supposed to get out. A 3-hour video to build a full LLM from scratch. A developer teaches you exactly how LLMs like ChatGPT and Claude are actually built. I watched it last night. Halfway through, I realized it's embarrassingly simple to break into an AI lab. Bookmark this and read the article below. • 00:00 - intro to LLMs • 05:43 - LLM transformer architecture • 40:24 - training the LLM • 1:30:27 - modernizing the LLM • 2:33:53 - scaling the LLM

My friend applied to 200 tech jobs in two years. No PhD. No Stanford. Last month Anthropic offered him $750,000. I asked him how he broke in from zero. He sent me a course that was never supposed to get out. A 3-hour video to build a full LLM from scratch. A developer teaches you exactly how LLMs like ChatGPT and Claude are actually built. I watched it last night. Halfway through, I realized it's embarrassingly simple to break into an AI lab. Bookmark this and read the article below. • 00:00 - intro to LLMs • 05:43 - LLM transformer architecture • 40:24 - training the LLM • 1:30:27 - modernizing the LLM • 2:33:53 - scaling the LLM

Roan

280,331 views • 3 months ago

Goldman Sachs paid $500K/year for the strategies of a 16-year-old Gave him $10M to manage on day one In 10 minutes he showed exactly how he reads macro and why his book holds while others collapse The AI pipeline in the article below automates exactly this Bookmark, watch the video and read the article

Goldman Sachs paid $500K/year for the strategies of a 16-year-old Gave him $10M to manage on day one In 10 minutes he showed exactly how he reads macro and why his book holds while others collapse The AI pipeline in the article below automates exactly this Bookmark, watch the video and read the article

Atlas

71,366 views • 4 months ago

Inference providers hate this man... Your LLM deployment can deliver 30 to 50% better performance after initial deployment. If your current inference provider isn't learning your workload and tuning the stack continuously after deployment, you need another provider. Wafer's continual inference agents do that work. Wafer's continual inference agents profile your production workload to identify where scheduling, decoding, or GPU memory pressure limits performance. The agents benchmark changes to batching, serving engines, and GPU kernels against your workload. Each candidate must preserve correctness and meet your reliability targets before it ships. The agents keep profiling as traffic changes, identifying new bottlenecks and further opportunities to improve performance per dollar. Watch emilio andere explain how that work continues after deployment in his conversation with theCUBE.

wafer

42,840 views • 1 month ago

My friend applied to 150 tech jobs in two years. No MIT. No Stanford. Last month Anthropic offered him $750,000. I asked him how he broke in from zero. He sent me the exact video that got him in. A 3-hour course to build a full LLM from scratch. A developer teaches you exactly how LLMs like ChatGPT and Claude are actually built. I watched it last night. Halfway through, I realized I could break into an AI lab in weeks, not years. Bookmark this and read the article below. • 00:00 - intro to building LLMs • 17:58 - LLM tokenization • 01:06:03 - LLM embedding vectors • 02:46:55 - LLM transformer architecture • 03:25:05 - GPT architecture

My friend applied to 150 tech jobs in two years. No MIT. No Stanford. Last month Anthropic offered him $750,000. I asked him how he broke in from zero. He sent me the exact video that got him in. A 3-hour course to build a full LLM from scratch. A developer teaches you exactly how LLMs like ChatGPT and Claude are actually built. I watched it last night. Halfway through, I realized I could break into an AI lab in weeks, not years. Bookmark this and read the article below. • 00:00 - intro to building LLMs • 17:58 - LLM tokenization • 01:06:03 - LLM embedding vectors • 02:46:55 - LLM transformer architecture • 03:25:05 - GPT architecture

Roan

52,929 views • 2 months ago

I wrote an interactive article explaining the geometric intuition behind Rectified Flows. I visually explain why flow-models tend to learn curved trajectories, why this is bad for sampling latency, and a relatively simple technique for mitigating it. Check it out! Link 👇

I wrote an interactive article explaining the geometric intuition behind Rectified Flows. I visually explain why flow-models tend to learn curved trajectories, why this is bad for sampling latency, and a relatively simple technique for mitigating it. Check it out! Link 👇

Alec Helbling

159,423 views • 9 months ago

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

h100envy

61,470 views • 3 months ago

Anthropic Engineer Andrej Karpathy dropped a full 6-hour course on how to build LLMs from scratch and use it: • 00:00 - Deep dive into LLMs • 03:31:23 - Building ChatGPT from scratch • 05:27:43 - How to use LLMs (Karpathy method) This course can replace a $150K Stanford LLM senior degree. Start watching today, then read article below

Anthropic Engineer Andrej Karpathy dropped a full 6-hour course on how to build LLMs from scratch and use it: • 00:00 - Deep dive into LLMs • 03:31:23 - Building ChatGPT from scratch • 05:27:43 - How to use LLMs (Karpathy method) This course can replace a $150K Stanford LLM senior degree. Start watching today, then read article below

Morty

41,267 views • 2 months ago

This is the First video in MPC Web3 Wallets Series. We explained how traditional and MPC wallet works. What is Entropy and why it is important. Hope you like the video.

This is the First video in MPC Web3 Wallets Series. We explained how traditional and MPC wallet works. What is Entropy and why it is important. Hope you like the video.

Crypto Point Hindi 🇮🇳💎💫

16,283 views • 2 years ago