Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Batching strategies in LLM inference, clearly explained! (bookmark it) - Static - Dynamic - And continuous batching I wrote a detailed article explaining how each works and why serving LLMs is a different problem from traditional ML inference. The article is quoted below.

Akshay 🚀

285,744 subscribers

29,245 views • 3 days ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

I built an interactive simulator that shows why LLM inference capacity is hard. Batching, queues, KV cache, GPU decode, priority tiers - all coupled. Change one knob and the bottleneck moves.

I built an interactive simulator that shows why LLM inference capacity is hard. Batching, queues, KV cache, GPU decode, priority tiers - all coupled. Change one knob and the bottleneck moves.

Khushboo Verma

24,726 views • 2 months ago

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Avi Chawla

16,140 views • 22 days ago

Your Mac is about to run inference like a datacenter. Coming soon to MLX-Swift: Continuous batching: the fastest way to handle multiple inference streams locally. It starts with regular inference and seamlessly upgrades to batched mode when new requests arrive. The best of both worlds. Based on the work of and Awni Hannun

Your Mac is about to run inference like a datacenter. Coming soon to MLX-Swift: Continuous batching: the fastest way to handle multiple inference streams locally. It starts with regular inference and seamlessly upgrades to batched mode when new requests arrive. The best of both worlds. Based on the work of and Awni Hannun

Ronald Mannak

60,570 views • 10 months ago

Met a guy making $1.6 million/year as an LLM engineer. I asked him how he learned LLMs from scratch. He sent me the exact video that got him in. A 1 hour course on how LLMs actually work. He shows how transformers inside LLMs like ChatGPT & Claude are actually built. I watched it last night. Halfway through, I realized LLM architecture is way simpler than they make it look. Bookmark this and read the article below. • 00:00 - LLM foundations • 04:21 - LLM tokenization • 05:43 - LLMs vector embeddings • 22:16 - attention mechanism of LLM • 43:42 - LLM multi head attention

Met a guy making $1.6 million/year as an LLM engineer. I asked him how he learned LLMs from scratch. He sent me the exact video that got him in. A 1 hour course on how LLMs actually work. He shows how transformers inside LLMs like ChatGPT & Claude are actually built. I watched it last night. Halfway through, I realized LLM architecture is way simpler than they make it look. Bookmark this and read the article below. • 00:00 - LLM foundations • 04:21 - LLM tokenization • 05:43 - LLMs vector embeddings • 22:16 - attention mechanism of LLM • 43:42 - LLM multi head attention

Roan

80,961 views • 1 month ago

Every AI product running in production depends on an inference system someone had to engineer, optimize and more. Scheduling, batching, routing, cost per token. This is a craft. The Inference Frontier Program spotlights the builders behind that work. 💡 Watch the video and nominate a team:

Every AI product running in production depends on an inference system someone had to engineer, optimize and more. Scheduling, batching, routing, cost per token. This is a craft. The Inference Frontier Program spotlights the builders behind that work. 💡 Watch the video and nominate a team:

Nebius

54,701 views • 5 months ago

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Andrew Ng

104,727 views • 2 years ago

I wrote a technical demo article explaining how you can build a seamless interactive web experience using the GSAP animation library alongside other techs like WebGL and page transitions. The article is live on Codrops ↓

I wrote a technical demo article explaining how you can build a seamless interactive web experience using the GSAP animation library alongside other techs like WebGL and page transitions. The article is live on Codrops ↓

chakib

36,885 views • 7 months ago

My friend applied to 200 tech jobs in two years. No PhD. No Stanford. Last month Anthropic offered him $750,000. I asked him how he broke in from zero. He sent me a course that was never supposed to get out. A 3-hour video to build a full LLM from scratch. A developer teaches you exactly how LLMs like ChatGPT and Claude are actually built. I watched it last night. Halfway through, I realized it's embarrassingly simple to break into an AI lab. Bookmark this and read the article below. • 00:00 - intro to LLMs • 05:43 - LLM transformer architecture • 40:24 - training the LLM • 1:30:27 - modernizing the LLM • 2:33:53 - scaling the LLM

My friend applied to 200 tech jobs in two years. No PhD. No Stanford. Last month Anthropic offered him $750,000. I asked him how he broke in from zero. He sent me a course that was never supposed to get out. A 3-hour video to build a full LLM from scratch. A developer teaches you exactly how LLMs like ChatGPT and Claude are actually built. I watched it last night. Halfway through, I realized it's embarrassingly simple to break into an AI lab. Bookmark this and read the article below. • 00:00 - intro to LLMs • 05:43 - LLM transformer architecture • 40:24 - training the LLM • 1:30:27 - modernizing the LLM • 2:33:53 - scaling the LLM

Roan

278,173 views • 1 month ago

Goldman Sachs paid $500K/year for the strategies of a 16-year-old Gave him $10M to manage on day one In 10 minutes he showed exactly how he reads macro and why his book holds while others collapse The AI pipeline in the article below automates exactly this Bookmark, watch the video and read the article

Goldman Sachs paid $500K/year for the strategies of a 16-year-old Gave him $10M to manage on day one In 10 minutes he showed exactly how he reads macro and why his book holds while others collapse The AI pipeline in the article below automates exactly this Bookmark, watch the video and read the article

Atlas

71,366 views • 3 months ago

My friend applied to 150 tech jobs in two years. No MIT. No Stanford. Last month Anthropic offered him $750,000. I asked him how he broke in from zero. He sent me the exact video that got him in. A 3-hour course to build a full LLM from scratch. A developer teaches you exactly how LLMs like ChatGPT and Claude are actually built. I watched it last night. Halfway through, I realized I could break into an AI lab in weeks, not years. Bookmark this and read the article below. • 00:00 - intro to building LLMs • 17:58 - LLM tokenization • 01:06:03 - LLM embedding vectors • 02:46:55 - LLM transformer architecture • 03:25:05 - GPT architecture

My friend applied to 150 tech jobs in two years. No MIT. No Stanford. Last month Anthropic offered him $750,000. I asked him how he broke in from zero. He sent me the exact video that got him in. A 3-hour course to build a full LLM from scratch. A developer teaches you exactly how LLMs like ChatGPT and Claude are actually built. I watched it last night. Halfway through, I realized I could break into an AI lab in weeks, not years. Bookmark this and read the article below. • 00:00 - intro to building LLMs • 17:58 - LLM tokenization • 01:06:03 - LLM embedding vectors • 02:46:55 - LLM transformer architecture • 03:25:05 - GPT architecture

Roan

51,597 views • 17 days ago

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps. request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve. That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week. LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack. Watch and save it, then wire the KV-offload into your inference stack.

h100envy

61,470 views • 1 month ago

I wrote an interactive article explaining the geometric intuition behind Rectified Flows. I visually explain why flow-models tend to learn curved trajectories, why this is bad for sampling latency, and a relatively simple technique for mitigating it. Check it out! Link 👇

I wrote an interactive article explaining the geometric intuition behind Rectified Flows. I visually explain why flow-models tend to learn curved trajectories, why this is bad for sampling latency, and a relatively simple technique for mitigating it. Check it out! Link 👇

Alec Helbling

159,259 views • 7 months ago

Anthropic Engineer Andrej Karpathy dropped a full 6-hour course on how to build LLMs from scratch and use it: • 00:00 - Deep dive into LLMs • 03:31:23 - Building ChatGPT from scratch • 05:27:43 - How to use LLMs (Karpathy method) This course can replace a $150K Stanford LLM senior degree. Start watching today, then read article below

Anthropic Engineer Andrej Karpathy dropped a full 6-hour course on how to build LLMs from scratch and use it: • 00:00 - Deep dive into LLMs • 03:31:23 - Building ChatGPT from scratch • 05:27:43 - How to use LLMs (Karpathy method) This course can replace a $150K Stanford LLM senior degree. Start watching today, then read article below

Morty

40,640 views • 22 days ago

This is the First video in MPC Web3 Wallets Series. We explained how traditional and MPC wallet works. What is Entropy and why it is important. Hope you like the video.

This is the First video in MPC Web3 Wallets Series. We explained how traditional and MPC wallet works. What is Entropy and why it is important. Hope you like the video.

Crypto Point Hindi 🇮🇳💎💫

16,274 views • 2 years ago

My friend applied to 300 tech jobs in two years. No MIT. No Stanford. Last month Anthropic offered him $750,000. I asked him how he broke in from zero. He sent me the exact video that got him in. Andrej Karpathy's 3-hour course on building a full LLM from scratch. Anthropic's own researcher shows you how LLMs like ChatGPT & Claude are actually built. I watched it last night. Halfway through, I realized it's embarrassingly simple to break into an AI lab. Bookmark this and read the article below. • 00:00 - intro to LLMs • 12:41 - LLM training pipeline • 31:58 - LLM tools & plugins • 41:14 - LLM transformer architecture • 2:19:02 - scaling the LLM

My friend applied to 300 tech jobs in two years. No MIT. No Stanford. Last month Anthropic offered him $750,000. I asked him how he broke in from zero. He sent me the exact video that got him in. Andrej Karpathy's 3-hour course on building a full LLM from scratch. Anthropic's own researcher shows you how LLMs like ChatGPT & Claude are actually built. I watched it last night. Halfway through, I realized it's embarrassingly simple to break into an AI lab. Bookmark this and read the article below. • 00:00 - intro to LLMs • 12:41 - LLM training pipeline • 31:58 - LLM tools & plugins • 41:14 - LLM transformer architecture • 2:19:02 - scaling the LLM

Roan

286,854 views • 1 month ago

Ilya Sutskever: “My document gently flowed to where AI is today - and agents are the future” He wrote the paper that made LLMs like Claude and ChatGPT possible - and explains it from scratch in 20 minutes Never touched AI or use it every day - you need to watch this course Watch - bookmark, then put the entire lecture into practice with the article below ↓

Ilya Sutskever: “My document gently flowed to where AI is today - and agents are the future” He wrote the paper that made LLMs like Claude and ChatGPT possible - and explains it from scratch in 20 minutes Never touched AI or use it every day - you need to watch this course Watch - bookmark, then put the entire lecture into practice with the article below ↓

codila

49,696 views • 20 days ago

This video explained clearly about that assault part and why it is there in the movie. People with pea-sized brain should stay away from such cinema. Look how they quoted a Gender Studies professor too to explain it.

This video explained clearly about that assault part and why it is there in the movie. People with pea-sized brain should stay away from such cinema. Look how they quoted a Gender Studies professor too to explain it.

Taani Partner

85,262 views • 9 months ago

Don't waste 2 years learning to build LLMs like Claude & ChatGPT. MIT dropped a 3 hour course on building LLMs from scratch for free. Training → Architecture → Scale. It covers everything about LLMs. • 00:00 - training an LLM • 1:20:46 - LLM architecture from scratch • 2:34:56 - scaling laws for LLMs Anthropic pays $750,000/year to AI engineers in 2026 who master this exact knowledge. Bookmark this & skip Netflix. Give it 4 hours today. Then read the article below.

Don't waste 2 years learning to build LLMs like Claude & ChatGPT. MIT dropped a 3 hour course on building LLMs from scratch for free. Training → Architecture → Scale. It covers everything about LLMs. • 00:00 - training an LLM • 1:20:46 - LLM architecture from scratch • 2:34:56 - scaling laws for LLMs Anthropic pays $750,000/year to AI engineers in 2026 who master this exact knowledge. Bookmark this & skip Netflix. Give it 4 hours today. Then read the article below.

Roan

85,618 views • 24 days ago

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

Madison Kanna

119,268 views • 5 months ago

Prolactin is a nasty thyroid inhibitor. It directly lower T4 production in the thyroid gland, while also preventing the liver from converting T4 into T3. It acts as a double edge sword, attacking one of our most crucial hormone systems, more info on how it works and how to lower it is in the article below.

Prolactin is a nasty thyroid inhibitor. It directly lower T4 production in the thyroid gland, while also preventing the liver from converting T4 into T3. It acts as a double edge sword, attacking one of our most crucial hormone systems, more info on how it works and how to lower it is in the article below.

Oxidative State

28,730 views • 19 days ago