Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Day 12/90 of Inference Engineering What is chunked prefill within vLLM? In continuation of yesterday's post on the high level architecture of vLLM, I want to dive deeper into vLLM core engine starting with the mechanics of chunked prefill. In this post, I will closely follow the original blog...

29,556 görüntüleme • 1 ay önce •via X (Twitter)

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Day 11/90 of Inference Engineering How does vLLM work and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.

max fu

70,797 görüntüleme • 1 ay önce

vllm-exl3 v0.3.0 is LIVE with custom native CUDA kernels for 2-bit EXL3 on NVIDIA DGX Spark GB10. GLM-5.3-Flash-EXL3-K2 jumped from 16.9 → 24.6 tok/s average single-stream decode, a +45.6% gain. Coding hit 27.6 tok/s, +85.6%. 🚀 The previous ExLlamaV3-backed path inside vLLM was leaving a lot of GB10 bandwidth on the table. So I rewrote the hot path specifically for EXL3 on Blackwell sm_121: → in-register Trellis dequantization → native fused MoE decode → power-of-two chunked prefill GEMM → parallel NVMe pre-warm Then I tested it side-by-side on physical DGX Spark hardware using my GLM-5.3-Flash-EXL3-K2 pack and live vLLM HTTP streaming. 🚀 𝗗𝗘𝗖𝗢𝗗𝗘 𝗧𝗛𝗥𝗢𝗨𝗚𝗛𝗣𝗨𝗧 Single-stream C1: Coding 14.9 → 27.6 tok/s +85.6% Prose 13.7 → 24.6 tok/s +79.3% Reasoning 18.9 → 25.1 tok/s +32.7% Summary 17.1 → 25.6 tok/s +50.0% Format 16.3 → 24.0 tok/s +47.7% Average: 16.9 → 24.6 tok/s 𝗡𝗘𝗧 𝗚𝗔𝗜𝗡: +45.6% ⏱️ 𝗙𝗜𝗥𝗦𝗧-𝗧𝗢𝗞𝗘𝗡 𝗥𝗘𝗦𝗣𝗢𝗡𝗦𝗜𝗩𝗘𝗡𝗘𝗦𝗦 Coding TTFT: 2,344 ms → 859 ms That is a 63.3% reduction, or about 2.7× faster to first token. Follow-up turn with prefix cache hit: 5,608 ms → 3,588 ms 1.56× faster. ⚡ 𝗪𝗛𝗔𝗧 𝗖𝗛𝗔𝗡𝗚𝗘𝗗 𝗢𝗡 𝗧𝗛𝗘 𝗚𝗣𝗨 40 routed-MoE layers: 19.9 ms → 11.5 ms per token Per-layer MoE compute: 497 μs → 287.8 μs That removes 8.4 ms of MoE compute from every generated token. Total per-step wall time: 59.2 ms → 40.6 ms -31.4% The key is `p2b_fused_moe`. Instead of expanding EXL3 weights through a traditional intermediate path, the new kernel performs Trellis dequantization in-register while executing the routed expert computation. The weights stay compressed until the GPU actually needs them. 🔥 𝗣𝗥𝗘𝗙𝗜𝗟𝗟 𝗚𝗢𝗧 𝗔 𝗡𝗔𝗧𝗜𝗩𝗘 𝗣𝗔𝗧𝗛 𝗧𝗢𝗢 The new `exl3_gemm` uses power-of-two chunked prefill GEMM. Measured: 7.85 TFLOPS 13.0× faster than the legacy prefill kernel 1,875 tok/s cold prefill sustained across 65K context 💾 𝗧𝗛𝗘 𝗕𝗢𝗢𝗧 𝗣𝗔𝗧𝗛 𝗡𝗘𝗘𝗗𝗘𝗗 𝗪𝗢𝗥𝗞 𝗧𝗢𝗢 Loading a ~91 GiB model is part of the user experience. Standard shard loading is mostly serial. The updated recipe parallelizes NVMe pre-warm across 8 workers so the storage controller gets used properly instead of feeding a ~100 GiB model one shard at a time. That turns boot-time storage into another optimization target instead of something we simply accept. 💡 𝗧𝗪𝗢 𝗦𝗘𝗥𝗩𝗜𝗡𝗚 𝗙𝗟𝗔𝗚𝗦 𝗪𝗢𝗥𝗧𝗛 𝗞𝗡𝗢𝗪𝗜𝗡𝗚 `--long-prefill-token-threshold 1024` Prevents giant prefill chunks from monopolizing step budgets and starving parallel decode sessions. `--enable-prefix-caching` Avoids paying for the same conversational prefix again on follow-up turns. 📦 𝗘𝗩𝗘𝗥𝗬𝗧𝗛𝗜𝗡𝗚 𝗜𝗦 𝗢𝗣𝗘𝗡 vllm-exl3: GLM-5.3-Flash one-Spark recipe: Model: This is why I like working at the kernel level. The model did not change. The quant did not change. The hardware did not change. The execution path did. 16.9 → 24.6 tok/s. 🛠️ vLLM turboderp

Cruz

20,136 görüntüleme • 7 gün önce

While working on a new video with solutions to the previous one, I found ChatGPT's new UI struggles even more with concurrent updates: entries lose state and stick around for too long (see video). If this was a LiveView app, we would be getting so much flak.😅 --- I believe part of the problem here is having separate mutate and fetch requests on every deletion. The first fetch is cancelled when the second one comes up, causing items to stick around for longer. Many said yesterday that you could do the mutation and fetch as a single request, but that leads to other problems, such zombie entries. For example, imagine you delete link1 and link2 within a brief period of time. There is no guarantee the deletion order in the database will match the order the client receives the response, so you may end up with this: 1. (client) request to delete link1 sent 2. (client) request to delete link2 sent 3. (server) deletes link1 and loads a new list (includes link2) 4. (server) deletes link2 and loads a new list (no link1 or link2) 5. (client) receives link2 response 6. (client) receives link1 response So if you choose to use the latest response (link1), you brought link2 back to life. If you say you will use the response from the last request, events 3-4 can be swapped, and now you bring link1 back to life. Another way to solve this is by basically not allowing concurrent requests at all but that can affect the user experience drastically in other ways. Next week I should publish a video explaining how LiveView tackles this. Stay tuned!

José Valim

23,050 görüntüleme • 2 yıl önce

50% cheaper Claude inference with just one line of code change! - Remove → model="claude-opus-4-8" - Add → model="ship-like/claude-opus-4-8" I verified the cost saving in my own terminal by invoking the same Anthropic model with the same prompt. The underlying engineering by Ship is actually interesting, and the patterns can be used in any production LLM stack. Essentially, a trained model is a frozen artifact. Every request performs the same forward-pass, whether it extracts a date or refactors a module, because the compute decision was made at training time, before the request existed. Ship makes that decision at inference time instead. After seeing a request, it searches over executions, involving single models, cascades, ensembles, or harnesses with tools, and serves the cheapest one that will match the reference model's quality. This is not a basic router, because picking a cheaper model per query doesn't ensure the cheaper model preserves the original's behavior, like output shape, tool-call patterns, and refusals. Ship measures this equivalence directly. Outputs stay distributionally indistinguishable from the reference model, not token-identical, since two calls to the same model already differ, but they are indistinguishable in capability and behavior. Of course, some requests execute cheaply and some cost Ship more than the customer pays, but the price per request is still a flat 50% off either way, so the execution-cost variance moves off the application's bill entirely. The video below depicts the cost savings and output in my real invocation, and I partnered with the team to put this together.

Akshay 🚀

63,725 görüntüleme • 1 ay önce

Gemma 4 26B A4B MoE - 500+ t/s decode - Single RTX 4090 (24 GB VRAM) - Llama.cpp concurrency 24 - q8 kv cache How many API users can you simultaneously host on a single RTX 4090 (24 GB VRAM) before it crashes? Yesterday, I proved you can host 14 active users using unquantized memory. Today, I used 8 bit KV Cache Quantization to hack the VRAM footprint. I successfully scaled to 24 concurrent users without a single dropped connection. A 71% server capacity boost for free. By adding the -ctk q8_0 -ctv q8_0 flags to llama.cpp, you compress the KV cache context memory from 16 bit to 8 bit. This unlocks massive concurrency limits on Gemma 4 26B (MoE) on a single 24GB consumer GPU. Here is the exact telemetry from pushing 8 bit quantization to its absolute physical edge: # TEST 1: The 24 User Concurrency Max Server Config: 24 slots (np 24) | 4,096 context per slot | 98,304 Total Context Client Load: 24 simultaneous requests (2,000 token prompt per user) Unquantized KV cache for this load requires 28GB+ VRAM (Instant OOM). Quantized to Q8, it allocated safely at 23.35 GB. The C++ engine crunched the entire batch in 28.5 seconds. Decode Speed: 21 t/s (Per User) | 500 t/s (Agg) # TEST 2: The 48 User Queue Overload What happens to a compressed cache during a traffic spike? Server Config: 24 slots (np 24) | 4,096 context per slot | 98,304 Total Context Client Load: 48 simultaneous requests (2k token prompt per user) Zero queue drops. The scheduler flushed and hot swapped the 8 bit memory flawlessly on the fly, completing all 48 users in 66.0 seconds (a perfect 2.3x queue scaling multiplier). Decode Speed: 18 t/s (Per User) | 430 t/s (Agg) # TEST 3: The 8 User RAG Slam Server Config: 8 slots (np 8) | 60,000 context per slot | 480,000 Total Context Client Load: 8 simultaneous requests (30k token prompt per user) It allocated 23.83 GB VRAM and chewed through ~240,000 prefill tokens in 46 seconds under massive memory pressure. Prefill Speed: 6,200 t/s (Agg) Decode Speed: 22 t/s (Per User) | 175 t/s (Agg) # The Engineering Alpha (The Quantization Tradeoff): You gain a massive 71% increase in server capacity, but what do you lose? Compute latency. Because the cache is stored in 8 bit, the GPU's cores have to dequantize the memory back to 16 bit on the fly during every single prefill step. In my unquantized tests yesterday, single slot prefill was hitting ~1,500+ t/s. Today, under the heavy 48-user Q8 load, prefill dropped as low as ~750 t/s. You trade a few seconds of initial prefill latency to essentially double your API hosting capacity. For production high volume SaaS, this is the ultimate unit economics cheat code. Here is the exact command to run a 24 user Q8 continuous batching server on your own single 4090, single 3090 or any 24gb vram rig: ./build/bin/llama-server -m gemma-4-26B-A4B-it.gguf -c 98304 -np 24 -b 2048 -ub 2048 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 --port 8080 (Note: -c 98304 allocates exactly 4,096 tokens of context per user across 24 slots). Hugging Face links to the Unsloth Gemma 4 26B QAT quants along with performance graphs available in the replies. Would you trade 3 seconds of Time To First Token latency to double your active user capacity?

Alok

17,465 görüntüleme • 1 ay önce

A very good morning. Welcome to The Council Benji This marks the third Skull in a little run. The first went to a fund I've never met. The second: through Eli Scheinman to a new collector/foundation who has been quietly entering the space in a very significant way across a number of collections whom I’ve never spoken to. Their new entrance enabled a wedding and start of a new married life for Conviction. In my very first conversation with him, we spoke about curses and commitments to the people we love. Since meeting got to talk through each step on that path, from letting go, what is imbued in the ring and ceremony of it all, a proposal, and on the way to the most important of the steps in pursuit of a blessed life. It is easy to get a little cynical on the over-leveraged exit stories that spring up from time to time, so it is a treat to watch one go towards a celebration that’s been building up in his life since the Skull was first acquired. And now: this. The third Skull and the first I can really write about as a shared story across both source and destination. An exit and an entrance. The exit: The Skulls of Luci were awarded as gifts 4 years ago. But before I'd minted Birth of Luci or painted the other 49, the first person in this space I showed the sketch of The Blueprint Skull to was actually Casey💎, when he was working at SuperRare . Casey was the very first person who onboarded me to NFTs, helping me navigate the early days of whatever it meant to even mint something. I explained the idea of gifting one to each person who bid in my first auctions. Though most of the Skulls went to the bidders, Casey's didn't. He didn't ask for one. I didn't tell him I'd give him one. But he helped me take my first steps here, and it's hard to imagine any of this making sense, or unfolding the way it has, without him. Since then, we've broken bread across continents, seen quite a lot of chortling margarita consumption, watched the rise and fall of a lot around us, weathered inter-Council dramas. He brought Laura El into The Monument Game, played as a Player, wore a Mask. Most of the vibe that started all of this, the wild west of it, feels faded in the broader space at times. But every Skull has a story and a person who helped us get here. Casey will always be the one who was there before any metric muddled the reason to care. The entrance: Last fall, Benji came over for a studio visit. We walked through Luci, the works, structure, and dream, as anyone who visits does. But we mostly talked about being a father and having a father. We discussed the very idea of "collection" stripped of accumulation, value, or signal, located more in the act or ceremony of it. What it was to grow up with a curious father who studied the edges of each thing he saw to know the next layer beneath why anyone might look or ignore it. That to pass this on is to pass on questioning, more than it is to pass on any kind of answer. The process of collecting can be perceived as an individual act of hoarding. For some it is maybe. But at its best, it's a way to bind through shared questioning, to bond in cooperation and competition with friends and family, it is the swapped story and meme of it all, and each object gathered along the way carries some shared memory that can, often does, and with intent: should; drift out of the object entirely. All in the psalm, always has been. The studio visit came and went. Soon after, a package arrived in the mail with two of the softest stuffed animals added to my daughter's own collection, now among her favorites. The Skull is a bonus to that, in the scheme of shared memory. For Rachel and I, while we are heads down making a body of work that unsettles us and excites us but demands unknown time to accomplish, it means a great deal to have this kind of support from long term people in the quiet process of making work we want to leave behind ourselves. Enormously grateful to Casey for the many years of support and friendship, to Benny for being a true patron, and to Benji for entering the arena for what I'm working on next. Welcome.

Sam Spratt

20,786 görüntüleme • 4 ay önce

Researchers made KMeans 200x faster. And the new technique also beats approaches like cuML and FAISS. Flash-KMeans is an IO-aware implementation of exact KMeans that redesigns the algorithm around modern GPU bottlenecks. By attacking the memory bottlenecks directly, Flash-KMeans achieves: - 33x speedup over cuML - 200x speedup over FAISS This speedup comes from how it moves through GPU memory. Standard KMeans runs in two steps, and both are bottlenecked by reads and writes to GPU memory: 1) The first step matches every point to its nearest centroid. Standard KMeans computes the full point-to-centroid distance matrix, writes it out to GPU memory, then reads it back to find each nearest centroid. That write-then-read round trip is the bottleneck. Flash-KMeans combines the distance calculation with the nearest-centroid step, so the result is computed on-chip and the full matrix is never written out. 2) The second step recomputes each centroid by averaging the points assigned to it. Standard KMeans has thousands of threads writing into the same centroid slots at once, so they stall waiting for their turn. Flash-KMeans sorts points by cluster first, turning scattered writes into sequential reductions that read and write memory in one efficient pass. Using these two optimizations at the million-scale, Flash-KMeans completes a standard KMeans iteration in a few milliseconds. The video below depicts this in action. Several reasons why this is important: KMeans has always been an offline primitive. Something you run once to preprocess data and move on. These speedups make the approach viable in several runtime-critical systems. ↳ Vector indices like FAISS use KMeans to build search indices. Faster KMeans means you can re-index dynamically as data changes. ↳ LLM quantization methods need KMeans to find optimal weight codebooks, per layer, repeatedly. What takes hours could now take minutes. ↳ MoE models need fast token routing at inference time. Flash-KMeans makes it viable to run this inside the inference loop, not just in preprocessing. I have shared the paper in the replies. That said, memory is the real constraint Flash-KMeans solves, and the problem is not just limited to clustering. The vectors a RAG system stores after indexing create similar bottlenecks. I wrote a detailed walkthrough recently on cutting this vector memory by 32x with binary quantization, querying 36M+ vectors in a few milliseconds. Read it below.

Avi Chawla

89,234 görüntüleme • 2 ay önce

There's been an unfortunate incident in LA with a Uhaul plowing into a crowd of anti-Khameini protestors This man should never have been able to get near the crowd with a uhaul but some info about the situation seem to be - signage on the truck is anti both the shah and the current ayatollah. is this his actual position or is this camouflage to have gotten into the protest to perpetrate an attack? "no Shah, No regime, No Mullah" Mullah is a religious leader so possibly referring to the current leader and not a king like Pahlavi Timeline appears to be - anti-Khameini protestors try to rip signs off his vehicle and are bashing on the windows and eventually his passenger side window is broken - guy in uhaul then stutterstops forward into the crowd, eventually accelerating further, then stuttering again, then full stopping down the road - there is a man surfing on top of the uhaul in the 3rd video, below I have posted another video showing the man on top of the uhaul trying to take the posters off the side, so he is likely part of the anti-Khameini protestors - uhaul driver is taken into custody by police is this a case of police not having the street sufficiently blocked off and so a guy was able to get a uhaul in here? He should not have been able to drive a uhaul this close to a massive protest crowd There are a lot of people saying this is a terrorist attack, it is possible it could be one but I don't think there's enough information to accurately assert that at this time The chronology of events also shows it is possible that the driver was in fear of his life since protestors were banging on the uhaul, windows, and removing signs+ eventually breaking his window Whatever turns out to be the actual case, it is an unfortunate event and as of right now a seeming silver lining is that no deaths have been reported

Kirsche 🥥 🧁

41,247 görüntüleme • 8 ay önce