Day 12/90 of Inference Engineering What is chunked prefill... within vLLM? In continuation of yesterday's post on the high level architecture of vLLM, I want to dive deeper into vLLM core engine starting with the mechanics of chunked prefill. In this post, I will closely follow the original blog on the anatomy of vLLM. To start, let's define chunked prefill. It's a runtime inference optimization technique that splits a long input request so that it doesn’t monopolize the whole GPU. Keep in mind this is all within the context of vLLM. And since vLLM is an inference engine that's meant to serve a model to multiple concurrent users, having a GPU that’s fully monopolized on a single user's request means other users' requests would be in queue waiting to be processed. It isn’t too good to have the whole GPU occupied on a single request when the GPU is meant to be shared! So the key idea behind chunked prefill is to break the long request into smaller chunks, so that each chunk along with other users' requests gets processed and written into the KV cache together. Suppose we split up the long request into chunks and each chunk has 8 tokens. Now each memory block can hold 4 tokens. Therefore, 8 tokens can fit into 2 blocks of memory. After the first forward pass, 2 blocks are occupied, and after the second forward pass, 4 blocks of memory are occupied and so forth. Each forward pass handles a small chunk of the long request so that there's room in the same pass to keep serving other users' requests. Here's a small animation that I made today to fully visualize the idea behind chunked prefill when learning this topic~show more

Day 11/90 of Inference Engineering How does vLLM work... and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.show more

While working on a new video with solutions to... the previous one, I found ChatGPT's new UI struggles even more with concurrent updates: entries lose state and stick around for too long (see video). If this was a LiveView app, we would be getting so much flak.😅 --- I believe part of the problem here is having separate mutate and fetch requests on every deletion. The first fetch is cancelled when the second one comes up, causing items to stick around for longer. Many said yesterday that you could do the mutation and fetch as a single request, but that leads to other problems, such zombie entries. For example, imagine you delete link1 and link2 within a brief period of time. There is no guarantee the deletion order in the database will match the order the client receives the response, so you may end up with this: 1. (client) request to delete link1 sent 2. (client) request to delete link2 sent 3. (server) deletes link1 and loads a new list (includes link2) 4. (server) deletes link2 and loads a new list (no link1 or link2) 5. (client) receives link2 response 6. (client) receives link1 response So if you choose to use the latest response (link1), you brought link2 back to life. If you say you will use the response from the last request, events 3-4 can be swapped, and now you bring link1 back to life. Another way to solve this is by basically not allowing concurrent requests at all but that can affect the user experience drastically in other ways. Next week I should publish a video explaining how LiveView tackles this. Stay tuned!show more

$zkHive X $DONGO 🤝 We are excited to announce... our partnership with Dongo AI ! Dongo AI is a platform designed to simplify and connect the expansive world of cryptocurrency for everyday users. Using their product users can research their preferred tokens and pose questions about the project, leveraging our AI's technology. We’re happy to announce that #zkHive is going to be one of the first projects to be integrated into Dongo’s upcoming platform launch. In addition #HiveBot will be integrated into Dongo’s community, and our security APIs will be used to secure the thousands of users of the Congo ecosystem! Thanks to the AI platform that DONGO is building, $ZKHIVE is going to protect thousands of new users! This is another great step by both parties to promote AI and security in the space!show more

50% cheaper Claude inference with just one line of... code change! - Remove → model="claude-opus-4-8" - Add → model="ship-like/claude-opus-4-8" I verified the cost saving in my own terminal by invoking the same Anthropic model with the same prompt. The underlying engineering by Ship is actually interesting, and the patterns can be used in any production LLM stack. Essentially, a trained model is a frozen artifact. Every request performs the same forward-pass, whether it extracts a date or refactors a module, because the compute decision was made at training time, before the request existed. Ship makes that decision at inference time instead. After seeing a request, it searches over executions, involving single models, cascades, ensembles, or harnesses with tools, and serves the cheapest one that will match the reference model's quality. This is not a basic router, because picking a cheaper model per query doesn't ensure the cheaper model preserves the original's behavior, like output shape, tool-call patterns, and refusals. Ship measures this equivalence directly. Outputs stay distributionally indistinguishable from the reference model, not token-identical, since two calls to the same model already differ, but they are indistinguishable in capability and behavior. Of course, some requests execute cheaply and some cost Ship more than the customer pays, but the price per request is still a flat 50% off either way, so the execution-cost variance moves off the application's bill entirely. The video below depicts the cost savings and output in my real invocation, and I partnered with the team to put this together.show more

Gemma 4 26B A4B MoE - 500+ t/s decode... - Single RTX 4090 (24 GB VRAM) - Llama.cpp concurrency 24 - q8 kv cache How many API users can you simultaneously host on a single RTX 4090 (24 GB VRAM) before it crashes? Yesterday, I proved you can host 14 active users using unquantized memory. Today, I used 8 bit KV Cache Quantization to hack the VRAM footprint. I successfully scaled to 24 concurrent users without a single dropped connection. A 71% server capacity boost for free. By adding the -ctk q8_0 -ctv q8_0 flags to llama.cpp, you compress the KV cache context memory from 16 bit to 8 bit. This unlocks massive concurrency limits on Gemma 4 26B (MoE) on a single 24GB consumer GPU. Here is the exact telemetry from pushing 8 bit quantization to its absolute physical edge: # TEST 1: The 24 User Concurrency Max Server Config: 24 slots (np 24) | 4,096 context per slot | 98,304 Total Context Client Load: 24 simultaneous requests (2,000 token prompt per user) Unquantized KV cache for this load requires 28GB+ VRAM (Instant OOM). Quantized to Q8, it allocated safely at 23.35 GB. The C++ engine crunched the entire batch in 28.5 seconds. Decode Speed: 21 t/s (Per User) | 500 t/s (Agg) # TEST 2: The 48 User Queue Overload What happens to a compressed cache during a traffic spike? Server Config: 24 slots (np 24) | 4,096 context per slot | 98,304 Total Context Client Load: 48 simultaneous requests (2k token prompt per user) Zero queue drops. The scheduler flushed and hot swapped the 8 bit memory flawlessly on the fly, completing all 48 users in 66.0 seconds (a perfect 2.3x queue scaling multiplier). Decode Speed: 18 t/s (Per User) | 430 t/s (Agg) # TEST 3: The 8 User RAG Slam Server Config: 8 slots (np 8) | 60,000 context per slot | 480,000 Total Context Client Load: 8 simultaneous requests (30k token prompt per user) It allocated 23.83 GB VRAM and chewed through ~240,000 prefill tokens in 46 seconds under massive memory pressure. Prefill Speed: 6,200 t/s (Agg) Decode Speed: 22 t/s (Per User) | 175 t/s (Agg) # The Engineering Alpha (The Quantization Tradeoff): You gain a massive 71% increase in server capacity, but what do you lose? Compute latency. Because the cache is stored in 8 bit, the GPU's cores have to dequantize the memory back to 16 bit on the fly during every single prefill step. In my unquantized tests yesterday, single slot prefill was hitting ~1,500+ t/s. Today, under the heavy 48-user Q8 load, prefill dropped as low as ~750 t/s. You trade a few seconds of initial prefill latency to essentially double your API hosting capacity. For production high volume SaaS, this is the ultimate unit economics cheat code. Here is the exact command to run a 24 user Q8 continuous batching server on your own single 4090, single 3090 or any 24gb vram rig: ./build/bin/llama-server -m gemma-4-26B-A4B-it.gguf -c 98304 -np 24 -b 2048 -ub 2048 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 --port 8080 (Note: -c 98304 allocates exactly 4,096 tokens of context per user across 24 slots). Hugging Face links to the Unsloth Gemma 4 26B QAT quants along with performance graphs available in the replies. Would you trade 3 seconds of Time To First Token latency to double your active user capacity?show more

A very good morning. Welcome to The Council Benji... This marks the third Skull in a little run. The first went to a fund I've never met. The second: through Eli Scheinman to a new collector/foundation who has been quietly entering the space in a very significant way across a number of collections whom I’ve never spoken to. Their new entrance enabled a wedding and start of a new married life for Conviction. In my very first conversation with him, we spoke about curses and commitments to the people we love. Since meeting got to talk through each step on that path, from letting go, what is imbued in the ring and ceremony of it all, a proposal, and on the way to the most important of the steps in pursuit of a blessed life. It is easy to get a little cynical on the over-leveraged exit stories that spring up from time to time, so it is a treat to watch one go towards a celebration that’s been building up in his life since the Skull was first acquired. And now: this. The third Skull and the first I can really write about as a shared story across both source and destination. An exit and an entrance. The exit: The Skulls of Luci were awarded as gifts 4 years ago. But before I'd minted Birth of Luci or painted the other 49, the first person in this space I showed the sketch of The Blueprint Skull to was actually Casey💎, when he was working at SuperRare . Casey was the very first person who onboarded me to NFTs, helping me navigate the early days of whatever it meant to even mint something. I explained the idea of gifting one to each person who bid in my first auctions. Though most of the Skulls went to the bidders, Casey's didn't. He didn't ask for one. I didn't tell him I'd give him one. But he helped me take my first steps here, and it's hard to imagine any of this making sense, or unfolding the way it has, without him. Since then, we've broken bread across continents, seen quite a lot of chortling margarita consumption, watched the rise and fall of a lot around us, weathered inter-Council dramas. He brought Laura El into The Monument Game, played as a Player, wore a Mask. Most of the vibe that started all of this, the wild west of it, feels faded in the broader space at times. But every Skull has a story and a person who helped us get here. Casey will always be the one who was there before any metric muddled the reason to care. The entrance: Last fall, Benji came over for a studio visit. We walked through Luci, the works, structure, and dream, as anyone who visits does. But we mostly talked about being a father and having a father. We discussed the very idea of "collection" stripped of accumulation, value, or signal, located more in the act or ceremony of it. What it was to grow up with a curious father who studied the edges of each thing he saw to know the next layer beneath why anyone might look or ignore it. That to pass this on is to pass on questioning, more than it is to pass on any kind of answer. The process of collecting can be perceived as an individual act of hoarding. For some it is maybe. But at its best, it's a way to bind through shared questioning, to bond in cooperation and competition with friends and family, it is the swapped story and meme of it all, and each object gathered along the way carries some shared memory that can, often does, and with intent: should; drift out of the object entirely. All in the psalm, always has been. The studio visit came and went. Soon after, a package arrived in the mail with two of the softest stuffed animals added to my daughter's own collection, now among her favorites. The Skull is a bonus to that, in the scheme of shared memory. For Rachel and I, while we are heads down making a body of work that unsettles us and excites us but demands unknown time to accomplish, it means a great deal to have this kind of support from long term people in the quiet process of making work we want to leave behind ourselves. Enormously grateful to Casey for the many years of support and friendship, to Benny for being a true patron, and to Benji for entering the arena for what I'm working on next. Welcome.show more

Researchers made KMeans 200x faster. And the new technique... also beats approaches like cuML and FAISS. Flash-KMeans is an IO-aware implementation of exact KMeans that redesigns the algorithm around modern GPU bottlenecks. By attacking the memory bottlenecks directly, Flash-KMeans achieves: - 33x speedup over cuML - 200x speedup over FAISS This speedup comes from how it moves through GPU memory. Standard KMeans runs in two steps, and both are bottlenecked by reads and writes to GPU memory: 1) The first step matches every point to its nearest centroid. Standard KMeans computes the full point-to-centroid distance matrix, writes it out to GPU memory, then reads it back to find each nearest centroid. That write-then-read round trip is the bottleneck. Flash-KMeans combines the distance calculation with the nearest-centroid step, so the result is computed on-chip and the full matrix is never written out. 2) The second step recomputes each centroid by averaging the points assigned to it. Standard KMeans has thousands of threads writing into the same centroid slots at once, so they stall waiting for their turn. Flash-KMeans sorts points by cluster first, turning scattered writes into sequential reductions that read and write memory in one efficient pass. Using these two optimizations at the million-scale, Flash-KMeans completes a standard KMeans iteration in a few milliseconds. The video below depicts this in action. Several reasons why this is important: KMeans has always been an offline primitive. Something you run once to preprocess data and move on. These speedups make the approach viable in several runtime-critical systems. ↳ Vector indices like FAISS use KMeans to build search indices. Faster KMeans means you can re-index dynamically as data changes. ↳ LLM quantization methods need KMeans to find optimal weight codebooks, per layer, repeatedly. What takes hours could now take minutes. ↳ MoE models need fast token routing at inference time. Flash-KMeans makes it viable to run this inside the inference loop, not just in preprocessing. I have shared the paper in the replies. That said, memory is the real constraint Flash-KMeans solves, and the problem is not just limited to clustering. The vectors a RAG system stores after indexing create similar bottlenecks. I wrote a detailed walkthrough recently on cutting this vector memory by 32x with binary quantization, querying 36M+ vectors in a few milliseconds. Read it below.show more

The animated scene of the new background + logo... is so lovely. They didn't have to make it but they did and I think that reflects the level of extra care that is going into the game now. For the first time in a long time, the future of the game is bright.show more

We just released a new marketing site and I... wanted to talk through some of the changes that we made and why. Design thinking is the other side of the feed, it's usually more nuanced than the flashier parts and typically doesn't hold up as well to a quick scroll. Doesn't bode well for this thread! But, this will be my attempt to talk through the changes made and why. First up, this is our new hero. At Loops.so, our goal right now is to be the only platform you need to email your users The H1 matches that intention and the subheader reinforces it. We also wanted it to be clear we ship fast and often, so we added a pill showing the latest featured changelog. Agents are part of how people build software today, so we wanted to give that the second most important placement on the page, right after signup. Finally, the logo strip should not distract too much, but it should still help people understand these are users, not just logos. So we added the date of first payment from Stripe next to each logo. Next up, more things!show more

I finally landed on a memory-optimal way to render... audio waveforms in the browser. The full audio is decoded once, then chunked into 2s bins. Each bin stores peaks at 800 peaks/sec in a Uint8Array. Bins are persisted individually in IndexedDB, which keeps the data as raw Uint8 arrays and is faster to read than OPFS. Local or session storage wouldn’t work here. A 2h video ends up as ~5.76 MB cached on disk. While scrolling or zooming the timeline, I load only the visible bins and downsample them to the required resolution. Everything runs async and I never keep more than a single bin in memory at once, ~800 bytes. Rendering supports mixed resolutions in the same pass since bins arrive async. By caching a read index and sorting bins ascending, each peak lookup stays O(1). On refresh, I wipe the IndexedDB so cache size never gets out of hand.show more

Heroes never die—they remain in our hearts. 😔🕯️I recently... received a request to honor the memory of a fallen soldier – Danylo Unfortunately, it is impossible to bring him back to life, but we can save the lives of other soldiers. To do this, I ask you to support the Armed Forces of Ukraine with your donations. One such option is the charity project “Revenge,” where, for a donation, we write your message on artillery, tank, mortar, and other shells. All funds go directly to the soldiers for their immediate needs. So don't pass by, write to us in private messages and place your order, thereby supporting the Armed Forces of Ukraine!show more

There's been an unfortunate incident in LA with a... Uhaul plowing into a crowd of anti-Khameini protestors This man should never have been able to get near the crowd with a uhaul but some info about the situation seem to be - signage on the truck is anti both the shah and the current ayatollah. is this his actual position or is this camouflage to have gotten into the protest to perpetrate an attack? "no Shah, No regime, No Mullah" Mullah is a religious leader so possibly referring to the current leader and not a king like Pahlavi Timeline appears to be - anti-Khameini protestors try to rip signs off his vehicle and are bashing on the windows and eventually his passenger side window is broken - guy in uhaul then stutterstops forward into the crowd, eventually accelerating further, then stuttering again, then full stopping down the road - there is a man surfing on top of the uhaul in the 3rd video, below I have posted another video showing the man on top of the uhaul trying to take the posters off the side, so he is likely part of the anti-Khameini protestors - uhaul driver is taken into custody by police is this a case of police not having the street sufficiently blocked off and so a guy was able to get a uhaul in here? He should not have been able to drive a uhaul this close to a massive protest crowd There are a lot of people saying this is a terrorist attack, it is possible it could be one but I don't think there's enough information to accurately assert that at this time The chronology of events also shows it is possible that the driver was in fear of his life since protestors were banging on the uhaul, windows, and removing signs+ eventually breaking his window Whatever turns out to be the actual case, it is an unfortunate event and as of right now a seeming silver lining is that no deaths have been reportedshow more

Adiyogi represents the absolute unity of the existence. To... be able to experience this unity within oneself is Yoga. For too long human beings have turned their differences into discriminations and in turn conflict. It is time everyone embraces all there is as life, and be in Yoga or union with the existence. In this brief life, this possibility is the only way forward. Adiyogi and the technologies of wellbeing that he has offered to us in the form of Yogic sciences, belong not to the past, but to the future and the future wellbeing of humanity on this planet. -Sg #HarGharTiranga #InnerEngineering Ministry of Cultureshow more

Dear The Right When you think of the DIRECTION... you want our country to go in, how brilliant it is that so many are now facing that way. If we turn on each other as angry individuals, we are fighting the wrong enemy Let us keep moving forwards, together, in the DIRECTION we need the country to go. I believe good people wish to maintain their integrity when they vote. A unified party will reveal itself. And that is a wonderful thing.show more

The part of PixVerse that stood out to me... was not just the AI fan host, but the chance to turn a football cheer into something personal. Users can interact with Reina, create personalized cheer videos, and enter a lucky draw for PixVerse service credits. The credits can only be used on PixVerse. They are non transferable and have no cash value. It is a playful early experiment, but I like that it gives users a simple way to make something of their own and share it with others. That mix of creating a cheer moment and joining the credit draw is what I would want to try 👉show more

OpenCode Go is now wired into Codex!! The pricing... is insane. $10 gets you 10,000 DeepSeek requests every 5 hours (no weekly limits). I converted that into DeepSeek API dollars because I thought I was reading it wrong, and the same 5 hours of usage would run somewhere between $10 and $30 depending on how big your context gets. So one afternoon of use already covers the whole sub. It comes with Kimi K3 as well. Both sit in the picker next to my other models now. Next time we hit a limit in the middle of a loop, we can grab it and keep going.show more

This Chinese developer launched Llama 70B locally on a... MacBook on a plane and for a full 11 hours without internet ran client projects. He was sitting by the window on a transatlantic flight with a MacBook Pro M4 with 64 GB of memory. WiFi on board cost $25 for the flight. He declined. No cloud API, no connection to Anthropic or OpenAI servers, no internet at all. Just a local Llama 3.3 70B on bf16 and his own orchestrator script. The model runs through llama.cpp. Generation speed, 71 tokens per second. Context around 60,000 tokens. Memory usage, 48.6 GiB out of 64. Battery at takeoff, 3 hours 21 minutes. And he gave the orchestrator this system prompt before takeoff: "You are an offline orchestrator running on a single MacBook. There is no network. The only resources you have are local files in /Users/dev/work, the Llama 70B inference server at localhost:8080, and a battery budget of 3 hours 21 minutes. Process the queue at /Users/dev/work/queue.jsonl (one client task per line). For each task: draft → run local evals → save artefact to /Users/dev/work/done/. Save context checkpoints every 12 tasks so you can resume after a battery swap. Stop only on empty queue or when battery drops below 5%." So the system knows exactly what resources it is running on. It knows it has no connection to the outside world for the next 11 hours. It knows it has finite memory and a finite battery. It knows the human will not intervene until the plane lands. The system runs in 1 loop. Takes a task from the queue, runs it through inference, saves the artifact, writes a checkpoint. Task after task, just like that. And only when the battery drops below 5% does the orchestrator automatically pause, waits for the laptop to switch to the backup power bank, and continues from the last checkpoint. Here is what the system actually writes in his log during the flight: "saved context checkpoint 8 of 12 (pos_min = 488, pos_max = 50118, size = 62.813 MiB)" "restored context checkpoint (pos_min = 488, pos_max = 50118)" "prompt processing progress: n_tokens = 50 / 60 818" "task 37016 done | tps = 71 s tokens text → /Users/dev/work/done/proposal_westside.md" Outside the window, clouds, blue sky, and no WiFi. On the tray, 1 MacBook, an open terminal on 2 screens, and an inference server on localhost. From what I have observed, this is the cleanest offline AI workflow I have seen in the past year: 11 hours of flight, $0 for WiFi, and the entire client queue closed before landing.show more