max fu's banner

max fu

@maxxfuu • 3,399 subscribers

writing gpu kernels…

Shorts

Day 11/90 of Inference Engineering How does vLLM work and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.

Day 11/90 of Inference Engineering How does vLLM work and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.

70,797 просмотров

Day 12/90 of Inference Engineering What is chunked prefill within vLLM? In continuation of yesterday's post on the high level architecture of vLLM, I want to dive deeper into vLLM core engine starting with the mechanics of chunked prefill. In this post, I will closely follow the original blog on the anatomy of vLLM. To start, let's define chunked prefill. It's a runtime inference optimization technique that splits a long input request so that it doesn’t monopolize the whole GPU. Keep in mind this is all within the context of vLLM. And since vLLM is an inference engine that's meant to serve a model to multiple concurrent users, having a GPU that’s fully monopolized on a single user's request means other users' requests would be in queue waiting to be processed. It isn’t too good to have the whole GPU occupied on a single request when the GPU is meant to be shared! So the key idea behind chunked prefill is to break the long request into smaller chunks, so that each chunk along with other users' requests gets processed and written into the KV cache together. Suppose we split up the long request into chunks and each chunk has 8 tokens. Now each memory block can hold 4 tokens. Therefore, 8 tokens can fit into 2 blocks of memory. After the first forward pass, 2 blocks are occupied, and after the second forward pass, 4 blocks of memory are occupied and so forth. Each forward pass handles a small chunk of the long request so that there's room in the same pass to keep serving other users' requests. Here's a small animation that I made today to fully visualize the idea behind chunked prefill when learning this topic~

Day 12/90 of Inference Engineering What is chunked prefill within vLLM? In continuation of yesterday's post on the high level architecture of vLLM, I want to dive deeper into vLLM core engine starting with the mechanics of chunked prefill. In this post, I will closely follow the original blog on the anatomy of vLLM. To start, let's define chunked prefill. It's a runtime inference optimization technique that splits a long input request so that it doesn’t monopolize the whole GPU. Keep in mind this is all within the context of vLLM. And since vLLM is an inference engine that's meant to serve a model to multiple concurrent users, having a GPU that’s fully monopolized on a single user's request means other users' requests would be in queue waiting to be processed. It isn’t too good to have the whole GPU occupied on a single request when the GPU is meant to be shared! So the key idea behind chunked prefill is to break the long request into smaller chunks, so that each chunk along with other users' requests gets processed and written into the KV cache together. Suppose we split up the long request into chunks and each chunk has 8 tokens. Now each memory block can hold 4 tokens. Therefore, 8 tokens can fit into 2 blocks of memory. After the first forward pass, 2 blocks are occupied, and after the second forward pass, 4 blocks of memory are occupied and so forth. Each forward pass handles a small chunk of the long request so that there's room in the same pass to keep serving other users' requests. Here's a small animation that I made today to fully visualize the idea behind chunked prefill when learning this topic~

29,449 просмотров

Videos

Anya Rossi

sweetdream.ai

SweetDream.ai•Sponsored•Livecam

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

Exclusive private shows

1.2k viewers online

Private Show

Join now for exclusive access

Free preview available • Premium content

Day 14/90 of Inference Engineering I didn't finish the MNIST in CUDA project today, which made me hesitant to post since I didn't want to put out some slop. But in the spirit of being consistent and showing up, here's a concept I've revisited today now that I've started writing CUDA again. This is an animation on the CUDA memory life cycle. Generally speaking, when writing CUDA, memory management goes through the following stages: 1. Allocate memory on the host 2.Initialize / populate the memory 3.Allocate device memory 4.Copy host data to the device 5. Launch the kernel 6. Copy device data back to the host 7. Free the memory - On a brief reflection, I'm starting to write the matmul kernels in CUDA instead of in C. So this in itself is another paradigm shift in thinking. Excited to hone my technique over the coming week. I'll update you guys when I've completed the project; otherwise I'll just make a post like usual on what I've learned during the day!

Day 14/90 of Inference Engineering I didn't finish the MNIST in CUDA project today, which made me hesitant to post since I didn't want to put out some slop. But in the spirit of being consistent and showing up, here's a concept I've revisited today now that I've started writing CUDA again. This is an animation on the CUDA memory life cycle. Generally speaking, when writing CUDA, memory management goes through the following stages: 1. Allocate memory on the host 2.Initialize / populate the memory 3.Allocate device memory 4.Copy host data to the device 5. Launch the kernel 6. Copy device data back to the host 7. Free the memory - On a brief reflection, I'm starting to write the matmul kernels in CUDA instead of in C. So this in itself is another paradigm shift in thinking. Excited to hone my technique over the coming week. I'll update you guys when I've completed the project; otherwise I'll just make a post like usual on what I've learned during the day!

42,808 просмотров • 1 месяц назад

Day 9/90 of Inference Engineering I built a visualization tool for CUDA kernels and C code to view the buffers for each step through. Check out Yesterday, I was terramogged by a simple implementation of a basic matmul between three matrices in 1D vector form. There were essentially three 1D vectors, each vector representing a corresponding matrix (A, B, C). In the implementation, each matrix is just just a pointer to the first element of a 1D array. You are to compute the output matrix through A^T @ B and store it in C. This morning I built a visualizer to help me see and understand why the triple for-loops work the way they do. In the coming days, I’m confident I can make sense as well as backpropagation! As for it does not include a built-in compiler. It's just an interpreter; the visualization shows exactly what the code does, bugs included. The main thing I wanted to visualize is matmul with a triple for-loop between a sequential program to that of a parallel one. Expect bugs and other inconsistencies within I’ll polish this mini-project in the coming days. As for the rest of today, I’ll be trying to understand backpropagation in C and CUDA .

Day 9/90 of Inference Engineering I built a visualization tool for CUDA kernels and C code to view the buffers for each step through. Check out Yesterday, I was terramogged by a simple implementation of a basic matmul between three matrices in 1D vector form. There were essentially three 1D vectors, each vector representing a corresponding matrix (A, B, C). In the implementation, each matrix is just just a pointer to the first element of a 1D array. You are to compute the output matrix through A^T @ B and store it in C. This morning I built a visualizer to help me see and understand why the triple for-loops work the way they do. In the coming days, I’m confident I can make sense as well as backpropagation! As for it does not include a built-in compiler. It's just an interpreter; the visualization shows exactly what the code does, bugs included. The main thing I wanted to visualize is matmul with a triple for-loop between a sequential program to that of a parallel one. Expect bugs and other inconsistencies within I’ll polish this mini-project in the coming days. As for the rest of today, I’ll be trying to understand backpropagation in C and CUDA .

12,746 просмотров • 1 месяц назад

Больше нет контента для загрузки