Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Absolutely recommend using Cache Control in Claude Code Probably one of the Mods that will save you the most tokens and help keep your sessions going much longer It adds this bar for Claude’s 5 minute cache and sends notifications to help keep the cache warm Install command: npx...

202,376 görüntüleme • 1 gün önce •via X (Twitter)

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Day 11/90 of Inference Engineering How does vLLM work and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.

max fu

70,797 görüntüleme • 2 ay önce

Claude Design + Shopify is f*cking ridiculous 🤯 You can now publish pages from Claude Design → Claude Code → Shopify. Built 100% with Claude Design, Claude Code, and the Shopify CLI. Perfect for DTC brands and agencies who want to skip the design → dev handoff entirely. Here's how it works: → Design any landing page in Claude Design → Export as a zip and drop it into Claude Code → Install the Shopify + Shopify AI Toolkit plugins → Prompt Claude to convert the HTML into a Shopify page template + push to live theme → Claude uploads the images, deploys the files, and creates a published page No more handing designs off to a dev and waiting 2 weeks for a Shopify page. What you get: - A workflow that turns any Claude Design page into a real Shopify page template - Editable sections so your marketing team can swap copy, images, and CTAs without code - Images uploaded straight to Shopify Files automatically - A files-only deploy that only touches what's new in your live theme - A repeatable pipeline you can use every time you design a new landing page This is essentially the design-to-deploy pipeline brands have been waiting for. I put together a step-by-step playbook for going from Claude Design → published Shopify page. Every install, every plugin, every command, and the exact prompt that runs the whole thing. Want the playbook for free? > Like this post > Comment "SHOP" And I'll send it over (must be following so I can DM)

Mike Futia

59,314 görüntüleme • 4 ay önce