You asked for it, so here it is. Visualizing... CPU cache speeds relative to RAM. Cache optimization is important too!show more

Some of the biggest news today is Cloudflare Worker's... Cache. So I built this small catalog to experience it. It allows you to purge the cache for certain pages, or the entire catalog. It also shows you the exact caching headers used. See it in action or try it yourself:show more

Submitted Easy Vertex Anim for #UE5.8 too. You can... create pivot painter 2 effects and bake chaos cache to vat, 100% NO AI, check it out!show more

MeidasTouch It was still in my cache

WanGP's latest MiniMax H3 update is huge: - Way... faster Generation - 2X faster Ref2VA - Lower-RAM & new accelerators - Long videos with smoother visual/audio transitions. - Huge speed gains with Sol-Attention, First Block Cache, etc. And Pinokio launcher is optimized for it 👇show more

A developer just squeezed this 284B-parameter DeepSeek model into... 64GB of regular DDR5 RAM. 🤯 And it's running with 0 GPUs! Very cool that with just a CPU, 64GB RAM, and some intelligent engineering, you can run DeepSeek. This is llama-modchip. GitHub dev synchopate has published a modified llama.cpp inference stack capable of running DeepSeek-V4-Flash entirely in system memory. Here's the setup 🧠 DeepSeek-V4-Flash 🔥 284.33B total / ~13B active 💾 64GB DDR5 RAM 🎮 ZERO GPUs 📚 256K configured context ⚙️ Custom AVX2 / AVX-512 CPU kernels 🔓 Source code available on GitHub Here are some developer-reported numbers Memory 🐌 Previous quant → 76.87 GiB 🚀 Optimized quant → 56.56 GiB 💾 Actual process RAM → 57.61 GiB 26.4% reduction in model storage, enough to bring it below the 64GB RAM limit. And the inference performance? ⚡ CPU decode → 9.1–14.7 tps (very usable) 📥 Prompt processing → 44.36 tps Prefill before optimization: 🐌 25.05 tps Prefill after optimization: 🚀 44.36 tps That's 77.1% FASTER prompt processing, damn So how did they do it? The developer combined these 🧠 Low-bit quantization 🔧 Specialized AVX2 / AVX-512 CPU kernels ⚡ Better reuse of weights across prompt tokens 💾 CPU cache tiling across registers, L1, L2 and L3 🔬 MoE expert and attention optimizations Making hardware do things it was never supposed to do. 🔥show more

A few of you asked if you could buy... the glass recorder, so I polished it, here it is:show more

A lot of people asked me to post FOOTage... of Iuno's burst It features her, you guessed it, feet So here it is Enjoy!show more

cache is out so use this easy b site... utility and win the round (hopefully)show more

Want the latest, flashiest skins? Lucky Cache is here... for you every week! Every Friday to Sunday, Step into the Lucky Cache and challenge destiny! (level 80 needed) 🎁 One free draw every week A single draw could change everything and rewards can refresh 3 times a week — are you ready? Luck doesn't wait!show more

cache-cache à paris

Day 11/90 of Inference Engineering How does vLLM work... and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.show more

Seniors discussing Architecture: Redis vs In-Memory Cache Meanwhile, the... Vibe Coder😂 wait for itshow more

RAM and CPU is DEAD ☠️

Some of you guys asked for a part two... so here it is also shout out to Joe Bart clips##joe#joebart#joebartolozzishow more

RAM and CPU is DEAD ☠️😱

Is it too much for you to handle?

🚨 Today I published my own Stream Deck plugin... on @Elgato Marketplace . I call it SimpleStats. It's a very lightweight plugin for viewing CPU usage, Ping, Net Throughput, GPU temps, even what is using up all your RAM!!!!!!! The most important part for me: Everything is included.show more

But the KV cache is created for each transformer... layer. By sending each layer’s KV cache after it’s computed, we overlap communication with computation. We stream the KV cache and hide the network delay. We achieve a 4x speedup in prefill & 3x in decode, with 0 network delay.show more