KV caching is the fundamental optimization underpinning autoregressive LLM inference. Transformer layers store keys and values from earlier tokens, then reuse them as new tokens are generated. This avoids recomputation, but at the cost of memory capacity and bandwidth.

I kind of expected a twist that "we can do better than KV cache."

Which software you used for this animation?

I made it from scratch using JavaScript.

@ThisIs_KroY What JavaScript frameworks did you use?

@ThisIs_KroY I use svelte and a combination of d3 js as well as the canvas api and custom shaders.

The part that bites in agent workloads is not the memory, it is invalidation. Reuse only happens on an exact prefix, so anything that shifts near the front, a reordered tool list, a timestamp in the system prompt, a retrieved doc pasted above the history, throws the whole thing away and you recompute a context you already paid for. Keeping the stable parts first and appending only at the tail buys more in practice than most of the tuning that gets attention.

I mean… That’s the reason linear attention exists, no? Memory becomes O(1), so does bandwidth (per token).

the memory-bandwidth tradeoff is the real bottleneck most teams ignore. we switched to paged attention (vLLM style) and cut our KV cache memory by 60% without touching the model. the inference cost problem is mostly an infrastructure problem, not a model problem.

This is closest animation i have seen how kv cache is actually goes through attention layers

KV cache growth is the real scaling bottleneck once you push context length up. Techniques like multi-query and grouped-query attention exist basically to trade off a bit of quality for a much smaller cache footprint.

The trade-off between memory capacity and bandwidth is often overlooked. It's a balancing act.

Can you please share the prompt you used to generate this animation?

the bandwidth half is doing more work than it looks: decode reads the whole kv cache per token, so tokens/sec tracks cache bytes, not flops. it's why mqa became gqa became mla — when the cache is the bill, you shrink the cache instead of buying more bandwidth.

That memory cost is where PagedAttention pays off: vLLM keeps the KV cache in blocks, so far less fragmentation and many more concurrent requests.

Good framing for LLM research work. The practical part is not just trying a stronger model, but logging baselines, data splits, task-specific metrics, and failure cases so the result is reproducible.

Exactly. Once KV is treated as a first-class execution artifact, inference stops being “run the model on a GPU” and becomes a routing problem: where to prefill, where to decode, when to move state, and when transfer cost makes staying put cheaper.

The memory/bandwidth tradeoff is the real bottleneck in practice. Techniques like PagedAttention and sliding windows help, but they shift the pressure to scheduling. Curious how you see the balance between cache size and throughput evolving as context windows grow.

Everyone quotes FLOPs. The invoice says memory.

This animation breaks down the memory vs. compute trade-off so well. Saved! 👏

the interesting part is that KV caching turns autoregressive decoding into a memory bandwidth-bound problem rather than a pure FLOPs problem. With long contexts, KV cache size can dominate GPU memory, i think thats why KV quantization is becoming increasingly important