i just beat Google DeepMind's turboquant introducing Shard. 10x KV cache compression on Llama-3.1-8B. zero quality loss - 10x @ 8K context, 11.2x @ 32K - NIAH recall 1.000 across 4K-32K - LongBench Δ ≈ 0 vs FP16 turboquant tops out at 4-6x at the same quality. we doubled... it. read more: Kirrishow more

github:

@AIBuddyK @GoogleDeepMind Cool work! but nobody is using Llama-3.1-8b. Would love to see this working with Qwen3.6 MoE or 27b

Interesting direction. The asymmetry is the important part here, not the headline compression ratio. K and V behave fundamentally differently under attention, and treating them identically leaves performance on the table. The memory caveat matters too: a compressed cache is only half the story if decode still needs a second working copy. If you store a tiny cache but also build dense scratch/reconstruction buffers to actually use it, the real RAM bill is closer to “compressed cache + working cache,” not just the advertised stored-cache size. That is where these systems get interesting: decode-time distribution shift, materialization overhead, transient workspace growth, and attention-weighted error behavior rather than raw reconstruction MSE. Related: * Asymmetric K/V Cache Compression: * Sparse V Dequantization: * Why MSE Fails for KV Quantization:

@GoogleDeepMind This needs a KVTC baseline

@GoogleDeepMind NIAH is basically a solved metric at this point. Run this through SWE-bench and let's see if the attention heads actually survive a 10x squeeze.

@GoogleDeepMind I explored this but it only works with llama? Can you get it to work with Qwen3.6 Moe because then its a Game changer!

@GoogleDeepMind The benchmark I'd want to see: does Shard hold quality across 50-turn agent loops, not just single-shot evals? KV bloat is the actual bottleneck once tool calls + retries start stacking.

@GoogleDeepMind Yep, low rank tensors are everywhere. The tricky part has always been, as you found, the decomp kills throughput

Love what you're doing, and really enjoyed the well written write up. One thought - Turboquant is data-oblivious - Shard fits a per-layer PCA basis + DP bit allocation on prefill activations. Have you compared it to other data-dependent baselines in same regime like KVTC, AsymKV etc?

@GoogleDeepMind

@GoogleDeepMind 10x with NIAH 1.000 is wild. Curious how Shard handles attention head heterogeneity — flat ratio across all heads feels suboptimal. Real agent workflows with tool calls stress KV patterns differently than LongBench prose. Would love multi-turn agentic benchmarks.

@GoogleDeepMind ur tuff

@GoogleDeepMind genuinely the tuffest

@GoogleDeepMind @grok how he have done this , process , and got this accurately

@GoogleDeepMind @grok how efficient is this in respect to deepseek's context compression? on a scale of 1-10, two decimal digits allowed

@GoogleDeepMind 10x compression, zero loss? Every compression paper says this on its own eval. The real test is the 'compression cliff' where NIAH fails.

@GoogleDeepMind @TheAhmadOsman Check this out big man.

@GoogleDeepMind interesting that the compression ratio varies with context length (10x at 8K vs 11.2x at 32K). pure quantization would be constant. suggests eviction, structural pruning, or a hybrid approach. what breaks first when you push beyond 32K?

@GoogleDeepMind Pretty legit. But cool kids are processing in the compressed space, too. I could teach you, but i'd have to charge. 🔔🥛

@GoogleDeepMind Need the repo. 10x KV compression at zero loss? Big claim. DeepMind didn't ship turboquant overnight. Show the work.

@GoogleDeepMind Congrats on this breakthrough! Really cool to see this happen! Waiting for the 20x eagerly now haha

@GoogleDeepMind Thank you, today will try and post here my experience!

@GoogleDeepMind @0xSero 👀

@GoogleDeepMind

@GoogleDeepMind Solo dev beating a DeepMind benchmark is the kind of thing that happens more now

@GoogleDeepMind Please create a PR on llama.cpp repo

@GoogleDeepMind @anirudhbv_ce 👀👀👀👀

@GoogleDeepMind Cool, can it work on Qwopus ?

@GoogleDeepMind Love this. the case I'd want benched is code completion at 32K with retrieved file context. that's where tool-call traces + repo Context blow past chat token entropyy

@GoogleDeepMind @grok can I test this with my 2x3090s

@GoogleDeepMind Interesting, waiting for more test on more models

@GoogleDeepMind Good Work 🫴

@GoogleDeepMind wow so tuff this is insane

@GoogleDeepMind What about prefill speed loss?

@GoogleDeepMind 👀