Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

i just beat Google DeepMind's turboquant introducing Shard. 10x KV cache compression on Llama-3.1-8B. zero quality loss - 10x @ 8K context, 11.2x @ 32K - NIAH recall 1.000 across 4K-32K - LongBench Δ ≈ 0 vs FP16 turboquant tops out at 4-6x at the same quality. we doubled...

157,252 görüntüleme • 5 ay önce •via X (Twitter)

35 Yorum

Krish profil fotoğrafı
Krish5 ay önce

github:

Sandro profil fotoğrafı
Sandro5 ay önce

@AIBuddyK @GoogleDeepMind Cool work! but nobody is using Llama-3.1-8b. Would love to see this working with Qwen3.6 MoE or 27b

Tom Turney profil fotoğrafı
Tom Turney5 ay önce

Interesting direction. The asymmetry is the important part here, not the headline compression ratio. K and V behave fundamentally differently under attention, and treating them identically leaves performance on the table. The memory caveat matters too: a compressed cache is only half the story if decode still needs a second working copy. If you store a tiny cache but also build dense scratch/reconstruction buffers to actually use it, the real RAM bill is closer to “compressed cache + working cache,” not just the advertised stored-cache size. That is where these systems get interesting: decode-time distribution shift, materialization overhead, transient workspace growth, and attention-weighted error behavior rather than raw reconstruction MSE. Related: * Asymmetric K/V Cache Compression: * Sparse V Dequantization: * Why MSE Fails for KV Quantization:

Will Bui profil fotoğrafı
Will Bui5 ay önce

@GoogleDeepMind This needs a KVTC baseline

Sakura Yuki profil fotoğrafı
Sakura Yuki5 ay önce

@GoogleDeepMind NIAH is basically a solved metric at this point. Run this through SWE-bench and let's see if the attention heads actually survive a 10x squeeze.

TurtleWaxWins 🇺🇦 profil fotoğrafı
TurtleWaxWins 🇺🇦5 ay önce

@GoogleDeepMind I explored this but it only works with llama? Can you get it to work with Qwen3.6 Moe because then its a Game changer!

Anoop Thiparala profil fotoğrafı
Anoop Thiparala5 ay önce

@GoogleDeepMind The benchmark I'd want to see: does Shard hold quality across 50-turn agent loops, not just single-shot evals? KV bloat is the actual bottleneck once tool calls + retries start stacking.

Dhruv Mangtani profil fotoğrafı
Dhruv Mangtani5 ay önce

@GoogleDeepMind Yep, low rank tensors are everywhere. The tricky part has always been, as you found, the decomp kills throughput

Ashley Hirst profil fotoğrafı
Ashley Hirst5 ay önce

Love what you're doing, and really enjoyed the well written write up. One thought - Turboquant is data-oblivious - Shard fits a per-layer PCA basis + DP bit allocation on prefill activations. Have you compared it to other data-dependent baselines in same regime like KVTC, AsymKV etc?

Nicholas Motamedi profil fotoğrafı
Nicholas Motamedi5 ay önce

@GoogleDeepMind

geekfox charlie profil fotoğrafı
geekfox charlie5 ay önce

@GoogleDeepMind 10x with NIAH 1.000 is wild. Curious how Shard handles attention head heterogeneity — flat ratio across all heads feels suboptimal. Real agent workflows with tool calls stress KV patterns differently than LongBench prose. Would love multi-turn agentic benchmarks.

shams haroon profil fotoğrafı
shams haroon5 ay önce

@GoogleDeepMind ur tuff

umar profil fotoğrafı
umar5 ay önce

@GoogleDeepMind genuinely the tuffest

Soham Patra profil fotoğrafı
Soham Patra5 ay önce

@GoogleDeepMind @grok how he have done this , process , and got this accurately

Quantum Voyager profil fotoğrafı
Quantum Voyager5 ay önce

@GoogleDeepMind @grok how efficient is this in respect to deepseek's context compression? on a scale of 1-10, two decimal digits allowed

DC|use.fo profil fotoğrafı
DC|use.fo5 ay önce

@GoogleDeepMind 10x compression, zero loss? Every compression paper says this on its own eval. The real test is the 'compression cliff' where NIAH fails.

Mohamad Al-Zawahreh profil fotoğrafı
Mohamad Al-Zawahreh5 ay önce

@GoogleDeepMind @TheAhmadOsman Check this out big man.

Guilherme O'Tina profil fotoğrafı
Guilherme O'Tina5 ay önce

@GoogleDeepMind interesting that the compression ratio varies with context length (10x at 8K vs 11.2x at 32K). pure quantization would be constant. suggests eviction, structural pruning, or a hybrid approach. what breaks first when you push beyond 32K?

gpu go brr... profil fotoğrafı
gpu go brr...5 ay önce

@GoogleDeepMind Pretty legit. But cool kids are processing in the compressed space, too. I could teach you, but i'd have to charge. 🔔🥛

Elon Orbit profil fotoğrafı
Elon Orbit5 ay önce

@GoogleDeepMind Need the repo. 10x KV compression at zero loss? Big claim. DeepMind didn't ship turboquant overnight. Show the work.

Piquo profil fotoğrafı
Piquo5 ay önce

@GoogleDeepMind Congrats on this breakthrough! Really cool to see this happen! Waiting for the 20x eagerly now haha

Erman Eroglu profil fotoğrafı
Erman Eroglu5 ay önce

@GoogleDeepMind Thank you, today will try and post here my experience!

L0GYKAL.gwei profil fotoğrafı
L0GYKAL.gwei5 ay önce

@GoogleDeepMind @0xSero 👀

Albert Axyonov profil fotoğrafı
Albert Axyonov5 ay önce

@GoogleDeepMind

AI Mastery Guide profil fotoğrafı
AI Mastery Guide5 ay önce

@GoogleDeepMind Solo dev beating a DeepMind benchmark is the kind of thing that happens more now

Jayakrishnan R profil fotoğrafı
Jayakrishnan R5 ay önce

@GoogleDeepMind Please create a PR on llama.cpp repo

luthira profil fotoğrafı
luthira5 ay önce

@GoogleDeepMind @anirudhbv_ce 👀👀👀👀

@alexdolbun profil fotoğrafı
@alexdolbun5 ay önce

@GoogleDeepMind Cool, can it work on Qwopus ?

Pochi profil fotoğrafı
Pochi5 ay önce

@GoogleDeepMind Love this. the case I'd want benched is code completion at 32K with retrieved file context. that's where tool-call traces + repo Context blow past chat token entropyy

Erman Eroglu profil fotoğrafı
Erman Eroglu5 ay önce

@GoogleDeepMind @grok can I test this with my 2x3090s

Luong NGUYEN profil fotoğrafı
Luong NGUYEN5 ay önce

@GoogleDeepMind Interesting, waiting for more test on more models

ZIAISTAN profil fotoğrafı
ZIAISTAN5 ay önce

@GoogleDeepMind Good Work 🫴

Ohm Patel profil fotoğrafı
Ohm Patel5 ay önce

@GoogleDeepMind wow so tuff this is insane

tryingEveryThing profil fotoğrafı
tryingEveryThing5 ay önce

@GoogleDeepMind What about prefill speed loss?

Ahmed Ibrahim Hamdy profil fotoğrafı
Ahmed Ibrahim Hamdy5 ay önce

@GoogleDeepMind 👀

Benzer Videolar

i spent 3 hours finding the sweet spot for hermes 4.3 36B on a single RTX 3090. saving you the trouble, anon. the model is 21.8GB at Q4_K_M. that leaves 2.2GB free on 24GB VRAM. not much room for KV cache. here's what actually happened: 4K: 35.3 tok/s 8K: 35.2 tok/s 16K: 34.8 tok/s 32K: 34.6 tok/s 64K: 6.4 tok/s 128K: 1.9 tok/s flat from 4K to 32K. then it falls off a cliff at 64K. the trick is quantized KV cache. without it you OOM at 16K. with quantized KV cache you get 32K at full speed. all 64 layers on GPU. at 64K something weird happens. ngl 99 (all layers on GPU) = 3.96 tok/s. the KV cache silently spills to CPU. drop to ngl 55 and speed jumps to 6.37. drop to ngl 48 and it gets worse again (3.46). there's an offload sweet spot where you free just enough VRAM for the cache without losing too much compute to PCIe transfers. 128K works at ngl 32 but you're at 1.95 tok/s. half the model on CPU. usable for batch work, not for interactive. the sweet spot command: llama-server -m hermes-4.3-36b-Q4_K_M.gguf -ngl 99 -c 32768 --cache-type-k q4_0 --cache-type-v q4_0 32K context. 34.6 tok/s. all on GPU. this is where dense 36B lives on 24GB. for comparison, qwen 3.5 (35B MoE, 3B active) holds 112 tok/s from 4K all the way to 262K on the same GPU. no speed drop. same total params, completely different architecture. hybrid linear attention means flat context scaling. dense pays for every token in the KV cache. code and quality comparison coming next. fast vs slow generation side by side in the videos below.

Sudo su

52,268 görüntüleme • 8 ay önce

Google's Gemma 4 26B A4B QAT hits 25+ tokens/sec and 320+ tokens/sec prefill on 8 GB VRAM (RTX 4060) + 16 GB RAM using TurboQuant Prefill just went from 200 → 320+ tok/s on the same 8GB card. 1.6x, no new hardware, no new quant, just a KV cache trick stacked on top of the Gemma 4 26B MoE setup from a few days ago. A few days ago I posted Gemma 4 26B A4B hitting 28 tok/s decode on 8GB VRAM using native MTP. prefill was stuck around 200 tok/s. fair callout by the community. So today I tested something I'd already been meaning to try: TheTom/llama-cpp-turboquant, the TurboQuant KV cache fork by Tom Turney (Tom Turney). (github link in the comments) thanks to him, the fork just got resynced to mainline, so MTP + TurboQuant now run together cleanly (I didnt see any meaningful gains by using MTP with this setup though but you can try). The flags (No MTP): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -c 64000 --cache-type-k q8_0 --cache-type-v turbo3 Results on the same RTX 4060 8GB, tested with a 27k token prompt at 64k context loaded: Prefill: 200 tok/s → 320+ tok/s Decode: stayed above 25 tok/s (without MTP) Why it works: TurboQuant uses walsh hadamard rotation + polar quantization on the KV cache. keys are sensitive to compression, values aren't much, so it splits the difference: K stays at q8_0, V drops to turbo3 (~3 bits). bonus from the memory savings: same 8GB card can now stretch to 100-120k context with minimal decode penalty. It should now be snappier with any agent harness such as hermes agent without compromise on intelligence. If you're already running Gemma 4 on a small card, this stacks on top for free. Try --cache-type-k q8_0 --cache-type-v turbo3 on your setup and report back what your prefill/decode split looks like. unsloth model gguf and llama.cpp turboquant fork links in the comments. what's your prefill number before vs after?

Alok

119,821 görüntüleme • 5 ay önce