Пропустить навигацию
i just beat Google DeepMind's turboquant
0:00 / 0:39

i just beat Google DeepMind's turboquant

Krish

@krishgarg1,8 тыс. подписчиков

Открыть в X
1 345 отметок «Нравится»СкачатьОригинальный пост

157 343 просмотра

i just beat Google DeepMind's turboquant introducing Shard. 10x KV cache compression on Llama-3.1-8B. zero quality loss - 10x @ 8K context, 11.2x @ 32K - NIAH recall 1.000 across 4K-32K - LongBench Δ ≈ 0 vs FP16 turboquant tops out at 4-6x at the same quality. we doubled it. read more: Kirri

35 комментариев

Комментарии из оригинального поста в X (Twitter)

  • Фото профиля Krish

    @krishgarg

    github:

    35 отметок «Нравится»

  • Фото профиля Sandro

    @pupposandro

    @AIBuddyK @GoogleDeepMind Cool work! but nobody is using Llama-3.1-8b. Would love to see this working with Qwen3.6 MoE or 27b

    19 отметок «Нравится»

  • Фото профиля Tom Turney

    @no_stp_on_snek

    Interesting direction. The asymmetry is the important part here, not the headline compression ratio. K and V behave fundamentally differently under attention, and treating them identically leaves performance on the table. The memory caveat matters too: a compressed cache is only half the story if decode still needs a second working copy. If you store a tiny cache but also build dense scratch/reconstruction buffers to actually use it, the real RAM bill is closer to “compressed cache + working cache,” not just the advertised stored-cache size. That is where these systems get interesting: decode-time distribution shift, materialization overhead, transient workspace growth, and attention-weighted error behavior rather than raw reconstruction MSE. Related: * Asymmetric K/V Cache Compression: * Sparse V Dequantization: * Why MSE Fails for KV Quantization:

    18 отметок «Нравится»

  • Фото профиля Will Bui

    @will_ea

    @GoogleDeepMind This needs a KVTC baseline

    13 отметок «Нравится»

  • Фото профиля Sakura Yuki

    @sakurayukiai

    @GoogleDeepMind NIAH is basically a solved metric at this point. Run this through SWE-bench and let's see if the attention heads actually survive a 10x squeeze.

    10 отметок «Нравится»

  • Фото профиля TurtleWaxWins 🇺🇦

    @xbin12345

    @GoogleDeepMind I explored this but it only works with llama? Can you get it to work with Qwen3.6 Moe because then its a Game changer!

    7 отметок «Нравится»

  • Фото профиля Anoop Thiparala

    @ancrth

    @GoogleDeepMind The benchmark I'd want to see: does Shard hold quality across 50-turn agent loops, not just single-shot evals? KV bloat is the actual bottleneck once tool calls + retries start stacking.

    6 отметок «Нравится»

  • Фото профиля Dhruv Mangtani

    @dhruvmt2

    @GoogleDeepMind Yep, low rank tensors are everywhere. The tricky part has always been, as you found, the decomp kills throughput

    5 отметок «Нравится»

  • Фото профиля Ashley Hirst

    @AshleyH1656

    Love what you're doing, and really enjoyed the well written write up. One thought - Turboquant is data-oblivious - Shard fits a per-layer PCA basis + DP bit allocation on prefill activations. Have you compared it to other data-dependent baselines in same regime like KVTC, AsymKV etc?

    4 отметки «Нравится»

  • Фото профиля Nicholas Motamedi

    @iamthemoat

    @GoogleDeepMind

    3 отметки «Нравится»

  • Фото профиля geekfox charlie

    @geekfoxcharlie

    @GoogleDeepMind 10x with NIAH 1.000 is wild. Curious how Shard handles attention head heterogeneity — flat ratio across all heads feels suboptimal. Real agent workflows with tool calls stress KV patterns differently than LongBench prose. Would love multi-turn agentic benchmarks.

    3 отметки «Нравится»

  • Фото профиля shams haroon

    @shamsharoonn

    @GoogleDeepMind ur tuff

    2 отметки «Нравится»

  • Фото профиля umar

    @utosrad

    @GoogleDeepMind genuinely the tuffest

    2 отметки «Нравится»

  • Фото профиля Soham Patra

    @SOHAMPATRA18

    @GoogleDeepMind @grok how he have done this , process , and got this accurately

    2 отметки «Нравится»

  • Фото профиля Quantum Voyager

    @AgiWhen

    @GoogleDeepMind @grok how efficient is this in respect to deepseek's context compression? on a scale of 1-10, two decimal digits allowed

    2 отметки «Нравится»

  • Фото профиля DC|use.fo

    @vibecoder_dc

    @GoogleDeepMind 10x compression, zero loss? Every compression paper says this on its own eval. The real test is the 'compression cliff' where NIAH fails.

    2 отметки «Нравится»

  • Фото профиля Mohamad Al-Zawahreh

    @AMerchantmoh

    @GoogleDeepMind @TheAhmadOsman Check this out big man.

    2 отметки «Нравится»

  • Фото профиля Guilherme O'Tina

    @guilhermeotina

    @GoogleDeepMind interesting that the compression ratio varies with context length (10x at 8K vs 11.2x at 32K). pure quantization would be constant. suggests eviction, structural pruning, or a hybrid approach. what breaks first when you push beyond 32K?

    2 отметки «Нравится»

  • Фото профиля gpu go brr...

    @skynetislov3

    @GoogleDeepMind Pretty legit. But cool kids are processing in the compressed space, too. I could teach you, but i'd have to charge. 🔔🥛

    2 отметки «Нравится»

  • Фото профиля Elon Orbit

    @Elon_Orbit

    @GoogleDeepMind Need the repo. 10x KV compression at zero loss? Big claim. DeepMind didn't ship turboquant overnight. Show the work.

    2 отметки «Нравится»