I implemented Google Research's TurboQuant as a CUDA-native compression engine on Blackwell B200. 5x KV cache compression on Qwen 2.5-1.5B, near-loseless attention scores, generating live from compressed memory. 5 custom cuTile CUDA kernels ft: - fused attention (with QJL corrections) - online softmax -on-chip cache decompression - pipelined TMA... loads Try it out: s/o Bryce, the CUDA Colonel and the cuTile team at NVIDIA for lending me Blackwell GPU access :) cc sunny madra Gavinshow more

The numbers: - 5.02x compression (3 bits/coordinate) ~0.985 cosine similarity vs FP16 attention (layers 7–27) - 144.7 tok/s on B200 - Fused attention kernel beats FP16 matmul at 4K+ sequence length Random rotation spreads outliers into a uniform bell curve → Lloyd-Max quantization bins where data is dense → QJL corrects the bias with 1-bit signs so attention scores come out lossless. The value cache gets 3-bit MSE, keys get 2-bit MSE + 1-bit QJL correction.

Started here, last week. Ended with ~2,000 lines of cuTile. Started with pseudocode and basic kernels. Profiled and iterated to optimize them. Shaved 6% compute time with fused V cache decompression and GPU memory swizzling.

@GoogleResearch I’m so happy I’m on this side of twitter

@GoogleResearch :)

@GoogleResearch did u start this from just writing it out by hand? holy

@GoogleResearch was j putting thoughts done on how a basic version should work lol

5x KV cache compression with near-lossless attention scores is the kind of result that directly translates to longer context windows at the same memory budget. The B200's TMA hardware making this practical on-chip is what separates it from software-only approaches that eat into throughput.

@GoogleResearch Couldn't have said it better. The newer chips can handle larger throughput- that's how they're designed. It's just a matter of recognizing it and adjusting your kernels accordingly.

@GoogleResearch Very cool

@GoogleResearch Thanks for checking my project out Sriram!

@GoogleResearch I did that

@GoogleResearch Cool stuff!!!

@GoogleResearch I did the same implementation on one of my projects, TQ makes it scalable for edge device LLMs

@humblefool @GoogleResearch Cool project!!!

@GoogleResearch Thanks

@GoogleResearch Cracked

@GoogleResearch Great work! Sent you a dm

@GoogleResearch 👋😁

@GoogleResearch This is impressive bro, I had to feed the papers into Notebook LM to understand what was going on. I'm relatively new into AI research and Model engineering, and seeing the speed at which you implemented this is mind blowing and a huge inspiration. Let's connect 🖖💪

@N_Ehxzel7 @GoogleResearch Yeah the paper was a lot to take in. I spent an entire day breaking down the math, tryna put the ideas together. Once things clicked, I put my ideas down on paper and began the kernels :)

@GoogleResearch Now that's some Tony Stark type shii Bro 🤣🤣🤣 Impressive.

@GoogleResearch My implementation on an A10 got me till here (I did 4 bit tho)

@GoogleResearch That's sick wtf Send your implementation!

@GoogleResearch This is sick

@GoogleResearch 🙏 following in your footsteps

@GoogleResearch Can you 10x compression ?

@GoogleResearch Worth taking a look 👀

@GoogleResearch I didn’t get it to compile to 4 bit - Claude got stuck in a loop - but 3bit results looked stellar - johndpope/llama-cpp-turboquant branch feature/planarquant-kv-cache, commit 20efe75 -

Insane work!!!

@GoogleResearch Anirudh, how many offers by how to join AI labs and start ups? :-) I’d wager >10!!

@GoogleResearch LOL 🙏

@GoogleResearch I implemented this from scratch on H100. Got 44-59% KV-cache reduction with exact prefill fidelity across 5 model families. Full benchmarks in my thread i found something that no one noticed

@GoogleResearch Thank for implementation! It helped us as we figured out how to beat @googleresearch’s provably near-optimal 5.02× compression ratio :-)

@GoogleResearch 🙏 this is so cool!!!

@GoogleResearch yo are you at shopify right now ?

@GoogleResearch Bro we literally went to the Sidekick team burst dinner together last month LOL Tall guy, intern who was from Turner lmao

@GoogleResearch yes i just remembered. check slack

@GoogleResearch 🦫bv ce

@GoogleResearch 🫡 you alr know

@GoogleResearch impressive work ! but i noticed in your video that the cosine similarity is not 0.9 across the board as declared, seems to be very low on some layers and some weights especially the first layers where it dips to 0.29 the max absolute weight error of 1 is also concerning

@GoogleResearch Absolute 🔥 Anirudh Combining this with Dynamo KVBM and 1000x memory capacity will unlock enormous agent swarm scale and efficiency 💪

@GoogleResearch 🙏 he knows

5x KV cache compression with near-lossless attention scores means you're fitting roughly 5x the context into the same GPU memory without degrading output quality. for anyone not tracking this, the practical implication is that models that currently need 8 GPUs to serve could potentially run on 2. the cost reduction isn't incremental, it's structural. this is the kind of work that changes who can afford to run large models

@GoogleResearch gonna try this, seems awesome

@GoogleResearch It is! I'm looking for feedback and ways to improve. Pls try and lmk 👍

@GoogleResearch this is so fire

@GoogleResearch 🙏

@GoogleResearch so sick!! you're killing it man

@GoogleResearch Yoo solid work. How does the chip cache decompression work?

Great question :) Instead of decompressing in a separate kernel, we do it inside the attention loop, right before they hit the Tensor Cores. This pushes our kernel more into compute-bound instead of memory-bound. A couple of things: 1.) I load the Pi (rotation matrix) beforehand. It's constant across all streamed KV blocks. 2.) For every value block, I use the indices and norms for Llyod-Max centroid lookup. 3.) Then, I unrotate k_mse (centroid-based coords) and multiply with the norm to turn it back into the full 128 dimension vector (not normalized anymore). Because we reconstructed V, I just accumulate it to finish our attention score. FP16 decompressed values never touch HBM.

@GoogleResearch @MeetSiddhapura

@GoogleResearch super cool anirudh

@GoogleResearch Thanks big bro 🙏

@GoogleResearch insanely cool work

@GoogleResearch Thanks! 🙏

A Waterloo undergrad just built the first working implementation of Google's most disruptive AI paper. In one week. Memory stocks crashed. Investors got it exactly backwards. TurboQuant compresses the KV cache — the part of an AI model that eats your GPU memory during long conversations — from 16 bits down to 3 bits per value. 5x compression. Near-zero quality loss. No retraining needed. Drop it into any existing model and it just works. Micron, SK Hynix, Samsung, SanDisk all dumped 3-6% in a single day. Wall Street's logic: if AI needs less memory, sell memory stocks. That logic has been wrong every single time in the history of computing. This is Jevons paradox — the 160-year-old principle that says when you make a resource more efficient, you don't use less of it. You use more. Coal engines got efficient, coal consumption exploded. Bandwidth got cheaper, data consumption went vertical. Storage got dense, we filled it with 4K video. Every time. Here's what TurboQuant actually does to demand. A 70B parameter model that used to need a $10,000 workstation to run at long context? Now runs on a $1,500 consumer GPU. A model that choked at 32K tokens? Now handles 100K+. A smartphone that could barely run a 3B toy model? Now runs 7-13B with thousands of tokens of usable context. You didn't reduce demand. You just made AI accessible to a billion more devices. Every one of those devices now needs memory. Every user who couldn't run local AI before now can — and will. And every developer who hits the new ceiling will immediately ask for more. Anirudh BV — a student at UWaterloo — took Google's paper, wrote 2,000 lines of CUDA, and had it running on Nvidia's Blackwell B200 within a week. From pseudocode to live generation from compressed memory. If a college kid can ship this before most corporate labs, the adoption curve isn't years out. It's already here. The investors who sold memory stocks on this news will be buying them back within two quarters. Efficiency doesn't kill demand. It creates the demand that didn't exist yet.

@GoogleResearch @nvidia

@GoogleResearch @nvidia Appreciate the lob parsaxa

@GoogleResearch Fire asf g🔥

@GoogleResearch thanks big bro :)

@GoogleResearch Implementing a cutting-edge paper like TurboQuant end-to-end on new hardware (Blackwell) is the real test of a system's potential. Moving from theory to live generation is impressive.

@GoogleResearch I appreciate it! Had lots of fun picking this one out

@GoogleResearch 😼🤘

@GoogleResearch 🙏 the one and only

@GoogleResearch solved the turboquant problem

@GoogleResearch Had to solve it bc @nengjiali already solved the drift problem and you solved the seo problem. Seo and drift are old news.