Reasoning LLMs generate very long chains-of-thought, so even small quantization errors add up. With AWQ, Qwen3-4B drops 71.0 → 68.2 on MMLU-Pro (~4% relative loss). 😬 ParoQuant fixes this! It keeps only the critical rotation pairs and fuses everything into a single kernel. Recovers most of the lost reasoning... accuracy with minimal overhead — so 4-bit models stay strong at reasoning. 💪💪show more

Accepted to ICLR 2026 📜 🔗 Blog: 📄 Paper: Great work led by my student @yesheng_liang, in collaboration with @HsChen335, @_SubSir, and @songhan_mit.

would love this tested on larger models, like 30B or above, since at that point quantization has a much smaller effect on intelligence

Testing on larger MoEs is on our roadmap!

We spent years shrinking models just to realize their internal monologues are more expensive than their final answers. Making a tiny model talk to itself for ten thousand tokens is the ultimate engineering paradox.

I'm all for better quantization methods, but it's well known that small models like Qwen3-4B are inherently susceptible to quality loss due to quantization...

Yes, but those models are really useful for local deployment, so their accuracy is also quite critical!

Compounding quantization error in long CoT has been the primary blocker for edge reasoning. Fusing critical rotation pairs into a single kernel to salvage 4-bit accuracy is a massive unlock.

Tracks with my experience: long reasoning amplifies tiny quant errors. AWQ helps, but I’d also watch long-context + KV-cache evals, not just MMLU.

Test our repo out! 😉

This looks soooo cool to me !

The real issue is that most quant benchmarks don't test on long CoT paths - they use short evals. When you run 10k+ token reasoning chains, error drift becomes the actual bottleneck, not compression ratio.

Congrats on the acceptance. this is great for local setups. expecting the FOSS community will engage with native GGUF export or direct llama.cpp support. (override-k-quants have limited accuracy gain).

W4A4 not converging is interesting. The Givens rotation may be at odds with per decoder optimization, and you may be on the cusp of an invariant we’ve yet to formalize

Congrats! Very impressive results 🎉🎉

Thanks, Megan!

That's really impressive progress on ParoQuant! It sounds like a huge leap forward in efficiency for those LLMs

very exciting Zhijian!!

This is a big step — quantization usually cripples reasoning models, but ParoQuant shows you can keep efficiency and accuracy with the right kernel design.

the repo is 4months old has it not gained any traction?

We've just made it public yesterday.

It‘s an interesting study! My m1 max will run well with MacBook, right? 😁

Hot take quantization quality is becoming product strategy not just infra detail. If ParoQuant holds under long horizon reasoning, it could reset what people expect from small local models.

Awesome work 😎👍

Dayum, so cool!

@zhuoyang_zhang 🫡

why doe the ParaQuant looks so much faster than tha AWQ?

We found that when quantization is more lossy, the model sometimes generate more tokens.

But I mean the speed

Token/sec is similar. Just more tokens for this example.

Dumb question, but is the comparitor in this image just for presentation or is this a public CLI that is available to test. I’d love to try a few prompts and see how ParoQuant compares to some of my other local models.

How’s memory usage for ParoQuant ? Would like to run this on mobile. @grok what’s the memory footprint?

How can i use paroquant models?

good

How does this method compare in precision and reasoning quality to QTIP used by exllamav3?

what did you build that UI with?