pip install spectralquant ✂️
ani
Visit on X@anirudhbv_ce7.6K followers
17,980 views
pip install spectralquant ✂️
Up to 6.62x KV cache compression for LLMs and transformers.
Same model. Faster outputs. Smaller KV cache.
Try now (2 mins):
- KV cache integration via Hugging Face's DynamicCache
- Three presets: 5.95x (paper), 6.55x (validated), 6.68x (edge)
- Mistral 7B / Qwen 2.5 7B / Llama 3.1 8B verified
- Pure PyTorch + future CUDA kernel support
- Auto-calibration from a bundled corpus
📰 Paper:
💻 Code, quickstart, and benchmarks:
#LLM #Inference #PyTorch #OpenSource #MachineLearning #LLM #KVCache #Inference
Comments
Comments from the original post on X (Twitter)
No comments yet
