With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀 At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public. What K3 brings: 🧠 2.8T-parameter... Mixture-of-Experts (16 of 896 experts active per token) 📚 1M-token context window 🛠️ Native multimodal understanding, including vision ⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable Huge thank you to Kimi.ai AI for the model release and partnership, Inferact for leading the vLLM optimizations, and to our partners at NVIDIA, AMD, and the broader vLLM community. 1/6show more

Fitting 2.8T parameters was not the hard part. Caching them was. Most of K3's layers hold fixed-size KDA state instead of a growing KV cache, so there is no per-token KV to hash. Prefix caching was rebuilt around that, and every hybrid model after K3 inherits the result. 2/6

✨ What you can turn on: 🔷P/D disaggregation, TP8 prefill to DP16/EP16 decode over NIXL 🔷MoE backend per topology, mega_moe for EP, trtllm for TP>1 🔷KV offloading, so the agent turns skip re-prefill 🔷Tool calling, reasoning, structured output 🔗 3/6

📈For ultra-low latency on a 2.8T model without accuracy loss, speculative decoding is the natural choice. @inferact trained and open-sourced a DSpark speculator for K3 that drafts multiple tokens in a single parallel pass. Perf improvement: 118 → 370 tok/s single-stream, ~3.14× on real reasoning workload datasets. 🔗 4/6

Kernels, recipes and the full write-up: 🔗 5/6

And celebrating a huge milestone, we have merged Kimi K3 as PR #50000! 🎊🎊

Was great to collaborate!

The best part of this is that whenever vLLM posts numbers, they are real, not inflated, and not cherry-picked. That's actually quite rare in the industry. If vLLM gives us a number, we can trust it's reproducible and real.

Gotta spin up my b300 for this

why would vllm vibe code a video?

Some free K3 credits to test:

Hehe, this must be your happiest model release, cause nobody can actually run it and put in bug reports 😅

Here is the Kimi K3 architecture breakdown such that anyone can digest in the form of an explainer video, one-shotted by Simi by Lamina Labs.

Day-0 support is quietly the best argument for composing engines unforked. NVIDIA's Molt carries zero vLLM patches, so an engine upgrade is a container pin and K3-class models reach RL training the day the weights land.

curious how many nodes it takes to even boot 2.8t

my local power grid is going to hate me the second those weights drop

🔥🔥

1m context is nice, but prefill cost will decide most startup deployments

Moonshot dropping 2.8T open weights with day-0 vLLM support. Closed labs must be thrilled

A 1M token context that's actually affordable is the real win here.