🚨 2.8 trillion parameters. Under 4GB VRAM. No quantization. An open source tool called AirLLM has changed how large language models run on consumer hardware. Instead of loading the entire model at once, it processes one layer at a time. This means your VRAM needs depend only on the... size of a single layer rather than the full model. No quantization. No distillation. No pruning. → DeepSeek V3 (671B) runs on just 12GB → Llama 3.1 405B runs on 8GB → Kimi K3 (2.8 trillion parameters) runs on under 4GB → Works with nearly every major open source model The largest model often needs the least VRAM. Kimi K3 is a sparse mixture of experts model, so it only loads the specific experts activated by each token instead of an entire dense layer. This is why a 2.8 trillion parameter model can run in less memory than a 70B model. AirLLM makes frontier scale AI practical on everyday hardware without any of the usual tradeoffs. (Link in the comments)show more

Repo:

great share

@HealthRanger

This is a very interesting development in the AI space.

Fantastic share

That's great

That's Excellent

Been looking for something that lets me run bigger models on my 16GB card without heavy quantization. Might test this.

Awesome share

Amazing Share

That's an excellent share.

I hadn’t thought about it this way before.

This is amazing share

The MoE part is the interesting bit. Most of the model never even touches VRAM. Still skeptical until I see real token speeds though.

MoE is the VRAM cheat code. Speed is the reality check: measured ~292 seconds per token on Kimi K3. Disk bandwidth is the new bottleneck.

The tradeoff it skips is speed, since every token streams the weights off disk while the GPU mostly waits. One AirLLM test on a 70B model clocked 292 s/token, which turns a 100-token answer into an 8-hour wait.

Thanks for noting the speed tradeoff. Sequential disk streaming keeps VRAM low but creates high latency, as seen in the 292 s/token result, making AirLLM suitable mainly for batch rather than interactive workloads.