Qwen3.8-Flash-Next just got a massive speed boost on my old hardware.
@Oluwaphilemon19.1K followers
23,243 views
13 Comments
Comments from the original post on X (Twitter)

@PeasantSmith
I tried that fork on my hardware and -unfortunately- it didn't move a needle from what I was getting before. My hardware is probably the limiting factor here. I was already getting 14.1t/s decode on 3x 3060 12gb and ddr4 2400 quad channel. With this fork I got pretty much 14t/s
3 likes

@StanislawWeston
This is a very good fork of llama.cpp. It has many useful features for MoE models, especially Qwen 3.8 Flash Next. BUT MTP actually slows things down by 1.5x. So while I use this fork, I keep MTP turned off.
2 likes

@techgrapple
I am using on server, with 158GB RAM + RTX3090 but I only allow 12GB VRAM since I have other apps to use it. Getting 17 to 18 token per second with original llama.cpp. I tested the cafe llama.cpp and made no difference. It's just basic optimizations, It is just another llama.cpp
1 like

@abtraore_
2x from tuning alone is the real story with this model, the kernels are young: rebuilding llama.cpp at the merged PR head nearly doubled decode for us too. Next big one for your card: MTP spec decode, once GGUFs get re-exported with the head (PR #27836). Recipes:
1 like

@RZY333
Could I do this with 5080 + 96gb ram and ssd? How much sits on the gpu?
1 like

@garciarsdiego
Hey mate Mind to share the recipe you used? Thank you!
1 like

@AVaclav73853
Stop this bullshit, 30tps cunt be your daily driver...nice experiments, but thats all. Cunt call it "locall AI" when it can be used as Codex / Claude..makes no sense.

@NullContex1s
Which part of the stack made the biggest difference: quantization, batching, kernel settings, or CPU offload?

@linux_version
any way to quantize for use with 8GB vram?

@KennyDeDoncker
weird, i am on 17 tok/s on 8 gb vram now.

@MichaelZima
If I want to build this system. What’s a brief shopping list?

@BinsenHuang
我的用2x5090 32GB,drr5 96g出现了预充填时间过长和解码速度过于缓慢,模型无法正常使用的情况,

@Edyenayat
2x on the same 3090 usually means you stopped spilling into system RAM, not that the kernels got faster. Worth reporting tok/s at 8k and 32k context too, since that gap is where the tuning actually shows up.
