Skip navigation
Qwen3.8-Flash-Next just got a massive speed boost on my old hardware.
0:00 / 0:50

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware.

FHILY👑

@Oluwaphilemon19.1K followers

Visit on X
117 likesDownloadOriginal post

23,243 views

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.

13 Comments

Comments from the original post on X (Twitter)

  • Peasant Smith's profile picture

    @PeasantSmith

    I tried that fork on my hardware and -unfortunately- it didn't move a needle from what I was getting before. My hardware is probably the limiting factor here. I was already getting 14.1t/s decode on 3x 3060 12gb and ddr4 2400 quad channel. With this fork I got pretty much 14t/s

    3 likes

  • Stanislaw Weston's profile picture

    @StanislawWeston

    This is a very good fork of llama.cpp. It has many useful features for MoE models, especially Qwen 3.8 Flash Next. BUT MTP actually slows things down by 1.5x. So while I use this fork, I keep MTP turned off.

    2 likes

  • Mohammad Zoheb's profile picture

    @techgrapple

    I am using on server, with 158GB RAM + RTX3090 but I only allow 12GB VRAM since I have other apps to use it. Getting 17 to 18 token per second with original llama.cpp. I tested the cafe llama.cpp and made no difference. It's just basic optimizations, It is just another llama.cpp

    1 like

  • Abdarahmane Traore's profile picture

    @abtraore_

    2x from tuning alone is the real story with this model, the kernels are young: rebuilding llama.cpp at the merged PR head nearly doubled decode for us too. Next big one for your card: MTP spec decode, once GGUFs get re-exported with the head (PR #27836). Recipes:

    1 like

  • LocalAIMaxxer // Niko's profile picture

    @RZY333

    Could I do this with 5080 + 96gb ram and ssd? How much sits on the gpu?

    1 like

  • Diego Garcia's profile picture

    @garciarsdiego

    Hey mate Mind to share the recipe you used? Thank you!

    1 like

  • Alois Vaclav's profile picture

    @AVaclav73853

    Stop this bullshit, 30tps cunt be your daily driver...nice experiments, but thats all. Cunt call it "locall AI" when it can be used as Codex / Claude..makes no sense.

  • why's profile picture

    @NullContex1s

    Which part of the stack made the biggest difference: quantization, batching, kernel settings, or CPU offload?

  • LinuxVersion's profile picture

    @linux_version

    any way to quantize for use with 8GB vram?

  • De doncker kenny's profile picture

    @KennyDeDoncker

    weird, i am on 17 tok/s on 8 gb vram now.

  • Zima's profile picture

    @MichaelZima

    If I want to build this system. What’s a brief shopping list?

  • Binsen Huang's profile picture

    @BinsenHuang

    我的用2x5090 32GB,drr5 96g出现了预充填时间过长和解码速度过于缓慢,模型无法正常使用的情况,

  • Idriss's profile picture

    @Edyenayat

    2x on the same 3090 usually means you stopped spilling into system RAM, not that the kernels got faster. Worth reporting tok/s at 8k and 32k context too, since that gap is where the tuning actually shows up.