From the Hamster Labs lab: Introducing Qwen3.8-Flash-Next-REAP-288-MLX-4bit, a 180B-class model running on just 39gb of memory - MLX-native 4-bit 60% smaller than stock q4 - Pruned 512→288 experts via REAP - 91.5% HumanEval (vs 93.9% stock) Hugging Face links below ↓

The folks at @Alibaba_Qwen recently released Qwen3.8-Flash-Next, a 180B-class sparse MoE with 48 layers, 512 routed experts each and top-10 routing (for 6B active params). Like most MoE models there's a distribution for how much work each expert tends to do overall. I charted and scored all 24,576 experts on real Claude Code sessions to find where the routing mass lives. Each row is one MoE layer and experts are sorted by saliency within their layer, brightest first. As expected, a thin band of experts does most of the work in every layer, and the long dark tail is what structural pruning removes. The goal was to remove as many experts as makes sense based on which stay mostly silent when using the model to generate code. Now that I knew which experts I could cut, the question then became: how many could I get rid of before accuracy took a real hit? My goal was to remain 90%+ accurate.

I measured an eleven-point pruning ladder and 288 is where quality-per-GB peaks among usable builds. Aggregate metrics (KL divergence against the stock model) suggest pruning deeper, but sampled rare-token reliability collapses below 288 At 256 experts, the model produces an intact rare name only in 1 of 10 seeded generations whereas the 288 expert build (this release) scores 9 of 10. At 288 experts, the model collapses from 97gb down to 68gb, but because the 25gb 51b n-gram table can be streamed from your hard drive, that 68gb goes down to 39gb, which opens an entirely new class of hardware that can run this incredible model.

Huggingface Links Model MTP drafter

+ 2 variants: Qwen3.8-Flash-Next-REAP-288-MLX-8bit - 8 bit, 90.9% HumanEval, 70 GB resident - q8 27% smaller than the q4 stock model Qwen3.8-Flash-Next-REAP-384-MLX-8bit - 4 bit with 384 experts, 92.1% HumanEval, 51 GB resident

This model could go fast. REALLY FAST. MLX inference today (mlx_lm, mlx_vlm and other friends) is constrained by components it depends on. That's why I'm writing a pure MLX inference engine - to get the most out of our machines. It opens up techniques and possibilities (i.e. kernel fusion) that would otherwise be impossible to develop. So far, two big wins in sight with the new engine: - Reached 65 tok/s with zero accuracy loss, aiming for 100+ - Expert paging brings the model down from 39gb to 20gb! We're exploring much of this at @usehamster and will house most of this research and future open source releases from the @hamsterresearch account, so make sure to follow it. Send this to a RAM-constrained friend. Help is coming.

@HamsterResearch @huggingface Thanks for the reap_kept_experts.json. Working on adding SSD expert streaming to oMLX and was wanting a useful base for expert pinning, and this is it.

@HamsterResearch @huggingface Have the 25gb resident working, will be an end to end example for ya

@HamsterResearch @huggingface Wow! Almost there. Let's see if I can run it on my 48GB Mac...

@HamsterResearch @huggingface stay tuned, i've got a goodie coming just for your setup :)

@_ARahim_ @HamsterResearch @huggingface wen pmlx, n-gram table in RAM

@_ARahim_ @HamsterResearch @huggingface it’s working, but needs some polish days not weeks

@HamsterResearch @huggingface Such a huge progress shrinking those weights

@HamsterResearch @huggingface and we're not done!

@HamsterResearch @huggingface Great work! Will you upload bf16 version of it by any chancd? I'd love to quantize it for my Strix Halo (using rocmfpx).

@HamsterResearch @huggingface I surely will, they should be up on hf sometime tonight

@HamsterResearch @huggingface Awesomre, TYVM 🙏!

@HamsterResearch @huggingface here you go

@HamsterResearch @huggingface oooh THANK YOU! It seems @rc_morano beat me at it and already did some ROCm quantizations, thank you too!

@HamsterResearch @huggingface @ItsmeAjayKV something like this might be nice on a 3090 + RAM + NVme

@HamsterResearch @huggingface Will you share your REAP recipe, and maybe you can share the diffusers version of the REAP checkpoint in native precision?

yes on both. the recipe (saliency calibration, kept-expert manifest, prune) is getting a proper writeup + the tooling released once the final version's out. the per-layer kept manifest already ships in the repo so the prune is reproducible today from the stock conversion. here btw REAP might not even be needed anymore based on my latest build where I'm expert paging to stream a % of experts from the disk and off residency (similar technique as n-gram streaming) residency is ratio-driven, so full-vs-pruned barely differ in ram and at equal memory the full model wins both quality and speed (26gb resident @ 19 tps stock vs 23gb resident @ 15 tps and 91.5% accuracy) So I likely won't do another pruned tiered release and my next release will likely subsume it. REAP still helps with disk footprint but that's hardly a benefit relative to the residency + decode speed with full experts re native-precision: doable, i can cut a bf16 REAP-288 for you, adding to my list

@HamsterResearch @huggingface Niiiiice, waiting for your updates on the matter of expert streaming. However, I want to try extra direction with this: to try to retain as much knowledge as possible by utilizing the REAM merge of the least experts.

@HamsterResearch @huggingface I'll definitely take a look at this good sir thanks!

followed through on the REAM idea: same 288 slots as your REAP manifest, but the 224 dropped experts get merged into their nearest kept neighbor (router-row cosine, saliency-proportional weights via a rank proxy since the npz isn't public). teacher-forced KL vs stock over a mixed corpus: merge 1.2752 vs prune 1.3066 (ru −6.6%, p95 −5.7%) — the win concentrates exactly where your card predicted degradation (non-code domains). drop-in replacements, same runtime/layout as yours (byte-identical backbone): (bf16 + campaign recipes) if you will publish saliency_full_505.npz I'll re-merge with true weights

@HamsterResearch @huggingface Still don't think there's much point in it since pruning is provably worse than the new engine's approach but I added the npz for you

@HamsterResearch @huggingface would try to prune only 25%, and quant the remaining 25% to q2, the next 25% to q3 and keep the important 25% q4 or so, should give a better result. You could also look into the experts themself and quant not often used neurons. Neuron aware quantisation

@HamsterResearch @huggingface i tried that - any q2 components I brought in created garbage not impossible though

@HamsterResearch @huggingface Thx bro 😁🍻

@HamsterResearch @huggingface FreeToken already achieved what you're trying to do

@HamsterResearch @huggingface what do you think that is?

I wanted to benchmark this but didn’t have 68GB available. @EyalToledano @HamsterResearch got a 180B-class @Alibaba_Qwen model to 39GB resident with a 2.4-point HumanEval hit. the 39GB claim depends on an unreleased NVMe offload patch. Stock MLX needs about 68GB resident, so it won’t run on my @Apple 64GB Mac @huggingface @Blaizzy @jundotkim.

@HamsterResearch @huggingface Why LM Studio is showing the size as 73.52GB?

@HamsterResearch @huggingface They probably have not yet added support for n-gram disk-load yet Support for that is pretty sparse right now and I've got a PR open with mlx-vlm to support it. some context here

@aitechgyan @HamsterResearch @huggingface Offloading works in oMLX

@aitechgyan @HamsterResearch @huggingface Nice! So it's good to go

@HamsterResearch @huggingface @Youssofal_ no idea what that means but we want that in mtplx plz ty

@HamsterResearch @huggingface @Youssofal_ it means qwen3.8 flash next which usually takes up 97gb of vram now can run on 39gb of memory instead with 91% accuracy compared to the model that takes up 97gb so you can run this beast of a model on a way larger variety of hardware like a 48gb or 64gb macbook

@HamsterResearch @huggingface @Youssofal_ sorry for the retarded questions but how does it compare with qwen 3.8 27b at that quantization? And how's the TPS? i got a m5 pro 64gb

legitimate question and easy to get confused so i'll answer in detail a dense model like qwen3.8-27b reads all 27b weights for every single token an MoE like flash-next has 125b total weights but only activates ~6b per token. so it decodes way faster than a dense 125b would, while being a much stronger model than a dense 27b. total size still matters though: the weights all have to live somewhere decode speed is basically bytes read per token vs your bandwidth. m5 pro bandwidth is 307 gb/s qwen3.8-27b at 4-bit: - only about 18gb of memory to begin with, amazing model - reads ~16gb per token so the bare decode ceiling is about 19 tok/s, prob ~15 (without MTP) - with a good MTP drafter maybe 35-50 tok/s flash-next REAP-288 at 4-bit: - reads only ~2.3gb per token (7x less), so bandwidth isn't the limit. on your chip i'd estimate ~15-20 tok/s bare, ~25-40 with the MTP drafter (which we ship) the model already runs on mlx-vlm, ollama and others today, but at 68gb resident, which your 64gb can't hold. I'll open a PR adding n-gram disk offload so it runs at 39gb out of the box. my guess is that patch/fix will make it across the different runtimes fairly quickly too when the mlx-vlm team merge my patch, it'll allow it to run the model at 39gb of memory (at ~35-50 tok/s) SORRY FOR WALL OF TEXT

@HamsterResearch @huggingface @Youssofal_ That PR's up now - @mlxvlmreleases it's in your hands now

@HamsterResearch @huggingface Congrats. Qwen seems to be killing it with releases recently!

@HamsterResearch @huggingface This is great! OMG testing right now!

@HamsterResearch @huggingface I was like "4 bit 39G" must be reap, yup it's reap

@HamsterResearch @huggingface Looks good! I'm on 3.8 27B now - not enough ram to run smth like this :D

@HamsterResearch @huggingface Soooooooon! I've got it down to 20gb off the new engine ;D

@antonlvovych @HamsterResearch @huggingface Can you give the exact details to get it down to 20Gb?

The general idea is tiered expert streaming from the nvme. A ledger can exist to assign the ideal expert to each layer and know to only activate those, so the rest stay on disk and resident ram goes way down I’ve got a working one at 91% accuracy and 25gb residence with a custom engine. I’ll release both the engine and the model at the same time

@antonlvovych @HamsterResearch @huggingface So essentially this, but running it on MLX framework instead of llama.cpp?

@antonlvovych @HamsterResearch @huggingface The mmap trick is a separate but also crucial way to get PLE tables off resident memory But the general idea of it is applied to the active experts Same same but different

@antonlvovych @HamsterResearch @huggingface Just to be clear the inference engine you’re working on would be doing both ngram streaming from NVME and non-active model weight streaming for MoE? If so you’re cooking HOT

@antonlvovych @HamsterResearch @huggingface Yep, AND tiered expert streaming

@HamsterResearch @huggingface Thanks. This is really interesting! Do you have benchmarks on this compared to the stock?

@HamsterResearch @huggingface yes, tok/s is unchanged and accuracy moved from 93.9% (for non-pruned) to 91% (pruned) on HumanEval that's just one benchmark though - still experimental but seems to retain remarkably well overall across context sizes

@HamsterResearch @huggingface I’m testing your Qwen3.8-Flash-Next-REAP-288-MLX-4bit build on a 96GB M3 Ultra. The ~39GB resident mode is the part I’m most interested in. Would you be able to share the row-granular NVMe n-gram offload patch you used for mlx-vlm, or point me to a branch/commit where it lives?

@HamsterResearch @huggingface fwiw I opened a PR for mlx-vlm and it was closed since the fix is apparently already part of a pending release so the 39gb residency is possible out of the box as soon as mlx-vlm upgrades and you update

@HamsterResearch @huggingface How does its perf compare to its smaller sibling, the dense 27b?

@HamsterResearch @huggingface

@HamsterResearch @huggingface 代价是损失百分之二？

@HamsterResearch @huggingface yep!

@HamsterResearch @huggingface You have a BF16 of this? Or fp8?

@HamsterResearch @huggingface Not yet - it's on the list

@HamsterResearch @huggingface Nice trade, though it is worth naming that this is lossy shrinkage, not the same game as exact storage. Our FCN work keeps weights bit exact and the ordinary single model wall sits near 1.52x, so the interesting question is which losses you can actually live with.

@HamsterResearch @huggingface What context window size can this handle?

Entirely dependent on the leftover memory after this is loaded in. Also depends on whether the runtime you use (ollama, mlx-vlm, etc) supports n-gram disk offload. Support for that is pretty sparse right now and I've got a PR open with mlx-vlm to support it.

@HamsterResearch @huggingface Got it. I’ll give it a go on my local benchmark (titanic simulator) M5Max 128GB this weekend.

@HamsterResearch @huggingface For science!

@HamsterResearch @huggingface

@HamsterResearch @huggingface 91.5% score at 60% smaller, solid tradeoff

@HamsterResearch @huggingface

@HamsterResearch @huggingface The tradeoff between pruning and benchmark dips makes me wonder if agentic self-correction could boost those scores later?

@HamsterResearch @huggingface Is it possible to run it with 3090+ 3080 and some RAM.🙏

@HamsterResearch @huggingface that'll be something! 24gb is possible but you'd have little to no space for context so i'm not sure it would be usable

@HamsterResearch @huggingface So not possible with 2 gpu + ram?

@HamsterResearch @huggingface The 2.4pt HumanEval drop is the cheap part. Pruning 512 experts to 288 changes which experts get routed on long-tail prompts, so the interesting eval is per-domain variance, not the aggregate. On unified memory the 39GB fit means you're bandwidth-bound at decode, not TOPS-bound.