Kydo

Qwen 3.8 27B on hit 3.3x faster decode in 7 days. Here's what happened and what we're thinking next. Result (so far) Median decode speed increased from 26 tok/s to 87.9 tok/s on the verifier M5 Max (33 to 93.1 tok/s across the eight prompts), with prefill around 971.8 tok/s. This came out of a collective effort: 31 solvers across 67 improvements. Most of the recent ones run custom MTP heads that draft and accept ~3.9 tokens per round while still matching serial output exactly. Why this matters Beyond the performance itself, two things stand out to me. (1) Dense models on Apple Silicon were supposed to be the hard case. "Everyone knows Macs are slow at dense models." But watching the community take it from the usual baseline to >3x in seven days shows the low-hanging fruit was still there. (2) Open-weight models have been small and effective for a while. This is the first time one is small and frontier. Qwen 3.8 27B is an extremely strong dense model, comparable in capability to Opus 4.6 (Max). Running it at usable speed (>45 tok/s) is a step change for local AI users. What we improved about the challenge itself This is our second challenge, and we took the feedback from the Laguna track and rebuilt a few core pieces. - Speculative decoding (native MTP) was available and editable on day one instead of bolted on later. - Scoring became the median of eight independent prompt speedups over pure serial decode (anchored at 1.0, floor 0.90, ceiling 3.0), so no single fixture could dominate. - The leaderboard now ranks total contribution rather than just the current record holder. - Every submission gets automated screening for gaming before it scores. I really appreciate folks who's provided feedback. Naming a few that came to mind Ivan Fioravanti TheDavidTai Morgan McGuire poly Takeshi7 Steven Gumbii.Digital Tanishq Dubey Arjun Ram Andrey 🦃 Petrov tiny edge David Zhang Jaime Rader Surf and many others on slack! We also widened the editable surface to include the MTP head weights themselves, the full draft/verify loop, and a large set of the underlying Metal kernels. How we got to the 3x speedup Here's a summary from Grok. Much of it is beyond my understanding, but I expect people (and agents) smarter than I am can take these insights and apply them in other contexts. Custom MTP heads + adaptive draft policy People stopped treating the head as fixed and started training or editing it for higher acceptance under the exact verify constraints. Combined with per-round draft counts that can adapt (0 to 8), this is what pushed average accepted tokens from ~1-2 up to 3.9 on the top runs. Tighter verify-block and KV rollback paths The Swift session code for assembling the verify pass, snapshotting KV, and rolling back on rejects got cleaned up a lot. Small latency wins here compound once you're drafting ~four tokens at a time. Metal kernel work on the hot paths SDPA, the MoE gather GEMM, RoPE, RMSNorm, and a few of the smaller element-wise ops saw targeted edits. Most of the gains only show up once the verify width is high and the memory traffic pattern changes. Fidelity-preserving residual handling Several submissions improved how residuals and acceptance decisions are managed, so that higher draft depth doesn't quietly degrade the token match rate. The gates stayed strict: every emitted token still has to equal serial, so these were real engineering wins rather than score hacks. What's next for Qwen 3.8 27B MLX. We plan to keep the track live a bit longer, then switch to Qwen 3.8's MoE version (rumored to be 35B-A3B). Given the recent DFlash 2 announcement, we're also looking at whether we can support broader speculative methods. The current surface already supports a lot of experimentation. The main gaps are better upstreaming for local usage and clearer docs on how the benchmark and verifier work. Multiplatform. In parallel, we're experimenting with running a similar effort around CUDA for Qwen 3.8 27B. A lot of people have asked for this, since the two communities overlap quite a bit. Our goal is to ship the CUDA version next week. We'd also love to partner with Qwen on it. If anyone has a connection there, please introduce us, and we'll see if they're down to match a bounty with us to push this out. What's most useful for the broader MLX community The improvements from the challenge are already upstreamed inside Darkbloom, and we're seeing ~2x faster decode in our production traffic for Qwen. Outside the challenge itself, something I've been thinking about deeply, and that a few community members have raised, is how to make these results useful to more people. There are many individual efforts happening across the MLX community, and honestly, the more I dig in, the more confused I get by the overlapping libraries and concepts. I'm sure I'm not alone, and newcomers probably feel the same. That's no one's fault, just the growing pains of an open source community. I don't expect I'm gonna come up with the answer, but I'd love to learn more about what different folks are working on and how they're thinking about their roadmaps. I'll share what I learn along the way, and hopefully someone smarter than me can turn it into a proposal for us to rally around.