Introducing JetSpec++: we find causal parallel drafting paired with hidden states predictor and anchor token decoder can effectively draft the next future block before the current one is verified, and spending the freed time on self refinement. This allows drafting to be pipelined with verification, hiding almost all of... its drafting cost. JetSpec++ reaches up to 1.54x end-to-end speedup over DSpark and 7.45x over autoregressive decoding while keeping lossless, at 1,361 TPS on Qwen3-8B and 367 TPS on Qwen3.8-27B in BF16. With NVFP4 and kernel optimizations, JetSpec++ further reaches over 4,000 TPS on Qwen3-8B and over 900 TPS on Qwen3.8-27B, on two GB200 GPUs. ⚡️ Project page:show more

[2/5] With JetSpec++, drafting time is almost removed from the critical path. Walltime reported below shows how latency reduces for each draft-verification round, and its breakdown on GSM8K: • Qwen3-8B: DSpark 9.92 ms (including 3.76 ms drafting time), JetSpec++ 6.84 ms (including 0.61 ms is drafting time) • Qwen3.8-27B: DSpark 31.88 ms (4.46 ms drafting time), JetSpec++ 28.47 ms (0.71 ms drafting time) The key is JetSpec++ has effective foresight drafting: it guesses the verification outcome correctly in 92.9% (8B) and 89.0% (27B) of rounds, so the next block is already drafted when verification ends.

[3/5] The rest of spare time waiting for verification outcome now goes into refining a better draft. While the target verifies, JetSpec++ corrects its own parallel draft: it redrafts on its own guesses, refreshes its predicted context, and fixes the first token where a decoder built from the target's last layers disagrees. 8.94 accepted tokens per round across 7 benchmarks, vs 6.89 for DSpark.

[4/5] Bigger targets leave more room. On Qwen3.8-27B, hiding the draft saves less, because verification dominates the round. But the same long window fits three refinement passes instead of one, so the gain moves to acceptance: +22% accepted tokens per round over DSpark on 27B, vs +6% on 8B (GSM8K, as served). It also extrapolates. Trained on blocks of 16, JetSpec++ drafts blocks of 32 and gains: 14.55 tokens per round on MATH-500 on 8B and 14.05 on 27B, while DSpark drops to 8.66 and 8.49. We would love to hear your feedback!

[5/5] JetSpec++ models, code and inference engine are fully open-sourced. Please give it a try and let us know what you think in the comments! Check out our paper and the link below for more details! Paper: Code: HF:

Once drafting overlaps verification, draft cost nearly disappears and acceptance becomes the whole game: a block drafted ahead is wasted whenever the current one gets rejected. A speedup breakdown by acceptance length would show how much of the 1.54x survives on harder prompts.

Comparison to dflash2 please

spec decoding finally stops billing you

Up to 7.45x faster with the same output distribution is the gain I actually want. For now it's a datacenter gain though, the README asks for two GPUs and only lists A100, B200 and GB200

NVFP4 plus the kernel work adds roughly another 2.9x on the 8B and 2.5x on the 27B. When the verifier runs in NVFP4, is lossless measured against the BF16 model or the quantized one? That changes what the quality claim means.

@ashxhart

I wish we get something similar for Mac devices.