While exploring diffusion language models (DLMs), we kept asking ourselves: given speculative decoding, a lossless acceleration technique, why do we still need DLMs? 🤔 Our latest work, ORL, tries to answer this question: AR optimizes intelligence 👆, speculative decoding optimizes speed 👉, and DLMs optimize the Pareto frontier of... speed and intelligence 🚀. Through co-design across the algorithms, the training and inference engine, and the kernels, ORL pushes out a new Pareto frontier of intelligence and speed. Similar performance, much faster than speculative decoding. You say lossy spec? ORL-DLM is better too! 😎 🌐 Project + live demos:show more

Three highlights for DLM communities 😄: 1. ORL points out that on-policy residual learning is the key to pushing a DLM's tokens per forward (TPF) to the limit. We release an RL training framework that implements exact on-policy residual capture, directly overcoming the state-staleness problem common to residual DLMs and recent popular loop transformers. Check out the GitHub: 2. ORL introduces an inference engine for DLMs. It is customized for low-latency, single-request DLM workloads. Check out the Github: 3. Block attention in DLMs has never matched FlashAttention-like speed in AR, so we redesigned the kernel for block-causal attention. Check out how we design the kernels: @Yifan_YYU, @jianyuqing1, @jisenli_ai , @yuezhouhu

ORL enables more parallelism at comparable quality: up to 6.2× TPF on QwenDLM (accuracy up on all four benchmarks), up to 1.5× TPF on LLaDA-2.1-mini, and up to 1.2× TPF on DiffusionGemma. With our engine and NVFP4, a single request decodes at 5,046 tok/s on one B200, 7,708 tok/s on 8× B200, and 2,788 tok/s on a consumer RTX 5090.

Insight 1: residual learning is inherently on-policy. A model's residuals decide which tokens get committed, and so which states it sees next. Offline residuals: TPF barely moves. Stale trajectories: accuracy drops. Fresh on-policy rollouts: both improve.

Insight 2: Lower cross-entropy loss ≠ refinement speed. Under standard training, entropy stays flat across denoising steps. ORL adds adjacent entropy contraction: each step must cut the uncertainty the previous one left. A block finishes in 22 steps instead of 41.

Insight 3: exact on-policy residual learning is a new systems workload. Considering per-step states, irregular trajectories, and heavy memory, we co-design async capture, memory-adaptive transport, packed compute-balanced training, and a staircase attention kernel: up to 5.2× faster training.

Result 1: faster than speculative decoding, lossless or lossy. Single request on one H100, avg GSM8K/HumanEval: AR (Qwen3-4B): 240 tok/s + EAGLE3: 353 strict / 405 relaxed + DFlash: 573 strict / 746 relaxed QwenDLM + ORL: 1,298 tok/s That's 1.74× faster than DFlash at matched accuracy. Even the lossiest DFlash setting we measured tops out at 827 tok/s, at 80% accuracy. Same on Gemma-4-26B: DiffusionGemma + ORL runs at ~1,400 tok/s, vs at most 663 tok/s for any Gemma speculative setting.

Result 2: ORL moves the whole quality–throughput frontier, not just one operating point. Sweeping the decoding threshold, ORL's curve sits above and to the right of the base model's: 5.6× higher throughput on QwenDLM and 2.95× on LLaDA-2.1-mini at comparable accuracy. The gain grows with block size: on HumanEval, ORL lifts LLaDA-2.1-mini's TPF from 11.4 to 15.0 at block size 32, and from 9.2 to 19.2 at block size 128, with accuracy up at every size.

Result 3: little data, little compute. ORL learns from the model's own rollouts. On LLaDA-2.1-mini, it needs only 4.2M self-generated tokens and 36 H100-hours of post-training, about 10× less data than the ~40M tokens used by d3LLM-LLaDA and dParallel.

Result 4: an engine built for a single request. Serving engines are built around batching. At batch size 1, a DLM's forward pass is short, so kernel launches and scheduling eat a large share of it. Our engine runs each block's denoising loop on the GPU as one CUDA graph: 1.22–1.56× faster than SGLang and vLLM on the same checkpoints, without changing the decoding rule (4,816 of 4,820 QwenDLM answers are identical to SGLang's).

Result 5: fast on real hardware, even consumer GPUs. ORL cuts the number of forward passes, and NVFP4 makes each pass cheaper, so the two compose. On one B200, NVFP4 is 2.3–2.9× faster than BF16 on one H100, with small accuracy changes. DiffusionGemma + ORL in NVFP4 is also 1.14–1.30× faster than NVIDIA's own NVFP4 DiffusionGemma at comparable accuracy. And the NVFP4 ORL models fit on a single 32 GB RTX 5090, matching or beating BF16 on an H100.

Code, models, and the engine are open. 🌐 Project + live demos: 📄 Paper: 💻 ⚡ Joint work with an amazing team: @Yifan_YYU , @jianyuqing1, @jisenli_ai , @yuezhouhu , Tianyi Zhang, @XiaoxiaWShirley , @RidgerZhu , @_junxiong_wang , @ZhongzhuZhou , Fan Lai, and @james_y_zou , across @togethercompute, UIUC, UC Berkeley, UC Santa Cruz, Stanford, and UT Austin.

Chenfeng !

Amazing 🤩🐐🐐

would love to see this tested on longer agent trajectories where decoding costs really add up