A single RTX 5090 producing 2,788 tokens/s with a diffusion LLM: the result shared by the Introspective Diffusion research group is not an academic exercise, but a concrete signal for teams working on local and self-hosted stacks. The ORL (On-Policy Residual Learning) project, fully open source, tackles a historical limitation of diffusion language models: the need for multiple forward passes to generate text, which has so far limited their use in low-latency inference.

The core idea is not to eliminate denoising passes, but to make the model converge more quickly. Instead of predicting everything from scratch at each step, ORL carries forward previous predictions as residual context and progressively refines them. Because these residuals are generated by the model itself and change during training, the team avoids the mismatch between training and inference by working on fresh trajectories produced by the current model. They add an Adjacent Entropy Contraction component that pushes each step to reduce uncertainty relative to the previous one, without over-constraining already confident predictions.

The reported numbers are peak results, not averages across arbitrary prompts, and refer to single-request decoding with batch size 1. On a single RTX 5090, LLaDA-2.1-mini with ORL and NVFP4 precision reaches 2,788 tokens/s. On 8 B200 GPUs, DiffusionGemma 26B MoE reaches 7,708 tokens/s. The QwenDLM-4B model, converted from Qwen3-4B, improves end-to-end throughput by up to 6.9× and raises HumanEval pass@1 from 82.9% to 89.6%. For LLaDA-2.1-mini, 4.2 million self-generated tokens and 36 H100 GPU-hours deliver up to 3× higher throughput at comparable quality.

In a direct comparison with speculative decoding on a single H100, QwenDLM-4B + ORL reaches 1,298 tokens/s, versus 746 tokens/s for DFlash and 405 for EAGLE3, at matched measured accuracy. The comparison is end-to-end: ORL runs on dlmrt, while the speculative decoders use vLLM.

On the infrastructure side, the team built dlmrt, an inference engine optimized for low-latency diffusion decoding. Instead of launching and synchronizing every denoising step from the CPU, dlmrt captures the entire loop in a CUDA graph and executes it on the GPU. It also supports NVFP4 inference on Blackwell GPUs, including the consumer RTX 5090. In single-H100 comparisons with identical models and decoding settings, dlmrt runs 1.2 to 1.6 times faster than SGLang/vLLM.

This is where the news stops being just a benchmark and becomes a structural indicator. The focus on single-request, rather than aggregate throughput for concurrent requests, shifts the optimization center of gravity. Teams managing on-premise or self-hosted deployments often do not need to serve hundreds of simultaneous users, but need to reduce latency for a single stream. Diffusion LLMs with ORL attack exactly that constraint: if the model converges in fewer passes, a high-end consumer GPU can become a credible inference node without sizing clusters for aggregate throughput.

The open-source release of models, training code, evaluation pipeline, and inference engine further lowers the barrier. Teams working on local stacks can reproduce results, test different workloads, and contribute to the engine. This puts pressure on existing serving ecosystems: vLLM and SGLang, designed mainly for autoregressive models and batch optimization, must adapt to a different decoding paradigm. At the same time, speculative decoding methods such as DFlash and EAGLE3, which require an additional draft model, now compete with an approach that does not need one, because the residual serves as internal context.

Another signal concerns quantization. Using NVFP4 on a consumer Blackwell GPU like the RTX 5090 shows that weight compression is not only a data center lever, but also a tool for self-hosted deployments without saturating VRAM. In on-premise scenarios, combining aggressive quantization with a dedicated inference engine can shift TCO calculations: fewer GPUs for the same single-user service level, or the ability to move local workloads that previously required more expensive hardware.

It should be restated that the cited numbers are peak results on specific benchmarks, not a guarantee of average performance on real workloads. But the value of the work is not only in tokens/s: it is in showing that a diffusion LLM can operate in realistic conditions for local inference, with an open-source engine designed for that type of load.

The group asks the community whether single-request latency is the main bottleneck or whether there are other aspects to optimize. The question is legitimate: for those evaluating on-premise deployments, single-request is often the deciding parameter, but memory, checkpoint management, and orchestration framework integration remain open. The direction, however, is clear: diffusion LLMs are no longer just a research curiosity, but a concrete alternative for those who want to keep models under their own control.