Putting a 35-billion-parameter Large Language Model (even in Mixture-of-Experts guise) on a gaming graphics card and having it generate over 500 tokens per second for extremely long texts: until yesterday, that sounded like science fiction. Yet NInfer, an inference engine built entirely from scratch in C++ and CUDA, does exactly that, driving Qwen3.6-35B-A3B to 543 tok/s on a single RTX 5090, over a 65,536-token decode window. This is not a one-off benchmark: the open-source project, published on GitHub, is designed to squeeze every last drop of performance out of two very specific checkpoints, Qwen3.6-27B (dense) and Qwen3.6-35B-A3B (MoE).

NInfer is the antithesis of generic serving engines. It doesn’t support dozens of architectures, doesn’t add continuous batching, doesn’t promise extreme flexibility. Instead, it’s a sharp Swiss Army knife for a single task: making those two models fly on an RTX 5090 (or, with some tuning, on an RTX PRO 6000). The secret lies in an end-to-end optimization pipeline that starts from custom quantization (5.03 bpw for the 27B, 4.97 bpw for the 35B-A3B, resulting in 16.29 and 20.84 GiB on disk respectively), moves through weight layout design, per-operation kernel optimization, kernel fusion, and a dedicated LM-head draft path. The upshot is that both models fit comfortably in the RTX 5090’s 32 GB VRAM, and with INT8 KV cache, they reach the full native context length of 262,144 tokens.

The numbers speak for themselves. On long-reasoning prompts, with a Multi-Token Prediction (MTP) draft window of 3 and the optimized draft path, NInfer sustains 542.8 ± 12.5 tok/s for 65,536-token completions, with MTP acceptance rates of 73%. On shorter sequences (8,675 tokens) it climbs to 634 tok/s with 82.7% acceptance. Even on mixed workloads – code, translation, story generation, and structured output – speed remains high (from 396 tok/s for stories to 661 tok/s for structured output). The 27B dense model is no slouch either, delivering 174 tok/s on long reasoning and 189 tok/s on structured output, again with MTP active. Capability scores, measured on AIME25, AIME26, and GPQA-Diamond, confirm that compression doesn’t sacrifice quality: the 35B-A3B scores 27/30 on AIME25 and 169/198 on GPQA, in line with unquantized models.

There’s a strong structural message in this feat, beyond the engineering tour de force. NInfer signals that the hardware barrier for frontier-model inference is dropping, down to consumer GPUs. An RTX 5090 costs less than a tenth of an A100 or an H100, consumes far less power, requires no data-center cooling, and can sit under a desk. For anyone with privacy requirements, GDPR compliance, or simply a desire to avoid recurring cloud costs, a hyper-specialized engine like this makes self-hosting viable even for models with tens of billions of parameters, with extremely low latencies and throughputs that until recently demanded far more expensive infrastructure.

The flip side is extreme specialization. NInfer supports no other models, does not batch concurrent requests, and works only on GPUs with sm_120a compute capability. It’s the classic trade-off between flexibility and raw performance: if your workload is serving a single model at low latency, giving up generality can be a bargain. If you need to scale to many users or swap models every week, engines like vLLM or TensorRT-LLM remain more suitable—but on this specific hardware-model pair, they lag behind. The author himself throws down the gauntlet: see another engine match or beat these numbers, with the same quantization size, on the same card.

This isn’t just a self-contained record. It’s proof that vertical optimization, weaving together algorithm, kernel, and hardware, can still claw out significant gains, and that the on-prem LLM market needs tailored solutions as much as universal platforms. For teams evaluating local deployments, the NInfer case suggests paying closer attention to niche, often open-source projects that can cut Total Cost of Ownership by an order of magnitude compared to cloud options. More broadly, it reminds everyone that the race for efficiency isn’t over—indeed, on high-end PCs, it’s just getting started.