Optimizations around NVIDIA’s NVFP4 format are starting to show real potential for those building local inference stacks. Just days after the Qwen3.6 models were released, the Unsloth team published a set of quantizations that promise to nearly double execution speed compared to NVIDIA’s own optimized versions, with almost identical accuracy benchmarks.

The core improvement lies in moving from a W4A16 scheme to W4A4. NVIDIA’s NVFP4 quantizations keep weights at 4 bits but leave activations at 16 bits, only partially engaging tensor cores designed for reduced-precision operations. Unsloth, instead, applies 4-bit to both weights and activations, allowing matrix multiplications to fully leverage FP4 tensor cores. The result is a measured speedup of 2.5x on the 27-billion-parameter Qwen3.6, and 1.56x–1.79x on the 35B-A3B variant (with an additional trade-off between maximum quality and maximum speed offered by the two versions, NVFP4 and NVFP4-Fast).

These gains do not come at the expense of accuracy. Tests on MMLU-Pro, GPQA, and AIME 2025 show negligible deviations from BF16 and FP8 baselines, and in some cases a slight improvement. For deployments aiming to balance operational cost and response quality, such margins are often enough to favor a quantized model, especially when the speed difference translates into better user experience or fewer GPUs needed to serve the same load.

It’s not just about throughput. The FP8 KV cache calibration built into the quantizations automatically doubles the manageable context length compared to a standard setup—a tangible advantage for applications like retrieval-augmented generation or extensive document analysis, where cache memory is often the primary bottleneck.

This move marks a notable development for those evaluating on-premise stacks. On one hand, it shows how the open-source ecosystem can surpass first-party implementations, unlocking latent hardware performance; on the other, it makes it more realistic to bring high-end models onto compact infrastructure, reducing TCO without sacrificing data sovereignty. AI-RADAR tracks these developments because they directly influence the choices of whoever is designing local inference architectures, where every watt and every gigabyte of VRAM matters.

The overall picture remains one of extremely rapid innovation. Formats like NVFP4, still young, are already seeing optimizations capable of shifting the economics of self-hosting, and it’s not hard to imagine further refinements in the coming months, with direct consequences for the viability of keeping data and control inside one’s own data centers.