The numbers come from a Reddit user sharing the output of an experimental build: 28.50 tokens per second, 60.70 GiB of VRAM, 122 billion total parameters but only 10 billion active per token – the classic Mixture of Experts structure of the Qwen3.5 model. What makes it possible is FP4 quantization and a fork of llama.cpp using ROCmFPX kernels, built for AMD GPUs with the ROCm stack and not yet part of the main project.

The most striking takeaway for those evaluating local deployments is the performance profile: a 36.89% speed boost in decoding and a 13.47 GiB savings over higher-precision or unquantized versions. The BF16 KLD of 0.041366 points to high fidelity to the original model, quieting initial doubts about the quality of four-bit floating-point representations.

It is no coincidence the test comes on AMD hardware. The ROCm platform is slowly narrowing the gap with CUDA, but the software ecosystem remains fragmented and often reliant on community efforts. Yet optimized kernels for FP4 – a format still rare in mainstream frameworks – hint at a direction: if quantization can pack a 122B model into just over 60 GiB, two high-end consumer GPUs or a single professional card can run a fully autonomous local instance.

For organizations focused on data sovereignty or operating in air-gapped environments, this opens the door to using model sizes that were unthinkable until recently without expensive NVIDIA infrastructure or cloud APIs. The MoE architecture, with its 10B active parameters, adds efficiency: the computational cost per token is tied to that fraction, not the full 122B, making inference affordable even on less exotic hardware.

Of course, the current limitation is the need for an external fork and some manual assembly of the stack. But exactly this kind of community experimentation paves the way for stable integrations into llama.cpp or inference servers like vLLM. Should FP4 consolidate as a standard for on-premise deployment, the hardware equilibrium that has so far favored NVIDIA might crack, opening room for genuine competition on pricing and accelerator availability.