The post is a rant, but it captures a precise tension. Every day a new optimized configuration or inference engine promises excellent performance. The problem appears at the bottom: it only works with NVFP4 or MXFP4. The author, on r/ROCm, puts it bluntly: you made the fastest option slightly faster, but the output is probably broken and hallucinations come from everywhere.

The distinction the author draws is between large and small models. On a Large Language Model with billions of parameters, 4-bit can still work: internal redundancy absorbs some of the noise introduced by quantization. Forgetting a spaghetti Bolognese recipe is not an operational disaster. But a small dense model, the kind that often runs on a single GPU in a closet or in an air-gapped environment, does not have that margin. When precision drops to FP4, the model can produce serious logical errors: the post's example is 1+1=3. At that point you are no longer optimizing; you are replacing the model with something that produces fast but unreliable answers.

This detail matters for anyone choosing an on-premise stack. ROCm, AMD's open software platform for GPU compute on local hardware, has become a testing ground for alternative inference engines. Many projects are born to squeeze more tokens per second out of GPUs with limited VRAM. The temptation is understandable: if a low-precision format lets you load a larger model or reduce energy costs, the TCO seems to improve. But real TCO also includes the cost of errors. If the model gets a calculation wrong, extracts data inconsistently, or generates code with subtle bugs, someone has to verify, correct, or rerun the work. That operational cost does not show up in throughput benchmarks.

There is a less visible second-order effect. The proliferation of FP4-optimized engines shifts attention from output quality to pure speed. Numbers circulating on forums and repositories become the selection criterion, while accuracy on specific tasks stays in the background. The losers are teams that use small models for deterministic automation, data extraction, or coding assistance: there an error is not a creative hallucination, it is wrong data entering a pipeline. The winners are projects that can show marginal gains on consumer or datacenter cards and make them look like general progress.

The structural lesson is not that FP4 should be thrown away. On large models and tolerant tasks, compression remains a useful tool for reducing VRAM requirements and energy consumption. The point is that numerical precision is not a neutral detail: it interacts with model size and with the nature of the task. For those evaluating on-premise deployment, this means speed benchmarks need to be paired with reliability tests on their own use case. AI-RADAR offers analytical frameworks at /llm-onpremise to evaluate these trade-offs. The next time you read about a miraculous inference engine, check the quantization format first. The devil is in the bits.