The final bill is not really about GPU hours but about tokens produced. The test published on books.vizuara.ai and discussed on r/LocalLLaMA starts from an aggressive setup: eight B300 GPUs on Modal, vLLM with tensor parallel 8 and native MXFP4 quantization for a 2.8 trillion parameter model. The system reaches 92 tokens per second in steady decode, 83 tokens per second on average across four prompts, with a time-to-first-token between 0.92 and 1.02 seconds. Rental cost is about $56.79 per hour, which translates into $190 per million output tokens. One clean run takes about $36 of GPU time; leaving the system warm raises the daily cost to $1,363.

The comparison with the cheaper variant is what flips the usual interpretation. The same author ran the model with Unsloth Dynamic GGUF at 1 bit, a 594 GB file that fits on eight A100-80GB GPUs via llama.cpp. The hourly rental drops to $19.99, 2.8 times less than the B300 setup. But decode collapses to about 9 tokens per second, the TTFT swings between 7 and 60 seconds, and the cost per token rises to about $620 per million: 3.3 times more expensive. In other words, the cheaper GPU per hour is the most expensive when paying per result.

This paradox is the technical core of the story. Token throughput, not the hourly price list, determines the unit cost of inference. Extreme 1-bit quantization lets the model fit into the VRAM of A100s, but it adds neither memory bandwidth nor compute capacity. Slow decode and long initial latency make the configuration at most suitable for batch or non-interactive workloads, not for a service with latency requirements. The quality of responses, according to who ran the test, remains acceptable for arithmetic and coherent prose, but it does not offset the final bill.

For anyone evaluating on-premise or self-hosted deployment, the test adds an often underestimated operational constraint. Cold boot on the B300 configuration takes about 27 minutes, with 1.56 TB of model loading, JIT compilation and 51 CUDA graph captures. This is not a lab detail: a powered-off system pays that time before producing the first token; a warm system costs $1,363 per day even during idle periods. This shifts the TCO calculation toward maintenance cost and service window, not just the hardware purchase price. Native MXFP4 support on B300s signals that the new generation of accelerators is designed to relieve memory pressure while keeping high throughput, whereas A100s, while able to load a compressed model, show the limits of shrinking the footprint alone.

The picture that emerges is not a confirmation that every LLM needs eight B300s. It is proof that the economics of inference must be measured in tokens per dollar and latency, not GPU hours. Anyone comparing infrastructure for local models should be wary of apparently convenient hourly prices and ask which throughput and TTFT are compatible with their use case. For those evaluating these scenarios, AI-RADAR collects analytical frameworks at /llm-onpremise to compare trade-offs between hourly costs, latency and data sovereignty.