The quantization of the KV cache is one of the most debated levers in local LLM deployment. The new benchmark on Qwen3.8 27B adds a rare data point: not a speed comparison, but a measure of token-by-token KL divergence against a bf16 reference. The result shifts the debate: the 8-bit format is almost indistinguishable from full-precision computation in the GGUF and EXL3 engines, while 4-bit introduces a penalty that grows with context depth.

The author tested 35 weight quantizations with llama.cpp, ExLlamaV3 and vLLM. Each configuration was evaluated with 16-bit, 8-bit and, where supported, 4-bit caches. The dataset mixes agentic tasks generated by the model at full precision and different domain texts, with contexts up to 64k tokens per row.

8-bit: almost a free cache

Memory numbers explain why the topic is so heated. At 64k tokens, a 16-bit cache weighs 4 GiB. Dropping to 8-bit brings the footprint to 2.13 GiB with llama.cpp and ExLlamaV3 and to 2 GiB with vLLM FP8: a saving of 1.9–2 GiB. In KL divergence terms, the effect on llama.cpp and ExLlamaV3 is below 0.00012, a move the author considers within numerical noise. vLLM adds 0.0004 to 0.0006, still small but systematic, possibly linked to the switch from FlashAttention to FlashInfer when the FP8 cache is enabled.

4-bit is not a linear shortcut

With a 4-bit cache the picture changes. In llama.cpp, q4_0 adds from +0.0016 to +0.0027 of KL. The relative effect depends on weight precision: on Q3-class quantizations the increase is 6–10 percent, on UD-Q4_K_XL it rises to 39 percent, and on Q8_0 it reaches 390 percent. In the latter case divergence rises from 0.0007 to 0.0034, surpassing a UD-Q5_K_XL with a 16-bit cache. ExLlamaV3 shows a similar pattern: the penalty grows with weight precision, from about +0.0002 on models at 2.0–2.5 bpw to +0.0032–+0.0033 on weights at 6–8 bpw.

In long documents the damage is not uniform. On Q8_0 with q4_0 cache it goes from +0.0035 in the first 4k tokens to +0.0041 in the 16–32k range; on UD-Q5_K_XL from +0.0030 to +0.0038; on ExLlamaV3 8.0 bpw from +0.0041 to +0.0050. The relative increase is 15–30 percent compared with the first 4k. The same quantizations with a 16-bit cache show no variation, so the signal is clear: it is the 4-bit cache that degrades with position. The author expects that at 128k+ token contexts the picture will worsen further.

The comparison with vLLM checkpoints adds another variable. All tested vLLM checkpoints are beaten by smaller GGUF or EXL3 quantizations, partly due to activation quantization. NVIDIA's NVFP4 checkpoint runs MLP layers as W4A4 and attention as FP8 W8A8; Qwen's FP8 is W8A8 across the network. Rerunning NVFP4 and FP8 with vLLM's Marlin kernels, which keep activations in bf16, drops NVFP4's KL divergence from 0.0256 to 0.0117 and FP8's from 0.0039 to 0.0028. Even then, the result only ties a UD-Q5_K_XL that is 8.7 GiB smaller.

For those evaluating on-premise deployment, the structural point is that memory savings cannot be read as a single parameter: they must be considered together with weight type and context length. The 8-bit cache is nearly transparent and offers double the usable context or frees VRAM for larger models. The 4-bit cache, by contrast, penalizes precisely the models with more precise weights and the long contexts where local inference promises the most value. In agentic workloads with windows of tens of thousands of tokens, 4-bit cache corruption is not uniform noise: it accumulates as tokens move further from the prompt, hitting decisions most sensitive to long-range coherence. AI-RADAR provides analytical frameworks on /llm-onpremise for comparing these trade-offs.