KLD has become a kind of sacred number in some discussions about quantized models. On Reddit, many treat it as a metric capable of ranking every quantized LLM by quality. But the acronym measures something much narrower: the divergence between the probability distributions produced by the logits of the base model and the quantized one, computed token by token and then averaged. It is not an evaluation of how well the model performs on real tasks.

The first crack is the dataset. If KLD is calculated on random text, such as Wikipedia articles, you get a number that does not represent what actually happens during inference. When a model generates, a growing portion of the context consists of tokens produced by the model itself, not external text. A more sensible value would require generating a dataset with the base model and computing divergence only on the generated tokens, giving zero weight to prompt tokens. This reverses the most common practice.

The second crack is even more relevant for anyone evaluating quantized models on-premise. Quantization methods often rely on calibration data. If KLD is calculated on the same data used for calibration, the numbers look artificially better. Even a claimed held-out dataset is not enough: if calibration and evaluation come from the same source or have similar distributions, the divergence remains inflated. Some teams push fine-tuning of quantization scale parameters on that very data, obtaining very low KLD scores on models that then struggle to generalize.

Here is the structural issue. Inflating benchmarks requires hardware, time, and compute; manipulating KLD can be much cheaper. This changes the incentives for those publishing quantized models: a low KLD number can be achieved with few tricks, while a robust benchmark is expensive. The result is that vendor-reported KLD is losing signal value. For those doing self-hosted deployments, where quantization serves to reduce the required VRAM and contain overall cost, the risk is selecting a model based on a metric that does not predict real behavior.

This does not make KLD useless: it remains a diagnostic tool for understanding how much quantization has altered the base model's distributions on a limited dataset. But it is not a universal leaderboard. A higher value does not imply the model will fail at the user's tasks, just as a low value does not guarantee quality. On-prem evaluation must compare KLD with tests on real tasks and verify the effective separation between calibration and evaluation data. Without that hygiene, the number circulating on Reddit is more noise than signal.