The discussion around new quantization types for llama.cpp focuses on a proposal introducing IQ2_K, IQ3_K, IQ4_K, IQ5_K and IQ6_K. According to the source, these formats keep the same file sizes as the well-known Q-quants while substantially reducing KL divergence from the original model. The cost is twofold: inference performance drops slightly and creating the quantized files is rather slow. The data, however, cover a single model and should be read as a preliminary signal.

For anyone running LLMs locally, quantization remains a key lever for fitting increasingly large models into limited VRAM budgets. Q-quants have long been a reference point in the GGUF format distributed by llama.cpp: they reduce the numerical precision of weights, lowering memory footprint and often improving efficiency on consumer CPUs and GPUs. The key difference of the IQ variants lies in how they approximate weights: they aim to keep a distribution closer to the original, measured through KL divergence. If the reported gain were confirmed across more architectures, it would be a step forward for self-hosted deployments that cannot accept a marked quality drop.

The trade-off is not free. Same file sizes mean VRAM requirements do not change: those who cannot run a Q4_K model today will not find extra room by switching to IQ4_K. In return, the slight performance degradation translates into lower token generation speed, which on machines with little memory or older CPUs can make the difference between a usable assistant and a frustrating one. The slow creation process, by contrast, weighs mainly on those producing and distributing custom quantizations: for end users downloading a ready-made file it is a marginal issue, but it slows test and release cycles for teams maintaining internal pipelines.

The proposal points to a structural direction: quantization research is shifting attention from simple space savings to approximation quality. This directly concerns on-premise and air-gapped deployments, where precision loss cannot be compensated by larger cloud-hosted models. If IQ formats manage to deliver a better quality profile at the same footprint, the threshold for self-hosting capable models lowers further. The caveat stated in the source remains: the analysis is limited to a single model and we do not know whether the observed improvements generalize. Before treating them as a breakthrough, systematic evaluations across different model families are needed.