LLM quantization explained: GGUF vs GPTQ vs AWQ vs EXL2

Why quantization is the single most important lever in local AI

A model's weights are billions of numbers. Stored the default way — 16 bits each (FP16/BF16) — the arithmetic is brutal: every billion parameters costs ~2 GB of memory. A 7B model needs ~14 GB, a 70B needs ~140 GB, and that's before the KV-cache and runtime overhead. On that math, local AI would end at 13B for almost everyone.

Quantization changes the equation. By mapping each weight onto a smaller set of values — 8, 4, even 2 bits — the same 70B model shrinks to ~40 GB at 4-bit. Suddenly it fits on two consumer GPUs, or one workstation card, or a Mac Studio. Nearly everything interesting happening in on-premise and local LLM deployment sits downstream of this one technique.

There's a second effect that surprises people: quantized models are usually faster. Single-user LLM inference is memory-bandwidth bound — the GPU spends most of its time streaming weights from VRAM, not doing math. Halve the bytes per weight and you roughly halve the bytes moved per token, so tokens/sec goes up even though extra dequantization work is added. On bandwidth-starved hardware (consumer GPUs, Apple Silicon), a Q4 model can be nearly twice as fast as its FP16 parent.

How it actually works (in five sentences)

Take a group of weights — say 32 or 128 of them. Record the range they span (a scale, sometimes plus a zero-point). Snap each weight to the nearest of 2^N levels within that range, storing only the level index (N bits) plus the shared scale. At inference, multiply back: index × scale ≈ original weight. The art is in the details — group sizes, keeping a few "outlier" weights at higher precision, and using calibration data to decide which errors matter least — and those details are exactly what separates the formats below.

Two families exist: post-training quantization (PTQ) — quantize a finished model, minutes to hours, what you use 99% of the time — and quantization-aware training (QAT), where the model is trained knowing it will be quantized. QAT gives the best low-bit quality but requires the model creator to do it; some modern releases (e.g. QAT editions of open-weight models) ship this way, and they're worth preferring when available.

Quantization levels: the quality/size trade-off

Sizes are indicative (weights only, before KV-cache); "quality" describes typical behaviour on modern dense models — MoE models and very small models (≤3B) degrade faster at low bits.

The golden rule: bigger-quantized beats smaller-full

The most useful practical result in the whole field: down to 4-bit, a larger quantized model almost always beats a smaller full-precision one at equal memory. A 70B at Q4 (~40 GB) comfortably outperforms a 13B at FP16 (~26 GB) on essentially every benchmark and subjective test. When you're deciding what to run, first pick the biggest parameter count your hardware holds at Q4, then spend any leftover VRAM on a better quant or more context — not on a smaller model at higher precision.

The rule has two limits. Below ~3-bit, degradation accelerates and can erase the size advantage — a 70B at IQ2 may lose to a 32B at Q4 that uses the same memory. And for tiny models (≤3B), even 4-bit hurts noticeably, because there's less redundancy to absorb the error.

The GGUF quant zoo, decoded

Download any GGUF model and you'll face a wall of suffixes. They're all the same model at different quality/size points:

• Q8_0 — plain 8-bit. Near-lossless, biggest file. For archival or quality-paranoid use.
• Q6_K, Q5_K_M, Q4_K_M, Q3_K_M, Q2_K — the K-quant family: smarter group-wise quantization that protects the most sensitive tensors with extra bits. The M (medium) variants are the balanced defaults; S (small) shaves size at a quality cost, L (large) the reverse.
• IQ4_XS, IQ3_M, IQ2_M… — the newer i-quants: importance-matrix-based methods that squeeze better quality out of very low bit-widths. At 2–3 bits, always prefer IQ over plain Q if available.
• imatrix quants — any level built using an importance matrix computed on calibration text, so the quantizer knows which weights matter. Meaningfully better at Q3 and below, mildly better at Q4.

Practical picks: Q4_K_M as default; Q5_K_M/Q6_K when you have VRAM to spare; IQ3/IQ2 imatrix only when nothing bigger fits; Q8_0 when quality is critical and memory isn't. If perplexity numbers are published for the specific quant, trust them over folklore — differences between well-made Q4 and Q5 are often smaller than differences between two fine-tunes of the same base.

GPU-serving formats: GPTQ, AWQ, EXL2, FP8

• GPTQ — the long-standing GPU 4-bit standard. Uses calibration data to minimize layer-by-layer error. Well supported everywhere (vLLM, TGI, TensorRT-LLM). Quality slightly behind AWQ at the same bit-width in most modern comparisons.
• AWQ (activation-aware) — observes which weights see large activations and protects them. Usually the best quality/speed balance for 4-bit GPU serving, and the default choice on vLLM in production today.
• EXL2 — the ExLlama format, with fractional average bit-rates (e.g. 4.65 bits): it mixes precision per layer to hit an exact memory budget. The power-user choice for maxing a single consumer GPU (fastest single-stream speeds, finest size control); less common in multi-user servers.
• FP8 — on Hopper/Ada and newer GPUs (H100, RTX 40/50), 8-bit floating point for weights and sometimes activations/KV-cache. Near-lossless with hardware acceleration; increasingly the enterprise default for serving when VRAM allows.
• bitsandbytes NF4 — load-time 4-bit quantization inside the Hugging Face ecosystem. Convenient for experimentation and the foundation of QLoRA, but slower than pre-quantized AWQ/GPTQ/EXL2 for serving.

Don't forget the KV-cache

Weights are only half the memory story. The KV-cache — the attention keys/values kept for every token in context — grows linearly with context length and can dwarf the weights: a 70B-class model at 128K context can need tens of GB of cache at FP16. If you've ever wondered why your "40 GB" Q4 model won't load with long context on a 48 GB card, this is why.

Modern runtimes let you quantize the cache too: Q8 cache is essentially free (negligible quality impact, halves cache memory) and Q4 cache is usable with a small quality cost, mostly felt in long-context recall. Turning on Q8 KV-cache is often the single easiest way to double your usable context — check your runtime's flags (llama.cpp --cache-type-k/v, vLLM's KV-cache dtype options).

What actually degrades, and how to judge it

Quantization error doesn't hit all abilities equally. In rough order of fragility: multi-step reasoning and math go first, then code generation precision, then instruction-following nuance, long-context recall, and finally fluent prose — chat "feel" survives almost anything, which is why casual testing at Q2 can fool you into thinking nothing was lost.

Perplexity deltas are the standard lab measure (a good Q4_K_M typically sits a few percent above FP16), but they compress everything into one number. If your use case is agentic tool-calling, structured JSON output, or RAG over long documents, test that before standardizing on a quant: a model that chats perfectly at Q3 can start emitting malformed JSON or missing tool-call syntax at the same level. Rule of thumb for anything precision-sensitive: stay at Q4_K_M or above, prefer Q5/Q6 for agents.

Quantizing your own model

You rarely need to — for any popular model, community quants (e.g. the well-known GGUF repositories on Hugging Face) appear within hours of release. You do need to when you've fine-tuned your own model or need a size nobody published:

• GGUF: convert_hf_to_gguf.py (in llama.cpp) turns HF weights into FP16 GGUF, then llama-quantize produces any level — minutes on CPU, no GPU needed. Add an imatrix step (computed on a few MB of representative text) for low-bit targets.
• AWQ/GPTQ: AutoAWQ / GPTQModel libraries; needs a GPU and a small calibration set (a few hundred samples of text that resembles your workload — using wildly unrepresentative calibration data measurably hurts quality).
• Fine-tuning connection: QLoRA trains LoRA adapters on top of a frozen NF4-quantized base — this is why you can fine-tune a 70B on a single 48 GB card. Quantization isn't just a deployment trick; it's what makes local fine-tuning affordable at all.

Common pitfalls

• Comparing quants across different models. "Model A at Q4 vs model B at Q5" confounds two variables; judge quant levels only within the same model.
• Grabbing day-one quants of a brand-new architecture. Runtime support often lags; early GGUFs of new model families have shipped broken (wrong tokenizer, wrong rope scaling). If a fresh model behaves strangely, try the runtime's latest build and a re-made quant before blaming the model.
• Ignoring the cache when budgeting VRAM. Weights-fit-so-it-fits is wrong at long context; budget weights + cache + ~1–2 GB runtime overhead.
• Assuming quantization always speeds things up. On compute-bound batch serving (many concurrent users on datacenter GPUs), low-bit formats can bottleneck on dequantization; FP8 often wins there instead.
• Quantizing tiny models hard. A 1–3B model at Q3 is usually a worse deal than a cleanly-quantized 7B — small models have no redundancy to spare.

How to choose — decision matrix

• Mac / CPU / desktop, easy setup → GGUF Q4_K_M via Ollama or LM Studio; raise to Q5/Q6 if RAM allows.
• Single consumer GPU, maximum context/speed → EXL2 at a bit-rate tuned to your VRAM, or GGUF fully offloaded; Q8 KV-cache on.
• Production multi-user GPU serving → AWQ (or FP8 on H100/Ada+) on vLLM/TGI.
• Quality-critical, VRAM to spare → 8-bit (Q8_0 / FP8), or Q6_K as the pragmatic compromise.
• Barely fits at all → IQ3/IQ2 imatrix quants — and consider whether a smaller model at Q4 serves you better.
• Fine-tuning locally → QLoRA (NF4) for training; then quantize the merged result to your serving format.