What is the KV-cache in an LLM?

The KV-cache stores the key and value vectors for every token already processed, so the model does not recompute the whole sequence for each new token. It is what makes generation fast — and it grows linearly with context length, consuming VRAM on top of the model weights.

Why does long context need so much VRAM?

Because the KV-cache size scales with context length times layers times heads times two (keys and values) times bytes-per-value. At 128K tokens the cache can rival or exceed the model weights themselves, which is why VRAM calculators that count only weights mislead you.

How do I reduce KV-cache memory?

Quantize the cache (Q8 roughly halves it vs FP16, Q4 quarters it with more quality risk), reduce the context window to what you actually need, use models with grouped-query attention, and limit concurrency — each simultaneous request needs its own cache.

KV-Cache & Context Memory: The Real Math (2026)

What the KV-cache is, in one paragraph

A transformer generates one token at a time, and each new token "attends" to every previous token. Without a cache, producing token 1,000 would mean recomputing the key and value vectors for the 999 tokens before it — every single step. The KV-cache stores those key/value vectors so each is computed once and reused. It's the single optimization that makes generation practical — and the price is memory that grows with every token in the context.

The formula

KV-cache bytes =
   2  ×  context_length  ×  n_layers  ×  n_kv_heads  ×  head_dim  ×  bytes_per_value  ×  batch

Reading each term:

2 — one set for keys, one for values.
context_length — tokens held in context. This is the term you control, and it scales the whole thing linearly.
n_layers × n_kv_heads × head_dim — the model's architecture. Note n_kv_heads, not attention heads — this is where GQA saves memory (below).
bytes_per_value — 2 for FP16 cache, 1 for Q8, ~0.5 for Q4. This is the quantization lever.
batch — number of simultaneous requests. Each concurrent user needs a full separate cache — this is the term everyone forgets.

A worked example: Llama 3.3 70B

Llama 3.3 70B has 80 layers, 8 KV heads (it uses grouped-query attention) and a head dimension of 128. For a single request at FP16 cache:

per token = 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes ≈ 0.31 MB

  8K context  →  ~2.5 GB
 32K context  →  ~10 GB
128K context  →  ~40 GB   ← as big as the Q4 weights!

That last line is the whole point. The Q4 weights are ~40GB; at full 128K context the cache doubles your memory need to ~80GB. This is why our VRAM guide insists you size for weights plus cache — and why "it fits" from a weights-only calculator is a trap. Now multiply by concurrency: ten simultaneous users at 32K context each need ten caches — ~100GB of cache alone.

Why GQA changed everything (and old formulas lie)

Older KV-cache formulas use the number of attention heads. Modern models use grouped-query attention (GQA), where many query heads share a small number of key/value heads — Llama 70B has 64 query heads but only 8 KV heads. Since the cache stores KV heads, GQA cuts cache size by that ratio (8× here). A pre-GQA formula would predict eight times the memory and scare you off a model that actually fits. Always use n_kv_heads, which you'll find in the model's config.json as num_key_value_heads.

The two levers that make long context affordable

1. Cache quantization. The cache doesn't have to be FP16. Storing it at Q8 (1 byte/value) halves cache memory; Q4 quarters it. In Ollama, enable it with the OLLAMA_KV_CACHE_TYPE=q8_0 environment variable; in llama.cpp use --cache-type-k/--cache-type-v; vLLM and SGLang have FP8 cache options. Q8 cache is nearly lossless in practice and is the single highest-leverage setting for long-context local work — it turns our 128K Llama example from ~40GB of cache into ~20GB.

2. Right-size the window. A model advertising 256K context does not force you to allocate 256K. Set the context to your real need: a chat assistant rarely uses more than 8–16K; a document-RAG system sized to its chunk budget might need 32K, not 256K. In Ollama this is num_ctx; in vLLM, --max-model-len. Allocating headroom you never use is pure wasted VRAM — the flip side of the "raise num_ctx" advice in our coding-assistant and Qwen3-Coder pages, where the mistake runs the other way.

The practical checklist

Budget VRAM as weights + KV-cache + ~1–2GB overhead — never weights alone.
Compute the cache at YOUR context and YOUR concurrency, not the model's maximum.
Turn on Q8 cache first — it's nearly free quality-wise and buys the most room.
For serving, remember each concurrent request multiplies the cache — this is what caps your user count, not the weights.
Prefer GQA models (virtually all current ones) and read num_key_value_heads from config.json for exact math.