How much VRAM do you need to run Llama 70B?

The VRAM a model needs is driven by three things: how many parameters it has, how many bytes each parameter takes (quantization), and how much working memory the context (KV cache) consumes. Get the first two and you have the weight footprint; add the third and you have the real requirement. This guide works all three out for Llama-70B-class models — and gives you the method for any model.

70B at every quant level

Weights only — add KV-cache and ~1–2GB runtime overhead before deciding what "fits". A 70B at IQ2 on a single 24GB card is a party trick, not a daily driver: quality and context both suffer.

The sizing formula (any dense model)

VRAM (GB) ≈ params(B) × bytes/weight × 1.15

Bytes/weight ≈ 0.5–0.6 (4-bit), 1 (8-bit), 2 (FP16). The ×1.15 covers runtime overhead and a modest context. Examples:

• 7B @ 4-bit ≈ 4GB — runs on an 8GB card. 13B @ 4-bit ≈ 7.5GB.
• 34B @ 4-bit ≈ 20GB (fits 24GB with context to spare).
• 70B @ 4-bit ≈ 40GB weights → ~44–48GB with a working context.
• 123B @ 4-bit ≈ 70GB → the 80GB tier; 8×22B MoE: see the MoE section.

The KV-cache, with real numbers

Every token in context stores attention keys and values for every layer. The formula:

KV bytes ≈ 2 × layers × kv_heads × head_dim × bytes × tokens

For a Llama-70B-class model (80 layers, GQA with 8 KV heads, head_dim 128) at FP16 that's ~320 KB per token — ~2.6GB at 8k context, ~10GB at 32k, ~40GB at 128k. At 128k the cache rivals the Q4 weights themselves. Three practical consequences:

• Two setups running "the same 70B" can need wildly different VRAM. A 4k-context chatbot sits comfortably in 48GB; a 128k document-analysis workload wants 80GB+ or aggressive cache management.
• Quantize the cache. Q8 KV-cache halves those numbers at negligible quality cost (llama.cpp --cache-type-k/v q8_0, vLLM FP8 cache). It's the cheapest context you'll ever buy.
• Concurrency multiplies the cache. Each parallel request has its own context: 10 concurrent 8k sessions ≈ 26GB of FP16 cache on top of the weights. When sizing a server, cache × users is often the real constraint, not the weights.

Silver lining: modern architectures use grouped-query attention (GQA) precisely to shrink this — older 70Bs without GQA had caches ~8× larger. When evaluating any new model, check its KV-cache-per-token, not just its parameter count.

If it doesn't fit: the offloading cliff

llama.cpp can keep part of the layers in system RAM and run them on CPU (--n-gpu-layers). It works — but understand the speed curve before planning around it. Because every token must traverse all layers, the slow CPU layers gate the whole pipeline:

• 100% on GPU: a Q4 70B on 48GB ≈ 15–20 tok/s (bandwidth-dependent).
• ~80% on GPU: already roughly halved.
• ~50% on GPU: ~3–5 tok/s — usable for batch, painful for chat.
• Mostly CPU (64GB DDR): 1–2 tok/s. Technically running; practically a demo.

Treat offloading as a bridge (trying a model before buying hardware) or a batch-job tool — not the production plan. If you're chronically offloading, the honest fixes are: a smaller quant, a smaller model, KV-cache quantization, or more VRAM.

MoE models: a different sizing rule

Mixture-of-Experts models advertise two numbers — total and active parameters — and VRAM follows the total: all experts must be resident even though only a few fire per token. A "120B-total / 12B-active" MoE needs ~120B worth of memory (≈65–70GB at 4-bit) but generates at roughly the speed of a 12B, because only the active parameters are read per token. That makes MoE models superb for high-unified-memory machines (Mac Studio, big multi-GPU rigs): huge capability, small per-token bandwidth. Rule: budget VRAM by total params, predict speed by active params.

Hardware paths for 70B, with expected speeds

• Single 80GB (A100/H100) — the clean solution: Q4 with very long context, or Q8. 25–40+ tok/s. Best for production; often cheaper to rent than own (see below).
• Single 48GB (RTX A6000-class) — Q4_K_M with moderate context on one quiet card. ~12–18 tok/s. The simplest owned setup.
• 2×24GB (dual 3090/4090) — the classic budget path: layer-split via llama.cpp (~13–18 tok/s) or tensor-parallel via vLLM/ExLlama (faster). Needs a big PSU and airflow, but roughly half the price of a 48GB card.
• Mac Studio (96–192GB unified) — holds a 70B at Q5–Q8 with giant context, silently. ~7–12 tok/s generation: usable, not fast. The best "no rack in my office" option.
• Rented 80GB by the hour — for occasional 70B needs, renting beats owning by a wide margin (see the cost guide's utilization math).

Common sizing mistakes

• Sizing to weights only. "40GB model, 48GB card, done" — then the first 32k-context request OOMs. Budget weights + cache + overhead.
• Ignoring concurrency. A card that serves one user comfortably can OOM at five. Cache scales per user.
• Chasing the biggest quant that technically loads. A Q5 that leaves 500MB free will fragment and crash under real use; leave 10–15% headroom.
• Assuming all 70Bs are equal. KV-cache size varies by architecture (GQA vs not); check the model card.
• Planning around offloading. The cliff is real — see above.