How to run Llama 3.3 70B locally

Llama 3.3 70B is the benchmark other open models are measured against: excellent instruction following, drafting quality and general knowledge, with the broadest ecosystem support of any family. The price is hardware: ~40GB of weights at Q4 means a 48GB card or a dual-24GB rig. If your tasks are RAG-grounded Q&A, test a 27B first — the 70B earns its silicon on nuanced writing and harder reasoning.

VRAM by quantization level

Weights only — add KV-cache (grows with context and concurrency) and ~1–2GB runtime overhead. Formula and cache math in the VRAM guide.

Quick start

Dual-GPU: llama.cpp/Ollama split layers automatically; vLLM with tensor-parallel 2 is faster for serving. Do not plan around CPU offloading — every 20% of the model on CPU roughly halves speed. On Macs, a 96GB+ Studio runs Q5 with big context, quietly, at ~7–10 tok/s.

Expected performance

Indicative single-user figures (llama.cpp/Ollama class runtimes); multi-user serving via vLLM multiplies total throughput 10–20× through batching.

Tips & gotchas

• Budget the KV-cache: 128K context can add tens of GB — quantize the cache to Q8 and size for your real context.
• Occasional 70B need? Rent an 80GB GPU by the hour instead of buying — the utilization math rarely favors owning for occasional use.
• Free for almost all business use; custom terms — have someone read it before productizing.

FAQ