How much VRAM do I need for local LLMs?

As a rule of thumb: VRAM (GB) must exceed the model size in billions of parameters times the bytes per weight (about 0.5 for 4-bit quant). A 4-bit 13B model needs ~8-10GB; a 4-bit 70B needs ~40-48GB.

Is a consumer GPU enough?

A 24GB RTX 4090/3090 runs most 7B-34B models comfortably at 4-bit and is the best value for individuals. 70B+ needs 48GB+ or multi-GPU.

AI-RADAR.IT · AI-RADAR.NET · AI-RADAR.TECH

News & analysis on local LLMs, stack & on-prem hardware.

GUIDEHARDWARE

Best GPUs for running local LLMs

Evergreen guide · updated 2026

Key takeaway

The right GPU is decided by one number: VRAM. For most people a 24GB RTX 4090/3090 is the sweet spot (runs 7B-34B at 4-bit). For 70B-class models you need 48GB+ (A6000, dual 3090/4090) or an 80GB datacenter card (A100/H100). Buy for the model size you actually run, not the biggest card you can afford.

Local LLM performance is gated almost entirely by VRAM: if the model plus its context fits in GPU memory, it runs fast; if it does not, you spill to system RAM and speed collapses. So choosing a GPU is really choosing a VRAM tier.

VRAM tiers

VRAM	Example cards	Runs (4-bit)
8-12GB	RTX 3060/4060, A2000	up to 7-8B
16GB	RTX 4060 Ti 16GB, A4000	up to 13B
24GB	RTX 3090 / 4090, A5000	7B-34B comfortably
48GB	RTX A6000, dual 3090/4090	up to ~70B
80GB	A100 / H100	70B+ and training

Best value picks

Individuals/hobbyists: a used RTX 3090 (24GB) is the price-performance king. Prosumers: RTX 4090. Small teams needing 70B: an A6000 (48GB) or dual-4090 rig. Renting an 80GB card by the hour often beats buying unless utilization is high.

Frequently asked questions

How much VRAM do I need?
Roughly params(B) × 0.5 for 4-bit. 13B ≈ 8-10GB, 70B ≈ 40-48GB.

No GPU yet? Rent one by the second to test:

⚡ RunPod →

🔗 Affiliate link.

Continue exploring