Best GPUs for running local LLMs

The two numbers that decide everything

Local LLM performance is constrained first by VRAM: if the model plus its context fits in GPU memory, it can avoid CPU offload; if it does not, offloading usually changes latency and throughput substantially, with the size depending on the runtime and workload. Choosing a GPU is therefore largely choosing a VRAM tier.

Within a tier, the differentiator is memory bandwidth. Token generation often streams a large part of the weights repeatedly, so bandwidth is a useful first order estimate; it is not a benchmark. The official RTX 3090 specification lists 24GB and 936 GB/s, while the RTX 5090 lists 32GB and 1,792 GB/s — compare those published figures with the model and runtime you plan to use. NVIDIA RTX 3090 specs · NVIDIA RTX 5090 specs. This is why a card with generous VRAM but a narrow bus can fit a model and still feel slow.

Compute (TFLOPS, tensor cores) also matters for prompt processing, fine-tuning and high-batch serving. For single-stream chat, bandwidth is often a useful predictor, but measure the target model because kernels, quantization and context can change the balance.

VRAM tiers — what each actually runs

The sizing formula

VRAM (GB) ≈ params(B) × bytes/weight × 1.15

Bytes per weight ≈ 0.5–0.6 (4-bit), 1 (8-bit), 2 (FP16). The ×1.15 covers the KV-cache at moderate context — long contexts (32k+) can need far more, and quantizing the cache to Q8 claws much of it back (see the quantization guide). Example: a 4-bit 34B ≈ 34 × 0.55 × 1.15 ≈ 21GB, so it fits a 24GB card with context to spare. A 4-bit 70B ≈ 40GB weights — hence the 48GB tier.

The used market: where the value is

The used RTX 3090 is often attractive because it combines 24GB of VRAM with 936 GB/s of bandwidth. Whether it is the best value depends on the actual local price, condition, power cost and warranty; this guide does not claim a universal price or speed ranking. What to check when buying used:

• Mining history — not automatically disqualifying (undervolted mining cards often lived gentle lives), but price it in. Ask for stress-test screenshots (e.g. a FurMark/occt run showing temps).
• VRAM thermals — the 3090's rear memory chips can run hot. Check the manufacturer's thermal limits and measure memory-junction temperature under a repeatable load; a reading near a limit is a reason to inspect cooling, not proof that pads are defective.
• Warranty reality — assume none. The discount vs a new card is your insurance budget.
• 3090 Ti / used 4090 — the Ti fixes the memory-thermal design; used 4090s rarely fall to compelling prices because of AI demand. If the gap to a used 3090 is large, take the 3090.

Ex-datacenter/workstation cards (A5000 24GB, A6000 48GB) appear on the used market too: blower coolers (louder but great for stacking), lower power, 2-slot width — often the smartest path to 48GB in a workstation.

NVIDIA, AMD, Intel — the honest state of play

NVIDIA is the default for a reason: every tool works, day one, no caveats — CUDA remains the moat. If your time has value or you want fine-tuning, image models, and serving stacks to just work, buy NVIDIA.

AMD is genuinely usable for inference: llama.cpp runs well via Vulkan or ROCm, and high-VRAM Radeons undercut NVIDIA per GB. The friction appears beyond basic inference — ROCm officially supports a narrow card list, fine-tuning stacks are CUDA-first, and many projects' AMD paths are community-maintained. Good for a budget dedicated inference box; frustrating as an experimentation platform.

Intel Arc offers aggressive VRAM pricing and improving llama.cpp support (SYCL/Vulkan), but the ecosystem is younger still. Viable for tinkerers; not yet a set-and-forget choice.

The Apple Silicon alternative

A Mac with unified memory plays a different game: an M-series Max/Ultra with 64–192GB can hold models no consumer GPU can — a 70B at Q5, even 100B+ MoE models — silently, at a fraction of the power draw. The trade-offs: memory bandwidth (~400–800 GB/s) is below high-end discrete GPUs, so big-model token speeds are usable-but-not-fast; prompt processing on very long contexts is notably slower than CUDA cards; and fine-tuning support (MLX aside) is limited.

Choose a Mac if you want the largest possible models on a quiet desktop and you already live in that ecosystem. Choose a GPU box for speed, serving, fine-tuning, or price at the 24GB tier. Many practitioners end up with both: Mac for daily driving big models, GPU rig for heavy lifting.

Multi-GPU: combining VRAM

For inference, frameworks shard a model across GPUs, so two 24GB cards hold a ~48GB model. Know the two modes: layer split (llama.cpp default — GPUs take turns, VRAM adds up but speed roughly equals one card) and tensor parallel (vLLM/ExLlama — GPUs work simultaneously; adds real speed but wants matched cards and fast interconnect). NVLink (3090s) helps tensor parallel and training; plain PCIe is fine for layer-split inference.

Budget the platform, not just the cards: check the motherboard's actual PCIe lane layout, calculate PSU headroom from the chosen cards and CPU, and verify case clearance and airflow. A dual-GPU build can hold a larger model, but capacity and speed depend on split mode, interconnect and runtime; a single 48GB workstation card trades assembly complexity for purchase cost.

Power: the invisible line item

A 350–450W GPU that runs often is a recurring cost — annual cost depends on actual power, hours and tariff. Treat any consumption figure as an estimate to verify with a wattmeter or machine telemetry. Two mitigations are power-limit or undervolt using vendor tools and matching the card to the duty cycle — if the box sits idle most of the day, a smaller card or renting bursts of cloud GPU may beat owning big iron (see the cost guide).

Best picks by buyer profile

• First local LLM box, tight budget — used RTX 3090 24GB. Nothing else comes close per euro.
• Enthusiast / developer daily driver — RTX 4090, or 5090 (32GB) if budget allows; its published bandwidth is higher, but verify target-model speed locally.
• Small team needing 70B on-prem — RTX A6000/6000 Ada 48GB (single-slot simplicity, pro drivers), or dual used 3090s for half the price and more assembly.
• Silent desktop, biggest possible models — Mac Studio with maxed unified memory.
• Budget pure-inference server — high-VRAM AMD via llama.cpp/Vulkan, eyes open about tooling.
• Production / training — A100/H100/Blackwell class — but rent-by-the-hour usually beats buying until utilization is consistently high.

Common buyer mistakes

• Buying VRAM on a narrow bus. 16GB on a 128-bit bus fits a 13B and then crawls. Check GB/s, not just GB.
• Buying for the model you'll "eventually" run. An 80GB card to occasionally chat with a 7B is money on fire; rent the big runs instead.
• Ignoring the rest of the box. The classic failure: a 4090 ordered, then the PSU, case clearance (3.5+ slots!) and airflow bill arrives.
• Assuming two cards = double speed. Layer-split doubles capacity, not speed. Tensor parallel adds speed but wants matched cards.
• Dismissing system RAM. 64GB of DDR lets llama.cpp offload overflow layers gracefully and cache models between runs — cheap insurance.