AI Hardware: GPUs, CPUs, and Accelerators

GPU Fundamentals for AI

GPUs excel at the parallel matrix math neural networks are made of. But for LLM inference specifically, the spec hierarchy surprises most buyers: single-user token generation is memory-bandwidth bound, not compute bound. Producing each token requires streaming essentially the whole model through the GPU once, so tokens/sec ≈ bandwidth ÷ model size. A card with 936 GB/s generating from a 20 GB model tops out around 40–45 tok/s; a 288 GB/s card manages ~13 with the same model. Compute (TFLOPS, tensor cores) matters for prompt processing, training and high-batch serving — third in line for a chat-style workload.

Key GPU specifications, in order of importance

• VRAM: the hard gate — if model + context don't fit, layers spill to system RAM and speed collapses ~10×.
• Memory bandwidth (GB/s): sets your tokens/sec. The most under-checked spec: some high-VRAM budget cards sit on narrow buses and fit big models only to run them at unusable speeds.
• Tensor cores / FP16-FP8 throughput: governs prompt ingestion, fine-tuning and batch serving.
• TDP: 350–575W cards are recurring electricity costs and cooling problems; power-limiting typically costs only 5–10% inference speed for 25–30% energy savings.
• Interconnect (NVLink/PCIe lanes): matters only for multi-GPU tensor parallelism and training.

NVIDIA vs AMD vs Intel — the honest state of play

Memory Requirements by Model Size

The sizing rule for any dense model: VRAM (GB) ≈ params(B) × bytes-per-weight × 1.15, where bytes/weight ≈ 0.55 at 4-bit, 1 at 8-bit, 2 at FP16. Quantization is what makes local AI possible at all — a 70B model drops from ~140 GB to ~40 GB at 4-bit with only small quality loss (see our quantization guide).

⚠️ The KV-cache trap: weights are only half the story. The attention cache grows linearly with context length — a 70B-class model can add ~10GB at 32k context and ~40GB at 128k, per concurrent session. Size for your real context and concurrency, quantize the cache to Q8 (nearly free), and keep 10–15% VRAM headroom. Full math in the VRAM sizing guide.

Consumer vs Workstation vs Datacenter

💡 Rule of thumb: buy the biggest parameter count your budget holds at 4-bit, then spend leftover VRAM on a better quant or more context — not on a smaller model at higher precision. And for occasional big-model needs, renting an 80GB GPU by the hour beats owning one (see RunPod vs Vast.ai).

The Used Market: Where the Value Is

The single best deal in local AI remains the used RTX 3090: 24GB of 936 GB/s VRAM at a fraction of new-24GB prices, delivering 70–85% of a 4090's chat speed (bandwidth-bound inference forgives the older compute). Buying checklist: ask for stress-test screenshots with temperatures; check VRAM junction temps (the 3090's rear memory chips run hot — many used cards want a €20 thermal-pad refresh; >100°C under load means the pads are done); assume no warranty and price it in. Ex-workstation cards (A5000 24GB, A6000 48GB) also surface used — blower coolers and 2-slot width make them the smart path to stacking. Full treatment in the GPU buyer guide.

Apple Silicon & Unified Memory

Macs (and unified-memory PCs like AMD Strix Halo machines) play a different game: an M-series Max/Ultra with 64–192GB can hold models no consumer GPU can — a 70B at Q5, even 100B+ MoE models — silently, at a fraction of the power draw. The trade-off is bandwidth (~400–800 GB/s vs 1,000–1,800 for high-end discrete GPUs), so big-model generation is usable-but-not-fast, and prompt processing on long contexts lags CUDA cards badly.

💡 The MoE synergy: Mixture-of-Experts models need memory for all experts but only read the active ones per token — huge capacity, low per-token bandwidth. That is exactly the unified-memory profile, which makes large MoE models the best model class for Macs and Strix-Halo-class machines. Details: MoE deployment.

CPU Considerations

For GPU inference boxes, the CPU is a supporting actor: it orchestrates, preprocesses, and feeds the GPU. Don't overspend — a mid-range 8-core CPU rarely bottlenecks single-GPU inference. Spend the savings on system RAM (64GB lets llama.cpp cache models between runs and absorb overflow layers gracefully) and NVMe.

CPU-only inference — honest expectations

Small models (≤8B, 4-bit GGUF via llama.cpp) run acceptably on modern desktop CPUs: roughly 5–20 tok/s depending on memory bandwidth (dual-channel DDR5 helps more than core count — CPU inference is bandwidth-bound too). A 70B on CPU produces 1–2 tok/s: technically running, practically a demo. CPU-only is right for edge boxes, background batch jobs and first experiments — not for interactive multi-user use.

Multi-GPU, Power & Storage

Multi-GPU: two modes, know the difference

Layer split (llama.cpp default): GPUs take turns — VRAM adds up, speed roughly equals one card. Fine over plain PCIe. Tensor parallel (vLLM/ExLlama): GPUs work simultaneously — real speed gains, but wants matched cards and fast interconnect (NVLink on 3090s helps). The classic budget 70B rig is dual used 3090s: half the price of a 48GB workstation card, more assembly — budget ≥8 PCIe lanes per card, a ~1200W PSU and airflow that doesn't cook the top card.

Power and cooling

High-end GPUs draw 350–575W under load; budget 1.3–1.5× GPU TDP for total system power on an 80+ Gold/Platinum PSU. Two habits pay for themselves: power-limiting (a 3090 capped at ~250–280W via nvidia-smi -pl loses only ~5–10% inference speed and saves 25–30% energy — bandwidth-bound inference wastes high power states anyway) and counting the whole chain — PSU losses, cooling, and in summer the air conditioning that removes the heat you paid to make. Full economics in the cost guide.

Storage

• Model storage: 5–80GB per model and they multiply fast — dedicate an NVMe (Gen 4+) volume, dedupe across tools, clean old quants.
• Loading speed: NVMe vs SATA is the difference between a 20-second and a 3-minute model swap.
• RAG/vector data + logs: plan 50–500GB working space for embeddings, caches and audit logs.

Build Recommendations (2026)

Compare full configurations in the interactive Hardware Matrix. Quick picks by budget:

Resources and Further Reading

Deep guides on AI-Radar

• Best GPUs for local LLMs — full buyer guide
• How much VRAM for Llama 70B (sizing math + KV-cache)
• LLM quantization explained (GGUF, AWQ, EXL2, FP8)
• The real cost of running LLMs locally (TCO)
• Interactive hardware comparison matrix
• LLM On-Premise Observatory

Recent Hardware Articles