Phi-4 is the proof that training data beats parameter count: a 14B trained on heavily curated and synthetic data that outperforms much larger models on math and reasoning benchmarks. It runs on a 12GB card at Q4 (and acceptably on CPU), making it the workhorse for edge boxes, high-volume cheap serving, and as a base for fine-tuned classifiers/extractors. Its one hard limit: a 16K context — not the model for long-document RAG.
VRAM by quantization level
| Quant | Weights | Fits on |
|---|---|---|
| Q4_K_M | ~8.5 GB | 12GB card; 8GB tight |
| Q6_K | ~12 GB | 16GB card |
| Q8_0 | ~15 GB | 16GB card |
Weights only — add KV-cache (grows with context and concurrency) and ~1–2GB runtime overhead. Formula and cache math in the VRAM guide.
Quick start
Small enough to fine-tune with QLoRA on a single consumer GPU — the classic base for distilling a big model's behavior into cheap production serving. Under vLLM, a single 24GB card serves dozens of concurrent Phi-4 users.
Expected performance
| Hardware | Generation speed |
|---|---|
| RTX 3060 12GB (Q4) | ~25–35 tok/s |
| RTX 3090/4090 (Q4) | ~50–70 tok/s |
| Modern 8-core CPU (Q4) | ~5–10 tok/s |
Indicative single-user figures (llama.cpp/Ollama class runtimes); multi-user serving via vLLM multiplies total throughput 10–20× through batching.
Tips & gotchas
- Mind the 16K context — chunk RAG tightly or pick a 128K model for long documents.
- As an agent tool-executor behind a bigger planner, it's the cheapest reliable option per call.
- Fully permissive.
FAQ
Can Phi-4 replace a 27B for my use case?
For focused tasks (math, extraction, classification, grounded Q&A) often yes; for nuanced long-form writing and broad knowledge, no. Its 16K context is the other gate.
Is CPU-only realistic?
For single-user, non-interactive or batch use, yes (~5–10 tok/s). For interactive multi-user work, use a GPU.