The R1 distills brought reasoning models to consumer hardware: this 32B thinks out loud in a <think> block before answering, buying large gains on math, hard code and planning. The trade is speed-to-answer — hundreds to thousands of thinking tokens per response — so it's a specialist for hard problems, not a daily chat driver. At Q4 it fits a 24GB card, and the MIT license makes it frictionless to build on.
VRAM by quantization level
| Quant | Weights | Fits on |
|---|---|---|
| Q4_K_M | ~19–20 GB | 24GB card (leave headroom for thinking-token context) |
| Q5_K_M | ~23 GB | 24GB tight; 32GB comfortable |
| Q8_0 | ~34 GB | 48GB card |
Weights only — add KV-cache (grows with context and concurrency) and ~1–2GB runtime overhead. Formula and cache math in the VRAM guide.
Quick start
Set num_ctx to 16K+ — the thinking block consumes context fast and truncated thinking ruins answers. Sampling: temperature ~0.6 works well; avoid system prompts that fight the thinking format. In pipelines, strip the <think>…</think> block before showing users or feeding downstream steps.
Expected performance
| Hardware | Generation speed |
|---|---|
| RTX 3090 (Q4) | ~25–30 tok/s |
| RTX 4090 (Q4) | ~30–40 tok/s |
| 48GB (Q8) | ~20–28 tok/s |
Indicative single-user figures (llama.cpp/Ollama class runtimes); multi-user serving via vLLM multiplies total throughput 10–20× through batching.
Tips & gotchas
- Don't judge it by wall-clock: token speed is normal, answers are slow because it writes its reasoning first. That's the feature.
- Route with a small model: send routine requests to a 7–14B, escalate hard ones to R1 — reasoning-per-euro is unbeatable used this way.
- Fully permissive — the most business-friendly license in open AI.
FAQ
Why are responses so slow if tok/s looks fine?
It generates a hidden reasoning chain before the visible answer — often 5–20× the answer's length. Total latency = thinking + answer.
R1 32B or Llama 3.3 70B?
Different jobs: R1 for math/code/planning where reasoning wins; Llama 70B for writing quality and broad knowledge. R1 also needs half the VRAM.