Mistral Small 3.1 is engineered for latency: a 24B tuned to produce near-27B-class quality with fewer layers and faster responses, plus vision support and a 128K window. At Q4 it squeezes into a 16GB card — the best model quality available at that tier — and flies on 24GB. For EU organizations with sovereignty-minded procurement, the combination of Apache 2.0 and a European lab is a quiet superpower.
VRAM by quantization level
| Quant | Weights | Fits on |
|---|---|---|
| Q4_K_M | ~14 GB | 16GB card — the best quality at this tier |
| Q5/Q6 | ~17–19 GB | 24GB comfortably |
| Q8_0 | ~26 GB | 32GB, or 24GB with tight context |
Weights only — add KV-cache (grows with context and concurrency) and ~1–2GB runtime overhead. Formula and cache math in the VRAM guide.
Quick start
Low-latency by design — a strong pick for interactive assistants and agent tool-steps. vLLM AWQ builds serve teams well on a single 24GB card. Note the 16GB tier: check the card's memory bandwidth (a 4060 Ti's narrow bus caps speed even though the model fits).
Expected performance
| Hardware | Generation speed |
|---|---|
| RTX 4060 Ti 16GB (Q4) | ~15–20 tok/s |
| RTX 3090 (Q4) | ~32–40 tok/s |
| RTX 4090 (Q4) | ~40–50 tok/s |
Indicative single-user figures (llama.cpp/Ollama class runtimes); multi-user serving via vLLM multiplies total throughput 10–20× through batching.
Tips & gotchas
- The go-to recommendation when the budget stops at a 16GB GPU.
- EU AI Act paperwork? A European model with Apache 2.0 simplifies the provenance conversation.
- Fully permissive — and European origin, which matters in EU sovereignty-sensitive procurement.
FAQ
Is it really usable on a 16GB card?
Yes — Q4 leaves ~2GB for context. It's the strongest model that genuinely fits that tier; mind bandwidth on budget cards.
Mistral Small or Qwen 27B on a 24GB card?
Qwen for peak quality and thinking mode; Mistral for lower latency, vision and EU provenance. Both fit — eval on your tasks.