The map: what "stack" actually means

"Running a local LLM" involves more layers than one tool. From bottom to top:

  1. Inference engine — the code that executes the model: llama.cpp (GGUF, CPU+GPU, runs anywhere), MLX (Apple Silicon), ExLlama (EXL2, consumer-GPU speed), TensorRT-LLM (max NVIDIA performance, max complexity).
  2. Runtime — wraps an engine with model management and an API: Ollama, LM Studio, llamafile, koboldcpp.
  3. Production server — engineered for concurrency: vLLM, TGI, SGLang.
  4. API layer — in practice, the OpenAI-compatible protocol everyone speaks; it's what makes the layers swappable.
  5. Application layer — chat UI (Open WebUI, LibreChat), orchestration (LangChain, LlamaIndex), vector DB (Chroma, Qdrant, pgvector) for RAG.
  6. Ops — Docker, GPU drivers, monitoring, model storage.

Most confusion ("Ollama vs vLLM?") dissolves once you see they live at different layers for different stages of the same journey.

The runtimes: Ollama, LM Studio and friends

Ollama is the developer default: ollama run llama3.3 pulls and serves a model; a Modelfile versions your configuration; it runs headless, in Docker, and integrates with everything. Watch its famous default: a small context window (num_ctx) that silently truncates long prompts — raise it explicitly. LM Studio is the GUI on-ramp: model discovery with fits-your-hardware hints, sliders, side-by-side comparisons — and on Macs its MLX engine often beats GGUF speeds. We compare the two in depth in the dedicated guide.

Also worth knowing: llama.cpp direct (all flags, newest features first — the power-user path), llamafile (model + runtime in one portable executable) and koboldcpp (creative-writing-oriented frontend). All are single-user tools at heart.

The production engines: vLLM and TGI

When several people hit the same GPU, single-stream runtimes collapse into a queue. vLLM's two inventions fix this:

  • Paged attention — the KV-cache (each session's working memory) is managed in small pages, like virtual memory, instead of big contiguous allocations. No fragmentation ⇒ far more concurrent sessions fit in the same VRAM.
  • Continuous batching — new requests join the running batch at the next token instead of waiting for the batch to finish. The GPU stays saturated ⇒ total throughput multiplies 10–20× vs a single-stream server, with modest per-user latency cost.

vLLM serves full-precision, AWQ, GPTQ or FP8 weights (note: not GGUF as its native path — a classic format mix-up), exposes an OpenAI-compatible endpoint plus Prometheus metrics, and does tensor-parallel across matched GPUs. TGI (Hugging Face) offers the same class of performance with tight HF-ecosystem integration; SGLang is the rising alternative, strong on structured output and prefix caching. For maximum single-model NVIDIA performance there's TensorRT-LLM — measurably fastest, dramatically more effort; it's what you graduate to when GPU count, not engineering time, is the binding constraint.

The application layer

  • Chat UI: Open WebUI is the de-facto standard front-end for self-hosted AI — multi-user with logins, model switching, built-in basic RAG (upload documents), works against any OpenAI-compatible backend. LibreChat is the alternative when you want a multi-provider, more "product-like" interface.
  • Orchestration: LangChain / LlamaIndex / Haystack shine for multi-step pipelines, many data sources and agent workflows. For a simple chat or one RAG flow, they're often overkill — plain OpenAI-client calls plus a vector-DB client are simpler to debug. Add the framework when the pipeline complexity demands it, not by default.
  • Vector DB (for RAG): start with Chroma (embedded, zero-ops) or pgvector (if you already run Postgres — one fewer system); move to Qdrant/Weaviate/Milvus when scale, filtering and multi-tenancy demand it.

Ops: the unglamorous layer that decides reliability

  • Containerize: official Docker images exist for Ollama, vLLM, TGI and Open WebUI; NVIDIA's container toolkit passes the GPU through. A docker-compose with ollama (or vLLM) + open-webui + a vector DB is the reference homelab/team deployment.
  • Pin versions. This ecosystem moves weekly; "latest" breaks things. Pin image tags and model versions; upgrade deliberately.
  • Monitor: vLLM exports Prometheus metrics (tokens/sec, queue depth, cache usage) — graph them from day one; nvidia-smi/DCGM for GPU health and VRAM headroom.
  • Model storage: models are 5–80GB each and multiply fast. Give them a dedicated disk/path, dedupe across tools (Ollama and LM Studio each keep their own copies), and clean old quants.

Three reference stacks

  • Solo developer: Ollama + Open WebUI (optional) + Chroma for RAG experiments. One box, one GPU, zero ceremony.
  • Team (≤50 users): vLLM (AWQ model) + Open WebUI with SSO + pgvector/Qdrant + Docker Compose + Prometheus. One 48–80GB GPU.
  • Production: vLLM fleet behind a load balancer (tensor-parallel where needed), dedicated embedding service, Qdrant cluster, k8s + autoscaling, full observability — plus small-model routing so cheap requests never touch the big GPU.

Common stack mistakes

  • Serving a team on Ollama. It queues; users blame the model. Concurrency is a vLLM job.
  • Format confusion. Downloading a GGUF for vLLM or an AWQ for LM Studio — check the expected format first (see the quantization guide).
  • Frameworks by default. Reaching for LangChain before the problem needs it adds a debugging layer, not capability.
  • No eval harness. Swapping models/engines without a fixed test set means judging by vibes. Keep ~50 known-answer questions.
  • Chat UI and engine competing for VRAM on one card — keep the UI on CPU or another node.