What you need

  • A Linux host (or Windows + WSL2) with an NVIDIA GPU, 24GB VRAM recommended
  • Docker + NVIDIA Container Toolkit installed (the toolkit is what passes the GPU into containers)
  • ~30GB free disk for the model cache

Step 1 — Verify GPU passthrough (2 min)

$ docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If you see your GPU listed, Docker↔GPU wiring works. If you get "could not select device driver", install the NVIDIA Container Toolkit (nvidia-ctk runtime configure) and restart Docker.

Step 2 — Choose the model: AWQ, not GGUF (2 min)

The classic mistake at this step: vLLM's native path serves full-precision, AWQ, GPTQ or FP8 weights — not the GGUF files Ollama uses (see the quantization guide). For a 24GB card, a ~27B-class AWQ build (≈16GB weights) leaves healthy room for the KV-cache. We'll use a Qwen3.6-27B AWQ build as the example — swap in your preferred model's AWQ repo from Hugging Face.

Step 3 — Launch vLLM (10 min incl. download)

$ docker run -d --name vllm --gpus all -p 8000:8000 \ -v vllm_hf_cache:/root/.cache/huggingface \ --restart unless-stopped \ vllm/vllm-openai:latest \ --model Qwen/Qwen3.6-27B-Instruct-AWQ \ --max-model-len 16384 \ --gpu-memory-utilization 0.92

The three flags that matter:

  • --max-model-len — the max context per request. vLLM pre-plans KV-cache memory around it: bigger = fewer concurrent sequences fit. 16K is a good team default.
  • --gpu-memory-utilization — fraction of VRAM vLLM claims (default 0.9). Raise to 0.92–0.95 on a dedicated card; lower it if the GPU also drives your display.
  • -v vllm_hf_cache — persists the downloaded model, so container recreation doesn't re-download 16GB.

Watch startup with docker logs -f vllm — you're ready when it prints the Uvicorn "running on 0.0.0.0:8000" line.

Step 4 — Test the API (2 min)

$ curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "Qwen/Qwen3.6-27B-Instruct-AWQ", "messages": [{"role": "user", "content": "Reply with one word: works?"}]}'

A JSON answer back = your team now has a private OpenAI-compatible endpoint. Any OpenAI SDK works against it by setting base_url to http://SERVER-IP:8000/v1 (api_key can be any string, or enforce one with --api-key).

Step 5 — Add Open WebUI with docker-compose (10 min)

For a durable setup, move both services into one compose file (docker-compose.yml):

services: vllm: image: vllm/vllm-openai:latest command: --model Qwen/Qwen3.6-27B-Instruct-AWQ --max-model-len 16384 --gpu-memory-utilization 0.92 ports: ["8000:8000"] volumes: ["vllm_hf_cache:/root/.cache/huggingface"] restart: unless-stopped deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] open-webui: image: ghcr.io/open-webui/open-webui:main ports: ["3000:8080"] environment: - OPENAI_API_BASE_URL=http://vllm:8000/v1 - OPENAI_API_KEY=local volumes: ["open_webui:/app/backend/data"] restart: unless-stopped depends_on: [vllm] volumes: vllm_hf_cache: open_webui:
$ docker rm -f vllm # remove the step-3 container first
$ docker compose up -d

Open http://SERVER-IP:3000, create the admin account, and invite the team. Both services now restart with the host automatically.

Step 6 — Monitor and size for concurrency (ongoing)

  • vLLM exports Prometheus metrics at http://SERVER-IP:8000/metrics — the ones to watch are running/waiting requests (queue building = saturation) and KV-cache usage. Graph them in Grafana from day one.
  • Concurrency math: KV-cache memory ≈ per-token cache × context × concurrent sequences. If users report queuing, either lower --max-model-len (frees cache → more concurrent sequences) or move to a bigger/second GPU. The full math is in the VRAM guide.
  • Keep an eye on power: a serving GPU runs hot 9-to-6 — a power limit (nvidia-smi -pl) trades ~5–10% speed for 25–30% energy.

Troubleshooting

  • CUDA out of memory at startup — the model + planned cache don't fit: lower --max-model-len (first lever), lower --gpu-memory-utilization, or pick a smaller AWQ model.
  • "Cannot load GGUF" / format errors — you pointed vLLM at a GGUF repo. Use the AWQ (or GPTQ/FP16) build of the model instead.
  • First request is slow — normal: CUDA graph warmup. Judge latency from the second request onward.
  • WebUI sees no model — check OPENAI_API_BASE_URL points at http://vllm:8000/v1 (service name, not localhost) and that the vllm container is healthy in docker compose ps.

Where to go next