How to build a private ChatGPT for your company

For many companies the goal is simple: the usefulness of ChatGPT, but answering from internal knowledge and without sending confidential data to a third party. That is entirely achievable today with open-weight models — and it has become the default path for regulated sectors. Here is the full architecture, the decisions that matter, and the failure modes that sink these projects.

The four building blocks

1. Model — an open-weight LLM (Llama, Mistral, Qwen, Gemma…) sized to your hardware and task.
2. Serving engine — Ollama for a prototype or small team; vLLM/TGI for production concurrency. Both expose an OpenAI-compatible API.
3. RAG layer — ingestion, chunking, embeddings and a vector DB, so the assistant answers from your knowledge and cites sources.
4. Chat UI + access control — an open chat front-end (Open WebUI, LibreChat) wired to SSO and per-team permissions.

Choosing the model (and the embedding model)

The counterintuitive truth of grounded assistants: with good RAG, retrieval quality moves answer quality more than model size. A modern 8–14B model quoting the right paragraph beats a 70B guessing from a bad one. Practical selection logic:

• 8–14B (Q5/Q8 on a 24GB card) — the right starting point for document Q&A, summaries, internal search. Fast, cheap, snappy under concurrency.
• 30–70B (48–80GB) — earns its cost for multi-step reasoning, drafting quality, code, and nuanced multilingual output. Upgrade when evaluation shows the small model is the bottleneck — not before.
• Language check: if your users work in Italian (or any non-English language), test candidate models in that language — instruction-following quality varies far more across languages than English benchmarks suggest. Qwen and Gemma families are often strong multilingually; verify on your own prompts.
• License check: "open-weight" ≠ do-anything. Most licenses (Llama, Gemma) allow internal business use, but some restrict fields of use or require attribution. Have someone actually read it before the pilot becomes infrastructure.
• The forgotten choice — the embedding model: retrieval lives or dies on it. Pick a strong multilingual embedder if your documents mix languages, keep it versioned (changing it means re-indexing everything), and prefer one you can also self-host — a "private" assistant that calls a cloud embedding API is leaking document contents.

The RAG pipeline, honestly

Reference flow: documents → parsing → chunking → embeddings → vector DB; at query time: question → retrieve top chunks (filtered by the user's permissions) → build prompt → local LLM → answer with citations. Each stage has a failure mode that will find you:

• Parsing is the swamp. Enterprise knowledge lives in PDFs with tables, scanned contracts, PowerPoints and wiki export formats. Bad text extraction poisons everything downstream — budget real time here, use OCR where needed, and spot-check what the parser actually produced.
• Chunking: split by structure (headings, paragraphs) not fixed character counts; keep chunks self-contained (a table row without its header is noise); attach metadata (source, date, department) for filtering.
• Retrieval quality: pure vector similarity misses exact terms (product codes, article numbers) — hybrid search (vector + keyword/BM25) plus a reranker is the standard fix and often the single biggest quality jump in the whole system.
• Citations are non-negotiable. Users forgive a wrong answer with a source they can check; they abandon a confident hallucination. Grounding + visible citations is your hallucination policy.
• Freshness: documents change. Schedule re-indexing, handle deletions (a revoked document must leave the index, not just the UI), and log index versions for audit.
• Vector DB: Chroma or pgvector to start (pgvector is ideal if you already run Postgres); Qdrant/Weaviate/Milvus when scale and filtering demands grow. This choice is rarely the bottleneck — don't over-engineer it.

The #1 enterprise failure: permission leakage

The moment you index the shared drive, you index the salary review folder somebody misfiled there. RAG then happily quotes it to anyone who asks the right question. ACL-aware retrieval — filtering candidate chunks by the requesting user's permissions at query time — is a launch blocker, not a roadmap item. Practical rules: mirror source-system permissions into chunk metadata at ingestion; filter at retrieval (not in the prompt — the model cannot be trusted to withhold what it has seen); re-sync permissions on a schedule; and start from an allowlist of clean, deliberately-chosen document sets rather than "index everything and restrict later" — later never comes.

Security: the threat model in brief

• Indirect prompt injection: a malicious instruction hidden inside a document ("ignore previous instructions and…") gets retrieved and executed. Mitigate: treat retrieved text as data (clear prompt separation), strip active content at ingestion, and never give the assistant tools that act (send email, call APIs) on retrieved content without human confirmation.
• Logging vs privacy: you want full query logs for audit and quality — but logs of an internal assistant contain PII and secrets by nature. Encrypt them, restrict access, set retention, and say so in your internal privacy notice (GDPR applies to your own employees too).
• Model supply chain: download weights from the official source, verify checksums, and pin versions. A model file is an artifact like any other dependency.
• Isolation: the serving box needs no internet access at runtime — that's the on-prem advantage; actually enforce it with egress rules.

Hardware by team size

Concurrency drives sizing more than headcount: 200 employees rarely means 200 simultaneous queries. Measure the pilot's real concurrency before buying the big card — and remember the KV-cache scales per concurrent session (see the VRAM guide).

Cost vs just buying subscriptions

Honest framing beats enthusiasm here. Per-seat AI SaaS at small scale is hard to beat on price: below roughly 30–50 users, a self-hosted stack (hardware + the engineer-hours that are its real cost) usually costs more per user. The self-hosted case is won on different axes: data control (nothing leaves; auditors and DPOs sleep), unlimited usage (no per-seat or per-token anxiety, so usage actually grows), customization (your RAG, your models, your workflows), and — at larger scale — raw cost, where one fixed rig replaces hundreds of subscriptions. Run the numbers with our cost guide's TCO method and present both scenarios; the strongest deployments are chosen with eyes open, not sold on hype.

What it won't do (set expectations early)

• It knows your documents, not yesterday's news — open-weight models have a knowledge cutoff, and your assistant is deliberately offline.
• It will still occasionally be wrong — grounding + citations manage this; they don't eliminate it. Ban it from binding decisions (HR, legal, pricing) in policy.
• It is not maintenance-free: model updates, index refreshes, permission syncs and quality evaluation are a recurring job — a realistic budget is a fraction of one engineer, permanently.

A pragmatic rollout

1. Pilot (weeks 1–4): one GPU, an 8–14B model, Ollama + Open WebUI, RAG on one clean, well-permissioned document set. One friendly team.
2. Evaluate honestly: build a test set of ~50 real questions with known answers; measure grounded-answer rate and citation accuracy before and after every change. This tiny harness will guide every future decision (bigger model? better retrieval?).
3. Productionize: move serving to vLLM, add SSO, ACL-aware retrieval, logging with retention, and an egress-blocked host.
4. Scale deliberately: expand document sets one at a time (permissions first), grow model/GPU when the eval says the model is the bottleneck, add small-model routing when load grows.