What you need

  • Python 3.11+ and basic FastAPI familiarity
  • An LLM: Ollama locally (any 7B+ model works) or an API key (DeepSeek/Gemini-class pricing keeps this nearly free)
  • No GPU required — embeddings run fine on CPU at this scale

Step 0 — Understand the pipeline (5 min)

RAG (retrieval-augmented generation) is a search engine bolted to an LLM. Everything else is detail:

documents → chunks → embeddings → ChromaDB ↑ question → embedding → top-k similar chunks ┘→ prompt → LLM → cited answer

The quality of the final answer is decided almost entirely on the left side of that diagram. In months of production logs, nearly every bad answer traced back to bad retrieval — the wrong chunks — not to a weak LLM. Invest your effort in chunking and retrieval before you shop for a bigger model. (For when RAG is the right tool at all, see RAG vs fine-tuning.)

Step 1 — Install and chunk (15 min)

$ pip install chromadb sentence-transformers fastapi uvicorn httpx

Chunking rule that survived production: split on structure first, size second. Paragraph and heading boundaries carry meaning; blind fixed-size windows cut sentences in half and poison retrieval. Target ~300–500 words per chunk with a 10–15% overlap:

def chunk_text(text: str, target_words=400, overlap_words=50): paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()] chunks, current, count = [], [], 0 for p in paragraphs: words = len(p.split()) if count + words > target_words and current: chunks.append("\n\n".join(current)) # keep the tail as overlap so context bridges the cut tail = " ".join(" ".join(current).split()[-overlap_words:]) current, count = [tail, p], overlap_words + words else: current.append(p); count += words if current: chunks.append("\n\n".join(current)) return chunks

Step 2 — Embed and index into ChromaDB (15 min)

For Italian (or any non-English) content, use a multilingual embedding model — the default English MiniLM quietly ruins cross-language retrieval. paraphrase-multilingual-MiniLM-L12-v2 is small, fast on CPU, and what this site runs:

import chromadb from chromadb.utils import embedding_functions client = chromadb.PersistentClient(path="./chroma_store") embed_fn = embedding_functions.SentenceTransformerEmbeddingFunction( model_name="paraphrase-multilingual-MiniLM-L12-v2") collection = client.get_or_create_collection( name="knowledge", embedding_function=embed_fn, metadata={"hnsw:space": "cosine"}) def index_document(doc_id: str, title: str, url: str, text: str): chunks = chunk_text(text) collection.upsert( ids=[f"{doc_id}:{i}" for i in range(len(chunks))], documents=chunks, metadatas=[{"title": title, "url": url, "doc_id": doc_id} for _ in chunks])

Two production notes. First, upsert with deterministic IDs (doc_id:chunk_n) makes re-indexing idempotent — you can re-run ingestion without duplicates. Second, put the source URL in the metadata now: citations are impossible to retrofit if you didn't store provenance.

Step 3 — Retrieve and build the prompt (15 min)

def retrieve(question: str, k: int = 5): res = collection.query(query_texts=[question], n_results=k) hits = [] for doc, meta, dist in zip(res["documents"][0], res["metadatas"][0], res["distances"][0]): if dist < 0.65: # similarity floor — tune on YOUR data hits.append({"text": doc, "title": meta["title"], "url": meta["url"]}) return hits PROMPT = """Answer the question using ONLY the sources below. If the sources do not contain the answer, say so — do not invent. Cite sources as [n]. SOURCES: {sources} QUESTION: {question}"""

The similarity floor is the detail most tutorials skip and the one that matters most in production. Without it, an off-topic question still returns the k "least dissimilar" chunks, and the LLM will gamely hallucinate an answer from irrelevant context. With it, you can honestly answer "I don't have that information" — which is what separates a trustworthy internal tool from a liability.

Step 4 — The FastAPI endpoint (15 min)

import httpx from fastapi import FastAPI from pydantic import BaseModel app = FastAPI() class Ask(BaseModel): question: str @app.post("/ask") async def ask(body: Ask): hits = retrieve(body.question) if not hits: return {"answer": "No relevant sources found.", "sources": []} sources = "\n\n".join(f"[{i+1}] {h['title']}\n{h['text']}" for i, h in enumerate(hits)) async with httpx.AsyncClient(timeout=120) as cx: r = await cx.post("http://localhost:11434/api/chat", json={ "model": "qwen3.6:27b", # or any Ollama model / swap for an API call "messages": [{"role": "user", "content": PROMPT.format(sources=sources, question=body.question)}], "stream": False}) return {"answer": r.json()["message"]["content"], "sources": [{"title": h["title"], "url": h["url"]} for h in hits]}

Run it with uvicorn main:app and you have a working private chatbot: question in, grounded answer plus clickable sources out. Swapping Ollama for an API provider is a five-line change to the HTTP call — the RAG layer doesn't care who generates.

Step 5 — The production lessons (the part nobody tells you)

Everything below comes from running this stack live on the public internet:

  • Index incrementally, not from scratch. Re-embedding the whole corpus on every update is the first thing that stops scaling. Embed only new/changed documents at ingest time (our pipeline indexes each new article the moment it's created).
  • Bots will find your endpoint and hammer it. Every /ask call costs real compute (retrieval + LLM). Within weeks, crawlers were burning ours with junk questions. Minimum defence: rate limiting per IP, Disallow in robots.txt, and never exposing the endpoint unauthenticated if it's internal.
  • Log question + retrieved chunks + answer. This triple is your debugging goldmine and your evaluation set. When an answer is wrong, the log tells you instantly whether retrieval or generation failed.
  • Answer language ≠ document language. With a multilingual embedder, Italian questions retrieve English chunks and vice versa — great for coverage, but tell the LLM explicitly which language to answer in, or it will mirror the sources.
  • ChromaDB is enough for longer than you think. Up to hundreds of thousands of chunks on one box, it just works. Don't reach for a distributed vector database until you've measured a real limit.

Where to go next

Two upgrades earn their complexity when basic RAG plateaus: hybrid retrieval (add keyword/BM25 alongside vectors — exact names and codes are where pure embeddings fail) and a reranker (a small cross-encoder reordering your top-20 into a sharper top-5). Add them only after logging shows retrieval misses; both are pure additions to this architecture.