The challenge of web access in on-premise agentic pipelines

When building an AI agent that interacts with the outside world, the ability to search and read web pages quickly becomes essential. For those adopting local, on-premise stacks, however, using commercial APIs like Tavily, Serper, or Firecrawl introduces external dependencies, recurring costs, and potential data exfiltration. The solution? An entirely self-hosted setup combining two open-source tools: a metasearch engine (SearXNG) and a page extractor with optional headless browser (Scrapling + Trafilatura).

Search without leaving home: SearXNG

SearXNG is a metasearch engine you can run in a Docker container, exposing a JSON endpoint. A simple GET request with the query and json format returns a list of results. The user kept it simple, returning only title, URL, and snippet (a short description, not page content). One important detail: you must add "json" to the search.formats list in SearXNG's settings.yml, otherwise the endpoint won't respond. The user advises against using public SearXNG instances for programmatic use: running your own keeps you in control.

From snippet to full text: Scrapling and Trafilatura

Snippets aren't enough for an agent that must reason about content. To extract full text, the setup uses Scrapling with two paths: a "fast" path (fetch without a browser, impersonating Chrome) for normal pages, and a "stealth" path with a real headless browser that tries to bypass Cloudflare protections and anti-bot challenges. If the page remains blocked, the system flags a partial block rather than returning fake content. Once HTML is obtained, Trafilatura converts it to Markdown, preserving links and tables – a format far more palatable for an LLM. There's also a visible-text fallback if Trafilatura under-extracts. PDFs are handled via pypdf; pages with CAPTCHAs are detected and marked; an SSRF guard prevents requests to internal/private addresses. An optional summarization step can shrink very long pages before passing them to the main model.

Honest trade-offs and known limits

The approach has clear compromises. The stealth browser path is slow and best kept as a fallback. SearXNG's search quality depends on configured upstream engines and rate limits – it's not a match for a paid service, but it's good enough for many use cases. The setup isn't impenetrable: queries still go through external search engines (SearXNG forwards the request), so it's not "zero third-party contact," but you avoid sharing API keys or paying per call. The author also asks for lighter alternatives to a headless browser for tough pages, signaling the community is looking to reduce friction.

What it means for on-premise adopters

This combination shows it's possible to build an AI agent with internet access without surrendering data sovereignty or racking up API bills. In a context where TCO and infrastructure control matter, tools like SearXNG and Scrapling fit into an increasingly mature self-hosting ecosystem. AI-RADAR has been observing how local-first solutions are evolving – not just for LLM inference, but for the auxiliary services that feed agents. The fact that such a framework can run entirely in Docker, without external keys, reduces attack surface and simplifies deployment in air-gapped or GDPR-regulated environments.

The search quality challenge remains, but the direction is clear.