On the surface, it looks like productivity. In reality, it's the latest incarnation of a problem as old as corporate IT: shadow IT. Every organization lives with two technology environments: the one IT built, documented, and oversees, and the one employees assembled on their own to get things done faster. The difference today is that the second environment now hosts language models capable of chewing through confidential documents, business strategies, and customer data.

When a marketing team uploads a financial report to an external generative AI service because “it's convenient and there's no time to open a ticket,” we're witnessing not an exception, but the rule. Security teams, accustomed to chasing unauthorized apps, suddenly face a structural threat: data no longer leaks only via Excel spreadsheets attached to personal webmail, but through prompts and improvised fine-tuning.

This isn't about malice. Employees seek tools that work instantly, and cloud-based services deliver a frictionless experience. The problem surfaces when that immediacy sidesteps data governance. In Europe, where GDPR mandates a clear chain of responsibility, every request that transits a server outside the organization's control becomes a concrete legal risk.

This landscape accelerates a reasoning that until recently was the domain of a few highly structured organizations: if data is the asset to protect, then inference must stay in-house. Not just for compliance, but to avoid gifting third parties with the memory of everything the company knows.

Self-hosting open-source models, served through frameworks like vLLM, Ollama, or TGI on bare-metal or virtualized on-premise infrastructure, moves from a tinkerer's exercise to an architectural safeguard. It doesn't erase shadow IT, but shrinks its attack surface: if employees can access a corporate LLM with the same performance and no friction, the urge to seek external shortcuts drops sharply. And that LLM, running on IT-managed nodes, brings credentials, logs, and token flows back under control.

Of course, on-premise deployment isn't a magic wand. It demands skills, GPUs with enough VRAM, TCO evaluations that pit CapEx and energy consumption against cloud OpEx. But it changes the nature of the trade-off: it's no longer just a cost comparison, but a strategic positioning choice around data sovereignty. It's no surprise that sectors like finance, manufacturing, and public administration are moving their proof-of-concepts precisely toward self-hosted architectures, even evaluating air-gapped solutions for extreme scenarios.

The proliferation of quantized models in INT8 or FP16, designed to run on less extreme hardware, widens the audience. You don't necessarily need a cluster of A100s to offer a decent internal service. This shifts the barrier: IT can't just say “we have no budget,” because an internal LLM today can be built with resources many companies already own, provided they have the right architecture.

In the end, the lesson of shadow IT in the Large Language Models era is simple: you don't fight it only with policies and sanctions. You shrink it by providing a powerful, equally-capable alternative inside the perimeter. Those who grasp this begin to see on-premise not as a legacy hangover, but as a freedom choice.