The UK's Financial Conduct Authority is posing an uncomfortable question for the tech industry: should Large Language Models be regulated directly? Sheldon Mills, an FCA executive director, has stated that the current rulebook must evolve as tools like ChatGPT, Claude and Gemini begin to shape consumer financial decisions.

This is not a one-off. The European Union has already charted a course with the AI Act, while US federal agencies are exploring transparency obligations for algorithms. But Mills' point carries specific weight: it's not about standard software, but opaque statistical models whose behavior emerges from billions of parameters and training data that is not always accessible.

Audit and accountability: the challenge for deployers

For those running LLMs in-house, regulation touches raw nerves. When a model's output influences a credit decision, an investment or an insurance policy, who is accountable? The base model provider, the company that fine-tuned it, or the team that wrote the prompts? In a cloud setting, the chain of responsibility frays: logs are scattered, weight access is indirect and an independent audit trail is often missing.

On-premise deployment – or tightly controlled hybrid setups – introduces a layer of auditability that public cloud can't always guarantee. It's not a silver bullet, but it lets you know exactly where data resides, how prompts are processed and who can access the models. In regulated sectors like finance, this traceability is becoming a requirement, not a luxury.

Beyond cloud: sovereignty and total cost

The FCA's conversation fits into a wider reassessment of compliance costs. Moving an LLM on-premise comes with upfront expense: specialized hardware, high-VRAM GPUs, serving pipelines and maintenance. Yet when privacy violation fines or loss of data control can dwarf the savings of a cloud API, the TCO calculus shifts dramatically. And it's not just about risk: the ability to run inference without sending sensitive data to third parties is a competitive advantage in client relationships.

Of course, direct LLM regulation doesn't mandate on-premise. But it makes it harder to settle for black-box solutions, where the model is a remote service and compliance rests on contractual trust. Mills did not propose specific rules, but his intervention signals that regulators are starting to look inside the black box.