On July 25, 2023, Discord admitted that a bug in its AI-powered moderation system had wrongfully banned hundreds of users for sharing harmless images. The anomaly was not a one-off: according to the company, the malfunction had been affecting accounts since May and spiked over the weekend before the fix, with more than 200 additional users impacted in a matter of days. This detail raises questions not only about algorithmic robustness, but also about the governance of automated systems running on centralized cloud infrastructure, far from the oversight of those who use them.

Automated moderation often relies on Large Language Models (LLMs) trained to detect prohibited content. These models, running in the cloud, process hundreds of thousands of messages per second. Discord’s bug illustrates what happens when a false positive goes off silently: for months the system misclassified innocuous images as offensive, without the error being caught. The absence of an immediate feedback loop from users — many of whom took time to realize the ban was unjust and even longer to obtain a review — magnified the damage to trust.

Who benefits from such an architecture? Large platforms and cloud providers, which can scale moderation while keeping operational costs low. The users and communities pay the price, facing opaque disciplinary actions with no transparency or real-time recourse. Yet there is also a lesson for organizations that handle sensitive data — law firms, financial institutions, healthcare providers — for which a false positive is not just a social inconvenience but can trigger serious legal or reputational fallout. In these settings, the cloud-only model leaves the organization at the mercy of algorithmic decisions over which it has no control.

This is where the discourse on data sovereignty and on-premise deployment gains concrete ground. Running LLM inference for moderation on local infrastructure means retaining the ability to set policies autonomously, audit every decision through detailed logs, and intervene the moment an anomaly appears. It is not without costs: managing adequate compute infrastructure requires hardware investment and ongoing maintenance. Still, for those weighing a move to on-premise deployment, significant trade-offs exist between control and Total Cost of Ownership (TCO), which AI-RADAR explores in detail through the analytical frameworks available at /llm-onpremise.

Structurally, the Discord incident signals that the industry has yet to solve the challenge of algorithmic transparency. As AI mediates more social interactions, audit mechanisms and the ability to run models locally to verify their behavior become increasingly critical, reducing reliance on centralized black boxes. In the meantime, the platform has fixed the bug and promises more rigorous testing processes. The open question remains: how much time and how many more false positives will it take before the very architecture of automated moderation is rethought?