How can companies defend against malicious AI if the LLMs that should assist them in offensive security refuse to do so, gagged by ever-tightening safety measures? This paradox is at the heart of the debate after an attack on the Hugging Face platform reminded the world that any AI system, closed or open, can behave unexpectedly and potentially harmfully.

The principle is well established in cybersecurity: white-hat hacking uses exactly the same techniques as black-hat attackers to find and patch vulnerabilities before they are exploited. If models refuse to perform these operations because they are blocked by safety filters, companies are left without the tools to conduct effective penetration testing and red-teaming on their own systems. It's like forbidding network engineers from simulating a DDoS attack to test defenses: prevention is sacrificed in the name of superficial security.

The Hugging Face attack highlighted that even seemingly harmless models can generate unexpected outputs or behaviors, with consequences ranging from data leakage to remote control. Real security does not lie in prohibition, but in allowing equally capable models to explore the perimeter of vulnerabilities. This is where the issue of open-weight models comes in: Anthropic, in a recent regulatory message, acknowledged that restricting open models is also a way to protect closed providers from competition. The rhetoric of “safe open models” – those with built-in restrictions – raises an essential question: safe for whom?

The implications go beyond corporate clashes. If the models usable for defense activities are shackled by policies designed to prevent abuse, the entire on-premise and self-hosted AI ecosystem loses a fundamental piece: the ability to independently validate the robustness of one's own implementations. Those managing local deployments, perhaps in regulated sectors or air-gapped environments, need tools that do not depend on an external vendor's choices about what is permissible to test. Data sovereignty and security posture require unfettered models that can be used for offensive red-teaming, just as has been done for decades with traditional penetration tests.

Anthropic's stance and that of other players pushing for “tamed” models risks creating a two-speed market: on one side, cloud providers with reassuring closed models, on the other, attackers – whether criminal groups or nation-states – who will not hesitate to use unrestricted models, including those developed outside regulated circuits. In this scenario, companies investing in defense would find themselves disarmed, with testing tools less powerful than the very offensive tools they are supposed to counter.

The lesson from the Hugging Face attack is that the only effective defense comes from the ability to think and act like the adversary, without chains imposed by commercial interests disguised as technological paternalism. For those evaluating on-premise LLM deployment, this short-circuit is not a philosophical debate but an operational choice. It is exactly the kind of trade-off that AI-RADAR explores for those designing or migrating to on-premise stacks: data sovereignty and security posture require unrestricted models, not declarations of intent.