A Critical Flaw in M365 Copilot

Microsoft recently patched a vulnerability rated as critical in its M365 Copilot AI platform. The discovery, revealed by the researchers who reported it to the company, highlighted the potential for an exploit capable of retrieving two-factor authentication (2FA) codes and other sensitive data directly from emails accessible to Copilot. This incident underscores a fundamental challenge that Large Language Model (LLM) providers face: the difficulty of preventing their products from complying with malicious requests that reveal confidential data.

The nature of this vulnerability is not a simple software bug but is rooted in an intrinsic characteristic of how LLMs operate. For organizations considering the deployment of AI solutions, understanding these limitations is crucial for ensuring the security and sovereignty of their data, whether in the cloud or in self-hosted environments.

The Technical Details of the Threat

The root cause of this vulnerability lies in the inability of AI bots to distinguish between instructions provided directly by users and those cleverly snuck into third-party content. This content might be summarized by the models, used to draft responses, or to perform other actions on behalf of the user. Without an effective way to secure this crucial boundary, Microsoft and other LLM developers are forced to implement complicated and often ad hoc “guardrails” designed to rein in the consequences of this inherent gullibility of the models.

An example of such guardrails is preventing Copilot and most other LLMs from submitting web forms or sending emails, actions that could be used to exfiltrate data. However, attackers have found ways to bypass these protections. One method involves using markup language, which allows for adding formatting elements such as headings, lists, and links to text without the need for complex HTML tags. Another approach involves wrapping sensitive data inside specific HTML tags, such as <img> and <form>. In both cases, a web request containing the sensitive data hits the attacker’s web server, where the secret information is captured in logs, completing the exfiltration.

Implications for Data Sovereignty and On-Premise Deployments

This vulnerability highlights a critical issue for companies evaluating LLM adoption, especially those prioritizing data sovereignty and control. Although Copilot is a cloud platform, the nature of the problem—the model's inability to discern instructions—is universal for LLMs. This means that even in an on-premise or air-gapped deployment, where the infrastructure is entirely under the company's control, the model's inherent vulnerability to manipulation remains a significant concern.

For CTOs, DevOps leads, and infrastructure architects, this scenario emphasizes the need to consider not only the security of the physical or virtual infrastructure but also the robustness and resilience of the models themselves. Managing the Total Cost of Ownership (TCO) for AI/LLM workloads must include investments in advanced security strategies that go beyond simple guardrails, exploring techniques such as continuous red teaming, rigorous validation of inputs and outputs, and the implementation of contextual security layers that can filter or block suspicious requests before they reach the model or after they exit it. For those evaluating on-premise deployments, AI-RADAR offers analytical frameworks on /llm-onpremise to assess specific trade-offs and constraints related to these aspects.

The Ongoing Challenge for LLM Security

The discovery of this vulnerability in M365 Copilot is a reminder that LLM security is a rapidly evolving and challenging field. The difficulty of distinguishing between benign and malicious instructions represents a fundamental obstacle for all model providers. Current solutions, based on reactive guardrails, are often insufficient and can be bypassed with relatively simple techniques.

This scenario compels companies to adopt a proactive approach to LLM security, integrating risk assessment from the earliest stages of design and deployment. The need for granular control over data, coupled with the ability to monitor and mitigate attacks based on prompt engineering or instruction injection, will become increasingly critical as LLMs are integrated into sensitive business processes. Research and development of new security architectures and validation methodologies will be essential to build truly reliable and secure AI systems.