The Shadow of Generative AI in Courtrooms

In recent years, the legal world has witnessed a growing number of incidents where attorneys have been caught including citations, references, and even entire sections of text fabricated by generative artificial intelligence in their filings. These errors, often discovered by opponents or judges themselves, have led to severe reprimands and, in some cases, significant sanctions, highlighting a growing concern about the reliability of AI tools in critical professional contexts.

This issue is not merely one of negligence but raises fundamental questions about the integrity of information produced by automated systems. While generative AI offers undeniable advantages in terms of efficiency, its propensity to "hallucinate" – that is, to produce plausible but unfounded information – represents a concrete risk that requires careful management and verification, especially in sectors where precision is non-negotiable.

The Intense Confrontation in the Appellate Court

A recent episode, captured live-streamed during an appellate hearing at the Supreme Court of the State of New York, Appellate Division, offered a vivid insight into this problem. Justices Valerie Brathwaite Nelson and Hector LaSalle severely reprimanded an attorney and his opposing counsel for over twenty minutes, calling the situation "striking, concerning, disappointing, and saddening." Attorney Michael Sanders, representing plaintiff Judith Landberg in a lawsuit against the City of New York, had submitted a document containing at least three court cases and related citations that, upon careful review by the court, proved to be non-existent.

Justice Nelson pointed out that not only were the citations fictitious, but ten other cited cases appeared to misrepresent the law. Faced with the judges' pressing questions, Sanders admitted he was unprepared to discuss the specific citations, attempting to apologize. Justice LaSalle promptly interrupted, recalling Rule 3.3 A of the rules of professional conduct, which requires lawyers not to knowingly make false statements of fact or law to a tribunal and to correct any errors. The episode revealed not only the negligence of the attorney who produced the document but also the failure of the opposing counsel to verify, who was also reprimanded for not reporting the inaccuracies.

Implications for Professional Integrity

This emblematic case extends far beyond the single hearing, touching the core of trust and integrity that are fundamental pillars of the legal profession. The judges expressed deep disappointment, highlighting how the judicial system relies on the loyalty and diligence of attorneys. The production of false citations, although not explicitly attributed to AI by the judges in court, fits into a broader context of increasing reliance on artificial intelligence tools for drafting legal documents, where the temptation to fully delegate research and verification can lead to disastrous consequences.

For companies and professionals evaluating the adoption of Large Language Models (LLM) to support decision-making processes or the production of critical content, this episode serves as a warning. An LLM's ability to generate coherent and convincing text does not equate to its factual accuracy. It is imperative to implement robust verification pipelines and significant human oversight to validate outputs, especially in areas where errors can have severe legal, financial, or reputational repercussions. Trust in AI tools must be built on a solid foundation of validation and transparency.

Beyond the Specific Case: The Challenge of AI Reliability

The "growing epidemic" of fake citations, as Justice LaSalle called it, is not an isolated problem for the legal sector. It represents a broader challenge for the responsible adoption of artificial intelligence in every professional field. The ability of LLMs to "hallucinate" is an intrinsic characteristic that must be managed through technical and procedural strategies. This includes the use of fine-tuning techniques on proprietary and verified data, the implementation of retrieval-augmented generation (RAG) systems to anchor responses to authoritative sources, and the integration of human review cycles.

For organizations considering the deployment of LLMs on-premise or in hybrid environments, the issue of reliability takes on an additional dimension. While direct control over infrastructure and data can offer greater guarantees in terms of data sovereignty and security, the responsibility for validating outputs falls entirely on the organization. It is crucial to invest in frameworks and processes that ensure the quality and accuracy of generated information, ensuring that AI tools support human intelligence and are not an uncritical substitute for professional diligence.