Addressing the LLM hallucination problem: The Copilex approach
With the increasing integration of AI-driven tools in the legal sector, the emergence of hallucinated references (i.e. fabricated citations or legal authorities generated by AI systems) has become a substantial concern for lawyers and organizations.
Recent articles from the Center for Internet and Society at Stanford Law School and 404 media, building on ongoing research by French scholar Damien Charlotin, point to a rise in legal filings containing non-existent legislation, regulations, or precedents.
Analyzing 114 U.S. matters in which lawyers’ filings contained AI‑caused inaccuracies, the authors found that the vast majority involved solo or small‑firm practitioners, relying not only on some general AI providers like ChatGPT or Anthropic’s Claude, but also specialized Legal AI assistants offered by well-established editors.
In all cases, the implications for practitioners encompass professional, financial, and reputational risks such as sanctions, erosion of client trust, and exposure to malpractice claims.
Maintaining the reliability and integrity of AI-generated legal content is of paramount importance in the context of legal work. It is, in fact the first concern expressed by our customers when we discuss the integration of Legal AI tools in their workflow.
In this article, we address the nature of hallucinated references, the limitations of current mitigation approaches, and Copilex’s approach to strengthening confidence in legal AI for our clients.
The LLM hallucination problem
Large Language Models (LLMs), widely deployed in today’s legal applications, generate text based on their training data rather than drawing from real-time legal databases. This means that LLMs only contain a partial collection of laws, regulations, precedents, limited to the data that was compressed into the model weights at the time of the training. They do not know anything about legal references that were not in the original training dataset and/or that arose after the model’s training. Yet, because they are trained to generate plausible textual answers, LLMs can produce references that appear legitimate but have no basis in actual law. This phenomenon is commonly referred to as “hallucination”.
While LLMs excel at summarization, drafting, and synthesis, they are not citation systems or truth databases. When a prompt calls for a specific legal authority and none is present in the model’s context, an LLM may still produce plausible but nonexistent citations, or misstate real ones.
This characteristic arises out from the probabilistic design of LLMs: they generate text that aligns with linguistic norms without the intrinsic ability to verify factual accuracy unless supplemented by up-to-date, authoritative information.
In legal practice, the consequences of such errors can obviously be substantial. Fabricated or misattributed sources can lead to improper legal advice, adverse case outcomes, regulatory and professional ethic breaches.
The limits of source-restricted approaches
To address these risks, many legal publishers have introduced LLM-based systems which they claim produce no hallucination because they operate exclusively within their proprietary databases.
While this approach effectively limits the frequency of hallucinations by confining the AI’s outputs to a verified documentary corpus, it does not completely solve the problem.
Indeed, although reduced, hallucinations are not eliminated. LLMs may still misquote, misconstrue, or misinterpret even vetted content. And even in well-funded Legal AI solutions from established legal publishers, lawyers can find hallucinated references, and unfortunately rely on them in their work if they do not pay attention.
Furthermore, these systems restrict users to materials selected by a single publisher, which may not encompass all relevant legal sources, particularly in matters requiring cross-jurisdictional or comprehensive research.
For practitioners engaged in complex or novel legal research, the balance between accuracy and source inclusivity remains unresolved.
The Copilex approach: Multi-modal Legal AI
Recent advancements in AI, particularly the development of agentic architectures that integrate multiple LLMs, web search tools, and verification modules, offer expanded potential for both accuracy and breadth.
At Copilex, we have adopted this direction from day-one, developing a multi-layered system segmented for the dynamic requirements of legal professionals.
Copilex’s Legal AI system relies on tools that notably (i) perform web searches in order to find reliable sources to support statements, with a specific focus on a vetted corpus of websites, selected by our team of lawyers as reliable sources, and (ii) verifies automatically all references to legal sources to make sure that not only the reference is correct, but also that the content matches exactly what the point it is supposed to support. And ultimately, remove points for which Copilex cannot find supporting evidence.
These features collectively reduce the prevalence of hallucinated references and empower legal professionals to audit the AI’s output efficiently.
However, no AI system can claim infallibility. While Copilex’s technologies substantially mitigate the risk of hallucinated references, occasional inaccuracies remain possible, and the evolving legal environment can present new challenges that can pose difficulties to our system.
We are transparent about these limitations with our users, and work continuously with them to refine our models and verification processes, incorporating user feedback and the latest research.
Nonetheless, even the most sophisticated Legal AI solutions like Copilex must be employed with expert oversight. Human judgment and diligence remain indispensable in all high-stakes legal work.
Accompanying organization in their adoption of Legal AI
Effective adoption of legal AI demands attention beyond technical deployment.
Sustained onboarding, regular training of practitioners, are essential. So are the adoption of internal AI use policies and governance rules within organizations, to ensure that teams make the best use of AI while maintaining the risk at the lowest level possible.
That is why we accompany our customers beyond the mere deployment of Copilex’s Legal AI assistant, helping them integrate AI responsibly, ensure teams understand both AI’s capabilities and boundaries, to guarantee true performance gains.
To learn more about how Copilex can support your organization in safely adopting legal AI, contact us or arrange a demonstration.