2.2.1. Transparency, Public Trust, and Acceptance

Many GenAI systems used today are often considered “black boxes.” (Cheng and Liu, Reference Cheng and Liu2024) This is because their decision-making processes are complex and not easily interpretable, with the systems giving responses based on an amalgamation of vast amounts of often unidentifiable sources, leading to the unexplainability of results (Yampolskiy, Reference Yampolskiy2019). As explained by Yampolskiy, “[t]o explain…a decision, which relies on literally billions of contributing factors, AI has to either simplify the explanation and so make the explanation less accurate/specific/detailed or to report it exactly but such an explanation elucidates nothing by virtue of its semantic complexity, large size and abstract data representation” (Yampolskiy, Reference Yampolskiy2019).

This is particularly problematic in judicial decision-making processes, which require clear thought pathways that consider ethics, reasonability, and fairness as core foundations. The lack of transparency in current GenAI-supported judicial decision-making can undermine accountability and due process, as judges, legal experts, and parties to judicial processes, may find it puzzling to understand and challenge how a specific decision was reached, and whether biases or errors were introduced into the final rationale provided.

Due to the complexity of some human circumstances and the nuanced understanding of the context required for fair and ethical legal judgments, GenAI systems may struggle to reliably inform decisions with significant legal consequences. Human oversight or involvement may remain crucial to ensure both the fairness of the outcome and the public’s perception of fairness (Morison and Mclnerney, Reference Morison, Mclnerney, Turenne and Moussa2024). Relying on GenAI systems for judicial decisions, either in whole or without proper safeguards, could reduce such human articulable intercession in the legal process, potentially reducing the transparency of justice served, thus bringing a potential consequent reduction in fairness, quality, and trust in the judicial system. The “black box” feature of GenAI’s processing is therefore impactful towards both the due transparency of legal reasoning that judicial processes owe to its parties, as well as to public trust. In modern democracies and jurisdictions with a strong rule of law, the legitimacy of a judiciary is dependent on public will, understanding, and in many ways, approval. Undermining this would jeopardize the integrity and foundations of the system itself.

GenAI implementation into judicial decision-making could therefore lead to a lack of explainability in process and outcome, which could (i) undermine transparency, reasonability, and fairness and (ii) erode public trust and acceptance of legal decisions, having adverse effects on how justice is administered, delivered, and perceived (Bühlmann and Kunz, Reference Bühlmann and Kunz2011; Versteeg and Ginsburg, Reference Versteeg and Ginsburg2017),^. Indeed, as U.S. Supreme Court Chief Justice Roberts noted, “[a]t least at present, studies show a persistent public perception of a ‘human-AI fairness gap,’ reflecting the view that human adjudications, for all of their flaws, are fairer than whatever the machine spits out” (Roberts Jr., Reference Roberts2023).

2.2.2. Fairness, bias amplification, and precedent interpretation

GenAI systems can inadvertently perpetuate biases when these are present in the training data (Wachter et al., Reference Wachter, Mittelstadt and Russell2021). The problem of bias undermining fairness in GenAI stems from the way these systems are trained and the data they learn from. GenAI systems are trained on vast datasets retrieved from various sources—which can be as indiscriminate as the internet, like it is for ChatGPT (the GenAI language model utilized in the case studies discussed below in this article); or they can be specific and targeted datasets, like it would be if a jurisdiction were to incorporate a GenAI that included only a select group of cases and decisions from a specific court or tribunal (Wachter et al., Reference Wachter, Mittelstadt and Russell2021) (Wachter, Reference Wachter2022; Starke et al., Reference Starke2022). These foundational databases may contain inherent biases that can manifest in the form of racial, gender, ethnic, or cultural leanings, and can extend to other biases concerning socioeconomic status, religion, and more (Juzgado Civil Transitorio de San Juan de Miraflores, n.d.). Additionally, GenAI models and LLMs can learn patterns and associations from the data during training, which can establish a direct-learning and reiterative amplification of specific biases.Footnote ⁶ Even if the data were to be collected solely from previous judicial decisions, those decisions could have still been made with biased leanings—whether contemporary or from historical inertia—that may not be fully captured, understood, or disclosed by an LLM when using GenAI to support judicial decision-making.

If the training data contains biased data points, the models inadvertently amplify, propagate, and even obfuscate those biases in their generated outcome, leading to AI systems producing content that may reflect and reinforce societal biases (Floridi and Cowls, Reference Floridi and Cowls2022). Moreover, GenAI lacks a true understanding of the outcomes they generate: Responses are generated based on patterns in the data the models were trained on, and the model tuning performed by the developers, but these systems lack genuine comprehension or critical thinking abilities.Footnote ⁷ As the GenAIs to be deployed in judicial decision-making are in some ways limited to language emulation, this can result in the systems generating biased content without recognizing its implications. Additionally, the use of GenAI could exacerbate negative feedback loops given that the content that models create is increasingly difficult to distinguish from real data or non-synthetic data (Remolina, Reference Remolina2024), even if the systems were to be adjusted for case validity or precedential authority. For example, should content produced by AI be recycled into the training datasets for subsequent models (a process that takes place in open-sourced networks such as ChatGPT) when there is a risk of perpetuating and amplifying existing biases, this could result in a cascading effect, where each new generation of AI is increasingly more prejudiced than the last (Nicoletti and Bass, Reference Nicoletti and Bass2023).Footnote ⁸

These inherent challenges in GenAI systems raise ethical concerns about the fairness of decisions, especially in cases involving sensitive issues such as race, gender, socioeconomic status, or other historical sources of bias and discrimination. Ensuring fairness and addressing bias in AI-generated content is a critical ethical consideration in judicial decision-making. It requires stakeholders to consider whether solutions in terms of data selection and processing, algorithmic audits, fairness metrics, user feedback, content moderation, or other novel solutions should be implemented and adjusted in the context of GenAI-driven judicial decisions.

While the use of GenAI in the courtroom holds potential to improve efficiency and consistency, its effectiveness hinges on the quality of the data it is trained on, and the tuning performed by its developers. If this data and tuning harbors biases inherent in historical legal or non-legal practices (depending on the training data chosen), the GenAI system could unintentionally and without understanding, perpetuate and amplify these biases in its decision-making, potentially leading to discriminatory outcomes and undermining the principles of fairness and equality within the justice system. It is crucial to ensure that the training data and tuning for AI-powered legal tools are rigorously evaluated and unbiased to avoid perpetuating historical injustices and safeguard the fundamental rights of all individuals.

2.2.3. Data quality and hallucinations

The quality of the responses provided by GenAI is largely dependent on the quality of the data the system is trained on—whether legal or otherwise. When a GenAI system is trained on vast amounts of non-legal data, but then applied in a legal context (as is the case in the case studies discussed below in this article, in which the judges used ChatGPT), the tool can generate content based on non-legal sources like blogs, news reports, and academic papers. While these sources may be helpful in certain contexts, they do not carry binding weight like statutes, regulations, and case law, and cannot be relied upon as such. Even if GenAI systems are trained on legal data, there are still substantial risks. These risks include bias embedded in the data, as historical legal decisions may have favored certain demographics or socioeconomic groups, with the GenAI system inheriting and amplifying those biases. Other risks include incompleteness of the database that will inform the content, leading to inaccurate or misleading results; or inaccuracy of the data, which may contain factual errors or outdated information. Under these conditions, GenAI could provide unreliable legal reasoning or misinterpret the relevance or legal context of the precedents.

In addition to risks associated with the underlying data quality, there are also risks in how this data gets processed, and then returned to the user. This is particularly concerning because of the ability of GenAIs to confidently generate content that appears truthful despite factual inaccuracies, due to how these systems function, learn, and structure responses. For example, LLMs (a type of GenAI that generates text based on vast amounts of data)Footnote ⁹ operate by statistically and algorithmically predicting the most likely sequences of words, rather than based upon the “truth” of a statement (Garry et al., Reference Garry, Chan, Foster and Linda2024).Footnote ¹⁰ This, in addition to opaque and aggregate processing and response capabilities, can lead to a phenomenon known as “AI hallucination,” where the LLM generates seemingly coherent but factually inaccurate or misleading statements, potentially creating a false narrative (del Campo and Leach, Reference del Campo and Leach2022). These hallucinations degrade the reliability of the content GenAI tools create.Footnote ¹¹

Even the creators of these tools recognize and are open about the potential failures in the veracity of the technology’s outputs. Exemplifying this, Meta’s 2022 release of its LLM Galactica, lasted only 3 days online before being pulled down for, among other reasons, “not being able to distinguish truth from falsehood.” (Heaven, Reference Heaven2022) Further, OpenAI admitted that its tools should not be used as a primary decision-making tool (Kirchner et al., Reference Kirchner2023), and Google’s senior vice president stated recently that “[a]s we’ve said from the beginning, hallucinations are a known challenge with all LLMs—there are instances where the AI just gets things wrong.” (Raghavan, Reference Raghavan2024)

In the context of judicial decision-making, the fallibility of these systems carries an elevated risk, given that there is an inherent potential jeopardy of fundamental rights and freedoms. Because the confidence level associated with any particular response from GenAI tool is variable, a single unchecked instance of the AI “getting things wrong” may result in loss of property, imprisonment, or even separation from loved ones when a judge relies on a GenAI to inform their decision-making. This level of systemic error risk requires an abundance of caution when considering the integration of AI in the courtroom, requiring rigorous testing and implementation within robust ethical, legal, and supervisory frameworks. This inconsistency in content accuracy can also undermine public trust in the use of GenAI, which may have significant consequences in the judicial system and the rule of law. Further development is crucial to ensure LLMs and other AI systems are not only demonstrably unbiased, but also accurate and grounded in factual information, or at the very least, transparent when unsure as to the accuracy of its outputs.

2.2.4. Accountability and legal liability

Legal liability may become a contentious issue when AI systems are involved in judicial decision-making. This determination of liability refers to establishing who is responsible for defective GenAI outcomes in a judicial process—whether it is the AI developer, the judge-user, or any other party. Liability and the chain of responsibility is a legal space that currently requires significant clarification. For instance, there is a need for determining whether legal professionals and/or judges have a duty to review the accuracy and legality of the outcomes that algorithms produce, to what extent, and in which ways. There is a need to ascertain who, how, and through which processes different responsible offices, individuals, or judicial decision-makers may be held professionally responsible if they do not exercise due diligence (and to which level) in reviewing and verifying AI-generated content that leads to, informs, or supports, a judicial decision. Additionally, questions about product liability may also arise—a subject currently under discussion in some jurisdictions, such as the European Union.Footnote ¹² For example, if a third-party vendor or developer supplies a GenAI model used in legal decisions, they could be held liable for defects or damages caused by the system they produced. Particularly, developers may be held liable if their products are found to be defective or if they fail to meet any specific metrics or standards of performance claims.

The lack of a clear attribution of responsibility in AI-driven judicial decisions raises critical questions about accountability. Without a designated party responsible for any judicial errors in process or adjudication, ranging from material mistake to bias amplification, the potential for miscarriages of justice increases. This ambiguity could erode accountability, public trust in the legal system, and infringe upon individuals’ rights to due process, fair adjudication, and review of grievances such as in appeal processes.

2.2.5. Data protection challenges

GenAI systems are trained on vast datasets that often contain personal data input by developers or users (Gupta et al., Reference Gupta2023). This data includes personal identifiers such as race, gender, and age; it can even contain information provided by users when interacting with the GenAI through prompts. This has raised significant privacy concerns for individuals, industry, and government alike. In fact, a 2024 report by Cisco from 2600 security and privacy professionals in 12 countries (5 Europe, 4 Asia, and 3 Americas) found that many businesses are limiting the use of GenAI and 27% have temporarily banned the technology’s use given privacy concerns (Cisco, 2024).

The incorporation of GenAI into judicial decision-making, therefore, raises data privacy and protection challenges due to its need to collect, store, and analyze sensitive personal data in order to provide its responses and increase its efficiency and accuracy. These information processes, necessary to make any legal GenAI functional, require robust safeguards to ensure general compliance with data privacy regulations, and to protect individuals’ rights to privacy and data protection.

More specifically, these risks exist at various stages of the data processing progression. First, regarding the introduction of information into the system, the input of vast judicial and extra-legal data into a GenAI LLM raises concerns regarding the sourcing and handling of this data, including whether informed consent was obtained for its use and whether relevant privacy rights were respected. In the U.S. Supreme Court’s Year End Report, Chief Justice Roberts notes hesitations from legal scholars regarding “whether entering confidential information into an AI tool might compromise later attempts to invoke legal privileges.” (Roberts Jr., Reference Roberts2023) Second, the storage and safeguard of data used in training and fine-tuning these models is critical for both individuals’ rights and the efficiency of the relevant GenAI. Data breaches or unauthorized access to these datasets can result in the exposure of sensitive information, leading to potential data protection and privacy violations. Third, GenAI systems may inadvertently generate content that discloses personal or confidential information. This could occur due to the inherent limitations of GenAI LLMs in understanding the nuances of context and privacy. For example, a GenAI system used to analyze pre-trial documents might unintentionally generate summaries that reveal sensitive details about individuals involved in a case. These privacy violations may also be connected to human-machine interactions, as when users—for instance, judges or other judicial office-holders—interact with a GenAI system, they may provide sensitive information such as personal identifiers or confidential details, which the system would then incorporate into its database and responses. The lack of transparency in data processing and resultsFootnote ¹³ may make it challenging to pinpoint specific data origins for iterated generative output. This opaqueness could lead to difficulties in addressing potential data privacy breaches or ensuring individuals’ right to the elimination of personal data or rectification of inaccurate information.Footnote ¹⁴

To mitigate the data protection risks posed by GenAI in judicial decision-making, and ensure the responsible and ethical use of AI data retrieval, processing, and storage, it is imperative to establish robust data management protocols within the judiciary that prioritize transparency and implement comprehensive data governance strategies.

2.2.6. Judicial independence

GenAI poses a potential threat to judicial independence by introducing the risk of undue influence on legal reasoning. Judges who rely on GenAI-generated content without critical analysis may inadvertently allow underlying biases, historical data patterns, or specific ideological perspectives embedded within the algorithms or the training data to shape their decisions in ways that could undermine their independent judgment. This could lead to undue influence and subconscious bias in decision-making, undermining the fundamental principle of independent judgment and potentially resulting in rulings influenced by algorithmic inertia rather than objective legal interpretations (Gautam et al., Reference Gautam, Venkit and Ghosh2024).

Furthermore, the reliance on GenAI in judicial decision-making, particularly within the current trend of using non-legal technologies in the courtroom (i.e., ChatGPT use by judges),Footnote ¹⁵ raises concerns about the potential for an arbitrary homogenization of legal outcomes. Though non-arbitrariness is a core principle to be pursued in the courtroom, it is necessary to address each case independently and impartially, as well as according to its uniqueness—and the uniformity of AI-generated content could overshadow the richness of judicial thought that arises from diverse perspectives and context-analysis. As previous scholarship has pointed out, not everything in the law could and should be automated (Deakin and Markou, Reference Deakin and Markou2020). Uniformity and non-legality of sources and GenAI responses have the potential to undermine the core principles of independent, case-by-case adjudication, and judicial discretion, which relies heavily on nuanced interpretations and impartial consideration of unique circumstances, as well as on very precise information sourcing (like jurisdictional hierarchies and precedent considerations) to be the basis of decision-making (Re and Solow-Niederman, Reference Re and Solow-Niederman2019). These challenges to rationale independence and case individuality are raised when predictive AI is used in judicial decision-making. However, based on the case studies detailed in this paper, we argue that, without a proper framework and guidance to use AI in the courtroom, judges may fall in the trap of technological solutionism (Morozov, Reference Morozov2013) given the user-friendly features of tools such as ChatGPT and the promises of the technology.

2.2.7. Access to justice

Intuitively, we tend to think that the use of advanced information technologies increases efficiency across sectors. In the context of AI, this has been advocated by consultants, investors, user companies, and even proven by academics in different contexts (Tarafdar et al., Reference Tarafdar, Beath and Ross2019). Regarding GenAI in judicial decision-making, however, these theoretical benefits of cost reduction and efficiency need to be analyzed under a different perspective, one that is inclusive of more metrics than processing efficiency—as the desired output of a judicial process is not just delivering an outcome, but a fair one both formally (in process) and materially (in adjudication). The increased use of AI by judges and attorneys can increase disparities among and within legal systems (Grossman et al., Reference Grossman, Grimm, Brown and Xu2023), threatening access to formal and material justice, with the potential to have macro and micro-scale impacts. At a macro-scale, nations with more resources can develop and integrate GenAI tools at a faster rate, increasing their efficiency, and therefore, also increasing the gap between different jurisdictions’ factual access to the system. On a micro-scale, individuals accessing the justice system without GenAI tools can be hindered in contrast to those with resources to access these tools and utilize them at the service of their legal arguments. Contrary to perceived ideals of increased access to justice, jurisdictions struggling with poor access to justice may suffer even more due to these increased competitive costs or lack of technical expertise. For instance, the state usually provides defense attorneys at no cost for defendants who lack defense counsel. However, the increased hidden costs (Grossman et al., Reference Grossman, Grimm, Brown and Xu2023) of deploying GenAI could frustrate efforts to build the necessary capacity for providing technological aid to defendants who need it. Hence, jurisdictions that use GenAI unsystematically, without full consideration of the plethora of challenges posed by GenAI, are likely to face these challenges in the near future.

For GenAI to have a positive impact on access to justice, it is necessary to take a conscious and unified strategy towards deploying GenAI within the context of each jurisdiction. However, this also raises concerns about differences in capability for GenAI implementation being paired with resource-capabilities. For instance, Singapore is testing the use of GenAI in the small claims tribunal first—which sees about 10,000 cases a year—with an eye to expanding it to other areas, such as divorce maintenance and civil claims (Ocampo, Reference Ocampo2023). Singapore is adopting a unified strategy on the use of GenAI in courts, experimenting with the technology while pondering the potential risks it could create, and developing its own tailored GenAI system under a two-year memorandum of understanding with Harvey AI, an American legal tech start-up (Ling, Reference Ling2023). The model being developed will advise users on their rights, guide them through the processes of a legal trial, and even estimate how much a user may be able to claim (Ling, Reference Ling2023).

Another jurisdiction actively exploring the use of AI in the administration of justice is Estonia, the second-fastest country in Europe in terms of court proceedings (Invest in Estonia, 2022). The Estonian Ministry of Justice has expressed keen interest in exploring AI projects to identify areas where AI can be beneficial. While AI-powered judicial decisions or the use of GenAI tools are not currently deployed in Estonian courts, the Ministry has expressed its openness to exploring these possibilities in the future to better support judges and court officials.

Well-resourced countries like Singapore and Estonia are taking a proactive, cohesive, and state-led approach. They are investigating and investing in GenAI development and implementation, ensuring robust data quality control, and tailoring the technology for the legal system. This approach has the potential to improve efficiency and fairness in the legal processes. On the other hand, the case studies analyzed below, localized in Mexico, India, Colombia, and Peru, exhibit a less cohesive, ad-hoc approach in their GenAI implementation strategies. In these jurisdictions, judges have deployed GenAI on a case-by-case basis, often without considering the potential risks and with little regulatory guidance. This disparity raises concerns about resource availability and its impact on access to justice: Wealthier nations, like Singapore, can invest in robust data quality control and develop tailored GenAI systems, potentially leading to fairer and more efficient legal processes. However, resource-constrained countries might struggle to implement all relevant safeguards, hindering their ability to integrate GenAI responsibly, or at all. This creates a worrying scenario where access to justice becomes increasingly dependent on a nation’s financial resources, potentially widening the gap among jurisdictions’ judicial efficiency, and for those already struggling within the legal system.

3.1.1. The specialists: A targeted approach to the use of Gen AI

One approach to integrating GenAI into the courtroom involves creating targeted regulations specifically designed to address the unique risks and challenges the technology presents within the judicial system. To this effect, jurisdictions such as the United Kingdom (UK) and New Zealand have published guidelines for judicial office-holders to counsel and instruct them in the use of AI.

The UK’s “Artificial Intelligence Guidance for Judicial Office Holders” (December 2023) approves the use of AI in courts, overall encouraging caution (Courts and Tribunals Judiciary of the United Kingdom, 2023). The document provides general direction for judges and judicial office-holders on the potential uses and risks of AI in Courts and Tribunals, while specifically advising against AI use in certain cases, including legal research and analysis, due to limitations in verification and inferential thinking capabilities, as these systems “do not produce convincing analysis or reasoning” (Courts and Tribunals Judiciary of the United Kingdom, 2023).

New Zealand’s “Guidelines for use of generative artificial intelligence in Courts and Tribunals” (December 2023) takes a more targeted approach. Addressing judges and judicial staff, these guidelines focus on risks associated with GenAI chatbots, such as ChatGPT, Bing Chat, or Google Gemini (Courts of New Zealand, 2023). The guidelines acknowledge severe limitations of publicly available GenAI chatbots, stating that “[e]ven with the best prompts, the output may be inaccurate, incomplete, misleading, or biased” (Courts of New Zealand, 2023). To mitigate these risks, the guidelines restrict judges from feeding data into an AI chatbot that is not already in the public domain, preventing them from “enter[ing] any information that is private, confidential, suppressed or legally privileged” (Courts of New Zealand, 2023). Notably, these guidelines do not currently require disclosure of GenAI use to the public when utilized in judicial decision-making.

3.1.2. The generalists: A guidelines-based approach to the use of AI

A number of jurisdictions have introduced broad guidelines for the use of AI and automation tools in courts, but these documents often do not specifically address the unique risks posed by recent advancements in GenAI. This generalist approach to AI regulation avoids focusing solely on judicial applications, opting instead for broader frameworks that apply across sectors. While this may encourage innovation and flexibility, it also risks overlooking the specific challenges and ethical concerns raised by GenAI, particularly in the context of sensitive judicial decision-making.

The EU for example, has taken a generalist approach to regulating AI in judicial decision-making through the risk-based regulatory framework applicable to AI systems. The EU’s Artificial Intelligence Act (AIA) (European Commission, 2024) regulates high-risk AI systems across different industries. While the AIA does not explicitly target courts, its categorization of AI based on risk could indirectly impact how courts utilize this technology. For instance, if a court uses an AI system for risk assessment in criminal sentencing, it might need to comply with stricter regulations under the AIA, such as robust data quality control or ensuring human oversight in the decision-making process (European Commission, 2021).

It is important to note the existence of the European Ethical Charter on the use of Artificial Intelligence in judicial systems and their environment (The European Commission for the Efficiency of Justice, 2018). This soft law instrument, issued by the Council of Europe, outlines ethical principles for AI use in courts. This document provides foundational ethical principles that are consistent with the AI Governance principles published by several jurisdictions, agencies, and the private sector globally (Habuka & Socol, 2024).Footnote ¹⁶ The document also explains the need for an in-depth public debate on these tools prior to the implementation of public policies for their development (The European Commission for the Efficiency of Justice, 2018). Even though GenAI may not have been as popular or developed as it is presently when the charter was published, the text encourages the use of chatbots to facilitate access to the various existing sources of information and also the use of AI to generate document templates like court applications or lease agreements (The European Commission for the Efficiency of Justice, 2018). However, the charter is not legally binding and predates the recent surge in GenAI technology. While this document is a valuable starting point for ethical discussions, it may not fully address the specific considerations surrounding GenAI in judicial settings. Therefore, a generalist approach relying solely on existing regulations might need to be supplemented with additional guidance or regulations specifically tailored to address the unique risks exacerbated and opportunities presented by GenAI in judicial decision-making, particularly as the technology continues to evolve.

3.1.3. The deployers of GenAI

One approach to AI governance goes beyond regulation, emphasizing the creation of clear guidelines alongside the proactive development and integration of responsible AI systems within specific sectors. Notably, as aforementioned,Footnote ¹⁷ Singapore is in the process of developing its own customized GenAI system through a two-year memorandum of understanding with Harvey AI. This model incorporates built-in features designed to mitigate risks and enhance rights protection, including advising users on their rights, guiding them through legal processes, and even estimating the value of potential claims (Ling, Reference Ling2023). Given the short timeline since the announcement of these measures, which were made public in 2024, the full impact of these developments on the administration of justice and the effectiveness of risk-mitigation strategies remain to be seen at the time of this article publication.

3.1.4. The deterrent approach

Some jurisdictions have opted for a more cautious strategy, advocating for a temporary ban or discouragement on the use of GenAI in courts until a more robust regulatory framework, or certifiable precautionary measures, can be established. In December 2023, the Canadian Federal Court banned the use of AI in making “judgments and orders” until a public consultation on the matter occurred (Federal Court of Canada (Cour fédérale), 2023). In its reasoning, the Court acknowledged “the potential for AI to impact adversely on judicial independence” (Federal Court of Canada (Cour fédérale), 2023). Further, the Court recognized the risk that “public confidence in the administration of justice might be undermined by some uses of AI” (Federal Court of Canada (Cour fédérale), 2023). This cautious approach, exemplified by the Canadian case, prioritizes mitigating potential risks to principles like judicial independence and public trust, over swiftly embracing GenAI’s promise. It also underscores the importance of public consultation, ensuring civil society has a voice in shaping how, or even if, GenAI will be integrated into the judicial system.

3.2.1. Colombia

Colombia has had two high-profile cases involving the use of GenAI in judicial decision-making. The first of these cases revolves around a court document released on January 30, 2023, in the case of Salvador Espitia Chávez v. Salud Total EPS (2023). In this case, Judge Juan Manuel Padilla issued a ruling on whether a child with autism should be eligible for medical treatment coverage. Believed to be the first documented case of a judge utilizing GenAI in the courtroom, the judge openly acknowledged that he used ChatGPT to guide his decision rationale and published the questions and answers of the interaction that he had with the GenAI LLM. The judge asked ChatGPT four questions related to the following items:

1. (1) Related to whether paying moderating fees is exonerated in the case of therapy for a child with autism.

   Response summary: According to Law 1753 from 2015, minors with autism are exonerated from moderating fees, binding private and public entities.

2. (2) Related to granting tutelage processes.

   Response summary: Tutelage processes may be granted so that the entity providing services may be compelled to provide therapy.

3. (3) Related to whether the moderating fee may be a barrier to access health services.

   Response summary: Moderating fees may be a barrier for minors with autism in accessing health services.

4. (4) Related to jurisprudence of the constitutional court in similar cases.

   Response summary: There have been favorable cases from the constitutional court (one of the sources of legitimate decision-making in this regard) made in favor of minors with autism within this context.

While the judge emphasized that the LLM did not replace his own judgment, and that it was deployed to optimize time spent writing sentences with previous corroboration of the information provided by the GenAI system, the unregulated use of the technology still raises concerns. The primary concern lies in the potential for biased or unreliable data to influence critical judicial decisions. Without clear legal or ethical guidelines, GenAI tools could perpetuate existing biases or generate flawed outputs based on inaccurate information. This is especially true in instances with heightened sensitivity such as this process, as it involves the medical care of a minor with autism. Especially in cases where minors, persons with disabilities, and other marginalized or vulnerable groups are concerned, the use of GenAI that may affect ruling outcomes, potentially signaling that they should be especially scrutinized. Additionally, using a GenAI LLM like ChatGPT, which does not disclose the origins of its responses and that is connected to extra-legal sources (i.e., encompassing information that extends beyond laws and previous cases), can introduce inaccurate or non-legal information into a response, potentially undermining the legality and accuracy of the response. This is a concern that permeates throughout these case studies, as all judicial office-holders alluded to utilizing ChatGPT. Furthermore, Judge Padilla’s case highlights a secondary risk—the erosion of public trust. The media’s criticism surrounding the use of GenAI (Gutierrez, Reference Gutierrez2023), even for efficiency purposes, underscores the public’s desire for transparency and established safeguards when such technology is integrated into the justice system.

Ten days later, in a second Colombian case, Magistrate María Victoria Quiñones from the Administrative Tribunal of the Magdalen issued a court order in a case where Union Temporal de Servicios Integrados y Especializados de Tránsito y Transporte de Santa Marta—SIETT filed a direct remedy claim (reparación directa) against Colombia’s National Police. As in Salvador Espitia Chávez v. Salud Total EPS, ChatGPT prompts were again transcribed (2023). In this case, Magistrate María Victoria Quiñones used ChatGPT to address technical questions that assisted in planning and conducting a judicial hearing within the Metaverse. The AI’s responses helped in understanding certain concepts related to the metaverse and in organizing the hearing. The questions asked were the following:

1. 1. What is an avatar?

2. 2. What is the most effective method to verify the authenticity of those connecting to a meeting and/or virtual hearing?

3. 3. Methods for verifying the authenticity of the avatar in the metaverse.

On February 15, 2023, the relevant hearing was held through Meta’s Horizon Workroom (Meta, 2024), and was livestreamed through YouTube (Victoria en linea-Vicky Quiñones, 2023). In this case, despite serious concerns over privacy and authenticity when conducting legal proceedings in the Metaverse, there is no indication of vetting or verifying the questions directed to ChatGPT. To date, no other judge in Colombia has replicated this approach, and no other audiences have taken place in the Metaverse (Ritterbusch and Teichmann, Reference Ritterbusch and Teichmann2023), or other virtual reality (VR) platforms.

The judge’s reliance on ChatGPT to navigate the complexities of a virtual courtroom raises potential risks, including the LLM’s limited understanding of procedural nuance, its potential focus on efficiency over security, and the overlooking of other technical legal implications. First, LLMs excel at processing vast amounts of information, but they may struggle with the legal system’s inherent subtleties and the ethical preservation of rights. The complexities of legal systems and due process could be misinterpreted by an LLM, leading the judge to overlook crucial steps for ensuring a fair trial. Second, LLMs are often optimized for efficiency and may prioritize completing tasks over rigorously evaluating security protocols. For example, an LLM might suggest avatar verification methods that seem efficient but lack robustness, leaving the court vulnerable to identity theft or manipulation. Finally, an LLM might not anticipate secondary potential legal risks of using avatars, including risks in evidence presentation, adversarial procedural issues, or witness independence, comprehensive treatment, and potential intimidation, among others. These risks highlight the danger of relying on LLMs for judicial decision-making, even in focused procedural contexts, as these may have significant consequences on procedural rights, adjudicatory outcomes, and the overall fairness of the judicial process.

3.2.2. Peru

On March 27, 2023, Judge Frank Paul Flores Garcia used GenAI to support his sentencing rationale and legal basis in a civil law case related to child support obligations for a mother and father with respect to their daughter (Juzgado Civil Transitorio de San Juan de Miraflores, n.d.).

To determine the proportion of child support that each parent should bear according to their capacity, the judge consulted ChatGPT for guidance on the relevant mathematical operations. The judge did not share any specific details about the prompt(s) used or the answer(s) he received from the system. He merely noted that he was assisted by “ChatGPT,” and that it was “corresponding to apply the technique of mathematical proportion to the effects of establishing which is the amount corresponding to each parent, according to their income, to satisfy the expenses of maintenance of their daughter” (Juzgado Civil Transitorio de San Juan de Miraflores, n.d.). The judge alluded to ChatGPT as a foundation for his legal reasoning, and footnoted a transcribed URL (Juzgado Civil Transitorio de San Juan de Miraflores, n.d.). Judge Flores Garcia may not have been aware that this footnote provided no insight into his consultation, given that readers cannot access his personal record with ChatGPT.

This raises concerns about transparency and visibility in decision-making mechanisms, alongside legal education and technological competency matters. Integrating AI into judicial decision-making without a proper understanding of its inner workings and limitations can lead to opaque processes that infringe upon individuals’ rights to due process and fair adjudication. The potential for the manifestation of risks such as unintended bias amplification, judicial independence, data protection, or simply judicial error within such AI systems, necessitates robust frameworks to safeguard the fundamental rights and liberties of all individuals that interact with the justice system—requiring solutions that include enhancing legal education surrounding GenAI, its uses in the courtroom, and standard transparency and use-case disclosures.

3.2.3. Mexico

During a hearing on March 29, 2023, Magistrate Reyes Rodríguez Mondragón, president of the Superior Chamber of the Electoral Tribunal of the Mexican Judiciary, shared that he had consulted ChatGPT on his phone on the matters concerning the hearing discussion (Tribunal Electora TEPJF, 2023). Magistrate Reyes disclosed his consultation with ChatGPT and read some of the outputs that he obtained to illustrate (i) the material substance of his point, and (ii) how ChatGPT could be of help in being exhaustive and complete in legal arguments.

In this appeal, the Electoral Tribunal was discussing the use of the expression “you know who” (in Spanish, “ya sabes quién”) (López Ponce, Reference López Ponce2023) when used within the context of an advertisement forming part of an electoral pre-campaign of the Morena political party. The Electoral Tribunal was deciding whether the use of the expression “you know who” could be interpreted as a signal of support by President Andrés Manuel López Obrador. Magistrate Rodríguez noted that a cursory analysis on ChatGPT had quickly allowed him to ascertain the relationship between the expression and its political references. The Magistrate argued that these new technologies may be a useful tool in satisfying certain judicial duties—particularly in pursuing seemingly comprehensive and thorough legal research. In fact, he noted that the decision subject to appeal had a judicial defect and should be revoked because it was incomplete, given that there was no contextual analysis of the “you know who” expression and thereby failed to be legally “complete.”

Magistrate Rodríguez stated that “the Electoral Tribunal of the State of Mexico reiteratively is not complying with the principle and duty of exhaustiveness (…) it is not providing sufficient reasons (…) when even technology now facilitates a series of information, obviously processing databases and all the knowledge that is available to the courts.” This implies that, in order to (i) satisfy the duty of exhaustiveness and (ii) provide sufficient reasoning, Magistrates may need to utilize modern technologies in their adjudication process and legal reasoning substantiation.

However, in this same hearing, Magistrate Vargas contested Magistrate Reyes with regard to the inclusion of ChatGPT as a foundation for legal completion. Magistrate Vargas noted: “I would like to think that what you have just told us is simply an isolated example and is not a forecast of what will be the jurisprudence of this court (…) because I would be concerned that now our resolutions will be taken based on what ChatGPT says.” He also inquired as to whether, in the future, courts will remain relevant by that logic since “ChatGPT [could] tell us everything” (Tribunal Electora TEPJF, 2023). Magistrate Vargas further noted how ChatGPT may contradict the principles of presumed innocence, presentation of proof, and valuation of evidence, all of which are central to the fair functioning of courts (Tribunal Electora TEPJF, 2023).

While AI in legal reasoning promises to be a resource for exhaustive information retrieval and thorough research, it raises ethical concerns regarding fundamental legal principles, potential bias, the erosion of judicial discretion, and the displacement of the human element in interpreting complex legal nuances. This case from the Mexican Judiciary highlights these concerns in sharp relief. Magistrate Rodríguez’s use of ChatGPT during a hearing, while intended to pursue comprehensiveness, underscores the potential for such tools to undermine core legal principles. Magistrate Vargas’ critique raises valid concerns about relying on AI outputs as a substitute for critical analysis and independent judicial reasoning.

3.2.4. India

In 2023, High Court judge Anoop Chitkara refused bail to a man accused of rioting, criminal intimidation, criminal conspiracy, and murder (Times of India, 2023). Judge Chitkara recognized that he had turned to ChatGPT to help guide his reasoning, though he noted that he did not rely on ChatGPT to decide his ruling. Instead, in substantiating his legal basis, he consulted with the GenAI tool to help determine if he was relying too heavily on his own “consistent view” developed from past cases, concerning allegations involving an unusually high level of cruelty, and whether they should count against granting bail (Smith et al., Reference Smith, Moloney and Asher-Schapiro2023). In this regard, it seems that there is a claim made that by using GenAI, the Judge was not only supporting his rationale but also enhancing its impartiality, ensuring that his decision-making was more balanced and less influenced by personal bias.

Judge Chitakara’s disclosed his consultation with ChatGPT, to which he presented the following prompt: “What is the jurisprudence on bail when the assailant assaulted with cruelty?” ChatGPT’s transcribed response stated that “If the assailants have been charged with a violent crime that involves cruelty, they may be considered a danger to the community and a flight risk. In such cases, the judge may be less inclined to grant bail or may set the bail amount very high to ensure that the defendant appears in court and does not pose a risk to public safety.” (Suryam, Reference Suryam2023) The LLM added that presumption of innocence is also a fundamental principle of the justice system and that bail is often granted if the judge is sure the accused does not pose a risk to public safety. Judge Chitkara decided that the accused acted with cruelty, subsequently denying the bail request.

This Court made it clear that any reference to ChatGPT was only intended to present a broader picture on bail jurisprudence where cruelty is a factor. Much like the case studies above, this approach on the use of AI in judicial decision-making has not been in accordance with any GenAI guidelines of judicial AI deployment or broader legal architecture in the country, nor is it consequent with a comprehensive strategy for all judges to consider its implications and correct use of the technology: GenAI was deployed, as in the above-noted cases, circumstantially and on an ad hoc basis. This ad hoc method of deployment of GenAI in judicial decision-making, without a clear or unified regulation for risk mitigation, a comprehensive framework, or guiding principles, poses numerous risks, including unequal application of the law due to a lack of standardization, the undermining of judicial discretion and potentially accuracy through reliance on non-legal algorithmic tools, the exacerbation of existing biases embedded within the algorithms themselves, and the creation of opaque decision-making processes—both at the judicial and algorithmic levels.

In addition to these concerns, this case, like the ones above, used ChatGPT, an open network that accesses all information on the internet. This raises concerns about data quality and the legal competence of the GenAI to present legal opinions. ChatGPT and similar open-source AI tools are not specifically designed for legal research or reasoning. They may process vast amounts of information, but they lack the ability to critically evaluate the legal relevance and accuracy of that information. This can lead to misleading or unreliable outputs, potentially influencing judicial decisions in unintended ways. Judge Chitkara’s reliance on ChatGPT to check his own “consistent view” exemplifies this risk. The GenAI’s response, while seemingly relevant, does not delve into the nuances of specific legal precedents in India generally or the specific court, or consider the unique circumstances or laws applicable to the case. This highlights the need for GenAI tools specifically designed for the legal domain, trained on high-quality legal data and capable of providing contextually relevant legal reasoning. The ad-hoc use of such tools, as seen in the case studies presented, further underscores the need for comprehensive frameworks. These frameworks should establish clear guidelines for data quality, address potential biases within the AI, and ensure transparency in GenAI application.

4.1.1. Capacity assessment

The potential of GenAI in courts is undeniable, but its effective integration hinges on a crucial factor: capacity. Before embracing GenAI, jurisdictions must undergo an assessment to determine their readiness for this complex technology. Key considerations for capacity assessment include infrastructure, expertise, and overall resource analysis to determine the capabilities met—and required—for responsible GenAI implementation. Utilizing GenAI necessitates a nuanced comprehension of the technology’s inherent risks, alongside a jurisdiction’s capability to not only factually implement the technology but mitigate these perils. Jurisdictions must assess whether they possess the requisite expertise within their judicial systems or if more investments in training, recruitment, or regulatory or infrastructural development are necessary. Furthermore, as implementation progresses, jurisdictions should conduct evaluations to ascertain the compatibility of their existing technology with GenAI systems, identifying necessary upgrades or modifications for seamless integration.

This evaluation process is of particular relevance for developing economies grappling with resource constraints in GenAI deployment. As part of capacity-building efforts, emerging markets could pursue strategies including in-house development, or fostering public-private partnerships aimed at driving digital government reforms and surmounting implementation challenges. Unfortunately, such collaborations are infrequent: According to the UN eGovernment Surveys covering 193 countries, only 42 countries participated in co-creating education e-services, while a small minority of 22 countries engaged in collaborative development of justice-related e-services (OECD, 2022). Capacity assessments and consequent development toward covering personnel, technology, or regulatory and infrastructure needs can help effectively integrate GenAI into the courtroom.

4.1.2. Stakeholder engagement

A proactive approach to stakeholder engagement in developing guidelines and AI use cases for courts offers significant long-term benefits. In fact, it can be considered a foundational element for implementing GenAI in the judicial system inclusively. Stakeholder engagement brings a valuable diversity of perspectives to the table. Judges, lawyers, court staff, litigants, GenAI developers, and the public all have unique insights into how GenAI could impact legal processes and outcomes. The importance of stakeholder engagement goes beyond mere practicality: The justice system relies heavily on ongoing public trust and legitimacy. Actively engaging stakeholders can help ensure that the implementation of GenAI fosters acceptance amongst those it affects and upholds the integrity of the judicial process. Stakeholder engagement strategies could include opening public forums and educational workshops, with sessions informing citizens about GenAI’s potential applications in court and addressing concerns about fairness, transparency, and data privacy; consulting with legal professionals and bar associations, engaging judges, lawyers, and legal experts who can provide valuable insights on how to integrate GenAI responsibly while maintaining ethical legal practices; and collaborating with civil society organizations such as academia, NGOs, and advocacy groups, ensuring a diverse range of voices are heard and potential negative impacts on marginalized communities are considered. This emphasis on stakeholder engagement is not limited to GenAI in courts; it extends to broader AI adoption. The call for public participation in AI governance aligns strongly with a public value management approach to public administration. This approach prioritizes transparency, accountability, and citizen participation, potentially influencing the consolidation of robust and inclusive AI governance frameworks and more informed decision-making by public authorities (Wilson, Reference Wilson2022).

4.1.3. Licensing and verification

To ensure only qualified GenAI systems enter courtrooms, this article presents an approach based upon an ex-ante verification process guaranteeing that any GenAI utilized in judicial decision-making meets a set of basic standards—both for third-party developers integrating a GenAI into the judicial system, as well as for GenAI developed in-house within the judiciary structure. While this proposal could be applicable to any use of AI by courts, it is particularly relevant in the context of GenAI given the extra-legal nature of the data sources in current models being utilized by judges, as explored in the case studies explained above.Footnote ²¹

A licensing and verification regime should be established to assess GenAI systems. This ensures a high bar for functionality, adherence to foundational legal principles, and minimizes potential risks in judicial processes and outcomes. The integration of a GenAI into judicial decision-making would require meeting the standards of elements such as the following:

• Data quality: A core focus should be on the quality and comprehensiveness of the data used to train GenAI algorithms. High standards for data inclusion are crucial, ensuring that GenAIs which will be used in court are trained on accurate, unbiased, and complete datasets that reflect the real-world situations these AI systems will encounter in judicial decision-making. This includes both legal data and any relevant extra-legal data (if any) that will be introduced into the algorithmic system and may inform its output.

• Algorithmic fairness and bias mitigation strategies: The verification process should scrutinize the algorithms for potential biases and ensure developers have implemented techniques like fairness testing and de-biasing methods to minimize discriminatory outcomes. For example, data gathered and processed to be utilized in assessing risk of recidivism should be representative and mindful of diverse populations and avoid perpetuating historical biases by means such as not algorithmically integrating biometrics within the context of judicial decisions, enhancing unbiased data, and further preserving data privacy.

• Explainability of the AI’s reasoning process: Judges and parties involved in legal proceedings need to understand how GenAI arrives at its conclusions. This could involve providing clear explanations of the factors, precedent cases, or laws and regulations considered by the relevant GenAI and the weight assigned to each factor in reaching a decision.

• Security and privacy safeguards for sensitive data: Robust security measures are essential to protect sensitive data used by GenAI systems from unauthorized access or manipulation. Additionally, privacy safeguards should be in place to ensure personal information is obtained, stored, handled, and processed according to legal data privacy requirements.

• Adherence to ethical guidelines for judicial AI use: A clear set of ethical guidelines should be established to govern the development and deployment of judicial GenAI. These guidelines could address issues like transparency, accountability, fairness, and the potential for human oversight at certain stages of training and deployment.

This licensing process should be overseen or ratified by a qualified independent body with expertise in AI, law, and ethics. Developers should be required to demonstrate their GenAI systems meet these standards before they can be used in courtrooms.

Additionally, courts that develop their own GenAI models should be subject to rigorous internal verification processes mirroring the external licensing regime that this article proposes. This internal verification procedure would ensure the in-house GenAI systems adhere to the same high foundational standards as those developed by third parties.

4.1.4. Trusted datasets and explainability

Integrating GenAI into judicial decision-making necessitates high-quality datasets, as a GenAI system’s potential is made possible, as well as limited, by the information it is trained on (Wilson, Reference Wilson2022). Biased or incomplete data can lead the system to perpetuate existing societal biases or generate inaccurate legal reasoning, potentially undermining the fairness and accuracy that are cornerstones of the justice system (Wilson, Reference Wilson2022). As such, when integrating GenAI into the courts, the following factors must be considered:

• • Data Quality: GenAI systems are only as good as the data on which they are trained. Robust measures must be implemented to ensure that the data used for judicial GenAI is accurate, unbiased, representative, and complete. This requires data quality monitoring and mitigation strategies for potential biases, comprehensive historical tracking accounting for court hierarchy and current legal developments, and weighted analytics on legal interpretation progressions. Examples of such strategies could include employing diverse data sources, using procedural and material fairness metrics during training, extricating biometric data from judicial decisions, and implementing human-in-the-loop review processes. Further measures may include establishing closed-network datasets, instead of utilizing open-source GenAIs in the courtroom—meaning, deploying only GenAIs trained on a defined and finite amount of trustworthy data (whether legal or extra-legal) that could be jurisdiction or even court-specific, and not the comprehensive universe of the internet.Footnote ²² This pre-emptive control must be focused on preventing the “garbage in, garbage out” phenomenon (Lew et al., Reference Lew2020; Bell et al., Reference Bell2022), by which low-quality input results in low-quality output. In essence, the success of integrating GenAI into judicial decision-making requires the review and implementation of high-quality datasets. By prioritizing high-quality data and implementing robust safeguards at the informational origin of a GenAI’s content, we can ensure that GenAI in the courtroom augments legal reasoning, reducing the risk of undermining the quality and accuracy of judicial processes and adjudications.

• • Data Access, Explainability, and Origin Visibility: Transparency and fairness demand clear protocols for data access. Judges and relevant parties should be able to understand the data sources used by the GenAI system and how it may have influenced its outputs. This could involve providing summaries of the data used, allowing for limited access to anonymized data for specific reasons such as tracing purposes, or requiring GenAIs to disclose the data sources and characteristics, inclusive of precedents, laws, and regulations used at arriving at a conclusion. Explainability empowers judges and litigants to understand the reasoning behind a GenAI’s outputs, fostering trust and transparency in the decision-making process, and ensuring a fair legal basis of judicial reasoning, while data access allows for clear and directed audits and verification of the data used to train the GenAI, mitigating potential biases and promoting fairness in the application of the technology.

4.1.5. Allocation of responsibility

A clear and consistent understanding of who bears responsibility for potential GenAI-related damages to the judicial process, its outcome, or its integrity, is critical when introducing GenAI into the courtroom. Responsibility allocation is crucial to account for, mitigate, or remedy potential miscarriages of justice, judicial errors, or any form of harm to judicial systems and processes caused by GenAI use. Accompanying its potential, GenAI includes high-risk features, including hallucination, inferential and reasoning mistakes, or the outputting of biased information, among other potential disruptions to the fairness of the judicial process. Striking a balance between incentivizing responsible GenAI, development, providing reliable and robust accountability networks, and empowering judges to make decisions effectively and efficiently, is key in this space. Overly stringent measures against developers—provided by, for example, a responsibility system that focuses solely on a GenAI’s algorithmic setup and its consequent input to judicial reasoning—could stifle innovation and deter companies from entering the field due to the high risk of extensive liability. Conversely, placing the burden of responsibility solely on judges—through for example, a responsibility system that focuses on the judge’s ultimate decision-making control on a GenAI’s output and its application—might disincentivize them from utilizing GenAI, or produce an overly burdensome review process for GenAI-created content, hindering potential efficiency benefits. Determining the exact level of responsibility for each party can be complex, and clear legal frameworks and guidelines are needed to navigate these challenges. Some considerations to strike this court-and-jurisdiction-dependent balance include:

• • Developer Accountability: Developers should be held accountable for the algorithmic design and functionality of their GenAI systems. This requires clear disclosures of their AI’s modeling and reasoning process, enabling scrutiny for potential bias or flaws in the algorithms. However, liability allocation networks must avoid overly punitive developer liability frameworks. These could create high barriers to access this sector, not presenting enough of an incentive for developers to pursue stepping into this space. Clear balancing of responsibility is crucial: Developers should be held accountable for functionality and failure to meet certain measurable pre-agreed upon standards, as well as demonstrably flawed algorithms that lead to harm, but not for unforeseen errors or damages in complex legal scenarios—especially those attributable to judges, or in which the judges would be responsible for accuracy or fairness review. A crucial tool to help balance incentivizing development and maintaining safe judicial thresholds is the ex-ante certification process. Under the certification model developed in this article, developers should be required to undergo a certification or licensing process that assesses their development practices and adherence to certain legal and ethical guidelines for judicial GenAI deployment. This would prevent latent unforeseen liabilities during deployment, and be able to “weed out” potential systemic shortcoming in GenAIs prior to liability scenarios.

• • Judicial Discretion: Judges retain ultimate decision-making authority, and thus bear responsibility for the process and outcome of their judicial processes. However, absolute responsibility for GenAI outcomes could discourage the judiciary from using GenAI, or slow down judicial processes with overly demanding review processes to GenAI content, thus missing out on leveraging the technology’s potential efficiency gains. To address this, judges should be equipped with comprehensive training and resources for critically evaluating GenAI outputs. Instead of full responsibility allocation, liability distribution should focus on a tiered approach integrating (i) training, allowing judges to exercise informed discretion when utilizing GenAI and understand its limitations; and (ii) risk-based responsibility. A risk assessment framework should be implemented, establishing that high-risk cases involving significant potential harm would necessitate a more rigorous review process, whereas low-risk cases would require lesser scrutiny from the relevant judicial office-holders. With this elevated review process and scrutiny obligations on GenAI’s output for high-risk cases, there should be an accompanying higher degree of responsibility for the judge, both in (a) the diligence exercised in the examination process of reviewing the GenAI output, and (b) the responsibility allocated in the relevant tiered-threshold. This may even involve discarding the use GenAI in high-risk cases altogether, or at least in specific portions of cases (e.g., not utilizing GenAI in sentencing, but potentially utilizing it in precedent review and summarization), and include semi-automatized or automatized-with-redress review for low-risk cases.Footnote ²³

• • Shared Responsibility Model: This article proposes shared responsibility models, where both developers and the court system bear some degree of liability depending on the specific circumstances of the case and the content of the algorithmic output. This could involve factors like judges’ GenAI training, the cases’ complexity and risk profile, and the transparency of the GenAI system used. Ultimately, the goal is to establish a framework that encourages responsible development and GenAI integration into the judicial system, empowers judges and allows for increased efficiency, while protecting the integrity of judicial decision-making.

4.1.6. Prompt engineering in the judicial system

A prompt is a natural language text that requests the GenAI system to perform a specific task (Marvin et al., Reference Marvin, Hellen, Jjingo and Nakatumba-Nabende2023). In all the case studies, it is judges that instruct the system to provide an output, interacting with LLMs via prompts. For GenAI to operate in a desirable manner, it is important to give context and detailed information through these interactions with the systems to produce relevant and effective outcomes. Therefore, prompt engineering is crucial to refine instructions until the judicial officer gets the desired outcomes from the system (Knoth et al., Reference Knoth, Tolzin, Janson and Leimeister2024). For instance, in the healthcare sector, prompt engineering has emerged as a promising approach to guiding model generation by providing targeted information-seeking mechanisms. This field proposes prompt design methods for different medical tasks. Namely, it explores how to choose and design prompt elements, how to use prompts to guide a model generation of text that meets medical requirements, and how to evaluate the impact of different prompt designs on model performance (Wang et al., Reference Wang, Shi, Yu, Wu, Ma, Dai and Yang2023).

In all case studies, all judges used human-designed prompts. Also, all guidelines and policies mapped in this paper assumed that a human will be the one giving the prompt to the GenAI system. Human-designed prompts depend on the human’s expertise in selecting relevant information and constructing prompts in a way that will produce valuable outcomes by the model (Wang et al., Reference Wang, Shi, Yu, Wu, Ma, Dai and Yang2023). According to the emerging literature on prompt engineering, human prompts are particularly effective for tasks with well-defined input data and structured output requirements (Wang et al., Reference Wang, Shi, Yu, Wu, Ma, Dai and Yang2023). However, designing effective human prompts requires the expertise of someone who understands the specific field where the GenAI model is being used, as well as human-machine interaction. Even though we assume that judges and judicial officers are experts, levels of competency, experience, and subject-matter expertise, as well as requirements to access the judicial profession, vary across jurisdictions and across fields.

Therefore, ensuring the quality of human-designed prompts could be a challenging task. Additionally, the parties affected by the judicial decision are currently not in a procedural position to assess such quality in order to challenge the decision as part of the due process. Due to these limitations of human-designed prompts, researchers are exploring automated prompt design methods to improve the efficiency and adaptability of prompt-based approaches. Automated prompts are generated using various algorithms and techniques, eliminating the need for human intervention, a technique that is already being deployed in sectors such as the medical field (White et al., Reference White, Fu, Hays, Sandborn, Olea, Gilbert, Elnashar, Spencer-Smith and Schmidt2023). Ultimately, the choice of prompt structure and overall systemic design will depend on the specific task and available resources—an area that needs further exploration in the context of judicial systems and the use of GenAI.

4.2.1. Updating ethical standards and enhancing continuous legal education

With the potential benefits of integrating GenAI into the courtrooms, also come risks—and with these risks, increased duties and responsibilities for judges.Footnote ²⁴ The inclusion of GenAI in the judicial system requires a review of ethical standards of the legal profession, alongside holistic legal training to educate current and future judges (and lawyers) on the use and misuse of GenAI in the courtroom. Such training should entail enabling practitioners to manage the existence, capabilities, and limitations of these technologies.

First, legal ethics requirements should be updated to include requirements regarding the use of GenAI. Such requirements could address issues like transparency and disclosure duties in AI use, data privacy considerations, and potential bias management and review responsibilities in AI outputs. Secondly, lawyers and judges should be mandatorily required to learn about these technologies under jurisdiction-wide strategies employed to bridge the gap in the tech savviness of stakeholders in the legal system. Continuous and updated legal education requirements could be a valuable tool for this purpose. However, this would necessitate building those capabilities from the ground for jurisdictions like Colombia—one of the court systems analyzed in the case studies above—which currently lack mandatory CLE programs.Footnote ²⁵

4.2.2. Case-based risk assessment

Not all legal cases are created equal when it comes to GenAI integration in the courtroom, and likewise, not all potential uses of GenAI by judicial office-holders create the same risks. There are multiple uses for GenAI that hold different risks. For example, using GenAI to draft decision texts after the legal reasoning has been decided has a different risk profile than assisting a judge in the decision-making process, or gathering context and determining the weight of certain proof around a general factual claim. A crucial aspect of developing a responsible judicial GenAI use is establishing a risk assessment framework that guides judges in (i) determining whether the application of the technology is warranted or appropriate, and (ii) what level of review or scrutiny a GenAI output may warrant. This framework should allow judges to exercise discretion based on a calculated risk analysis.

• • Risk-Based Deployment and Modular Accuracy Thresholds: For cases with significant potential consequences for a defendant’s rights, such as those involving criminal sentencing or loss of liberty, a heightened level of caution for the deployment of GenAI is warranted. In these high-risk use cases, stricter accuracy thresholds for GenAI outputs should be mandated, and a higher level of responsibility should be provided to the reviewing process within the judicial system. These thresholds could be based on metrics relevant to the specific legal domain, such as the predictive accuracy of risk assessment algorithms, or a more precise and narrow database sourcing of information to generate the system’s output. This should also affect the scope that GenAI can have on the specific case—from being excluded from deployment altogether given the high risk of infringing upon an individual’s rights; to limited application, ranging from reviewing case law or weighing evidence; to potentially providing comprehensive adjudication and decision for a process, to be reviewed by judges or judicial system staff.

• • A Graduated Approach to Review and Responsibility: The level of review and scrutiny for GenAI outputs, and the level of responsibility mandated, should also be commensurate with the case’s risk. Low-risk cases might require a basic review of the GenAI output by the judicial system. However, high-risk cases would necessitate a more thorough examination by the judge, clerks, or a specific organization dedicated to these reviews. These increased review requirements could include legal professionals delving deeper into the GenAI’s reasoning process, the data used (e.g., case precedents, laws, and extra-legal information), and potential biases. Correspondingly, as the risk of infringing upon rights and freedoms increases, the responsibility borne by judges (and developers) should be heightened. Judges must evaluate the GenAI’s outputs and ensure they align with their own legal reasoning before incorporating them into decisions. Developers, on the other hand, bear accountability for the algorithmic design and functionality of their GenAI systems in these high-risk scenarios.

By implementing a risk-based deployment model with escalating levels of accuracy thresholds, appropriately allocated review requirements, and shared responsibility, the judicial system can leverage the potential benefits of GenAI while prioritizing the protection of fundamental rights and freedoms.

4.2.3. Disclosure to the parties

When GenAI is deployed in a judicial process, this article proposes mandatory disclosure of its use to all relevant parties. This transparency serves several key purposes:

• • Empowering Parties, Ensuring Fairness: By knowing GenAI is involved, parties can make informed decisions about their legal strategy. For instance, they might choose to challenge the GenAI output or request a more traditional review process. Ultimately, transparency promotes fairness in the proceedings by ensuring all parties have a clear understanding of the tools being employed and the ability to shape their participation accordingly.

• • Promoting Trust, Procedural Awareness: Transparency builds trust in the judicial process. Knowing GenAI is being used allows parties to have awareness of their procedural status, the rights available to them, and with these, have the confidence that their case is being considered within the boundaries of due process.

To effectively fulfill these purposes, the disclosure of GenAI should follow robust and transparent methods. These information distribution methods should prioritize clarity, accessibility, and comprehensiveness. Information disclosed should be presented in a way that is understandable to all parties involved, regardless of their technical or legal expertise. Additionally, the disclosure should be comprehensive, detailing, among other items, the specific type of GenAI tool used, the stage(s) in the process where it will be employed, the limitations of the technology, and redress rights.

• • Methods and Content of Disclosure: The specific method of disclosure can be tailored to the risk profile of the case, the type of GenAI used, and the means and resources of the court. Some due inclusions and potential approaches follow:

  1. i. Timeline and Consent: The timing of disclosure can be adjusted to the case, court, and risk level associated with the case.

     1. a. A Priori Disclosure for Informed Consent: Early “a priori” disclosure should be considered preferred practice. Informing the parties of GenAI involvement prior to its deployment maintains procedural integrity and user awareness. Knowing how their case is being managed fosters a sense of agency and allows parties to, for example, challenge the GenAI output, advocate for limiting GenAI use to specific aspects of the case, or challenge the technology’s deployment in the proceedings altogether. Consent to the use of GenAI by the parties is a variable that should respond to the specific risk and case, as well as to other considerations that may inform a jurisdiction’s judicial policy.

     2. b. A Posteriori Disclosure for Urgent Situations: For unforeseen circumstances where GenAI becomes necessary, specifically in low-risk cases, “a posteriori” disclosure could be permitted. However, even in these situations, prompt disclosure after the fact is still essential. This ensures fairness and allows parties to understand how GenAI may have influenced the proceedings.

     3. c. Risk-Adjusted Disclosure Strategy: The timing of disclosure should be part of a risk-adjusted disclosure strategy. This strategy considers the potential consequences of the case and tailors the level of transparency accordingly. High-risk situations demand the utmost transparency, with early disclosure and detailed explanations being paramount. Conversely, low-risk cases allow for more flexibility in the timing of disclosure, as long as information is promptly distributed, and in some cases, consent is ultimately obtained. By implementing a risk-adjusted disclosure strategy, the judicial system can strike a balance between transparency and efficiency where parties are empowered with the information they need to participate in the process effectively and with awareness of their status, while the legal process maintains its momentum.

  2. ii. Method: The chosen method for disclosing GenAI use should be flexible and responsive to the specific court and case. Court orders might be appropriate for formal notification, while pre-trial motions could be a tool for parties to raise concerns and trigger disclosure discussions. Ultimately, the method should ensure timely and comprehensive information reaches all relevant parties.

  3. iii. Level of scrutiny: The disclosure information should outline the level of scrutiny the judge will apply to the GenAI output. This could range from a cursory review to a more in-depth analysis depending on the case’s risk and the GenAI’s role.

  4. iv. General or Specific Involvement of GenAI: The disclosure should clarify the extent of GenAI involvement in the case. This may include disclosures on whether GenAI will be deployed (a) generally throughout the process, or (b) for specific purposes (e.g., weighing evidence, identifying relevant case law, or structuring attenuating circumstances). This transparency allows parties to understand the potential impact of GenAI on their case, and exercise their rights accordingly.

  5. v. Explanatory Documents: The court could provide parties with explanatory documents outlining the nature and limitations of the GenAI system used.

  6. vi. Escalation and Review Rights: Disclosure should inform parties of their rights, including the right to challenge the use of GenAI in the case comprehensively, modify the scrutiny and risk basis, request a review of the decision-making process, or generally challenge for process or accuracy of outcome (among others); as well as secondary post-adjudication motions, if they believe the GenAI output is flawed or biased.

• • Balancing Disclosure with Efficiency: While transparency is crucial, it should be balanced with the efficiency advantages GenAI provides to the court system. A court-and-case-specific balance must be pursued, ensuring parties understand the role of GenAI, and their rights associated with said use, without hindering the enhanced efficiency of the legal process.

By establishing clear disclosure protocols, the judicial system can ensure transparency and fairness when GenAI is integrated into court proceedings. This fosters trust in the system and empowers parties to navigate the legal process effectively.

4.2.4. Verification system

While GenAI holds promise for enhancing judicial efficiency, ensuring the veracity, reliability, and non-biased nature of its outputs is paramount. As such, it is crucial to establish a verification network certifying that GenAI-generated content is reviewed for fairness and precision, sometimes requiring human oversight.

• • Scrutiny Tailored to Risk: The level of scrutiny applied to GenAI content should be commensurate with the risk associated with the case. High-risk cases, such as those involving criminal sentencing or significant impacts on rights and freedoms, necessitate a more rigorous verification process. Information generated by GenAI, particularly in high-risk scenarios, should be subject to independent verification by court personnel or qualified third parties. This could involve cross-referencing data sources, verifying legal citations, and ensuring factual accuracy. Conversely, low-risk cases may require a reduced review process, tailored to the needs and rights affected by each specific process.

• • Flexible Review Network: The verification process should be adaptable to the specific court and case. For instance, courts with limited resources could leverage standardized verification protocols and tools. Courts handling complex cases might require the formation of specialized review teams with expertise in AI and relevant legal domains. Essentially, this article proposes an adaptive review network, the standards of which can be decided by the relevant body of government of judges, and deployed via its network of clerks, staff, the judges themselves, potentially operating in collaboration with automated review technology.

• • GenAI as an Enhancer, not a Bottleneck: Verification should not become cumbersome to the extent it slows down the judicial system. The goal is to utilize GenAI as an efficiency enhancer while mitigating potential risks. Streamlined verification procedures and leveraging technology-assisted verification tools can help achieve this balance.

4.2.5. Specific procedural rights: Scope, consent, and escalation procedures

Integrating GenAI into the courtroom necessitates establishing clear procedures for individuals to safeguard their rights and challenge potential biases or errors. Some key rights and consequent considerations include:

• • Right to Opt-Out or Modify AI Use:

  • – Refusal of AI Integration: In certain situations, particularly those with the highest level of risk to infringe upon fundamental freedoms, individuals should have the right to refuse GenAI integration into their case entirely. This method would allow parties to opt for a traditional judicial process if they have concerns about fairness or transparency.

  • – Modification of AI Scope: Individuals should have the right to influence the scope of GenAI use within their case. This could involve choosing between:

    • ▪ Full Integration: GenAI is used for the entire trial, potentially analyzing evidence, suggesting relevant case law, and aiding in judicial reasoning.

    • ▪ Limited Application: GenAI use is restricted to specific tasks, such as analyzing historical precedents or weighing the strength of evidence. This allows for a more targeted approach and potentially mitigates risks associated with broader AI involvement.

• • Escalation and Appeals:

  • – Right to Appeal: Just as with traditional judicial procedures, individuals should retain the right to appeal. Within the context of GenAI application specifically, individuals should have the right to challenge (i) adjudicative or procedural decisions made with the aid of GenAI, and (ii) specifically appeal the results of GenAI-generated content. This ensures individuals have recourse if they believe the GenAI output has significantly impacted the outcome in a way that is unfair or inaccurate.

  • – Transparency in Appeals: Appeals processes involving GenAI-aided decisions should be transparent. Individuals should have access to explanations of how the GenAI was used and how it may have influenced the decision. This allows for a more informed, fair, and targeted appeal process.

4.2.6. Ongoing audits and evaluation

Algorithmic audits should also become a common tool of GenAI practice and a staple of regulation (Bandy, Reference Bandy2021). Even outside of the implementation of this regulatory proposal, AI models like ChatGPT do go through various forms of auditing and evaluation, notwithstanding the fact that the specifics of these processes are not fully disclosed to the public for proprietary and even reputational reasons.Footnote ²⁶ It is not currently clear how AI audits should be conducted, and the private sector has developed an unstandardized approach, which undermines interoperability. This is still a nascent field in which regulators need to think about the basic components of such audits before they can reliably address algorithmic harms (Goodman and Trehu, Reference Goodman and Trehu2023). Nonetheless, an adequate, iterative, and transparent AI audit framework for GenAI that is used in courts could increase the transparency and interpretability of these AI systems and overall enhance fairness and public trust.