This Article examines the “human in the loop” argument regarding the increasing use of artificial intelligence (AI) in court proceedings, challenging the intuition that AI assistance is inherently less problematic than full delegation. It argues that even limited AI support can risk blurring the lines between human and AI decision-making, posing significant dangers for human judgment in critical judicial functions.

The Article starts out by identifying the main arguments in favor of human oversight and categorizing them into technological, legal, and psychological reasoning in Section B. It then shifts focus to the potential dangers of applying the “human in the loop” concept to judicial decision-making in Section C. The analysis highlights the fundamentally different modes of operation between humans and AI, particularly in handling natural language and legal reasoning. Furthermore, it explores how human judges might become over-reliant on AI, effectively acting as “rubber stampers” and leading to eroding human vigilance, skills, and independent judgment. To illustrate the complexities of these dynamics, the Article categorizes various AI tools used in court proceedings based on their degree of involvement and effect on judicial decision-making in Section D.

The Article concludes by urging us to rethink our current understanding of “co-working” with AI as a universal remedy and putting emphasis on a clear division of labor instead, as discussed in Section E. For those scenarios where human and AI contributions are deeply interwoven, the Article stresses the need for making a conscious decision whether AI or a human judge should perform the underlying tasks in the future.

This paper examines the legal implications of the explicit mentioning of automation bias (AB) in the Artificial Intelligence Act (AIA). The AIA mandates human oversight for high-risk AI systems and requires providers to enable awareness of AB, i.e., the human tendency to over-rely on AI outputs. The paper analyses the embedding of this extra-juridical concept in the AIA, the asymmetric division of responsibility between AI providers and deployers for mitigating AB, and the challenges of legally enforcing this novel awareness requirement. The analysis shows that the AIA’s focus on providers does not adequately address design and context as causes of AB, and questions whether the AIA should directly regulate the risk of AB rather than just mandating awareness. As the AIA’s approach requires a balance between legal mandates and behavioural science, the paper proposes that harmonised standards should reference the state of research on AB and human-AI interaction, holding both providers and deployers accountable. Ultimately, further empirical research on human-AI interaction will be essential for effective safeguards.

Evaluating the algorithmic behavior of interactive systems is complex and time-consuming. Developers increasingly recognize the importance of accountability for their algorithmic creations’ unanticipated behavior and resulting implications. To mitigate this phenomenon, developers not only need to concentrate on the observable inaccuracies that can be measured quantitatively but also the more subjective outcomes that can perpetuate social bias, which are challenging to identify. We require a new approach that involves humans in scrutinizing algorithmic behavior. It leverages a combination of quantitative and qualitative methods to support an ethical, value-aligned design and a system’s lifecycle, informed by users’ perception and values. To date, the literature lacks an agreed-upon framework for such an approach. Consequently, we propose an oversight framework, Modular Oversight Methodology (MOM), which aids developers in assessing the behavior of their systems by involving a carefully crowdsourced society-in-the-loop. The framework facilitates the development and execution of an oversight process and can be tweaked according to the domain and application of use. Through such an oversight process, developers can assess the human perception of the algorithmic behavior under inspection, and extract valuable insights that will aid in assessing its implications. We present the MOM framework, as a first step toward tailoring more robust, domain-specific solutions to exercise human oversight over algorithms, as a means for software developers to keep the generated output of their solutions fair and trustworthy.

Human decisions are increasingly supported by decision support systems (DSS). Humans are required to remain “on the loop,” by monitoring and approving/rejecting machine recommendations. However, use of DSS can lead to overreliance on machines, reducing human oversight. This paper proposes “reflection machines” (RM) to increase meaningful human control. An RM provides a medical expert not with suggestions for a decision, but with questions that stimulate reflection about decisions. It can refer to data points or suggest counterarguments that are less compatible with the planned decision. RMs think against the proposed decision in order to increase human resistance against automation complacency. Building on preliminary research, this paper will (1) make a case for deriving a set of design requirements for RMs from EU regulations, (2) suggest a way how RMs could support decision-making, (3) describe the possibility of how a prototype of an RM could apply to the medical domain of chronic low back pain, and (4) highlight the importance of exploring an RM’s functionality and the experiences of users working with it.