A. Introduction: The Emergence of New Humans in Law
Digitalization has the power to both inspire and confuse. While it has been celebrated as a transformative development, it has also raised significant legal concerns. The EU’s technology regulations—such as the General Data Protection Regulation (GDPR), Digital Services Act (DSA), and Artificial Intelligence Act (AIA)—have proliferated, affecting businesses, public authorities, and citizens alike.Footnote 1 This Article focuses on public administration, acknowledging that the lines between public and private sectors are increasingly blurred in this context. In this Article, I argue that technology regulation has enabled the emergence of so-called “new humans” in law, and that further necessitates a reevaluation of legal concepts and practices. This reevaluation, in turn, will be approached through a new conception of legal technology.
First, I suggest that in addition to seismic shifts, subtle changes in law are taking place. Though surreptitious, I argue that they are equally as important as the more dramatic ones. These subtle changes take place in legal concepts, doctrines, and mindsets. In this Article, I delve into one of these developments: A development I call the emergence of “new humans” in law. This means that instead of natural persons, clerks, bureaucrats, or civil servants, “humans” are assigned legal assignments and power. The emergence of these “new humans” can be detected from the concepts used in technology regulation and AI ethics discourse. Humans are required to be in the loop, on the loop, or in command. This is reflected in legal roles such as “human intervention,” “human oversight,” or “human judgment.” Additionally, humans have indirect expressions in law, as I will explain later.
Importantly, rather than education, status, or profession, the source of the authority of “new humans” seems to emanate from their sheer humanity.Footnote 2 Like in Robert Musil’s famous novel, these humans are men without qualities. These humans, I argue, stem from a legal attention that is directed towards digital technologies. Consequently, the role of humans as interactors with digital tools ends up being regulated. As a result, new humans make visible the “old” humans by questioning their monopoly to apply the law on the one hand and recasting them in new roles on the other. I argue that as for now, the emergence of the new humans and their concomitant authority remains, if not undetected, at least under analyzed and undertheorized.
As a reflection of this theoretical deficiency, the doctrine of new humans in public law doctrine has not properly developed. The lack of appropriate concepts and doctrines can have an impact on how law can capture and regulate the changes that digitalization brings about.Footnote 3 For example, the monopoly of humans applying the law has been transformed into a gold standard of humans. This means that on the one hand, the human touch is still something that is considered to affect the legitimacy of legal decision-making, regardless of quality.Footnote 4 On the other hand, some established human-centered concepts of public law such as official accountability, struggle to fit into the reality of automated decision-making, whereby an accountable human might be but a legal fiction.Footnote 5
Second, I propose that we could look at the new legal landscape through the lens of a legal technology. In this Article, I develop a blueprint for this conception. The idea is to marry the metalanguage (the terminology of the researcher) with the object language (the terminology of the research object; here: technology regulation) and see which phenomena can in this way be made visible. How do different legal practices, analog or digital, look through the lens of legal technology? As a working definition, I take legal technology to be a formalized process of creating a legal product, an artifact, such as an administrative decision. This is to say, I approach legal technology as a wider concept that is customary, in other words, as the use of software in the legal industry.Footnote 6
In this Article, my goal is thus to make two contributions. First, I show and problematize the emergence of “new humans” as a corollary of digital technologies in European technology regulation and ethical discourses. I argue that this notion of “human” is a conceptual transplant from the discourses of technology and its incorporation into law is uneasy. Second, I suggest that the emergence of new humans and the following reimagining of legal professionals give rise to a new framing that requires us to adopt a wider conception of legal technology. Through this conception, we could dismantle the distinction between humans and machines in public administration and approach all—digital tools, humans, their legal labor, and even mindsets—as legal technologies.
Therefore, the main goal in this Article is to introduce a novel idea, the first attempt to suggest that we could widen the concept of legal technology. As I explain, this would add value by making visible something I call naïve human exceptionalism—the uncritical reliance on human superiority—embedded in our legal thinking and would question its sustainability. I conclude that the superiority of humans seems to build on a suspicion of technology and the traditional roles assigned to humans rather than superiority in quality of decision-making, while these humans silently carry other legal technologies in themselves.
The Article is organized as follows. In the next section I look at human-centrism as the matrix through which the new humans infiltrate into our legal vocabulary and legal thinking. After that, in the third section, I move to discuss the idea of “human in the loop” as one aspect of human-centrism. I also look at examples from the AIA and the GDPR and argue that the emergence of new humans within them has taken place without sufficient analysis and critique. In the fourth section, I will move on to discuss legal technology. How is it understood in contemporary discussions, and should we think differently about it? In the fifth section, I apply the new conception of legal technology to humans and attempt to diagnose what kind of legal technology humans make in the bureaucratic machinery of the state. Finally, in the sixth section, I present brief conclusions.
B. Humans in the Center
I. Which Humans?
The superiority of humans in executing law is often assumed but rarely argued. According to this assumption, humans are good or at least predictable, whereas technology is unknown and potentially dangerous. This positive effect of the human touch reflects the so-called the anthropocentric bias: A condition whereby humans are privileged in applying the law without naming the source and nature of this privilege;Footnote 7 human work is just “something higher” than “lower” machines,Footnote 8 which make a dubious “other.”Footnote 9 This bias can be further argued to reflect and reproduce so-called human exceptionalism, the notion that humans are not only different from other species or agents but are also greater in moral value.Footnote 10 Human exceptionalism seems to enjoy self-evident value in our legal thinking, although a contrasting view, building on so-called flat ontology—for example, in discussions of new materialism and post-humanismFootnote 11 —is also gaining ground in legal scholarship.Footnote 12
Anthropocentric bias creates new hierarchies in law and legal practices.Footnote 13 The existence of such hierarchies could be expressed thus: Human decision-making is better than automated decision-making with a human in the loop, and automated decision-making with a human in the loop is better than automated decision-making without it (human > human in the loop > pure digitalization). In one way, this ranking makes humans superior to digital technologies, in another way it makes them auxiliary or subordinate to digital technologies. To illustrate: Why is human decision-making not subject to human oversight but to judicial control?Footnote 14 Sometimes, the role of humans is residual or exclusive: If automation is not allowed for certain tasks, humans may still be allowed to carry out those tasks. I suggest that this is due to the silent reliance of humans on law, although their dark sides and deficits are largely acknowledged.Footnote 15
However, in public law, the concept of a human has a different rationality. “Humans” are primarily conceptualized as carrying rights: Human rights. This means human rights protect their subjects from the misuse of governmental power or create rights to get something from the government, such as social benefits or education. Conceptually—if not historically—this “human” in human rights discourse can be argued to derive its meaning from two sources: belonging to the species homo sapiens,Footnote 16 and a legal relationship to governmental power. Moreover, this notion of human rights having universally “intrinsic worth” also refers to human dignity. Whether this “intrinsic worth” is legally guaranteed in practice is a different discussion that has been the subject of wide and multifaceted study. In any case, reflecting the Kantian categorical imperative, “human” in human rights discourse is constructed as an end himself, someone who is invaluable just by existing and therefore merits rights.Footnote 17
Importantly, “human” is something else than a “person” in law. The connection between legal persons and humans is complex, and at times, the concepts overlap in the point of reference. Unlike “human,” “person” is a functional concept that approaches different entities from the perspective of legal transactions. In the long tradition of European law—extending all the way back to Roman law—people are divided into natural and legal or artificial persons. The latter are typically corporations. Both can have legal rights and obligations. In the wake of the digitalization and the development of AI, the limits of artificial persons have become a topic of heated debate.Footnote 18
Due to the possibility of artificial or legal persons of law, as Britta van Beers argues, a person and a body can be separated from each other in law. On the one hand, a human would be equated with his free will.Footnote 19 On the other hand, legal persons are quite openly legal fictions. No one claims, for example, that a corporation should have a body to claim legal existence. In contrast, to date, our legal conceptual system does not recognize “artificial” or “legal” humans.Footnote 20 Humans are essentially biological beings, whose legal status is, in a way, anchored to their body. In other words, a body seems to be an inalienable part of what constitutes a human in law. In the future, this might change, as, in van Beers’ words, “the artificialization of human life is running parallel with the artificialization of law’s natural person.”Footnote 21
In this Article, I do not primarily concentrate on humans as carriers of rights, humans-as-an-end-themselves or free legal subjects. Neither do I concentrate on natural persons, although in some contexts, humans are specified to signify a natural person. Instead, I suggest that there is a new legal construction called a “human” which is in some contexts expressis verbis mentioned, and in some other contexts it is implicitly assumed. In the discourse on the new humans, humans are something other than humans in human rights discourse, although both are necessarily embodied, unlike persons. Instead of intrinsic worth legally vested in humans as the correlative of public authority, new humans derive their meaning from the contrast with, and relation to, digital technology.
That said, the new humans are not characterized by their intrinsic worth. Instead, in the discourse of new humans, a human becomes a means to an end, and that end is the control of technology. Importantly, the “humanness” of these new humans means primarily the identity of a non-machine;Footnote 22 while at the same time—and perhaps paradoxically—new humans are part of governmental machinery. The new humans exist both spontaneously (a naturalistic, embodied view of humans) and are constructed legally, (the emergence of new humans as opposed to the old ones). As I argue later, this framing as non-machine, in fact, facilitates our understanding of the covertly technological nature of humans.
No doubt, the new humans also possess dignity and rights. However, when understood as non-machines, this dignity becomes bracketed out, paving the way for a new, technologized “anthropogenesis”:Footnote 23 The creation of humans. The new humans are characterized by their ability to act as a backstop to digital technology. By governing technology, they exercise indirect power over subject-humans, humans as rights carriers, and function as a medium of governmental power. As in the case of legal persons, the meaning of the new humans resides in their functional or instrumental rather than existential value. However, as I show, the discourses on old and new humans intersect in technology regulation and AI ethics under the banner of “human-centrism.”
II. Human-Centrism as an Ideal
Today, human-centrism—the idea that humans should be in control of machines both in terms of goals and functioning—and human agency are emphasized perhaps more than ever in technology regulation and policy discourse. Human-centrism has a strong institutional support in technology regulation and the AI ethics guidelines that preceded it and still surround it. Since 2018, there has been a surge of AI ethics guidelines. One could argue that these guidelines are proto-legal in the sense that they have paved the way for hard regulation. Algorithm Watch is an NGO monitoring the development of automated society. Its inventory shows there were altogether 167 ethics guidelines on automated decision-making and AI, given by NGOs, governments, and companies in 2020.Footnote 24
One of the most influential of these guidelines is the Ethic Guidelines for Trustworthy AI, developed by the European Commisson’s Independent High-Level Expert Group (AI HLEG). The significance of these guidelines and the formulation of the key principles in the AIA clearly show that an understanding of human-centrism is worth a closer look. According to the introduction of the Guidelines, “[they] put forward a set of 7 key requirements that AI systems should meet in order to be deemed trustworthy.”Footnote 25 This means that the goal and execution of the principles is to create AI technology such that it induces trust towards it. To reach that goal, the Guidelines aspire to be more than just a list of principles. Rather, “these Guidelines seek to go beyond a list of ethical principles, by providing guidance on how such principles can be operationalised in socio-technical systems.”Footnote 26
The first of the AI HLEG principles is called Human Agency and Oversight: “AI systems should empower human beings, allowing them to make informed decisions and fostering their fundamental rights. At the same time, proper oversight mechanisms need to be ensured, which can be achieved through human-in-the-loop, human-on-the-loop, and human-in-command approaches.”Footnote 27
The idea behind this principle has significant implications as it emphasizes the inherent value of human agency: “AI systems need to be human-centric, resting on a commitment to their use in the service of humanity and the common good, with the goal of improving human welfare and freedom.”Footnote 28 The Guidelines also state that
The common foundation that unites these rights [dignity, freedoms, equality and solidarity, citizens’ rights and justice] can be understood as rooted in respect for human dignity – thereby reflecting what we describe as a ‘human-centric approach’ in which the human being enjoys a unique and inalienable moral status of primacy in the civil, political, economic and social fields.Footnote 29
Indeed, human-centrism is an umbrella term under which many different aspects and subprinciples gather. In addition to keeping humans in command or at least in or on the loop, the Guidelines call for the respect of human autonomy and human rights. The formulation laid down in the AI HLEG principles of human autonomy was later adopted into the AIA in its first recital. Among other goals, the Regulation is
To promote the uptake of human-centric and trustworthy artificial intelligence (AI) while ensuring a high level of protection of health, safety, fundamental rights as enshrined in the Charter of Fundamental Rights of the European Union (the ‘Charter’), including democracy, the rule of law and environmental protection, to protect against the harmful effects of AI systems in the Union, and to support innovation.Footnote 30
Interestingly, both in the AI HLEG principles and in the AIA recital, human-centrism is mentioned along with safety and trustworthiness, and human empowerment. This co-mentioning—their frequent collocation, as it is called in linguistics—creates an atmosphere of harmony and coherence. The connection between the mentioned goals is further strengthened by the fact that the AI HLEG principles are explicitly mentioned in AIA recitals 7 and 27. The latter recital lays down that: “According to the guidelines of the AI HLEG, human agency and oversight means that AI systems are developed and used as a tool that serves people, respects human dignity and personal autonomy, and that is functioning in a way that can be appropriately controlled and overseen by humans.”Footnote 31
Therefore, human agency and human oversight bring three things together: (1) AI’s function is to serve humans—it should be subject to human goals and design; (2) AI should respect human dignity and personal autonomy—letting people make informed choices without manipulation or exploitation; and (3) humans should control and oversee AI systems. This is to say that humans appear in the AIA in three registers: as goal setters of AI, as its dignified users or subjects, and as its overseers and controllers. This assumes that protecting the intrinsic value of humans-as-an-end-themselves requires specifically humans to do this, in other words, humans-as-a-means-to-an-end. I will question this assumption later in the Article.
Thus, the identity of a “human” has widened to cover not only the subject of human rights but also to the human in the loop. In this way, human-centrism couples the “old” concept of humans as rights-holders with the “new” concept of humans as non-machines. Consequently, the human rights discourse and the new legal discourse on humans intersect and intertwine, making the interests of humans the only legitimate concern in the design and use of digital technology. It seems that from centering human rights and dignity it is assumed to follow that humans are also the best in carrying out the best possible decisions—either in a supervisory role, or ideally, as the privileged decision-maker due to their nature as non-machines. The assumption is that humans can best empower humans.
At first glance, the emphasis on human-centrism looks like a desirable development. It promises human flourishing and human control while allowing us to enjoy the benefits that the new digital technologies entail. Humans’ interest would govern the development and governance of digital technologies. On closer inspection, however, there are two interesting implications worth mentioning.
First, we can assume that the growing recognition of human-centrism is symptomatic of the jeopardization of this very idea. What I mean by this is that the emphasis on human-centricity has arisen as a defense mechanism because human interests are in danger of becoming marginalized. In other words, emphasizing human centricity implies that other centrisms—such as techno-centrism—are also possible. As Lena Enqvist puts it, “[a]s the most pressing issues of AI deployment relate to the fear that technological development and rationalised efficiency will take place at the cost of human agency and safety or rights, ‘human-centrism’ is an obvious counterweight.”Footnote 32 The entire concept of human-centrism can thus be seen as a byproduct of digitalization and the concomitant reimagination of the role of humans. Indirectly, it manifests the underlying pursuit to gain economic profit through the adoption of AI technology.
Second, as technical concepts infiltrate legal vocabulary and doctrines, the digitalization of law is not only a matter of deploying technology, it is also a matter of conceptual change.Footnote 33 Conceptual borrowing is not neutral: as we have learned from Carl Schmitt, (“All significant concepts of the modern theory of the state are secularized theological concepts.”),Footnote 34 “[e]very technical term is part of a network of conceptual structures to which it remains linked, providing contextual constraints and exerting semantic influences and powers.”Footnote 35 Paradoxically, although the notion of a “human” seems the opposite of a technological concept, the emergence of the new humans can be argued to represent this very phenomenon. The new humans are humans from the discourses of technology and as such these concepts disrupt the way in which humans are traditionally understood in legal thinking,Footnote 36 making a “conceptual bottleneck” in law.Footnote 37 As already mentioned, this disruption may be connected to the anthropocentric bias and the hierarchies it produces.
The entanglement of old and new humans discourse matters because an essential power relation is changing or, at least, it appears to be changing. Instead of the relationship between a human as a subject of power/human rights and the government as the privileged exerciser of power, the relationship becomes redefined.Footnote 38 Now, the key boundary is positioned between humans—covering both old and new humans—and technology. The implication is deeply unpolitical: That the emergence of technology would put all humans, regardless of whether they represent power as cogs of the governmental machine or are subject to it as citizens, on one side of a power relationship and technology on the other.Footnote 39 As I explain later, this calls us to question what kind of technology has the power to jeopardize human dignity.
C. Humans in the Loop
I. Examples of Humans in the Loop
As explained, human-centrism brings together the discourse on new and old humans, making humans a boundary object for ethical, legal, and technological discourses. In the following, I discuss a few examples of mentions or assumptions on new humans in technology regulation.Footnote 40 The goal of this brief discussion is to illustrate how new humans have penetrated our law and legal thinking and what follows from that. This goal means that the following discussion is not a comprehensive analysis of the chosen provisions or exegesis of all the relevant literature. Instead, it serves as the formation of the general argument of the Article. Although humans are called for in the loop in various automated systems, for example, in the context of automated vehiclesFootnote 41 or weapons,Footnote 42 the following examples consider EU-regulated automated decision-making,Footnote 43 generally in public administrationFootnote 44 and its limitations. I now discuss human oversight, human intervention, and the limits of automation.
1. Human Oversight
The new humans have an array of legal duties in the AIA under the canopy of human oversight. Human oversight means that humans must be able to supervise the workings of AI systems and stop the system when necessary. In addition, human oversight should extend to the design of the AI systems.Footnote 45 Along with the adoption of the AIA, human oversight has become a legal concept, AIA 14 Article, and as such is a novelty in European law.Footnote 46 The AIA proposes a set of requirements for so-called “high-risk” AI systems to be designed and developed in such a way that they can be effectively overseen by natural persons during their use. As we can see, in the context of human oversight in the AIA, the role of a new humans can be filled with a “natural person.”Footnote 47
According to the AIA 14(2):
Human oversight shall aim to prevent or minimise the risks to health, safety or fundamental rights that may emerge when a high-risk AI system is used in accordance with its intended purpose or under conditions of reasonably foreseeable misuse, in particular where such risks persist despite the application of other requirements set out in this Section.Footnote 48
The ambition level is thus high, and much hope is vested in human oversight to prevent different risks from occurring:
The high-risk AI system shall be provided to the deployer in such a way that natural persons to whom human oversight is assigned are enabled, as appropriate and proportionate:
(a) to properly understand the relevant capacities and limitations of the high-risk AI system and be able to duly monitor its operation, including in view of detecting and addressing anomalies, dysfunctions and unexpected performance;
(b) to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias), in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons;
(c) to correctly interpret the high-risk AI system’s output, taking into account, for example, the interpretation tools and methods available;
(d) to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output of the high-risk AI system;
(e) to intervene in the operation of the high-risk AI system or interrupt the system through a “stop” button or a similar procedure that allows the system to come to a halt in a safe state.Footnote 49
The provision thus assigns many demanding assignments and attributes to overseers. Overseers must be highly capable and vigilant, and able to understand, monitor, remain aware, detect, interpret, decide, disregard, override, reverse, and intervene the operations of AI systems when necessary and ultimately, know when to push the stop button.Footnote 50 These requirements thus extend to the use of the AI system and the monitoring of its function, although Article 14 also includes design obligations. In other words, it has both ex-ante and ex-post elements, although human oversight is reactive rather than proactive.Footnote 51
2. The Right to Human Intervention
The new humans are also explicitly mentioned in the GDPR in the form of the right to human intervention. In this context, too, humans are a corollary of automation: Human intervention would not be needed were the original decision made by humans.Footnote 52 Instead, human decisions are subject to a legal corrective mechanism whereby humans are assumed but not mentioned. In other words, a similar layer of corrective mechanisms is lacking as it is with human decision-making.
In the GDPR Article 22(1), a data subject has the right not to be subject to automated decision-making with legal or similar effects. However, this right has many exceptions, and automated decision making is common. For example, ADM can be permitted under national law. If this is the case, certain legal safeguards must be guaranteed, one of which being the right to human intervention (GDPR recital 71).Footnote 53
According to the European Commission Data Protection Working Party, Article 22(1) applies only if “there is no human involvement in the decision process.”Footnote 54 In addition, the Working Party guidance document suggests that “[t]o qualify as human involvement, the controller must ensure that any oversight of the decision is meaningful, rather than just a token gesture.”Footnote 55 The right to human intervention requires that the original decision is fully automated. The assumption is that if the ADM fails, the problem is solved or mitigated by a human stepping in and reassessing the ADM’s decision. However, it remains unstated what it is that constitutes the legitimating effect of human intervention as a legal safeguard. Human intervention is an ex-post perspective that keeps humans in the loop.Footnote 56
As a result, people do not have a right not to be subject to human decisions, which makes human decisions legally privileged in law. Some authors, in turn, understand GDPR 22 as a right to a human decision and not an obligation to be subject to human decision-making.Footnote 57 Either way, the right to human intervention means that human involvement becomes a legal safeguard, and as such, carries an inherent promise of better-quality decision-making than that made by automated systems.
2.1. Limits of Automation
Humans in the loop is not always explicitly regulated, yet regulation may embody this idea indirectly. More accurately, the loop may not include human involvement but form an alternative loop with certain decisions. Sometimes the new humans make an appearance when certain practices are banned from automated practices. If this is the case, some practices may still be allowed for humans to carry out. If we look at the AIA and the GDPR alike, we notice that certain practices are forbidden or left unregulated. For example, the GDPR does not grant a right not to be subject to human decision-making, although human decision-making can fail in numerous ways.
AIA Article 5, in turn, lists several prohibited AI practices.Footnote 58 Article 5 is, however, unusual in the sense that some practices it prohibits would not be possible for humans. AI is potent in activities that are impossible for most humans. Thus, it does not mean much if humans were prohibited to manipulate subliminally if that kind of manipulation were impossible in practice, or possible only to some potentially malevolent experts. That is to say, all limits of automation do not mean that human decision-making is privileged, as human decision-making might be blocked due to its limitations.
Not all automated decision-making includes AI, and automation can be limited for various reasons. As mentioned, automated decision-making can be allowed under national legislation. For example, in Finland, automated decision-making is possible in public administration only with decisions that do not include case-by-case discretion. This norm includes an obligation to deliberate beforehand, which decision types include this kind of case-by-case discretion so that it can be ruled out from the scope of the automation.Footnote 59
In effect, discretionary decision types—for example, those that require the assessment of vague language, the role of legal principles, or individual circumstances—cannot be subject to automated decision-making regardless of the technology used. Instead, they must be subject to human decision-making. This implies that automation is limited due to the discretion that the decision-making requires, and this discretion is legally construed as compatible only with human decision-making.Footnote 60 I return to this point later.
II. Different Roles in Different Loops
The implicit and explicit limits of automation show a bias towards favoring human decision-making regardless of its quality. As shown, humans in the loop—one of the key tenets of the uptake of human-centric AI and other digital tools—is embedded in technology regulation in both explicit and implicit ways. However, what is left somewhat unclear is the question of what humans do in the loop. What kinds of roles do they perform? Clearly, as argued, they oversee, intervene and retain certain tasks to themselves. Yet, in different loops humans can do—or neglect—different things, and at times adding humans to the loop may be a questionable exercise.
I suggest that in public law doctrine, the different roles that new humans are cast in are not sufficiently recognized. Often, it is a question of reinterpretation—I would say demotion—of civil servants or other professionals. Today, a legal professional. Tomorrow, a non-machine that understands at least some facets of digital technology. In fact, Rebecca Crootof and her colleagues distinguish nine different roles, some even with sub-roles, for humans in the loop. Depending on the situation, they can function in corrective roles, resilience roles, justificatory roles, dignitary roles, accountability roles, stand-in roles, friction roles, “warm body” roles, and interface roles.Footnote 61 Additionally, I argue, the limits of automation would also create situations where humans would have an exclusive role, which Crootof and her peers do not discuss, perhaps because in that situation humans constitute the entire loop.
Lately, it seems that the problems of the mechanical addition of humans in the loop have gained increasing attention in legal scholarship. There are critical voices in the literature highlighting that humans in the loop are not always reliable, necessary, or legitimate. At times, they can even be detrimental and give a false sense of security. There are many, partially overlapping reasons for that.
First, humans may easily become rubber stampers who cannot really assess the quality of automated decision-making.Footnote 62 The technological systems they are supposed to oversee are simply too complex or require a significant amount of expertise. In the AIA, for example, this problem is not sufficiently acknowledged as humans are supposed to have almost superhuman powers.Footnote 63 The implication is that humans might be added into the loop as an effortless procedural safeguard that seems to take care of its own legitimacy. This effortlessness can, nonetheless, turn into a skeuomorph, an obsolete design featureFootnote 64 which involves little independent agency and has little bearing on the actual content of automated decisions.
Second, humans can affect the functioning of an automated system in many ways, and these different roles are not necessarily equally warranted or effective. Some roles may perform other functions than the one expressed or implied in legislation. As explained, sometimes the role of a human in the loop is only a warm body, which is there perhaps to protect the interests of the legal profession in an ever-automating environment. At the same time, the role of humans as adding dignity—implying that automated decision-making lacks that—might be something else than the expressed function of the reason why humans are in the loop.Footnote 65 Instead of the expressed or intended function in the loop, humans might end up performing some shadow function that our legal justification system does not recognize.
Third, humans are faulty in many ways and restricted by human needs and features. It is commonly acknowledged that humans have many characteristics and predilections that make them unreliable. The entire American branch of critical legal studies is premised on this unreliability and bias. Humans get tired and hungry, they have different value systems and political beliefs, different personalities, perhaps unacknowledged preferences and biases, and their attention span is hopelessly limited.Footnote 66 Simply put, humans do not always deliver exactly because they are human.
It seems that to some extent the problems of adding a human in the loop are also increasingly recognized in the normative discourses of human-centrism. Indeed, just adding a human in the loop is hardly a silver bullet to tackle all of automation’s risks and problems, but perhaps it is rather a psychological safety mechanism to gain a feeling of control. This is visible in the literature, which calls for the meaningfulness of human involvement. The call, just like the GDPR calls for giving the data subject access to the meaningful information on the logic involved in the ADM process, implies the risk of certain procedural characteristics becoming a box-ticking exercise of no substantial significance. The call for meaningfulness in the implementation of these procedural guarantees seems to be a signal of that.
To sum up, adding a human in the loop in technology regulation underpins and reproduces the human-machine dichotomy, and approaches humans as non-machines instead of professional or independent moral agents. The discourses of humans-in-the-loop seem to overlook the fact that the law has attempted to make humans something other than a human for a very long time. In fact, the human executing the law has been approached according to a machinic ideal in which the humanness of humans is deliberately forgotten. I return to this point later.
Could or should the human-machine dichotomy be dismantled? Science and technology studies (STS) have for years emphasized that humans and machines work in socio-technical hybrid forms.Footnote 67 Should we then (1) regulate human-machine hybrids instead of mechanically putting a human in the loop;Footnote 68 (2) widen the concept of a “person” to cover other agencies than those traditionally thought of;Footnote 69 or (3) change the legal vocabulary and mindset to cover both humans and digital tools and make them commensurable?
I acknowledge the previous scholarship whereby the role of a person, and with certain limitations, a human too, are legal fictions. The first option above is viable and is already the subject of a growing literature. However, instead of widening the category of a person or aspirations to regulation human-tech hybrids, my proposition is different. In the following sections, I delve into the last option and argue that humans could also be seen as legal technology in public administration.
C. What is Legal Technology – or What Could It Be?
One of the main arguments of this Article is that our current legal vocabulary is not sufficiently nuanced to systematize the role of new humans in law. As described above, humans are put in the center as rights-holders and as back-up to oversee technological processes and intervene when necessary. “New humans” are there to protect the old ones. Sometimes the role of human is exclusive, as was discussed in the context of the limits of automation. The entire idea of keeping humans in or on the loop, or in command, can be seen as a conceptual transplant that originates from technological discourse, which has through AI ethics codes Footnote 70 become part of legal vocabulary.Footnote 71 As such, its compatibility with how humans are traditionally understood by law may be illusory. Even if the same word, human, is used, the meaning of that word can differ in different discourses.Footnote 72 For example, humans as an end in themselves, as opposed to humans as a means to an end.
Therefore, I suggest a different approach to make sense of the meaning of the new humans in law by utilizing the concept of legal technology. This approach could be helpful in disentangling the new conceptual landscape that digitalization has brought about. It could also help reorganize the rationalities that these new concepts inevitably embody in law. For example, the naïve human exceptionalism that can be argued to underpin this conceptual borrowing from the discourses of technology could be challenged. This would further contribute to the conceptual renewal—necessary for public law to remain viable—that I claim is necessary.
To argue a wider concept of legal technology requires a careful approach and incremental proceeding. It is necessary to clarify what constitutes legal technology in contemporary discourses and then investigate on which grounds this concept could be widened. Admittedly, the attempt to reconceptualize legal technology is an ambitious undertaking. The philosophy of technology is a wide field, and I do not claim or aim to make a comprehensive summary of it, in general or as applied to law. I can only scratch the surface of this topic given the length of this Article.
Legal technology is a concept that typically refers to the use of software in the legal industry.Footnote 73 According to Ryan Whalen, legal technology is “all devices, capable of being used as a means for interacting with the substance of law or assisting its user to interact with the law, and the skills and techniques by which we use them.”Footnote 74
Oliver R. Goodenough, in turn, approaches legal technology through generations of automated tools. Legal technology 1.0 would mean tools that support legal professionals, such as digital document management systems, office organization software and legal research databases. Legal technology 2.0, in turn, would consist of automated legal services that perform specific tasks without human intervention, like e-discovery software and online dispute resolution platforms. Finally, legal technology 3.0 would cover advanced technologies, including smart contracts and AI that have the potential to transform the legal profession fundamentally.Footnote 75 Seen like this, legal technology consists of different tools that legal professionals use to facilitate their work; legal technology 3.0 may even subvert that work for good.
As we can see, legal technology is commonly understood as automated tools that lawyers can use. However, legal technology can be construed more widely to cover a significantly longer history of invention. As Whalen explains, the relationship between law and technology is much older than one could assume. In addition to computationally enabled information technologies, technological developments such as writing, papermaking, and citator organization could be seen as legal technologies. These “legal” technologies have profoundly influenced the development of law and legal systems.Footnote 76 Thus, seen widely enough, we can argue that legal technologies have always been a part of the effective execution of law and not a modern addition.
Technology regulation, in turn, is a recent phenomenon, and it has significantly proliferated in the wake of technological development during the past decade.Footnote 77 We only have a little legislation on how to use paper in law, or how to use writing in law. These legal technologies are not considered technologies in the eyes of law. Instead, they are a part of what constitutes legal practices; they have been embedded into the materiality of law.Footnote 78 Law has absorbed certain technologies as part of its operational logic instead of tools to make its execution more effective. Therefore, a tool may lose its nature as a tool and become a part of the user of the tool.
How widely can we understand technology, though? Can we even go beyond the most rudimentary technologies? I suggest that legal technology does not have to have a direct connection to inventions or tools. As Lyria Bennet Moses notes, much depends on whether all means are considered technology, not just tools and crafts. I suggest that the key is to see technology as a process, resembling what Bennet Moses called by “all means.” This, as such, is by no means a novel approach, quite the contrary. Aristotle, for example, described techne as a means to an end. In fact, technology has at its roots two Greek words: techne—craftsmanship, craft, or art—and logos—word, reason, discourse, and so forth.Footnote 79 The idea of techne is thus a human craft or skill whose goal is to produce something (poiesis), an artifact.Footnote 80
As such, techne can be contrasted with praxis, whereby something is exercised for its own sake instead of producing an artifact (instrumental activity). Play or sports would make illustrative examples (expressive activity). Techne is also different from phronesis, practical wisdom including ethical judgment.Footnote 81 Of course, one must be careful not to think in anachronistic terms, as the idea of technology in a strict sense did not exist in Aristotle’s time. Moving forward, I use the terms techne and phronesis for heuristic purposes, as functional placeholders to signify mechanical or discretionary decision-making. Yet, the concept of techne could be understood as the root of how we understand technology today, although no linear trajectory from Aristotle to present-day technology can be drawn.
To argue a wide conception of legal technology, it is not satisfactory to retain a structural understanding of technology. Technology is something that is peculiar to humans and human culture. Therefore, we must investigate how technology affects the human lifeworld, how it alters it, and becomes a part of it. Some theorists are quite skeptical or ambivalent about this “technologization” of the lifeworld and argue that this might have a detrimental impact on humans. One of the most prominent critiques of technology is Martin Heidegger’s in The Question Concerning Technology.Footnote 82 In the essay, he talks about how technology is a way for the world to reveal itself, while making people a “standing reserve.” He uses the concept of enframing or framework, Gestell. His view is critical as he sees technology reducing everything to a resource and thus impoverishing human experience.Footnote 83
Regardless of whether we share Heidegger’s pessimism or not, it is safe to argue that humans and technology co-shape each other. This idea is also prevalent in contemporary discourses on STS.Footnote 84 As Whalen argues, technology is not deterministic; it is a social construct and its implications are social. Therefore, technology is not only a tool with capacities that become evident when used. Rather, it is an entity which offers a variety of technological affordances which users may or may not avail themselves. This further reveals that even though technologies may have inherent affordances, they are subject to human agency and thus are both shaped by and shape the practices humans use in interaction with, around, and through them.Footnote 85 Therefore, technological development and its effects should be approached both as a function of technology’s affordances and of the social construction that is limited by those affordances.
Emphasizing the idea of the social and socially embedded nature of legal technology, we could also approach legal technologies more abstractly, such as legal education, legal mindsets, or even a general reverence of law. If we also interpret ideas and ideals as legal technologies, the wide conception of a legal technology becomes possible. Importantly, legal technologies are both internal and external to humans. This means legal technologies are embedded in those executing the law, making humans a collection of legal technologies, certain kinds of cyborgs. Andy Clark argues that we are natural-born cyborgs, combinations of the biological and the technological. In Clark’s view, this is the very thing that makes humans different from other species: the capacity to incorporate tools and cultural practices into our existence. The cyborg breaks conventional distinctions and categories, and as such mirrors our prejudices, and questions the assumptions on which these prejudices are based.Footnote 86
We can argue that legal technology can be seen as a means to an end, like techne, and not only as the use of tools. In public administration, techne is a form of power. Law can be seen as a technology itself, which needs to be coded into its executors to have a bearing.Footnote 87 For example, in public administration, a process of deciding over people’s rights, duties and benefits, techne, would be a poietic, technological practice of creating an administrative decision, artifact. Alternatively, one could argue that this kind of decision-making is a form of phronesis, practical wisdom. However, in the modern legally organized state, decision-making processes are highly regulated and organized into operationalized steps, leaving little room for independent moral agency. Which technology is at play, however, depends on the level of discretionary leeway that the provision in question allows, and in some ways techne and phronesis intersect.
We can conclude that not only do humans and technology co-shape each other but also that the same logic applies to technology and law. They, too, co-shape each other. This co-shaping relationship accentuates when we are talking about legal technologies of public administration and not only legal technology in general. Law is a technological practice where law makes its humans technological entities. Humans, in turn, require various kinds of technologies to perform legal actions.Footnote 88 In the following section, I examine how this techne of governmentFootnote 89 This will enable us to consider humans and human legal labor as legal technologies.
D. Humans as Legal Technology in Public Administration
I. Implications of the Bureaucratic Machine
As described, by legal technology, I mean—at its simplest—a poietic process, (techne), and different ways in which legal rules, concepts, and procedures are made real (artifact). To complete the reconceptualization of legal technology, we must still reassess the identity of the user or the deployer of legal technologies. Does it have to be an individual legal professional as is customarily thought? Or should we think about the issue more holistically?
I suggest that we approach the user of different legal technologies in public administration through the metaphor of the bureaucratic machinery of the state. This machinery may be argued to consist of both analog and digital, embodied or artificial cogs, such as humans and digital technologies. This further creates a portrait of the embodied cog—the human—who is a collection of internalized legal technologies. This makes the humanness of the human such that they can act as a cog in the bureaucratic machine.
Indeed, this context is important as the question is not only about legal technologies in the abstract, but also about legal technologies that serve the state. Moreover, the state and its bureaucratic machinery inevitably set the goal or the purpose that humans as a legal technology would serve other than controlling other legal technology. Why are legal technologies used in public administration in the first place? Why is their careful supervision important? As I will show, in the context of government and structures of power, the broad conception of techne and technology is, in fact, readily available.
The key is to understand “machine” or “machinery” widely. Lewis Mumford argued in his famous two-volume book The Myth of the Machine (1967-1970) that human tools and social organization developed in parallel mainly through language and rituals, convergence of science, economy, technics, and political power making a “megamachine.” He argued that throughout history, societies have created these large, rigid bureaucratic structures—such as in the building of the Egyptian pyramids or modern military-industrial complexes—that organize human labor and behavior in machine-like ways. To make a “human machine,” certain hierarchical structures must be established and discretion diminished. Megamachines are related to power as a synthesis of technical instruments—tools, infrastructure, science—and social organization—bureaucracy, administration, labor control.Footnote 90
Thus, talking about public administration—or human organization more widely—employing the operational logic of a machine is not rare. Instead, a machine seems to metaphorically capture some of the public administration’s key elements, at least seen in a traditional way.Footnote 91 In that way, looking at bureaucratic machines in a manner that also consists of human cogs has established intellectual roots. The idea of bureaucratic machinery more specifically is often attributed to Max Weber, particularly his ideas of legal rationality and his coining of the very term bureaucracy. He also called for civil servants who would work “sine ira et studio,” without anger or passion, thus making machines a regulative ideal for their behavior and thinking.Footnote 92
A machine, furthermore, can be used as a concept to make sense of how the state is executing its goals. Famously, Michel Foucault talked about governmentality, technologies of government, and technologies of the self, not to mention his more indirect technological discussions on biopolitics and disciplinary power. Foucault’s very understanding of technology is correlative of a subject and government. Foucault’s conception of technology is also procedural rather than tied to a material artifact, unless of course one interprets the subject as an artifact of power.Footnote 93 One could argue that these technologies of the self are one way that legal technologies can be internalized.
A somewhat machinic understanding of civil servants or bureaucrats is visible in contemporary discussions of their changing role in public administration. They revolve largely around Michael Lipsky’s famous book about street-level bureaucrats that was first published in 1980.Footnote 94 In the book, the key idea is that bureaucrats who oversee executing laws and government policies must use their own discretion and cope with unofficial practices in their workload. This creates instability and uncertainty in the execution of law.
The paradox is that although the administrative machinery should just execute the goals set in democratic processes, in fact, it ends up being interpreted in various ways due to the discretionary power and best practices that are at play on the street level. However, along with digitalization, the meaning of street-level bureaucrats—or screen-level, as the term has been updated—has relocated discretion to the design processes of automated systems.Footnote 95 As this argument has some merit, it does not consider the fact that some legal provisions are deliberately left vague to enable street-level discretion. Not all discretion can be removed or relocated to the system level.
The machinic thinking described above seems to form a point of reference that enables us to ask what roles should be reserved only for humans. It, however, makes itself visible only obliquely. In several scholarly contributions, the need for human judgment is placed in circumstances where one must deal with new or vague situations. In these situations, the complex legal questions go beyond the calls for non-discrimination or consistent practice and consider case-specific details instead.
This has been formulated, for example, as a matter of case-by-case judgment and individual justice,Footnote 96 complex deductions relating to the social dimensions of law or other norms,Footnote 97 a circumstance, where there is no tractable parameter amenable to prediction,Footnote 98 or the interpretation of legal language.Footnote 99 In other words, in these contributions legal discretion is considered technology-free and even essentially non-automatable.Footnote 100 That said, it is crucial to notice that technological vocabulary and thinking are not foreign even in analog administration, whereby increasing formalization and standardization of work practices have preceded digitalization in public administration and diminished the use of discretionary leeway.Footnote 101
The approaches discussed above—overt and covert applications of machinic thinking—pave the way for understanding humans as legal technology. Digitalization has put the discussion into a new gear and has utilized this underlying sympathy for the technological mindset. Yet, oversimplifications should be avoided as the connection between public administration, humans, and technology is manifold and their mutual relations are subject to different, changing interpretations, as I argue in this Article. Still, we can observe that it is possible to rethink the user or the deployer of legal technology by connecting this technology to the machinery of the state and the technologies it utilizes.
II. The Qualities of Humans as Legal Technology
In the context of deploying AI tools and other digital technology in public administration, the categorization of humans as legal technology could be summarized in the following fashion. Much of human legal labor can be seen as a techne in a traditional way. Although in some instances it does involve ethical judgment and deliberation, in modern bureaucracy, legal practices are increasingly mechanized, reflecting machinic thinking. At the same time, humans as cogs in the bureaucratic machine are not merely mechanistic rule followers. Instead, they are constructed as agents who have a meta-level understanding of the demands of special circumstances, allowing them to understand when automation is possible.
As I have argued, human superiority as a legal technology is not clearly argued in technology regulation but it is implicitly assumed to stem from human-centricity. Legal technology, in turn, can be understood in practical or in abstract terms, as either being opposed to humans or extending to them and their work. I am interested in how humans as a legal technology are silently approached in terms of contemporary technology regulation and perhaps public law more widely.Footnote 102 I suggest that a “pure” human free of all technology is a legal fiction whose function seems to be to legitimate the deployment of digital technology.
When we use the lens of legal technology, a human could be described in the following fashion. From an ideal point of view, humans make an embodied analogue legal technology which uses symbolic language and is attuned to social contexts and interpersonal considerations, traditionally framed as a question of discretion. This legal technology is generally reverent to law and is equipped to apply it flexibly, alternating adequately between two modes of reasoning, techne and phronesis. This legal technology is optimal for understanding legally vague situations and can navigate through the varying need for literal or liberal legal interpretation. At the same time, an understanding of the special human skills and abilities required would clarify which tasks could be given to machines.
From a practical point of view, the image of a human is more nuanced. It is commonly accepted that humans also suffer from several defects, such as a predilection to bias, thinking through fast heuristics instead of rational reasoning, bodily needs such as hunger and fatigue, and wavering moral stamina. In technology regulation, the vested value of human skills and abilities would outweigh the negative sides. In fact, these negative sides are mostly left unaddressed in law, and their expressions are indirect. However, to fight them, other legal technologies, such as legal education and the fear of consequences, are embedded in this “pure” human: through them, humans can be assumed to serve other purposes than immediate self-interest and social agendas.
All this means that the character of a human as a legal technology oscillates between two opposites: something that spontaneously is—humans as an embodied material agent—and something that ought to be—humans as a normative legal construction. The action expected from this character changes accordingly, between something fallible and self-interested, and perhaps incomputableFootnote 103 —the embodied human—and something almost superhuman in its capabilities, for example, potent in preventing AI-related harm and reassessing complex automated decisions—the legal construction of a human.Footnote 104 As I argue in this Article, in technology regulation, the normative image of a human prevails. Humans come across as wise deliberators who are willing and able to question what a machine does, freed from their all-too-human weaknesses.
The confusion follows when the embodied human serves as the interfaceFootnote 105 of the legally constructed one, mixing up their qualities. What is being called for is an embodiment who is called a “human” whose technological dimensions are hidden or ignored.Footnote 106 The “purity” of a human seems to be an important legal fiction to justify the use of digital technologies. It further produces and maintains the idea that law and humans can be purified from technologyFootnote 107 and the seemingly pre-technological law restored whenever needed. This means that the new humans are, however, covertly technologized humans, as their essence is dependent on the technology they are overseeing. Their technological function is embedded in law.Footnote 108
The seemingly “pure” human is a human to whom law and its functionality is programmed. To fulfill their normative function, humans are driven by the technologies of the self, including the mechanization of the mind and analog automation of their work practices. They are interconnected with bureaucratic machinery, and they are trusted to alternate flexibly between techne and phronesis. In effect, humans have a ritual purification function through which technologically tainted and dignity-jeopardizing law is kept seemingly human-centric. They work as the responsibilized nodes in legal practices or, in Madeleine Clare Elish’s words, moral crumple zones, scapegoats who are expected to take responsibility for technological failures.Footnote 109
By analyzing humans and digital technologies together, it is possible to make visible the underpinning assumptions of human exceptionalism on the one hand, and the technological function of humans on the other. Human-centricity hides the way in which law is attempting to make human as a legal technology by programming its rationality to legal thinking. The emergence of the new humans shows how humans are covertly technologized regardless of their seeming opposition to technology. In Connal Parsley’s words, “Can the law of this legal human prevail, or will it be reduced to the technological element it must pretend to eschew?”Footnote 110 Technology is thus also something internal, not only external to these humans. Therefore, a man without qualities is, in the end, a man full of legal technologies.
To summarize, I argue that a human can be seen both as a legal technology of public administration, and as a collection and an interface of other legal technologies. As such, these humans are no longer humans in a strict sense, but products of other legal technologies. To argue for the existence of such technological humans evokes many thorny questions. Could “sine ira et studio” lead to “robotomorphy,” humans becoming their own creations,Footnote 111 or are digital technologies necessary “psycho-morphs,” projection objects that help us make sense of technology?Footnote 112 Can humans be reduced to a cog in a machine without severe ethical problems or even represent an aspect of the banality of evil? Should the digitalizing administrative state be particularly concerned about empathy?Footnote 113 These questions cannot be answered within the scope of this Article, but it is to be hoped that further scholarship will tackle them.
E. Conclusions: Diminishing Human Cogs in the Bureaucratic Machinery?
In this Article, I have made two contributions: First, I have argued for the emergence of the so-called new humans in law and explained their explicit and implicit tasks. It would seem that the new humans have a special effect in public administration: they cause micro disruption as a borrowed concept and a doctrinal irritant, while continuing the macro rationality of public administration by reproducing the machinic ideal. The latter idea hinges on a paradox, namely dehumanizing humans by calling them humans. This means recasting humans as rights-holders to humans as a means-to-an-end. As humans become non-machines, their identity and meaning depend on the legal conception of those machines.
Second, I have suggested that instead of relying on and reproducing the problematic human-machine distinction, we could reimagine the concept of legal technology to cover both digital tools and human labor. I argued that we could approach the state and its bureaucratic machinery as the deployer of different legal technologies instead of looking at legal technology as a lawyer’s use of software tools. Humans as legal technology seems to have reserved a role which is superior to machines and this role is the overall governor of the machinic thinking that labels public administration. This was earlier referred to as the anthropocentric bias. The superiority of humans seems to build on a suspicion of technology and the traditional roles assigned to humans rather than superiority in quality of decision-making, while these humans silently carry other legal technologies in themselves.
To properly analyze humans as legal technology is an extensive undertaking, and that can only be touched upon in this Article. For now, my attempt has been to outline this field as a topic of discussion rather than say everything there is to say about it. Additionally, this field is in constant movement as AI and other digital technology continue to develop. As John Danaher puts it:
Therefore, although we may well continue to have human legislators and human politicians – no one is envisaging a robot president or king just yet – their role in the business of government will be much diminished. They will simply supervise the machinery of state – machinery that will be made up of fewer and fewer human parts!Footnote 114
The future will show whether Danaher got it right or not.
Anknowledgements
I would like to thank the University of Helsinki Legal Tech Lab members for helpful discussions on the earlier draft. I thank Mika Viljanen for reading through the entire manuscript and providing insightful comments. I also thank the anonymous peer-reviewers and GLJ editing team.
Funding Statement
No specific funding has been declared in relation to this Article.
Competing Interests
The author declares none.
Open access
