5.1 Introduction

Artificial intelligence (AI) is a broad church. Automated decision-making (ADM), a subset of AI, is the form of technology most commonly encountered in public administration of the social services, a generic term which includes income support (social security) and funding or provision of services such as disability support funding under Australia’s National Disability Insurance Scheme (NDIS). New public analytics is a label that nicely captures how ADM is deployed as the contemporary form of public administration.Footnote ²

ADM has long been an integral aid to the work of hard-pressed human administrators exercising their delegated social security powers in Centrelink (the specialist service delivery arm of the federal government department called Services Australia). Early digitisation of social security benefits administration not only resulted in considerable efficiency gains but provided the guide-rails that protected against the more egregious errors or decline in decision-making quality as staffing was drastically reduced in scale and shed higher levels skills and experience. Automation as such has not been the issue; the issue is a more recent one of a breakneck rush into a ‘digital first future’Footnote ³ and the abysmal failure of governance, design, ethics, and legal rectitude associated with the $1.8 billion robodebt catastrophe.Footnote ⁴ As Murphy J observed in his reasons approving the class action settlement, this was a ‘shameful chapter in the administration of the Commonwealth social security system and a massive failure of public administration [which] should have been obvious to the senior public servants charged with overseeing the Robodebt system and to the responsible Minister at different points’; a verdict echoed by the Royal Commissioner in her July 2023 Report.Footnote ⁵

ADM is only a technology. Like all new technologies, there are extremes of dystopian and utopian evaluative tropes, though a mature assessment often involves a more ambiguous middle ground.Footnote ⁶ Like the history of other new technological challenges to law, the answers may call for innovative new approaches, rather than extension of existing remedies. Robodebt was ultimately brought to heel by judicial review and class actions, but the much vaunted ‘new administrative law’ machinery of the 1970sFootnote ⁷ was seriously exposed. Merits review failed because government ‘gamed it’Footnote ⁸ while the other accountability mechanisms proved toothless.Footnote ⁹ So radical new thinking is called for.Footnote ¹⁰ AI for its part ranges in form from computational aids (or automation) to neural network ‘machine learning’ systems. Even agreed taxonomies of AI are still in development, including recently by the Organisation for Economic Co-operation and Development (OECD), with its four-fold schema of context; data and input; AI model; and task and output.Footnote ¹¹

The focus of this chapter on social security and social services is apt, because Services Australia (as the former Department of Human Services is now called) was envisaged by the Digital Transformation Agency (‘DTA’ formerly ‘Office’) as ‘the first department to roll out intelligent technologies and provide new platforms to citizenry, in accordance with the then DTA’s roadmap for later adoption by other agencies’.Footnote ¹² The focus of this chapter is on the associated risk of digital transformation in the social services, of three main forms. First, the risk due to the heightened vulnerabilities of clients of social services.Footnote ¹³ Second, the risk from inadequate design, consultation, and monitoring of ADM initiatives in the social services.Footnote ¹⁴ And finally, the heightened risk associated with particular ADM technologies.

The next section of the chapter (Section 5.2) reviews selected ADM/AI examples in social services in Australia and elsewhere. To draw out differences in levels of risk of various initiatives it takes as a loose organising principle Henman’sFootnote ¹⁵ observation that the risks and pitfalls of AI increase along a progression – lowest where it involves recognising ‘patterns’, higher where individuals are ‘sorted’ into categories, and highest where AI is used to make ‘predictions’. Section 5.3 discusses the harm inflicted on vulnerable clients of social services when ADM and AI risks are inadequately appreciated, and some options for better regulation and accountability. It questions both the capacity of traditional judicial and administrative machinery in holding AI to account, and the relevance and durability of those ‘values’ in the face of the transformational power of this technology to subordinate and remake law and social policy to instead reflect AI values and processes.

Restoration of trust in government is advanced in a short conclusion (Section 5.4) as being foundational to risk management in the social services. Trust is at the heart of the argument made for greater caution, more extensive co-design, and enhanced regulatory oversight of ADM in the social services.

5.2 Issues Posed by Automation and ADM

Three issues in particular stand out for social services in Australia. First, the comprehensibility or otherwise of the system for citizens engaging with it. Second, the compatibility or otherwise of ADM in case management. Finally, the risks and benefits of ‘predictive’ ADM in the social services.

5.3 Responding to the ‘Power’ of AI

ADM and AI technology in the social services carries a potential both to harm participants as well as to radically transform services by compressing the range of social policy options considered in program design much in the same way these technologies can change the bases of legal accountability (Section 5.3.2).

The power of poorly conceived ADM and AI to inflict unacceptable harm on vulnerable citizens reliant on social services is well established. The challenge here lies in finding ways of mitigating that risk, as discussed below.

5.4 Towards AI Trust and Empathy for Ordinary Citizens

Administration of social security payments and the crafting of reasonable and necessary supports under the NDIS are quintessentially examples of how law and government administration impact ‘ordinary’ citizens. As Raso has observed:

It is suggested here that the principal impression to be drawn from this review of Australia’s recent experience of rolling out ADM in Raso’s ‘everyday’ domain of the ordinary citizen, is one of failure of government administration. It is argued that the history so far of Australian automation of welfare – most egregiously the robodebt debacle – demonstrates both a lack of government understanding that the old ways of policy-making are no longer appropriate, and that public trust in government has seriously eroded. Automation of welfare in Australia has not only imposed considerable harm on the vulnerable,Footnote ¹¹⁶ but has destroyed an essential trust relationship between citizens and government.Footnote ¹¹⁷

Restoring trust is critical. Trust is one of the five overarching themes identified for consultation in February 2022 by the PM&C’s Digital Technology Taskforce and in the AHRC’s final report.Footnote ¹¹⁸ Restoration of trust in the NDIS was also one of the main themes of the recent Joint Parliamentary Committee report on independent assessments.Footnote ¹¹⁹ Consequently, if future automation is to retain fidelity to values of transparency, quality, and user interests, it is imperative that government engage creatively with the welfare community to develop the required innovative new procedures. A commitment to genuine co-design and collaborative fine-tuning of automation initiatives should be a non-negotiable first step, as stressed for the NDIS.Footnote ¹²⁰ Ensuring empathy of government/citizen dealings is another.

Emphasising in Chapter 9 about the potential for the automated state, wisely crafted and monitored to realise administrative law values, Cary Coglianese writes that

He warns that ‘[t]o build public trust in an automated state, government authorities will need to ensure that members of the public still feel a human connection’. This calls for a creative new administrative vision able to honour human connection, because ‘[i]t is that human quality of empathy that should lead the administrative law of procedural due process to move beyond just its current emphasis on reducing errors and lowering costs’. That vision must also be one that overcomes exclusion of the marginalised and vulnerable.Footnote ¹²¹ Another contribution to building trust is to be more critical of the push for automated administration in the first place. An American ‘crisis of legitimacy’ in administrative agencies has been attributed to the way uncritical adoption of ADM leads to the loss of the very attributes that justify their existence, such as individualisation.Footnote ¹²² Framing the NDIS independent assessor episode in this way demonstrated a similar potential deterioration of citizen trust and legitimacy.

Building trust and empathy in social service administration and program design must fully embrace not only the mainstream human condition but also the ‘outliers’ that AI standardisation excludes.Footnote ¹²³ At the program design level this at a minimum calls for rejection of any AI or ADM that removes or restricts inclusion of otherwise appropriate elements of personalisation, subjective human judgement, or exercise of discretion relevant to advancing agreed social policy goals. This extends to AI outside the program itself, including being sensitive to indirect exclusion from discriminatory impacts of poorly designed technological tools such as smartphones.Footnote ¹²⁴

Half a century ago in the pre-ADM 1970s, the ‘new administrative law’ of merits review and oversight bodies was touted as the way to cultivate citizens’ trust in government administration and provide access to administrative justice for the ordinary citizen, though even then the shortfall of preventive avenues was recognised.Footnote ¹²⁵ Overcoming the ability of government to game first-tier AAT by keeping adverse rulings secret, and arming it with ways of raising systemic issues (such as a form of ‘administrative class action’) might go a small way to restoring trust and access to justice. But much more creative thinking and work is still to be done at the level of dealing with individual grievances as well.Footnote ¹²⁶

In short, this chapter suggests that the conversation about the ADM implications for the socioeconomic rights of marginalised citizens in the social services has barely begun. Few remedies and answers currently exist either for program design or for individual welfare administration.

6.1 Introduction: ADM and the Machinery of Government

The machinery of government are those structures, processes, and people that comprise departments and agencies, and through which governments perform their functions. The term is perhaps best known in the context of ‘MoG changes’ – the frequent adjustments made to the way departments and agencies are structured, responsibilities and staff are grouped and managed,Footnote ¹ and how agencies are named.Footnote ² For at least the last half century, the defining characteristic of the machinery of government has been public officials (the ‘bureaucrats’), structured into branches, divisions, and departments, operating pursuant to delegations, policies and procedures, and providing advice, making and implementing decisions, and delivering services for and on behalf of the government. Characterising governments as a ‘machine’ is both a metaphor and, like the term ‘bureaucracy’, can convey a somewhat pejorative connotation: machines (even ‘well-oiled machines’) are cold, unfeeling, mechanical things that operate according to the dictates of their fixed internal rules and logic.

This chapter examines a change brought about to the machinery of government that is increasingly permeating government structures and processes – the adoption of automated decision-making (ADM) tools to assist, augment, and, in some cases, replace human decision-makers. The ‘machinery of government’ metaphor has been extended to frame the discussion of this topic for three reasons. First, it more clearly focuses attention on the entire system that underpins any government administrative decision, and in which digital technology may play some role. Second, rather than assuming that new technologies must – because they are new – be unregulated, the role of new technology within the machinery of government should be considered, and therefore (at least as a starting point) the well-established laws and principles that already control and regulate the machinery of government need to be analysed. Finally, this chapter aims to consider whether there might be lessons to be learnt from the past when other significant changes have taken place in the machinery of government. For example, do the changes that are now taking place with the increasing digitisation of government decision-making suggest that we should consider a deeper examination and reform of our mechanisms of administrative review, in a similar way to what happened in Australia in the 1970s and 1980s in response to the upheavals then taking place?

In this chapter, some of the key themes addressed in detail in the NSW Ombudsman’s 2021 special report to the NSW Parliament, titled ‘The new machinery of government: using machine technology in administrative decision-making’Footnote ³ (Machine Technology report), are outlined. This chapter provides a brief context of the need for visibility of government use of ADM tools and the role of the Ombudsman, key issues at the intersection between automation and administrative law and practice, and broad considerations for agencies when designing and implementing ADM tools to support the exercise of statutory functions. The chapter concludes with a question of whether the rise of ADM tools may also warrant a reconsideration of the legal frameworks and institutional arrangements.

6.2 Context

6.3 Administrative Law and ADM Technologies

There is an important ongoing discussion about the promises and potential pitfalls of the most highly sophisticated forms of AI technology in the public sector. However, maladministration as described above can arise when utilising technology that is substantially less ‘intelligent’ than many might expect. The case studies in the Machine Technology Report illustrate a range of issues relating to administrative conduct, for example, the automation of statutory discretion, the translation of legislation into code, and ADM governance. Only some aspects of the technologies used in those case studies would be described as AI. In any case, the focus from an administrative law and good conduct perspective is not so much on what the technology is, but what it does, and the risks involved in its use in the public sector.Footnote ¹³

Mistakes made when translating law into a form capable of execution by a machine will likely continue to be the most common source of unlawful conduct and maladministration in public sector use of ADM tools. While of course unaided human decision-makers can and do also make mistakes, the ramifications of automation errors may be far more significant. The likelihood of error may be higher, as the natural language of law does not lend itself easily to translation into machine code. The scale of error is likely to be magnified. The detection of error can be more difficult, as error will not necessarily be obvious to any particular person affected, and even where error is suspected, identifying its source and nature may be challenging even for the public authority itself. A machine itself is, of course, incapable of ever doubting the correctness of its own outputs. Rectifying errors may be more cumbersome, costly, and time-consuming, particularly if it requires a substantial rewriting of machine code, and especially where a third party vendor may be involved.

6.4 Designing ADM Tools to Comply with the Law and Fundamental Principles of Good Government

To better manage the risks of maladministration in the use of ADM tools, there are at least five broad considerations that government agencies must address when designing and implementing ADM systems to support the exercise of an existing statutory function.Footnote ³⁹ Dealing with those comprehensively will assist compliance with the principles of administrative law and good decision-making practice.

6.5 Conclusion

Appropriate government use of ADM tools starts with transparency. The current lack of visibility means that it is not well known how many government agencies in NSW are using or developing ADM tools to assist in the exercise of administrative functions or what they are being used for. Nor is it possible to know who is impacted by the use of ADM tools, what validation and testing is being undertaken, whether there is ongoing monitoring for accuracy and bias, and what legal advice is being obtained to certify conformance with the requirements of administrative law.

Much of this chapter has focussed on how existing laws and norms of public sector administrative decision-making may control the use of ADM tools when used in that context. However, there are likely to be, at least initially, significant uncertainties and potentially significant gaps in the existing legal framework given the likely rapid and revolutionary changes to the way government conducts itself in the coming years. Government use of ADM tools in administrative decision-making may warrant a reconsideration of the legal frameworks, institutional arrangements, and rules that apply. It may be, for example, that existing administrative law mechanisms of redress, such as judicial review or complaint to the Ombudsman, will be considered too slow or individualised to provide an appropriate response to concerns about systemic injustices arising from algorithmic bias.Footnote ⁶⁰ Modified frameworks may be needed – for example, to require the proactive external testing and auditing of systems, rather than merely reactive individual case review. If a statute is to be amended to specifically authorise particular uses of ADM tools, this creates an opportunity for Parliament to consider scaffolding a governance framework around that technology. That could include stipulating certain properties the system must exhibit in terms of transparency, accuracy, auditability, reviewability, and so on.

However, an open question is whether there is a need to consider more generally applicable legal or institutional reform, particularly to ensure that ADM tools are subject to appropriate governance, oversight, and review when used in a government context.Footnote ⁶¹ There may be precedent for this approach. The machinery of Australia’s modern administrative law – the administrative decisions tribunals, Ombudsman institutions, privacy commissions, and (in some jurisdictions) codified judicial review legislation – was largely installed in a short period of intense legislative reform, responding to what was then the new technology of modern government.Footnote ⁶² Ombudsman institutions (and other bodies which perform similar and potentially more specialised roles, including, for example, human rights commissions, anti-discrimination bodies, or freedom of information (FOI) and privacy commissions) have proven useful in many areas where traditional regulation and judicial enforcement are inadequate or inefficient. Ombudsman institutions also have the ability to not only respond reactively to individual complaints but also to proactively inquire into potential systemic issues, and to make public reports and recommendations to improve practices, policies, and legislation.Footnote ⁶³ This ability to act proactively using ‘own motion’ powers may become increasingly relevant in the context of government use of ADM tools, partly because it seems less likely that complaints will be made about the technology itself – including if complainants are unaware of the role played by technology in the relevant decision. Rather, when people complain to bodies like the Ombudsman, the complaint is usually framed in terms of the outcome and impact on the individual. It must also be recognised that, if Ombudsman institutions are to perform this oversight role, there will be a need for capability growth. At present, it is likely they lack the in-house depth of technical skills and resources needed for any sophisticated deconstruction and interrogation of data quality and modelling, which may, at least in some cases, be required for effective scrutiny and investigation of ADM tools.Footnote ⁶⁴

7.1 Introduction: Two Tales of the Automated State

In his 1967 book, which partially shares its title with this edited collection (The Automated State: Computer Systems as a New Force in Society),Footnote ¹ Robert McBride anticipated that public authorities would be able to do ‘more’ thanks to the possibility of storing more detailed data combined with the increasing capacity of machines to process that data. He conjectured that this would create new legal problems. Fast forward half a century and the Automated State may (really) be on the brink of happening. AI can essentially change the state and the way it operates – note the ‘essentially’.

Public authorities, employing (or assisted by) machines to a large scale, could do more. What this ‘more’ is, is a matter of discussion,Footnote ² but, broadly speaking, it can mean two ideas: (i) doing things that humans could do, but more efficiently or to a larger scale; or (ii) doing things that could not be done before, at all or at a reasonable cost.Footnote ³ Therefore, the rules that regulate the action of public authorities need to be adapted. This chapter deals with the normative question of the type of regulatory reform that we should aim for.

It can be anticipated that changes within the immediate horizon – three to five years – will be marginal and starting at the points of least resistance, that is, in tasks currently done by humans that could be easily automated. In these cases, the preferred regulatory option is likely to be the creation of some lex specialis for the situations when public authorities are using AI systems. This approach to automating the state and the necessary changes to the administrative law are explored in the following section (Section 7.2).

The much bigger challenge for the regulation of the Automated State will come from structural changes in the way we design policy and decide on policy options. This is best illustrated with one example already in the making: digital twins, data-driven copies of existing real-life environments or organisms. Although the attention has primarily focused on digital twins of living organisms,Footnote ⁴ promising work is being undertaken in other types of real-life twins, such as factories or cities. One leading example is the work in Barcelona (Spain) to create a digital twin that will help make decisions on urban policy, such as traffic management or planning.Footnote ⁵

According to some reports, when one of the key planning initiatives of the local government – the superilles, which involved the creation of limited-traffic city-block islands – was run through the system to see the effects with and without its implementation, it showed that there was close to no improvement on air pollution levels, one of the drivers for the creation and implementation of the initiative.Footnote ⁶ In other words, the intervention failed to achieve one of its main goals. Does this matter for administrative law?

Section 7.3 considers these policy-oriented types of AI systems. The systems used to design policy and make decisions among policy options open the door to an intrinsically different automated state which may require completely new tools and approaches to regulate it. Although the word ‘automated’ could be misleading – it is better described by the periphrastic ‘AI-driven decision support system for policy design and creation’ – the outputs of these systems are within the scope of administrative law. They are part of processes that eventually generate administrative acts or decisions and, as such, can be the object of challenges on legal grounds in many jurisdictions.

A key part of that discussion is the problem of translating into law a procedure for legal administrative accountability for ‘objectives’ (a particular type of input for those AI systems) and ‘insights’ (outputs). AI systems are often developed to optimise a number of objectives set by humans or to autonomously find insights and interesting relations among the data fed into it. When these types of AI systems are used on data held by public authorities for policy-making purposes, they generate immediate challenges to administrative law: how do we regulate policy-making that is meant not to be about discretionary choices, but about data-driven optimisation?

Concepts such as ‘arbitrariness’ or ‘discretion’ mean very different things in regard to public authorities’ decisions which are an application of the law to individuals or groups, covered in Section 7.2,Footnote ⁷ and for decisions about how to best use public resources at a policy level, explored in Section 7.3.Footnote ⁸ Distinguishing between legitimate political (or policy) choices and unreasonable decisions will be challenging if at a given stage of the decision-making process there is a system that is considering one option preferable to another according to the parameters built into that system.

This type of problem may still be incipient. The technology may still be very far from reliable, but if we reach a point when some policies can be shown to be Pareto superior to others (i.e., not one of the indicators considered in the policy is worse-off, but at least one is improved), is the choice of the Pareto inferior option still legitimate or fair? Will it be legal? How much deference should then be given to the choices of decision-makers?

To solve some of these questions in Section 7.4, I suggest some preparatory work for this scenario. I develop some heuristics – or rules of thumb – to distinguish between both tales of the Automated State. On that basis, I explore whether democratic and liberal societies can create a new type of administrative law that can accommodate divergence of views and still ensure that the margin of discretion of policy choices is adjusted to this new reality.

7.2 The Administrative Law of AI Systems That Replace Bureaucrats

The use of AI for automating work currently done by humans – or creating systems that facilitate the performance of those tasks by humans – can be directly linked to previous investments by governments in information systems. These were generally associated with attempts to update the ways public organisations operated to enhance efficiency and policy effectiveness.Footnote ⁹ Those AI systems, if used for fully automated administrative tasks, could be ‘isolated from the organisational setting they originated from’Footnote ¹⁰ and, therefore even legally considered as ‘individual artificial bureaucrats’.Footnote ¹¹

In this context, the main consideration is that the system should be able to do its job properly. This view, therefore, naturally places the accent on testing the AI systems beforehand, particularly for impartiality and standardisation. This is something we are relatively familiar with and not conceptually dissimilar to the way Chinese imperial mandarins were subject to excruciating exams and tests before they could work for the emperor, or to the way the Spanish and French systems (and the countries in their respective areas of influence) still see the formalised gruelling testing of knowledge as a requisite to access a ‘proper’ bureaucrat position.

Therefore, administrative rules for the use of these AI systems are likely to focus on the systems themselves. As mentioned, the regulatory approach will then most likely emphasise ensuring that they are fit for purpose before starting operation, which is a type of legal reform already observed in several jurisdictions.

Commonly cited examples are the mechanisms already in place in Canada,Footnote ¹² which focus on the risks of AI systems employed by public authorities; the proposed general approach in the European Union,Footnote ¹³ which expands to high-risk systems in the public and private sector; or the light-touch intervention model, which creates some pre-checks for the use of certain AI systems by the public authorities, such as the recently introduced rules in the state of New South Wales in AustraliaFootnote ¹⁴ – although with no concrete consequences, in this case, if the pre-check is not done properly.

Generally speaking, these approaches place the stress on the process (or its automated part) and not on the outputs. It is the system itself that must meet certain standards, defined on the basis of actual standards or specifications (in the EU case as described in article 9 of the proposal) or an impact assessment of some kind (Canada model) or the considerations of ‘experts’ (New South Wales, Australia model). At a higher level, this makes sense if what we are concerned about is the level of risk that could be generated by the system. The question here is ‘how bad can it go?’, and the law mandates to undertake that check beforehand.

In my opinion, this deviates from the views of administrative law that see the action of the public authorities as a materialisation of values such as equality and fairness.Footnote ¹⁵ Instead, this Weberian machine bureaucracy would stress impartiality and standardisation,Footnote ¹⁶ values more intrinsically attached to procedural elements.Footnote ¹⁷

In the classic model of Peters, in which the public administration is a manifestation of a combination of societal, political, and administrative cultures,Footnote ¹⁸ the direct connection here is to the administrative culture, and only collaterally to societal or political elements. That type of Automated State does not need to be fair, it needs to be accurate. The fairness is meant to be embedded in the policy it implements and the legitimacy of outputs depends on whether the process correctly implements the policy.

However, as this approach incorporates elements of risk-based regulatory techniques, outputs are indeed considered in the process of conformity checks. Normally, most of these regulations of the use of AI in administrative law settings will mandate, or make a reference to, some kind of cost–benefit analysis of the social utility of the deployment and use of the system, in the way described by Sunstein.Footnote ¹⁹ The test to start employing automated systems in this context is one that compares an existing procedure in which humans participate against the efficiency, savings, reliability, risks of mistakes and harms, and other social and cultural aspects of the automated systems.

Probably the only real complication from a regulatory point of view for these systems is the decision to shift from one model to another. I have considered this problem with Vijeyarasa in relation to the VioGén, a computer-based system used for the assessment of the level risk of revictimisation of victims of gender-based violence in Spain.Footnote ²⁰ If an AI-based system is considered to be ready to deliver an output better than a human qualitative assessment or one based on traditional statistics, what is the degree of outperformance compared to humans, or the level of reassurance necessary to make that shift, and how much capacity should be left to bureaucrats to override the system’s decisions? These are not easy questions, but they are not difficult to visualise: should the standard for accepting automation be performing better than an average bureaucrat? Better than the bureaucrats with the best track records? Or when the risk of expected errors is considered as reduced as possible? At similar levels of performance, should cost be considered?

These are decisions that administrative law could explicitly leave to the discretion of bureaucrats, establish ex ante binding rules or principles, or leave it to the judiciary to consider it if a complaint is made. Again, not easy questions, but decisions that could be addressed within the principles that we are familiar with. In the end, the reasoning is not that dissimilar from a decision to externalise to a private provider a service hitherto delivered by the state.

To be clear, I am not suggesting that there is anything intrinsically wrong with focusing our (regulatory) attention on these issues. I believe, however, that this view encompasses a very narrow understanding of what AI systems could do in the public sector and the legal problems it can create. This approach is conceptualised in terms of efficiency and the hope that AI can finally deliver the (so far) unmet promise of the productivity revolution that was expected from the massive incorporation of computers in the public officials’ desks.Footnote ²¹

From that perspective AI could be a key element of that Automated State. AI systems could be optimised to limit the variance between decisions with similar or equal relevant attributes. Consequently, AI-driven systems could be the best way to reach a reasonable level of impartiality, while fulfilling mundane tasks previously performed by humans.

Obviously, this cannot happen without maintaining or improving the rights of those individually or collectively affected by these automated decisions. Administrative law would need to ensure that the possible mistakes of these ‘approved and certified’ systems can be redressed. The legal system must allow affected parties to challenge outputs that they believe do not correctly implement policy. This could be, at least, on the basis of a possible violation of any relevant laws for that policy or a lack of coherence with its objectives, or with other relevant rights of the person or entity affected by the output of the system.

Therefore, the only need for reforms (if any) for administrative law in this Automated State is to (i) create a path to pre-validate the system; (ii) create guidance or determine when to change to such a system; and (iii) enable parties affected by its outputs to complain and challenge these decisions.

Other chapters in this book look at this third point in more detail, but I see it as requiring affected parties to go ‘deeper’ into the automated (or machine-supported) decision. The affected party, alone or in conjunction with others affected by the same or similar decisions from that system, need to be able to – at least – (i) explore why their decision can be distinguished from similar cases deserving a different administrative response; (ii) be able to raise new distinguishing factors (attributes) not considered by the system; and (iii) challenge the whole decision system on the basis of the process of pre-certification of the system and its subsequent monitoring as the system learns.

Generally speaking, the type of legislative reform necessary to accommodate this change will not create excessive friction with the approaches to administrative law already in place in civil and common law systems.Footnote ²² Essentially, the only particularity is to be sure that the rights of the parties affected by administrative decisions do not get diluted because the administrative decision comes from a machine. The right to receive a reply, or to an intelligible explanation, or to appeal a decision considered illegal should be adapted, but not substantially changed.

Perhaps the concept of the ‘organ’ in civil law systems and the allocation of responsibility to the organ, which in practice makes administrative law a distinct area of law, with a different logic from the civil/criminal dichotomy still dominating the common law system,Footnote ²³ could make the transition easier in civil law systems. The organ, not the bureaucrats or their service, is responsible for its outputs. However, certain rules about the burden of proof and the deference towards the state in continental systems could make it more difficult to interrogate the decision-making process of a machine.

Finally, in terms of administrative law, it is even possible to envisage a machine-driven layer of supervision or control that could monitor human action, that is, using AI to supervise the activity of public officials. One could imagine a machine-learning system which could continuously check administrative outputs created by human bureaucrats alerting affected parties and/or bureaucrats when it detects decisions that do not appear to align with previous practice or with the application of the normative and legal framework. Such an Automated State could even increase the homogeneity and predictability of administrative procedures and their alignment with the regulatory regime,Footnote ²⁴ therefore increasing trust in the public system.

In this scenario, the Automated State will not (for the time being) replace humans, but work alongside them and only reveal itself when there is a disparity of criteria between the output of the human bureaucrat and the automatic one. The existence of this Automated State cohabiting with a manual one may require different administrative rules for human-made decisions. When decisions diverge, possible options may involve an obligation to notify affected parties of this divergence and, perhaps, granting them an automatic appeal to other administrative entities, or requiring reconsideration by the decision-maker, or imposing on the human decision-maker an obligation of more detailed and explicit motivations. In this state of automation, the human administrative decision will not be fully acceptable unless it aligns with the expected one from the Automated State. And, yet, we can still address these situations with a lex specialis for the automated decision, remaining within the logic and mechanisms of ‘traditional’ administrative law.

Having now covered the easier of the two transitions, it is now the time to consider the other Automated State, the one that liberal-democratic legal systems could find most difficult to accommodate. The tale of the Automated State that designs or evaluates policy decisions.

7.3 Regulating the Unseen Automated State

As noted in the Introduction, AI can be harnessed by public authorities in ways that have not been seen before. The idea of a digital twin, for example, alters the logic behind the discretion in the decision of public authorities, as it makes possible to envisage both states of a world, with and without a decision.

If we take another step in the same direction, one could even assume that in the future the design and establishment of policy itself could be delegated to machines (cyber-delegation).Footnote ²⁵ In this scenario, AI systems could be monitoring opportunities among existing data to suggest new policies or the modification of existing regulations in order to achieve certain objectives as defined by humans or other AI systems.

Yet, for the purpose of this chapter, we will remain at the level of the foreseeable future and only consider systems that may contribute to policy determination. The discussion below also assumes that the systems are correctly designed and operate as they are expected.

This type of automation of the state involves expert systems that are considered to provide higher levels of confidence about choices in the policy-making process. This view of the Automated State sees AI systems as engineered mechanisms ‘that generate[…] outputs such as content, forecasts, recommendations or decisions for a given set of human-defined objectives’, in line with current thinking in the global standardisation process.Footnote ²⁶

This corresponds to existing observations in governance theory that note that ‘the transfer of governmental decision-making authority to outside actors occurs along a continuum’.Footnote ²⁷ A public authority generally decides on policy through an output generated by one of its employees (elected or appointed) or a committee of them. How to reach that policy decision could be left to the employees of that public authority, reached through a system of consultation, or fully deferred to a committee of experts.

Regardless of how the decision is reached, the essential element is that the decision process is oriented towards the achievement of an implicit or explicit set of human-defined objectives. Achieving these objectives is the raison d’être of the policy decision, even if, from a social point of view, the ultimate motivation, and, therefore, the legitimacy of the decision to set these precise objectives, may have been spurious (e.g., to unjustifiably favour a certain service provider over others). If the advice to the decision-maker is assisted by an AI system, however, that objective needs to be explicit as it is what the system will try to achieve and optimise in relation to other factors.

Allow me, however, to explain the consequences of this statement, before exploring these objectives. The state, as an agent, does not act on its own behalf. The existence of the modern liberal state is based on the founding principle that it does not act on its own interest, but as a human creation for the benefit of its society. The human-defined objectives are the reason for its existence, the state being a tool to achieve them.

Leaving aside if this is actually the case – diverging from those who see the state as better described as a mechanism for preservation of certain parts of that society or more theoretical discussions about the role of the state – in this section I assume that decision-makers are honest about those objectives or boundary conditions.Footnote ²⁸ As noted in the previous sections, what matters for systems that merely apply policy to reach outputs is to correctly reflect that policy in those outputs. Broader objectives such as fairness through redistribution, or equality of opportunities must be embedded in the policy design, the outputs just being the automated application of that policy. Here, the policy is what is being created by the Automated State, so the system will design or propose a policy that optimises those objectives.

In societies that democratically elect its decision-makers, one can assume that some of these objectives can come from different sources, such as:

1. 1. Those determined by basic legal norms that constrain the action of public authorities. This is the case, for example, of constitutional rules, such as ‘no discrimination on grounds of age or socioeconomic grounds’, or a mandate to redress inequality derived from socioeconomic grounds or a ‘right to access a no-fee system of quality education until the age of 16’.Footnote ²⁹

2. 2. Those determined by the objectives hierarchically established at higher levels of decision-making. For example, one could consider the programme from a central government, or the priorities established at the ministerial level – and the principles explicated therein – as a restriction to the action of lower hierarchical levels, especially when materialised in formal directives. For example, in the fiscal context, one objective could be increasing the fight against fraud or, in the education context, improving the standardised results of students from disadvantage backgrounds.

3. 3. Those that are determined by the specific decision-maker (organ or individual), who is formally in charge of making that decision. For example, in the tax context it could be accepting that more exhaustive detection of fraud would be at the cost of more administrative complaints from honest taxpayers that would be incorrectly identified. In the education context it could be a limit in the amount of resources that could be allocated to improving educational standards overall.

In all three cases the objective is the key element for the development of policy. An Automated State in which AI systems are designed to optimise these objectives will, in principle, derive its legitimacy and legality from these objectives. More importantly, the sequence of objectives listed above can be seen as hierarchical, with policymakers assisted by these AI systems bound by the objectives established by the superior levels. As an example, a decision-maker on the lowest level of hierarchy who sets the level of expenditure at this lowest level (district, local council, federated state, or national level) for public (government paid) education could not accept any recommendations from the Automated State that could suggest as optimal interventions those expected to deliver a significant improvement for overall academic standards for 99.9 per cent of the students of that administrative level, but would not offer free education for the 0.1 per cent living in the most remote communities if there is a constitutional mandate to offer free education for all. A proposal that would involve the exclusion of even one person would not be acceptable. Similarly, an option that improves the academic results for all at a given cost, but forces students from the most deprived backgrounds to separate from their families would be a violation of a tier 1 objective, and, therefore, not acceptable either. A correct design of the AI system producing the recommendation should not even generate these options.

Obviously, not all objectives follow this neat hierarchical structure. Sometimes the systems could offer recommendations for policy options that are seen as trade-offs between objectives at the same level. Some other times, there could be enough flexibility in the language of the boundary conditions that, at least formally speaking, it would not require to build those boundary conditions into the system. This would allow systems to generate some proposals that would not be accepted under a stricter objective or a different reading of the wording of the objective.

For example, a system may be allowed by humans to suggest an education policy that is expected to achieve a significant improvement for 99.9 per cent of the students. In this case policymakers tasked with creating a policy to improve standardised scores may decide to allow systems to consider this option, if they knew that they could meet the formal requirement of providing free education for all students through other means or policies. That could for example involve providing untested remote self-learning program to students for free. This would be feasible in policy settings where the boundary condition is just ‘providing non-fee education’ without qualification of ‘(proved) quality’.

As we know, it is not unusual for general mandates to be unqualified, particularly at the constitutional level, and see the qualifications being derived by interpretation from other sources (human rights principles or meaningful interpretations from high-ranking courts). In any case, it is how humans decide to translate those mandates into the system objectives what matters here.

Yet, this kind of problem may still not be that different from what systems of administrative control are facing today. The level of discretion is still added into the systems by humans and this concrete human choice (the decision to place other options within the scope of analysis) is still the one that could be controlled by courts, Ombudsman, or any other systems of administrative checks.

A second type of problem appears when the system is showing that certain options are superior to others, but benefit some groups of people differently. For example, a system that is expected to improve the results of all students, but improve the results of students from advantaged socioeconomic backgrounds by 10 per cent and those from disadvantaged backgrounds by the same 10 per cent would not be generated as a recommendation by a system which is requested to produce only options that are also expected to redress inequality. However, the same system could recommend the next best option at the same cost, which is expected to improve the results of the first group by 7 per cent and the second by 9 per cent, as this option does address inequality, which was a requirement set by humans to the system.

Favouring the latter proposal may seem absurd from a (human) rational point of view. The first suggestion is clearly superior as it would see all students being better off overall in terms of academic performance. Yet, only the second system would meet the objectives manifested in boundary conditions. A correctly built system would respect the hierarchy of objectives. Given that redressing inequality is more likely to be a constitutional or general mandate and, therefore, trump improving results – which is more likely to be an objective set at a lower hierarchical level – the first option would never be offered as a suggestion to the policymaker.

In this case, a better approach would be to allow the Automated State to present the first option to policymakers as far as the expected outputs are clear and the violation of the boundary condition is explicit. This would allow policymakers to simultaneously intervene in other ways to redress inequality. AI systems do not live in a policy vacuum, so it is important to design them and use them in a way that allows for a broader human perspective.

A third type of problem could occur when the system is designed with an added level of complexity, presenting the options in terms of trade-offs between different objectives at the same level.Footnote ³⁰ For example, the choice could be offered to the decision-maker as policies that are expected to deliver overall improvements of educational standards, for all students, with a bigger gain for those from disadvantaged backgrounds (i.e., meeting all the boundary conditions and objectives), but expressed in terms of cost (in monetary units) and levels of overall improvement. Then it would be up to the decision-maker to decide which option of the many possible ones would be preferred. In this case, the main problem is one of allocation of resources, so this could initially be left to human discretion. However, as public resources are limited, if different AI systems are used to automate policy-making, setting a limit for one of these trade-offs would affect the level of trade-offs for other recommendation systems operating in other policy areas.

This could be intuitively grasped in the tax context. Imagine a public authority tasked with maximising tax revenue at the lowest cost within the legal boundaries. The system assessing anti-fraud policy may recommend an optimal level of investment in anti-fraud and establish the identified taxpayers that should be checked. Other system may be used to recommend possible media campaigns promoting compliance. This other system may suggest an optimal level of investment and the type of campaigns expected to give the highest return. Yet, it is possible that the level of resources available may not be enough to follow both suggestions. A broader system could be created to optimise both systems considered together, but what could not be done is considering each of the systems in isolation.

Looking together at these three types of problems gives us an idea about how this Automated State is different. For the systems discussed in Section 7.2, those that replace humans, I indicated that the most promising regulatory approach is the one that focuses on the systems and the testing beforehand and then shifts to monitoring of the outputs. As the bulk of the effects of each automated decision will be centred around a limited (even if large) number of individuals, the affected parties will have an incentive to raise their concerns about these decisions. This could allow for a human (administrative or judicial) review of these decisions according to the applicable rules. The automated outputs could be compared with what humans could do, according to the applicable administrative law, in those circumstances. This process would confirm or modify the automated decision and the automated systems could be refined to learn from any identified errors.

However, for the systems discussed in Section 7.3, that are used to do things that humans cannot do, especially in terms of policy design or supervision, it is impossible to proceed in such a way. Any challenge of a concrete decision could not be compared with what a human could do. Any disagreement about the reliability of the system would be too complex to disentangle.

Yet, there are aspects of the process that would still need to meet societal standards about adequate use of resources, fulfilment of superior principles of the state, or, more generally, the need to meet the state’s positive obligations to protect human rights, remove inequalities, and redress violations of rights of individuals or groups. At the very least there are three elements regarding how humans interact with the systems that generate the outputs that could be considered.

First, humans must test the systems. To grant some legal value to the recommendations of these systems – for example, to demand more from policymakers that deviate from their recommendations – this type of Automated State must be tested in real-life, real-time conditions. In the next section I explain in more detail what I mean by this point. Suffice to note here that systems tested only against data from the past may not perform well in the future and their legal usefulness as a standard for the behaviour of policymakers may therefore be undermined.

Second, humans must set the objectives that the system is meant to optimise (and suggest ways to achieve) and the boundaries that the suggestions are not meant to trespass. Which objectives and boundaries are incorporated into the system and how they are hierarchically placed and balanced can be explained and the legality of those choices controlled.

Third, humans must translate automated suggestions into policy. The example of the AI system used for assessing quality of teaching in the United States discussed in Chapter 10 of this bookFootnote ³¹ is a perfect example of this point. Even if we trusted that the system was correctly evaluating the value of a teacher in terms of improvement of the results of their students, the consequence attached to those findings is what really matters in the legal sphere. Policymakers using such a system to assess quality of teaching could decide to fire the lowest performing teachers – as it was the case in Houston – or to invest more in the training of those teachers.

7.4 Preparing for the Two Tales of the Automated State

In the previous sections, I discussed the two different tales of the Automated State and the distinct legal implications that each tale involves. This, however, was an oversimplification. Going back to the VioGén system presented above, one can today see a system of implementation, typical of the first tale of the Automated State, as it assesses each individual woman based on their risk of revictimisation. The suggested assessment, if accepted by the human decision-maker, automatically triggers for that victim the implementation of the protection protocol linked to her level of risk. Yet, VioGén could easily become a policy design tool. For example, it could be repurposed to collate all data for all victims and redeveloped into a system that allocates resources between women (e.g., levels of police surveillance, allocation of housing, allocations of educational programmes, suggestions about levels of monitoring of restraining orders for those charged with gender-based violence). If we consider every automated system a potential policy tool, we may be moving towards an excessive degree of administrative control of policy-making. As policymakers will have much more and richer data, administrative law could be used to question virtually any policy decision.

At the other extreme, one could think that it could be better to revert to almost complete deference to the discretion of policymakers. If we think of policy-making as a black box driven by criteria of opportunity or the preferences of high-ranking elected officials it is difficult to justify the need for a new type of administrative law for these situations, even if the policymakers are better placed to assess the consequences of their decisions. One can, for example, imagine the decision of a public authority to approve a new urban planning policy after a number of houses are destroyed by floods. The new policy may be so different to previous practice that its effects in case of another flood cannot be assessed by an AI-driven recommendation system. The system, however, can suggest several minor modifications that are expected to be enough to avoid a repetition of the situation. In this case the ultimate purpose of the new policy may be to increase resilience of the housing in case of new floods, but the real value of the initiative is to convey that public authorities are seen as reacting to social needs.

The expected evolution of the first type of the Automated State could also support deferring to the discretion of policymakers and ignoring the new tools of the Automated State from an administrative law perspective. As more decision-making is automated at the level of implementation, a reduction of variance should be expected. The effect in the world of these outputs could then be analysed in real time and the outputs will speak for the policies they implement. Public office holders would then be accountable if they fail to modify policies that are generating undesirable outputs. The effects of a change in policy that is implemented through fully automated means will be the basis to judge that policy. Policy design will not only refer to ‘design’, but also the choice and design of the automated tools that implement it.

In my view, none of these options are reasonable, so it is necessary to start developing new principles that acknowledge the legal relevance of these new tools in policy-making, without separating ourselves excessively from the process. The absolute deference to policymakers choices, even if tempting, would be a reversal of the positive ‘erosion of the boundaries separating what lies inside a government and its administration and what lies outside them’ or, in other words, of the transition from ‘government’ to ‘governance’.Footnote ³²

A way to illustrate this latter point would be to consider the French example, and its evolution from a black-box State to an administré-centred one.Footnote ³³ This transition, induced – according to a leading French scholar – by Scandinavian-, German-, and EU-driven influences, has forced administrative law to go beyond traditional rights in French law (to an intelligible explanation, to receive a reply, to appeal a decision considered illegal) into a regime where the administré can be involved in the decision-making process and is empowered vis-à-vis the State.Footnote ³⁴ It is not just the output, but also the logic behind the process that matters.

If the reasons for policy decisions matter, how can we then use the Automated State to demand better accountability for those decisions? Trusting this Automated State blindly or inextricably binding decision-makers to its decisions does not appear to be a good option, even if we have tested the AI systems according to the most stringent requirements. My suggestion is to develop a few principles or heuristics that could guide us in the process of reform of administrative law.

The first – and most essential from my point of view in a technology without historic track record of performance – is that systems designed to make predictions about impacts of public actions in the future need to have been tested in real conditions. This Automated State could only be relied upon for the purpose of legal assessments of policy decisions, if the predictions or suggestions of its systems have been proven to be reliable over a given number of years before the date of the decision.

Systems that are ‘refined’ and reliable when tested against the past cannot be a legal basis to contest policy decisions. Only real-life experiments for policy design without ‘the benefit of hindsight’ should matter. In these cases, deference should be paid to policymakers to the same degree as before. However, for learning and testing processes an adequate record of use should be kept – that is, systematically recording how the system was used (for testing purposes) in real-time conditions.

Secondly, we should be flexible about setting boundaries and objectives. Administrative rules should not impose designs that are excessively strict in terms of hierarchy of objectives, as some of the objectives can be addressed by different policies at the same time, not all covered by the automated systems. For those cases, the systems should be designed to allow for the relaxation of the boundary conditions (objectives) in a transparent manner, so policymakers can assess the need for other interventions. In the example above about the education systems, a rigid translation of legal principles into data could blindside us to policy options that could be adapted further to respect legal boundaries or even be the reason to adapt those boundaries.

Finally, decisions that deviate from those suggested by legally reliable automated systems should be (i) motivated by decision-makers in more detail than traditionally required; and (ii) the selected (non-recommended/Pareto-inferior) policy should be also assessed with the relevant AI systems before implementation. The results of that assessment, the policymaker motivation, and all connected information should be made – in normal circumstances – publicly available. This would allow the improvement of systems, if necessary (e.g., incorporating other considerations), and allow better administrative or judicial control of the decision in the future. Guidance could be extracted from decisions that override recommendations of environmental impact assessment, where an administrative culture that relied on discretion rather than law – for example in the English contextFootnote ³⁵ – has traditionally been an obstacle to the effective judicial control of those decisions. Discretion should be accepted as an option as far as it is explicitly justified and, hopefully, used for developing better automated systems.

8.1 Introduction

Catalonia is home to the largest Muslim communities of the Iberian Peninsula: a roughly 8 per cent of its population (617,453 out of 7,739,758) follows the Islamic tradition. Despite the neofascist natalist rhetoric of far-right parties speaking about a ‘great replacement’ (Aduriz, 2022), the number of Muslim students is consistent with the total number of Muslims. There are 1,337,965 non-tertiary education students in Catalonia,Footnote ¹ approximately 101,721 of them are Muslims (7.60 per cent).Footnote ² However, here the statistical consistencies end. The majority of Muslims work in precarious jobs or do not have jobs at all. Roughly 20 per cent of the migrant population is unemployed, compared to 8.19 per cent of general population in Catalonia.Footnote ³ They live in impoverished and deprived zones with less access to public resources and green areas. Traditionally migrant neighbourhoods such as la Barceloneta, el Raval, or Poblenou in Barcelona are among the most affected by the touristification and gentrification unleashed by foreign investment firms.Footnote ⁴ With scarce jobs, skyrocketing rents, and living costs, thousands of families are forced to live in slums and industrial areas with extremely poor living conditions, and are exposed to violent evictions and fatal accidents.Footnote ⁵ But the socioeconomic is just one of the areas where Muslim population face discrimination.

Muslim communities are targeted in relation to their beliefs, culture, and ways of socialising. Despite their demands, 90 per cent of Muslim students do not enjoy the same right to religious class in the public education system as their Christian-Catholic counterparts. Muslim communities often face fierce resistance from far-right organisations and public officers against their attempts of setting up and/or building mosques.Footnote ⁶ However, the situation is even worse within the welfare and the punitive systems. People of migrant origin, especially those from countries with Muslim majorities, are disproportionally present in the prison system. Despite being just 3.1 per cent of the population, people of Maghrebian background represent 16 per cent of the incarcerated population in Catalonia.Footnote ⁷ As has been pointed out by a large number of academics and activists, this is not a matter of rampant criminality among a very specific and identifiable segment of population, but the consequence of racial profiling among police agencies and social services who disproportionately target those produced as ‘enemies’.Footnote ⁸ These episodes of discrimination are not accidental, but rather functional elements of what we conceptualised as the Islamophobic Consensus.

From the early days of inquisition to the latest developments in automation the social construction of the Muslim as a social enemy has helped to shape both the Spanish identity and the Spanish state’s surveillance and repressive apparatuses. The subjectification of Muslims as a threat ranges from labelling them as job-stealers, and herein as a risk to the working class, to them being the ultimate enemy, the terrorist.Footnote ⁹ This racialisation process operates not only in relation to newcomers, but also towards the second and third generation of Muslims. As Suhaymah Manzoor-Khan has recently pointed out,Footnote ¹⁰ the pernicious characteristics attributed to the ‘Muslim culture’ rapidly evolved into a racially inherited condition that passes through generations.

The second decade of the twenty-first century has witnessed the proliferation of heavily racialised surveillance and carceral geographies. As the anti-immigrant raids in the United Kingdom, the United States, and Australia show, bordering technologies extend now to every territory, every street, and every working place.Footnote ¹¹ The ‘exceptional’ and ‘temporary’ powers to surveil and to punish delegated to public authorities in order to fight the ‘war on terror’ are now well-established practices affecting every area of the public life. In Catalonia, entire Muslim communities and mosques are targeted and surveilled by an expanding ‘preventive’ sociotechnical system.Footnote ¹² An army of educators, social workers, and police officers are now entrusted with gathering information from endless data points, and to report to their civil and police superiors the most subtle changes in individual and collective behaviour. For instance, teachers are taught by police agencies that the everyday manifestations of religiosity such as the adoption of ‘Islamic’ dress codes or collective prayer could could be indicators of ‘radicalisation’. This information is used to terrorise vulnerable communities who are routinely threatened with criminalisation, family separation, and even deportation.

The system to prevent terrorism envisaged by the Spanish multiagency initiative on national security operates as a self-fulfilling prophecy mechanism. The risk assessments tools may flag as a threatening symbol of radicalisation of mundane and often contradictory facts. For instance, either exercising too much or having an absolute sedentary life may induce vigilantes to believe that a young Muslim is up to something.Footnote ¹³ In the same vein, young Muslims following severe religious routines may signal fundamentalist tendencies, but also not following religious mandates may be, in the eyes of the police services, a worrying nihilistic symptom of latent lone-wolf tendencies. These instruments and the way they look, and produce Muslims have a profound impact on the lives of thousands. Are these individuals appropriate candidates for welfare benefits? Will they be subject of an investigation either by social services or by any of the multiple police agencies? Will they be released on parole? Will they remain in prison? Will they be processed under terrorism charges? A vast sociotechnical assemblage of analogical and digital technologies controls the lives of thousands of Muslim people in Catalonia.

However, these control and disciplinary technologies are not only aimed at limiting, cancelling, and governing subaltern people. Drawing on the structural comprehension of racism pinpointed by Eduardo Bonilla Silva,Footnote ¹⁴ we argue that these technologies are part of what here is coined as the Islamophobic Consensus, that is, the Southern European iteration of racial neoliberalism. A system of domination intended to reinforce structural gender, racial, and class inequalities, through a sociotechnical system encompassing all sorts of surveillance, repressive legal, political, economic, educational, and military instruments. Some may argue that the Spanish surveillance state has not reached full or high degrees of datafication or digitalisation as it may have been the case in countries such as the Netherlands.Footnote ¹⁵ And, perhaps, the digitalisation in Spain will never reach this level, given the characteristics of Southern European countries. However, as this chapter hypothesises, the vast surveillance apparatus deployed for gathering data of vulnerable populations, and the extensive use of actuarial and automated methods is leading to a form of datafied surveillance state.Footnote ¹⁶

This chapter has two objectives. First, to point to the necessity of building a non-Anglocentric theoretical framework from which to study the ideological and sociological fundamentals in which datafied forms of societal oppression stand. As we further develop, the datafication techniques underpinning contemporary automated governmentalities build on long-term historical, epistemological, and ideological processes. In the case of Southern Europe these techniques can be traced back to the sixteenth century genocidal biopolitics deployed against Muslims, Jews, Roma, and Indigenous peoples.Footnote ¹⁷ We aim to fill an important gap in race, sociolegal, and critical data studies. Despite Spain and Catalonia’s long and influential history of surveillance and racial oppression, its institutional surveillance apparatuses remain largely unknown and understudied. As the chapter demonstrates, the data surveillance state does not rely on the same technologies, focus on the same subjects, and pursue the same objectives in every context. On the contrary, it draws on contextual genealogies of domination, specific socioeconomic structures, and distinctive forms of distributing power. The second objective is to provide an empirical analysis on the ways the Islamophobic ConsensusFootnote ¹⁸ is being operationalised in Catalonia, and with it to expose the overlapped racist mechanisms governing the lives of racialised black and brown young adults.

Drawing on empirical and archival research, the first part of the chapter analyses the surveillance-governmental apparatus deployed over Islamic communities in Catalunya. The second part of the chapter frames the ideological, epistemological, and historical fundamentals of the Southern European way to racial neoliberalism, here labelled as the Islamophobic Consensus. Drawing on surveillance and critical race studies, we synthesise the defining features that distinguish this model of domination from other iterations of neoliberal racism. The section continues examining two dimensions of the Islamophobic Consensus: Islamophobia as an epistemology of domination and Islamophobia as a governmentality.

8.2 Datafying Islamophobia

Since 2016, Catalonia has been implementing the Catalan Protocol for prevention, detection and intervention in processes of Violent Extremism or PRODERAE in schools, local police stations, prisons, and social services. PRODERAE is part of the wider Special Counter Terrorism Policing Operational Program. Despite its relevance (and the persistent requests of the authors through official channels) most details of the PRODERAE remain unavailable to the public and hence hidden from democratic scrutiny due to ‘security reasons’.Footnote ¹⁹ However, a leak allowed us to get access to some documents and to a non-official recording of the PRODERAE training. On 18 May 2022, upon the requirement of the Catalan parliamentary group of the Candidatura d’Unitat Popular, we also obtained information on the training given on these instruments to public servants across different services. Specifically, the scarce data provided by the Catalan authorities accounts for the number of attendees and the number of courses given. We have crossed this documentation with the PRODERAE antecedent, the PRODERAI-CE Protocol de prevenció, detecció i intervenció de processos de radicalització islamista- Centres Educatius [Protocol for the prevention, detection and intervention of Islamist radicalization processes – Education centres] widely used over young Muslims. While not fully accurate, this analysis could provide a glimpse into the racist governmental strategies deployed over Muslim population in Catalonia.

Both instruments evaluate and assess the risk to individuals based on different elements such as their individual behaviour, the social, economic, professional, and educational contexts, or the ways they engage with beliefs, politics, and religion. In this regard the instruments used in Catalonia are similar to other predictive and preemptive tools used in the European context, such as the Dutch Violent Extremism Risk AssessmentFootnote ²⁰ and the British Structured Professional Guidelines for Assessing Risk of Extremist Offending.Footnote ²¹ Like the infamous British Prevent strategy,Footnote ²² the model proposed by the Spanish and Catalan authorities establishes a comprehensive although distributed surveillance regime over the population under risk of radicalisation (the entire Muslim community).

The PRODERAI-CE differentiates four areas from which the risk of a given subject will be evaluated: personal development, school context, family context, and social context. To obtain information the system relies on a vast array of agents, technologies, and points of data extraction that amalgamate under the securitarian prism – members of the community, educators, social workers, police officers, and intelligence services. To that end, the Catalan government has deployed considerable efforts and resources in providing training on the use of these tools to educators (3,118 since 2018), officers of the criminal justice system (CJS) including lawyers and social workers (2,013 since 2015), and police officers (30,902 since 2015). This has resulted in 667 thorough investigations of which 250 were conducted by police intelligence services. Herein, the boundaries between welfare and policing, street surveillance and cyberwarfare blurry in a diffuse although perceptible regime of racialised social control.

Among the factors related to personal development the instruments evaluate negatively ‘the difficulty of managing emotions’, ‘the difficulty of building a multiple identity’, the ‘proximity to radicalised peer groups’, and ‘low expectations of success’.Footnote ²³ Elements such as the dress code (hijab, niqab), personal appearance (beard), as well as dietary and leisure habits (halal, alcohol consumption), are surveilled with special interest. In the same vein public servants are instructed to follow closely religious beliefs and political attitudes towards specific issues. Besides the above elements, school educators are asked to pay special attention to ‘the lack of bonds between peers’ and ‘the difficulty of (the teacher) establishing bonds with students’,Footnote ²⁴ as these elements are considered risk indicators.

With regard to the family environment, ‘low family participation and involvement in school activities’ and ‘the [lack of] sense of belonging’ are also considered as elements to consider in measuring potential radicalisation processes.Footnote ²⁵ In terms of social context, the instruments evaluate negatively ‘the influence of social networks’, or if the individual belongs to ‘socioeconomically disadvantaged contexts’. Another element that may trigger an alarm is the ‘lack of attachment to the social environment’.Footnote ²⁶ The information collected by public servants is transferred to the Territorial Evaluation and Monitoring Board where police officers and education inspectors will decide the feasibility of the indicated risk. This could eventually lead to further investigation, wiretapping, raids, detentions, and deportations.

Given the opacity, secrecy, and the lack of transparencyFootnote ²⁷ guiding the Spanish and Catalan authorities’ operations with regard to cases of alleged radicalisation, it is utterly difficult for researchers, activists, and even politicians to access critical information. What data gathering tools, both analogical and digital, are currently being used? How is the data gathered across services being stored, processed, analysed, and by whom? Are these data sets feeding ADM systems used in the public sector? Who is entrusted with overseeing these data-intensive tasks? Had these instruments and technological tools passed any form of auditing and impact assessment? We have asked Spanish and Catalan authorities these and other questions, but have not received any response whatsoever. However, we can infer some of this information from: (1) The documentation related to RisCanvi, the risk assessment tool used in the Catalan prison system to assess the potential recidivism of inmates in order to determine paroles, and (2) the well documented usage of tools for preventing ‘radicalisation’ in the United Kingdom and the Netherlands.

RisCanvi is an automated tool used by prison authorities, psychologists, criminologists, and social workers in the Catalan prison system. So far only one official report has been published,Footnote ²⁸ which is consistent with the lack of transparency in other instruments and areas; however, the report and several academic works published by its designers gives a glimpse of the system. The tool provides a recidivism risk score that helps professionals to decide whether inmates can be paroled. For that it takes into account forty-five variables, encompassing behavioural, sociodemographic, biographical, educational, economic and social data. For instance, the system will measure whether an inmate belongs to a vulnerable group, their criminal history (and that of their peers), addictions, sexual behaviour, and so on. While necessarily overseen by humans, officers rarely disagree with the ‘algorithmic score’ (1 per cent) which given the 82 per cent false-positive rateFootnote ²⁹ leads to a situation of unfairness. The weight of each variable in the final score has not been revealed, however given the known items we can infer that a potential automated discrimination may be taking place. For instance, the tool negatively weighs a vulnerable economic situation, employment status, the criminal history of family and peers among others. Items like these have been used in other toolsFootnote ³⁰ as proxies to punish race and poverty, reinforcing social prejudices against vulnerable collectives. In addition, RisCanvi has been built upon historical data gathered by the prison system, the fact which raises important problems. As we have demonstrated elsewhere,Footnote ³¹ classism and racism run rampant across the Spanish and Catalan criminal justice systems. Racialised and poor subjects are more likely to be stopped, detained, arrested, and processed. Hence, the ‘dirty’ data setFootnote ³² feeding the system nurture a discriminatory feedback loop.

Britain’s Violent Extremism Preventing Program, popularly known as Prevent, is part of United Kingdom’s national counter-terrorism strategy CONTEST. It was launched in 2006 by the then governing UK Labour Party.Footnote ³³ Its reach has expanded from police and prisons, to child care, elementary and high schools, tertiary education institutions, and even the National Healthcare System (NHS). The Extremism Risk Guidelines 22+ (known as ERG 22+) developed by ‘Her Majesty’s Prison and Probations Service’ in 2011 is the inductive instrument that gathers the ‘radicalisation signals’ and backs-up the program with the risk assessment framework. The ERG22+ is presented as ‘a structured professional judgement (SPJ) tool that assesses individuals along 22 factors that are grouped into three domains; Engagement, Intent and Capability’.Footnote ³⁴ This has been replicated in the PRODERAE-PRODERAI-CE training, which uses terminology such as ‘identity, meaning and belonging’, ‘us and them thinking’, ‘overidentification with a group, cause or ideology’, ‘the need to redress justice’, or ‘the need to defend against threats’.

Many scholars have highlighted how the UK’s automated tools associate Muslims with terrorism, putting the entire Muslim population on the spot.Footnote ³⁵ Moreover, recent research highlights community surveillance is becoming universal surveillance.Footnote ³⁶ For instance, NHS’ public servants are now legally obliged to comply with their policing tasks, not only over ‘suspicious communities’, but also have to look for unpredicted new patterns of extremism in the entire patient population.Footnote ³⁷

In fact, as Heath-KellyFootnote ³⁸ points out the implementation in the national healthcare and education systems belongs to modalities of calculation derived from automated and big data tools that enable mass surveillance methods. She even argues that this kind of surveillance inductively produces the terrorist profile.Footnote ³⁹ Consequently, the outcome of this approach is the production of Islamophobic data associated with Muslim (pre)criminality. Even if the cases are dismissed, the details of the people that flagged the alert remain in the UK’s police database for seven years.Footnote ⁴⁰ ‘Prevent’ has been the target of profound critique in numerous reports from antiracist and anticolonial grassroots movements (Islamic Human Rights Commission, Cage UK), as well as international human rights organisations such as the Transnational Institute and Amnesty International.Footnote ⁴¹ One of the last reports not only pointed to its Islamophobic and discriminatory nature, but also to its ineffectiveness.Footnote ⁴² Despite the wide critique, the UK Home Office has only expressed that they ‘can find no evidence to support these claims’.Footnote ⁴³

Despite the limited information available, the PRODERAE and PRODERAI show important theoretical and operational flaws worth highlighting. First and foremost, both instruments are aimed at preventing radicalisation. However, there is a striking lack of theoretical consensus on its definition.Footnote ⁴⁴ Radicalisation takes shape when protocols such as PRODERAE are applied. It is thus a tool for producing ‘dangerous subjects’. The second problem is that many of the hidden indicators are expressions of religious practice. Changing the dressing code or adopting a more visibly Muslim expression, as wearing a hijab, putting henna on their hands, respecting prayer hours, demanding a halal menu, speaking or expressing opinions based on Islamic precepts or even expressing social discontent or pointing out Islamophobic or racist practices can all be indicators of radicalisation.

The tools analysed are embedded in vagueness and abstraction, if not falling in blatant contradictions. Factors and indicators that guide their implementation are left to the arbitrary interpretation of public officers. For instance, playing too many violent video games may indicate ‘military training’, although not playing video games at all may be a symptom of rejection of ‘westernisation’, in consequence both playing and not playing video games become a cause of suspicion. In the same vein, many of the ‘radicalisation symptoms’ indicated by the tools, such as troubles in navigating multiple identities, swift changes in appearance, friends, and habits, are most often processes inherent to the personal development of teenagers and young adults, and not ‘strange’ or ‘deviated’ as the tools make them to be. These tools embrace a hyper individualistic approach making individuals responsible for the consequences of complex socio-structural problems. For instance, individuals are accused of separatism and cultural isolation, ignoring the endemic economic crisis that, along with the racial division of labour, nurtures a growing racialised geography and school segregation. To illustrate, the chances of being flagged as a risky subject dramatically rise when students rely too much on ‘cultural and religious’ peers, because as the document states ‘the school has difficulties in promoting an inclusive environment’.Footnote ⁴⁵ The tools, far from helping the school to better understand these difficulties, seem to present them as elements of suspicion. As we can see, the pernicious consequences of the racial neoliberal project are datafied and hidden under an aura of false technological neutrality, just to be weaponised against its victims.

Finally, as multiple scholars have warned, predictive and preemptive tools used across the public sector (welfare, CJS, policing, and surveillance) entail considerable risks especially for already vulnerable and racialised populations.Footnote ⁴⁶ This has been demonstrated in recent scandals involving classist and racist sociotechnical systems deployed in Australia, the United Kingdom, and the Netherlands, to name a few. It was Bernard Harcourt who famously stated that these technologies can ‘create a vicious circle, a kind of self-fulfilling prophecy’Footnote ⁴⁷ contributing to ‘reinforce[ing of] stigmatisation, significantly undermining living conditions of certain population groups and restricting the possibilities of insertion of the individuals belonging to them’.Footnote ⁴⁸ Some have rightfully described the plans to prevent radicalisation in Spain as an example of neoliberal exceptionalism.Footnote ⁴⁹ A system that ‘employs surveillance technologies and situational crime control measures and that minimises or curtails a variety of social welfare programs’ against vulnerable people, producing it as dangerous population and criminalising it accordingly. Far from preventing any potential harm, the datafication processes triggered by tools like the ones analysed increase the occurrence of racial pre-criminality and reinforce the socially harmful policies. Our aim in the following sections is to contextualise the ongoing actuarial and datafication processes within a longer history of Islamophobia that far predates contemporary forms of datafied governance.

8.3 Southern European Neoliberalism Fundamentals

Multiple local organisations and antiracist grassroots movements such as the Asociación Musulmana por los Derechos Humanos [The Islamic Association for Human Rights],Footnote ⁵⁰ SOS Racisme CatalunyaFootnote ⁵¹ have denounced how institutional, political, and social Islamophobia narratives run rampant in Southern Europe. They are not alone in their criticism. Higher supranational instances have also pointed in the same direction. For instance, the UN Special Rapporteur on freedom of religion or belief released in 2021, a report on anti-Muslim racism informing how government-driven securitisation processes severely affect Muslim rights to freely exercise their religion, with intelligence services surveilling mosques, and governments such as the French restricting the ability of Muslim communities to stabilising charitable institutions.Footnote ⁵² However, these efforts can do little against the Islamophobic narrative deployed at every institutional and social level. In the media, a wide variety of actors, from so-called liberal philosophers to well-known white feminist writers have contributed to the production of the Islamic otherFootnote ⁵³ with labels such as ‘backwards’, ‘antimodern’, ‘violent patriarchal’, and ‘dangerous’.Footnote ⁵⁴ As the report highlights, these stereotyped narratives promoted by ‘prominent politicians, influencers, and academics’ who ‘advance discourses online on both social networks and blogs that Islam is innately antithetical to democracy and human rights, particularly gender equality, often propagating the trope that all Muslim women are oppressed’. Sociologist Sara Farris has coined this ideological, neoliberal political-economy convergence as Feminacionalism.Footnote ⁵⁵ Despite meaningful divergencies in other political arenas, neoliberal politicians, right- and far-right nationalist parties and feminist bureaucrats, or ‘femicrats’ seem to agree on the intrinsic dangers of Islam in general and male Muslims in particular.

Politically, far-right parties cashed the endless succession of crises caused by financial capitalism, becoming key political actors in Spain (third political party), Portugal (third political party), Greece (formerly third political party), Italy (first political party). The most impoverished and discriminated segments of populations were used by the far-right as a scapegoat of the 2008 and 2021 crises, and accused of stealing jobs, being responsible for an inexistent wave of criminality, and the destruction of moral values and social coexistence. Rising neofascist political parties such as Vox in Spain (a spin-off of the conservative Popular Party) have, for instance, proposed to reverse the already granted Spanish citizenship to ‘dubious migrants’ stating that ‘[c]itizenship is a privilege’.Footnote ⁵⁶ These discriminatory discourses permeate the political landscape across the political spectrum due to the modern transhistorical persistence of what Edward Said described as Orientalism.Footnote ⁵⁷ Islamophobia is indeed one of the defining features of the Southern European iteration of racial neoliberalism. Although sharing some common traits with its Global North counterparts, Southern European racial neoliberalism emerges from a different genealogy and is built upon different socioeconomic and ideological structures, presenting thus its own characteristics.Footnote ⁵⁸ While the main objective of the chapter is to focus on the Islamophobic Consensus, it is worth highlighting some distinguishable elements of Southern European neoliberalism.

First, Southern European racial neoliberalism does not stick to a single ideology, policy, technology, and regulation, nor univocally attached to exclusive forms of domination. Instead, it is composed by a baroquianFootnote ⁵⁹ multilayered structure encompassing traditional and latest technological developments (including ADM and AI) with colonial and postcolonial practices of racialised governmentality developed through centuries of colonialism. These proto-racistFootnote ⁶⁰ dynamics defined by pre- and capitalist cultural and religion discrimination practices, still inform the performativity of the Spanish racial formation. For instance, the colour-line created during the slave economy still works as a racialising technology in the current welfare, migration, and criminal policies.Footnote ⁶¹ As Deepa Kumar stated: ‘While race is dynamic, contingent, and contextual, the ideology of Islamophobia attempts to fix what it means to be Muslim and to create a reified Muslim whose behaviour can be predicted, explained, and controlled.’Footnote ⁶² Because of the above, racial politics, deeply bound with the legacies of coloniality, operate with significant differences from other countries in the Global North. For instance, Romani people, an extremely diverse and historically oppressed minority,Footnote ⁶³ is also celebrated as quintessential of the Spanish and Catalan popular cultures. As the global success of the Catalan singer Rosalia stresses (see, for instance, her video ‘Málamente’), folklorised values and aesthetics associated with Romani people are appropriated by individuals and institutions and commodified, while Romani people are discriminated at every level.Footnote ⁶⁴

Secondly, the public sector plays a key role in the societal, economic, and political dimensions. It controls significant aspects of key ideological apparatuses such as schools and media. It holds a vast influence over the workforce through direct employment of relatively significant segments of population.Footnote ⁶⁵ Unlike other polities such as the United States, Southern European countries have not fully privatised their criminal justice systems, retaining much of the organisational, operational, and designing sovereignty over these areas.

Thirdly, the privatisation of democracy described by Basque philosopher Jule GoikoetxeaFootnote ⁶⁶ as the hijack of public institutions and common assets by corporations and private interests, and the perceptible sacrifice of social rights for the sake of the capitalist class has not fully impacted the entire population. As a plethora of feminist researchers demonstrate, women, especially those belonging to racialised communities, have disproportionally paid a heavy price containing what would have been otherwise a societal tragedy.Footnote ⁶⁷ They have disproportionately sustained the family structures that have safeguarded the well-being of entire families, especially taking care of dependents. In the following section we will focus on two dimensions of what we identify as the Southern European path to neoliberal racism, what here is called the Islamophobic Consensus: Islamophobia as a racialised epistemic formation, and as a form of governmentality.

8.4 Conclusion

In 2017 a series of attacks shocked Catalonia. A van was driven in the centric Rambla of Barcelona, killing fourteen people and injuring hundreds. Hours later in Cambrils (Catalonia), another woman was killed, and several others injured. According to the PRODERAE and PRODERAV tools, one of the most relevant factors in any type of radicalisation relates to the perceived sense of belonging and connection with a territory. However, a social educator from Ripoll, the hometown of the young adults who committed the attacks, said: ‘[t]hese boys were integrated; they spoke perfect Catalan and they became terrorists’. The attacks demonstrated the uselessness of the protocols, indicators, and criteria for detecting radicalisation. As we have seen, the PRODERAI/PRODERAEV are preventive actuarial methods aimed at measuring and preventing radicalisation. For that, the instruments draw on classic social risk factors (personal development, school context, family context, social context) along with other ‘radicalisation indicators’ inaccessible to the public. The concealment of these indicators from public knowledge hinders social and political opposition, precisely because it hides the explicitly Islamophobic character of the automated tools used. However, the problem will not be solved just by making these sociotechnical systems more transparent and accountable.

As we have demonstrated, the digital and analogue technologies used to control, surveil, and punish young Muslims are not ends by themselves, but rather they mean for reinforcing a socially harmful system of oppression rooted in the darkest moments of the global European domination. The Islamophobic Consensus, that is, the Southern European iteration of neoliberal racism, stands on centuries of eurocentrism and white suprematism articulated through intricate institutional, legal, political, and economic developments transcending regional and national boundaries. Similarly, today’s astonishing data gathering, data management and data analysis capabilities, and the ‘magic’ behind predictive automated tools are not spontaneous outputs but the result of centuries of training, experimentation, and scientific developments. From the early colonial censuses and regulations designed to protect the healthy Christian population from depraved Muslims and Jews to more recent forms of predictive policing and digital surveillance, numbers, statistics, and dozens of other governmental tools have served the interests of the powerful.

There are no shortcuts, neither technical, legal nor magical solutions for a global problem rooted in centuries of oppression, domination, genocide, and deprivation. No solution will come from a political party, a corporation, a new legal instrument (either National or Universal). The long history of struggles against colonialism, racism, and fascisms demonstrates that the perversity of domination extends from the most obscene and crude forms of domination to highly sophisticated and subtle alienation. To fight such massive structures we need, undoubtedly, powerful communities, and meaningful relations, let alone the energising voices of empowered singers such as Huda who reminds us to Keep It Halal. But we also need adequate epistemic tools to be able to think politically and historically about the events surrounding us. Hopefully, this chapter could help radical researchers and other folks in such endeavour.