More recently, administrative agencies have introduced ‘new public analytics’ approaches, using data-driven technologies and risk models to reshape how commonplace administrative decisions are produced.Footnote 1
5.1 Introduction
Artificial intelligence (AI) is a broad church. Automated decision-making (ADM), a subset of AI, is the form of technology most commonly encountered in public administration of the social services, a generic term which includes income support (social security) and funding or provision of services such as disability support funding under Australia’s National Disability Insurance Scheme (NDIS). New public analytics is a label that nicely captures how ADM is deployed as the contemporary form of public administration.Footnote 2
ADM has long been an integral aid to the work of hard-pressed human administrators exercising their delegated social security powers in Centrelink (the specialist service delivery arm of the federal government department called Services Australia). Early digitisation of social security benefits administration not only resulted in considerable efficiency gains but provided the guide-rails that protected against the more egregious errors or decline in decision-making quality as staffing was drastically reduced in scale and shed higher levels skills and experience. Automation as such has not been the issue; the issue is a more recent one of a breakneck rush into a ‘digital first future’Footnote 3 and the abysmal failure of governance, design, ethics, and legal rectitude associated with the $1.8 billion robodebt catastrophe.Footnote 4 As Murphy J observed in his reasons approving the class action settlement, this was a ‘shameful chapter in the administration of the Commonwealth social security system and a massive failure of public administration [which] should have been obvious to the senior public servants charged with overseeing the Robodebt system and to the responsible Minister at different points’; a verdict echoed by the Royal Commissioner in her July 2023 Report.Footnote 5
ADM is only a technology. Like all new technologies, there are extremes of dystopian and utopian evaluative tropes, though a mature assessment often involves a more ambiguous middle ground.Footnote 6 Like the history of other new technological challenges to law, the answers may call for innovative new approaches, rather than extension of existing remedies. Robodebt was ultimately brought to heel by judicial review and class actions, but the much vaunted ‘new administrative law’ machinery of the 1970sFootnote 7 was seriously exposed. Merits review failed because government ‘gamed it’Footnote 8 while the other accountability mechanisms proved toothless.Footnote 9 So radical new thinking is called for.Footnote 10 AI for its part ranges in form from computational aids (or automation) to neural network ‘machine learning’ systems. Even agreed taxonomies of AI are still in development, including recently by the Organisation for Economic Co-operation and Development (OECD), with its four-fold schema of context; data and input; AI model; and task and output.Footnote 11
The focus of this chapter on social security and social services is apt, because Services Australia (as the former Department of Human Services is now called) was envisaged by the Digital Transformation Agency (‘DTA’ formerly ‘Office’) as ‘the first department to roll out intelligent technologies and provide new platforms to citizenry, in accordance with the then DTA’s roadmap for later adoption by other agencies’.Footnote 12 The focus of this chapter is on the associated risk of digital transformation in the social services, of three main forms. First, the risk due to the heightened vulnerabilities of clients of social services.Footnote 13 Second, the risk from inadequate design, consultation, and monitoring of ADM initiatives in the social services.Footnote 14 And finally, the heightened risk associated with particular ADM technologies.
The next section of the chapter (Section 5.2) reviews selected ADM/AI examples in social services in Australia and elsewhere. To draw out differences in levels of risk of various initiatives it takes as a loose organising principle Henman’sFootnote 15 observation that the risks and pitfalls of AI increase along a progression – lowest where it involves recognising ‘patterns’, higher where individuals are ‘sorted’ into categories, and highest where AI is used to make ‘predictions’. Section 5.3 discusses the harm inflicted on vulnerable clients of social services when ADM and AI risks are inadequately appreciated, and some options for better regulation and accountability. It questions both the capacity of traditional judicial and administrative machinery in holding AI to account, and the relevance and durability of those ‘values’ in the face of the transformational power of this technology to subordinate and remake law and social policy to instead reflect AI values and processes.
Restoration of trust in government is advanced in a short conclusion (Section 5.4) as being foundational to risk management in the social services. Trust is at the heart of the argument made for greater caution, more extensive co-design, and enhanced regulatory oversight of ADM in the social services.
5.2 Issues Posed by Automation and ADM
Three issues in particular stand out for social services in Australia. First, the comprehensibility or otherwise of the system for citizens engaging with it. Second, the compatibility or otherwise of ADM in case management. Finally, the risks and benefits of ‘predictive’ ADM in the social services.
5.2.1 Comprehensibility Issues
5.2.1.1 Early Centrelink Adoption of Digitisation and Decision Aids
Prior to robodebt, Centrelink clients concerns mainly centred on intelligibility of digitised social security records and communications, and the ability to understand automation of rate calculations or scoring of eligibility tools. The ADEX and MultiCal systems for debt calculations generate difficult-to-comprehend and acronym-laden print-outs of the arithmetic. This is because the measures were designed for convenience of internal inputting of data rather than ease of consumer comprehension.
The combination of deeply unintelligible consumer documentation and time-poor administrators often leaves too little time to detect less obvious keying or other errors. Internal review officer reconsiderations instead often focus on very basic sources of error such as couple status.Footnote 16 While external merits tribunal members do have the skills and expertise to penetrate the fogFootnote 17 this only rectifies a very small proportion of such errors (only 0.05% in the case of robodebts), and only for those with the social capital or resources to pursue their concern.
Lack of transparency of communications with run-of-the-mill social security clients remains problematic for want of investment in provision of the ‘public facing’ front-end interfaces (or correspondence templates) to convert an almost 100 per cent digital environment into understandable information for the public. Instead, new investment was initially in pilots to classify and file supporting documents for claims processing.Footnote 18 Only in recent years were expressions of interest sought for general customer experience upgrades of the MyGov portal,Footnote 19 reinforced by allocation of $200 million in the 2021–2022 budget for enhancements to provide a ‘simpler and more tailored experience for Australians based on their preferences and interactions’, but also including virtual assistants or chatbots.Footnote 20
Comprehensibility of debt calculations and other routine high incidence transactions to ordinary citizens surely should be the first reform priority. Transparency to citizens certainly hinges on it. Availability of accurate information to recipients of ADM-based welfare is fundamental to individual due process. This was demonstrated by the contrast between Australia’s failure to explain adequately the basis of yearly income variations under its unlawful ‘robodebt’ calculations, compared to the way case officers in the Swedish student welfare program provided explanations and an immediate opportunity to rectify inaccurate information.Footnote 21 Even review bodies such as the Administrative Appeals Tribunal (AAT) would benefit from comprehensibility of the basis of decisions. It would benefit from time freed up to concentrate on substantive issues, due to no longer having to pick their way through the morass of computer print-outs and multiple acronyms simply to create an accessible narrative of issues in dispute.Footnote 22
5.2.2 ADM Case Management Constraints
5.2.2.1 The (Aborted) NDIS Chatbot
The NDIS is seen as a pathbreaker for digitisation in disability services.Footnote 23 But the National Disability Insurance Authority (NDIA) was obliged to abort roll-out of its sophisticated chatbot, called Nadia.
Nadia was designed to assume responsibility for aspects of client interaction and case management. The chatbot was built as a machine learning cognitive computing interface, involving ‘data mining and pattern recognition to interact with humans by means of natural language processing’.Footnote 24 It was to have an ability to read and adjust to emotions being conveyed, including by lightening the interaction such as by referencing information about a person’s favourite sporting team. However, it did not proceed beyond piloting. As a machine learning system it needed ongoing access to a large training set of actual NDIS clients to develop and refine its accuracy. Rolling it out was correctly assessed as carrying too great ‘a potential risk, as one incorrect decision may disrupt a person’s ability to live a normal life’.Footnote 25
This risk of error is a serious one, not only for the person affected by it, but also to the confidence of people in public administration. Given the sophistication required of ‘human’ chatbots, it presently must be doubted whether a sufficient standard of performance and avoidance of risk can be attained for vulnerable social security or disability clients. As Park and HumphreyFootnote 26 suggest, that ability to give human-like cues to end users means that the chatbot ‘need[s] to be versatile and adaptable to various conditions, including language, personality, communication style and limits to physical and mental capacities’. This inability of ADM to bridge the ‘empathy gap’ is why it is so strongly argued that such administrative tasks should remain in the hands of human administrators.Footnote 27 Even smartphone digital reporting proved highly problematic for vulnerable income security clients such as young single parents under (now abolished) ParentsNext.Footnote 28 So, it was surely hoping too much to expect better outcomes in the much more challenging NDIS case management environment.
Such issues are not confined to Australia or to case management software of course. Ontario’s ‘audit trail’ welfare management software, deployed to curb a perceived problem of over-generosity, was found to have ‘decentred’ or displaced caseworkers from their previous role as authoritative legal decision-makers.Footnote 29 The caseworkers responded by engaging in complicated work-arounds to regain much of their former professional discretion. As Raso concluded, ‘[s]oftware that requires individuals to fit into pre-set menu options may never be sophisticated enough to deliver complex social benefits to a population as diverse as [Ontario’s welfare] recipients’.Footnote 30
A US federal requirement to automate verification of Medicaid remuneration of disability caregivers provides yet another example. The state of Arkansas adopted an inflexibly designed and user-unfriendly service app (with optional geo-location monitoring). This proved especially problematic for clients who were receiving ‘self-directed’ care. Care workers were unable to step outside the property boundaries on an errand or to accompany the person without triggering a ‘breach’ of the service being provided. Unlike Virginia, Arkansas had neglected to take advantage of the ability to exempt self-care, or remove problematic optional elements.Footnote 31
5.2.2.2 NDIA’s Aborted ADM Assessment and Planning Reforms
In 2021 public attention was drawn to an NDIA proposal to replace caseworker evaluations by objective rating ‘scores’ when assessing eligibility for the NDIS, and to also serve as a basis for providing indicative packages of funding support. This was shelved on 9 July 2021,Footnote 32 at least in that form.Footnote 33 The measure was designed to address inequities around access and size of packages. The stated policy objective was to improve equity of access between different disability groups and between those with and those without access to a good portfolio of recent medical reports, as well as reduce staffing overheads and processing time.Footnote 34 Subjective assessments of applicant-provided medical reports were to have been replaced by objective ‘scores’ from a suite of functional incapacity ‘tools’. Rating scores were designed not only to improve consistency of NDIS access decisions, but also generate one of 400 personas/presumptive budgets.Footnote 35
The rating tool and eligibility leg of this reform was not true ADM. That aspect mirrored the historical reform trajectory for Disability Support Pension (DSP) and Carer Allowance/Payments (CA/CP). Originally eligibility for DSP (then called Invalid Pension, IP) was based on showing that an applicant experienced an actual or real life 85 per cent ‘incapacity for work’.Footnote 36 In the 1990s this was transformed from an enquiry about the real human applicant to becoming an abstraction – assessing the theoretical ability or not of people with that class of functional impairment to be able to perform any job anywhere in the country – and requiring minimum scores under impairment tables rating functional impairment (leaving extremely narrow fields/issues for subjective classification of severity). These and associated changes significantly reduced the numbers found eligible for these payments.Footnote 37 Similar changes were made for CA and CP payments. The proposed NDIS assessment tools, distilled from a suite of existing measures and administered by independent assessors (as for DSP), followed the disability payment reform pathway. The risks here were twofold. First that the tool would not adequately reflect the legislative test; second, that the scoring basis would not be transparent or meaningful to clients of the NDIS and their family and advisers.Footnote 38
The reform did however have a genuine ADM component in its proposed case planning function. The assessment tool was intended not only to determine eligibility for NDIS access but also to then generate one of 400 different ‘template’ indicative funding packages. This leg of the proposed reform was criticised as robo-planning which would result in lower rates of eligibility, smaller and less appropriate packages of support, and loss of individualisation (including loss of personal knowledge reflected in medical reports no longer to be part of the assessment) along with a substantial reduction of human engagement with case planners.Footnote 39
This was a true deployment of ADM in social services, highlighting Henman’s middle range risks around ADM categorisation of citizens, as well as risks from devaluing professional casework skills, as further elaborated in the next section.
5.2.3 Predictive ADM
Risks associated with ADM are arguably most evident when it is predictive in character.Footnote 40 This is illustrated by the role predictive tools play in determining the level and adequacy of employment services for the unemployed in Australia,Footnote 41 and the way compliance with the allocated program of assistance to gain work is tied to retention of eligibility for or the rate of unemployment payments.Footnote 42 The accuracy or otherwise of the prediction is key to both experiences.
5.2.3.1 Predictive ADM Tools in Employment Services and Social Security
Predictive ADM tools to identify those at greatest risk of long-term unemployment operate by allocating people to homogenous bands according to predictors unemployment duration (statistical profiling). These statistical profiling predictions are much more accurate than random allocation, but still misclassify some individuals. They also fail to identify or account for causal reasons for membership of risk bands.Footnote 43 Human assessments are also liable to misclassify, but professional caseworkers lay claim to richer understandings of causal pathways, which may or may not be borne out in practice.
Predictive tools are constructed in two main ways. As an early pioneer, Australia’s Job Seeker Classification Instrument (JSCI) was developed and subsequently adjusted using logistic regression.Footnote 44 Other international designs are constructed using machine learning which interrogates very large data sets to achieve higher accuracy of prediction, as in the Flemish tool.Footnote 45 As with all ADM predictive tools, reflection and reinforcement of bias is an issue: ‘[b]y definition, high accuracy models trained on historical data to satisfy a bias preserving metric will often replicate the bias present in their training data’.Footnote 46
While there is a large literature on the merits or otherwise of possible solutions for unacceptable bias and discrimination in AI, statistical profiling poses its own quite nuanced ethical challenges. Membership of a racial minority is associated with longer durations of unemployment for instance. But the contribution of racial minority to allocation to a statistical profile band can be either bitter or sweet. Sweet if placement in that band opens a door to voluntarily obtaining access to employment services and training designed to counteract that disadvantage (positive discrimination). Bitter if band placement leads to involuntary imposition of requirements to participate in potentially punitive victim blaming programs such as work for the dole. This risk dilemma is real. Thus, a study of the Flemish instrument found that jobseekers not born in the country were 2.6 times more likely to wrongly be classified as at high risk of long-term unemployment.Footnote 47
Nor is the issue confined to the more obvious variables. It arises even with superficially more benign correlations, such as the disadvantage actually suffered from having a long duration of employment for a single employer prior to becoming unemployed. Its inclusion in the predictive algorithm is more acceptable if this results in accessing programs to help counter the disadvantage, such as projecting the human capital benefits of past loyalty to the previous employer compared to likely future sunk costs associated with other applicants with more varied employment histories. But its inclusion is ethically more problematic if it only exposes the person to greater likelihood of incurring income support or other sanctions. Other examples of predictive legal analytics also show that the normative aspect of the law is often supplanted by causal inference drawn from a data set, which may or may not reflect the relevant legal norms.Footnote 48
To a considerable degree, the contribution of statistical profiling hinges on the way it is used. The lack of engagement with causal factors and the arbitrariness or bias of some variables constituting the algorithm is magnified where caseworkers are left with little scope for overriding the initial band allocation. This is the case with Australia’s JSCI, a risk compounded by lack of transparency of the algorithm’s methodology.Footnote 49 These risks are lessened in employment services systems which leave caseworkers in ultimate control, drawing on assistance from a profiling tool. That is the way the tools are used in Germany, Switzerland, Greece, and Slovenia.Footnote 50
This analysis of the risks associated with predictive tools in employment services is consistent with findings from other areas of law. For example decisions grounded in pre-established facts, such as aspects of aggravating and mitigating criminal sentencing considerations may be more amenable to computation, overcoming perceived deficiencies of instinctive synthesis sentencing law.Footnote 51 Distinguishing between administrative decisions as either rule-based or discretionary may prove also useful, because ADM applied to discretionary decisions may result in a failure to lawfully exercise discretion.Footnote 52 Discretionary tasks high in complexity and uncertainty arguably fare better under human supervision and responsibility, such as by a caseworker.Footnote 53
For its part, Australia mitigates the risk of JSCI predictive errors in two ways. First, an employment services assessment may be conducted by a contracted health or allied health professional in certain circumstances. This is possible where it is shown that there are special barriers to employment, a significant change of circumstances or other indications of barriers to employment participation.Footnote 54 The weakness of this is that it occurs only in exceptional circumstances, rather than as part of routine caseworker fine tuning of the overly crude and harsh streaming recommendations resulting from application of the JSCI. So it is essentially confined to operating as a vulnerability modifier.
Second, a new payment system has been introduced to break the overly rigid nexus between the JSCI determined stream and the level of remuneration paid to employment providers for a person in that stream. The old rigid payment regime was exposed as perverse both by academic researchFootnote 55 and the government’s own McPhee Report.Footnote 56 Rather than encourage investment in assisting people with more complex needs it perversely encouraged parking or neglect of such cases in order to concentrate on obtaining greater rewards from assisting those needing little if any help to return to work. The New Enhanced Services model decouples service levels and rates of payment to providers for achieved outcomes, ‘which provides some additional flexibility, so a participant with a High JSCI but no non-vocational barriers can be serviced in Tier 1 but still attract the higher outcome payments’.Footnote 57
The obvious question is why Australia’s employment services structure and JSCI instrument survived with so little refinement to its fundamentals for nearly two decades after risks were first raised.Footnote 58 Davidson argues convincingly that path-dependence and cheapness were two main reasons why it took until the McPhee Report to effect systemic change.Footnote 59 It is suggested here that another part of the answer lies in a lack of appetite for and difficulty of realising processes of co-design with welfare clients and stakeholders. Certainly, recent experience with co-design in Denmark demonstrates that it is possible to construct a more sophisticated and balanced system which avoids the worst of the adverse effects of statistical profiling and welfare conditionality.Footnote 60 The case for co-production with users is not new to Australian public administration.Footnote 61 Co-design is particularly favoured where risks of discrimination and exclusion are present.Footnote 62
In theory co-design of employment services should also be possible in Australia, but the history of top-down and very harsh ‘work-first’ welfare-to-work policiesFootnote 63 suggests that its realisation is unlikely.
5.3 Responding to the ‘Power’ of AI
[I]n our current digital society, there are three phenomena that simultaneously connect and disconnect citizens from government and impede millions of individuals from exercising their rights on equal terms: bureaucracy, technology, and power asymmetries.Footnote 64
ADM and AI technology in the social services carries a potential both to harm participants as well as to radically transform services by compressing the range of social policy options considered in program design much in the same way these technologies can change the bases of legal accountability (Section 5.3.2).
The power of poorly conceived ADM and AI to inflict unacceptable harm on vulnerable citizens reliant on social services is well established. The challenge here lies in finding ways of mitigating that risk, as discussed below.
5.3.1 The Vulnerability Challenge in Social Services
Common to assessing all of these examples of automation and artificial intelligence in welfare is the impact on vulnerable clients. That vulnerability cannot be overstated. As Murphy J wrote in approving the robodebt class action settlement in Prygodicz:
It is fundamental that before the state asserts that its citizens have a legal obligation to pay a debt to it, and before it recovers those debts, the debts have a proper basis in law. The group of Australians who, from time to time, find themselves in need of support through the provision of social security benefits is broad and includes many who are marginalised or vulnerable and ill-equipped to properly understand or to challenge the basis of the asserted debts so as to protect their own legal rights. Having regard to that, and the profound asymmetry in resources, capacity and information that existed between them and the Commonwealth, it is self-evident that before the Commonwealth raised, demanded and recovered asserted social security debts, it ought to have ensured that it had a proper legal basis to do so. The proceeding revealed that the Commonwealth completely failed in fulfilling that obligation.Footnote 65
The pain and sufferingFootnote 66 from the abysmal failure of governance, ethics, and legal rectitude in the $1.8 billion robodebt catastropheFootnote 67 was ultimately brought to heel by judicial review and class actions. Yet as already mentioned, the much vaunted ‘new administrative law’ remedial machinery of the 1970s was seriously exposed. Merits review failed because government ‘gamed it’ by failing to further appeal over 200 adverse rulings that would have made the issue public.Footnote 68 Other accountability mechanisms also proved toothless.Footnote 69 Holding ADM to account through judicial remedies is rarely viable, though very powerful when it is apt.Footnote 70 Judicial review is costly to mount, gameable, and confined to those risks stemming from clear illegality. Robodebt was a superb but very rare exception to the rule, despite the November 2019 settlement victory in the AmatoFootnote 71 test case action, and the sizeable class action compensation settlement subsequently achieved in Prygodicz.Footnote 72 A test case launched prior to Amato was subject to government litigational gaming. That challenge was halted by the simple step of a very belated exercise of the statutory power to waive the ‘debt’. The same fate could have befallen Amato had the then government been less stubborn in refusing to pay interest on the waived debt.Footnote 73 For its part, the reasons approving the Prygodicz settlement makes clear how remote is the prospect of establishing a government duty of care in negligence, much less establishing proof of breach of any duty of care.Footnote 74
Administrative law judicial or merits review redress predicated on an ‘after-the-event’ interrogation of the process of decision-making or the lawfulness (and merits in the case of tribunal review) of the reasons for decisions is further undermined by the character of ADM and AI decision-making. This is because neither the decision-making processes followed, nor the thinned down/non-existent reasons generated by the ‘new technological analytics’Footnote 75 are sufficiently amenable to traditional doctrine.Footnote 76 For example, bias arising from the data and code underlying ADM together with biases arising from any human deference to automated outputs, pose evidentiary challenges which may not be capable of being satisfied for the purpose of meeting the requirements of the rule against bias in judicial review.Footnote 77 The ability to bend traditional administrative law principles of due process, accountability, and proportionality to remedy the concerns posed by ADM thus appears to be quite limited.Footnote 78
Outranking all of these concerns, however, is that neither merits review nor judicial review is designed to redress systemic concerns as distinct from individual grievances. So radical new thinking is called for,Footnote 79 such as a greater focus on governmentality approaches to accountability.Footnote 80 To understand the gaps in legal and institutional frameworks, the use of ADM systems in administrative settings must be reviewed as a whole – from the procurement of data and design of ADM systems to their deployment.Footnote 81 Systemic grievances are not simply a result of purely ‘mathematical flaws’ in digital systems, as opposed to the product of accountability deficiencies within the bureaucracy and structural injustice.Footnote 82
One possible new direction is through ADM impact statement processes designed to help prevent systemic grievances. An example is Canada’s Directive, modelled on the GDPR and largely mimicking administrative law values.Footnote 83 While this certainly has merit, it is open to critique as paying but lip service to risk prevention because it relies on industry collaboration and thus has potential for industry ‘capture’ or other pressures.Footnote 84 Other alternatives include a mixture of ex ante and ex post oversight in the form of an oversight board within the administrative agency to circumvent the barrier of a costly judicial challenge,Footnote 85 and the crafting of sector-specific legal mechanisms.Footnote 86
There is also theoretical appeal in the more radical idea of turning to a governance frame that incorporates administrative accountability norms as its governance standard. The best known of these are Mashaw’s trinity of bureaucratic rationality, moral judgement, and professional treatment, and Adler’s additions of managerialism, consumerist, and market logics.Footnote 87
However these innovative ideas presently lack remedial purchase. Incorporation of tools such as broadened impact assessments may give these norms and values some operational purchase, but the limitations of impact assessment would still remain.Footnote 88 A research impact framework for AI framed around concepts of public value and social value may hold greater promise.Footnote 89
Self-regulation against industry ethics codes, or those co-authored with regulators, has also proven to be weak reeds. They too are easily ‘subsumed by the business logics inherent in the technology companies that seek to self-impose ethical codes’,Footnote 90 or a form of ‘ethics washing’.Footnote 91 As Croft and van RijswijkFootnote 92 detailed for industry behemoths such as Google, this inability to curb corporate power is because it is systemic. As James and Whelan recently concluded:
Codifying ethical approaches might result in better outcomes, but this still ignores the structural contexts in which AI is implemented. AI inevitably operates within powerful institutional systems, being applied to the ‘problems’ identified by those systems. Digital transformation reinforces and codifies neoliberal agendas, limiting capacities for expression, transparency, negotiation, democratic oversight and contestation … This can be demonstrated by juxtaposing the AI ethics discourse in Australia with how AI has been implemented in social welfare.Footnote 93
The Australian Human Right Commission (AHRC) Report also delivered underwhelming support,Footnote 94 though academic work is continuing to boost the contribution to be made by ethics-based audits.Footnote 95
Consideration of how to mitigate risk of harm to vulnerable recipients of the social services cannot be divorced from meta-level impacts of ADM and AI technology on the character and design of law and social programs, as discussed below.
5.3.2 The Transformational Power of AI to Shape Social Policy and Law
Lawyers and social policy designers are rather accustomed to calling the shots in terms of setting normative and procedural standards of accountability (law) and formulating optimally appropriate social service programs (social policy). Digitisation, however, not only transforms the way individual citizens engage with the state and experience state power at the micro-level, but also transforms the nature of government services and modes of governance. The second of these, the transformation of governance by ADM and AI technologies,Footnote 96 is perhaps better known than the first.
Public law scholars have begun to recognise that it may not simply remain a question of how to tame ADM by rendering it accountable to traditional administrative law standards such as those of transparency, fairness, and merits review, but rather of how to avoid those values being supplanted by ADM’s values and ways of thinking. The concern is that ADM remakes law in its technological image rather than the reverse of making ADM conform to the paradigms of the law.Footnote 97
The same contest between ADM and existing paradigms is evident in other domains of government services. Contemporary advances in design of social services for instance favours ideas such as personalisation, social investment, and holistic rather than fragmented services.Footnote 98 But each of these policy goals is in tension with ADM’s design logic of homogenisation and standardisation.Footnote 99 Personalisation of disability services through case planning meetings and devolution of responsibility for individual budgets to clients, in place of top-down imposition of standard packages of services, is one example of that tension, as recently exemplified in the NDIS.Footnote 100 The mid-2022 roll-out of algorithmic online self-management of employment services (PEPs) to all except complex or more vulnerable clients is anotherFootnote 101 despite introduction of requirement for a digital protection framework under s 159A (7) and (9) of the Social Security Legislation Amendment (Streamlined Participation Requirements and Other Measures) Act 2022.
Initiatives across the health and justice systems, such as ‘social prescribing’ designed to address the contribution of socioeconomic disadvantage to disability and health issues such as by coordinating income support and health services,Footnote 102 or integration of human services and justice systems through justice reinvestment or therapeutic ‘problem-solving’ courtsFootnote 103 are two other settings where the same tension arises. In the case of social prescribing, the rigid ‘quantification’ of eligibility criteria for access to the disability pension, together with strict segregation of social security and health services, compounds the issue. In the second instance, predictive criminal justice risk profiling tools threaten to undermine the central rationale of individualisation and flexibility of justice reinvestment interventions to build capacity and avoid further progression into criminality.Footnote 104
What is able to be built in social policy terms depends in no small part on the available materials from which it is to be constructed. Rule-based materials such as the algorithms and mechanisms of ADM are unsuited to building social programs reliant on the exercise of subjective discretionary choices. Just as the fiscal objective of reducing staff overheads to a minimum led to enactment of rules in place of former discretionary powers in Australian social security law,Footnote 105 government policies such as ‘digital first’ inexorably lead to push back against policies of individualisation and accommodation of complexity. Those program attributes call for expensive professional skills of human caseworkers or the less pricey discretionary judgments of human case administrators. ADM is far less costly than either, so in light of the long reign of neoliberal forms of governance,Footnote 106 it is unsurprising that social protection is being built with increasing amounts of ADM and AI,Footnote 107 and consequently is sculpted more in the image of that technology than of supposedly favoured welfare policies of personalisationFootnote 108 or those of social investment.Footnote 109
There are many possible longer-run manifestations should ADM values and interests gain the upper hand over traditional legal values. One risk is that ADM systems will create subtle behavioural biases in human decision-making,Footnote 110 changing the structural environment of decision-making. For example the facility of ADM to ascertain and process facts may lead to lesser scrutiny of the veracity of these facts than would be the case in human decision-making. Abdicating the establishment of fact and the value-judgements underlying factfinding to ADM substitutes digital authority for human authority.Footnote 111 This raises questions of accountability where human actors develop automation bias as a result of failing to question outputs generated by an automated system.Footnote 112
Other manifestations are more insidious, including entrenchment of an assumption that data-driven decision-making is inherently neutral and objective rather than subjective and contested, or overlooking the contribution of surveillance capitalism discourse around the business practices that procure and commodify citizen data for a profit.Footnote 113 This criticism has been levelled at Nordic governmental digitalisation initiatives. The Danish digital welfare state, for example, has drawn academic scrutiny for an apparently immutable belief that data processing initiatives will create a more socially responsible public sector, overlooking the consequences of extensive data profiling using non-traditional sources such as information from individuals’ social networking profiles. The public sector’s embrace of private sector strategies of controlling consumers through data suggests a propensity for rule of law breaches through data maximisation, invasive surveillance, and eventual citizen disempowerment.Footnote 114
This is not the place to do other than set down a risk marker about the way ADM and AI may change both the architecture and values of the law as well as of the very policy design of social service programs. That resculpting may be dystopian (less accommodating of human difference and discretions) or utopian in character (less susceptible to chance variability and irrelevant influences known as decisional ‘noise’). The reciprocal power contest between the power of AI technology on the one hand and law/social policy on the other is however a real and present concern, as the NDIS example demonstrated.
5.4 Towards AI Trust and Empathy for Ordinary Citizens
Administration of social security payments and the crafting of reasonable and necessary supports under the NDIS are quintessentially examples of how law and government administration impact ‘ordinary’ citizens. As Raso has observed:
As public law scholars, we must evaluate how legality or governance functions within administrative institutions in everyday and effectively final decisions. As we develop theories of how it ought to function, we must interrogate how decision making is functioning.Footnote 115
It is suggested here that the principal impression to be drawn from this review of Australia’s recent experience of rolling out ADM in Raso’s ‘everyday’ domain of the ordinary citizen, is one of failure of government administration. It is argued that the history so far of Australian automation of welfare – most egregiously the robodebt debacle – demonstrates both a lack of government understanding that the old ways of policy-making are no longer appropriate, and that public trust in government has seriously eroded. Automation of welfare in Australia has not only imposed considerable harm on the vulnerable,Footnote 116 but has destroyed an essential trust relationship between citizens and government.Footnote 117
Restoring trust is critical. Trust is one of the five overarching themes identified for consultation in February 2022 by the PM&C’s Digital Technology Taskforce and in the AHRC’s final report.Footnote 118 Restoration of trust in the NDIS was also one of the main themes of the recent Joint Parliamentary Committee report on independent assessments.Footnote 119 Consequently, if future automation is to retain fidelity to values of transparency, quality, and user interests, it is imperative that government engage creatively with the welfare community to develop the required innovative new procedures. A commitment to genuine co-design and collaborative fine-tuning of automation initiatives should be a non-negotiable first step, as stressed for the NDIS.Footnote 120 Ensuring empathy of government/citizen dealings is another.
Emphasising in Chapter 9 about the potential for the automated state, wisely crafted and monitored to realise administrative law values, Cary Coglianese writes that
[i]n an increasingly automated state, administrative law will need to find ways to encourage agencies to ensure that members of the public will continue to have opportunities to engage with humans, express their voices, and receive acknowledgment of their predicaments. The automated state will, in short, also need to be an empathic state.
He warns that ‘[t]o build public trust in an automated state, government authorities will need to ensure that members of the public still feel a human connection’. This calls for a creative new administrative vision able to honour human connection, because ‘[i]t is that human quality of empathy that should lead the administrative law of procedural due process to move beyond just its current emphasis on reducing errors and lowering costs’. That vision must also be one that overcomes exclusion of the marginalised and vulnerable.Footnote 121 Another contribution to building trust is to be more critical of the push for automated administration in the first place. An American ‘crisis of legitimacy’ in administrative agencies has been attributed to the way uncritical adoption of ADM leads to the loss of the very attributes that justify their existence, such as individualisation.Footnote 122 Framing the NDIS independent assessor episode in this way demonstrated a similar potential deterioration of citizen trust and legitimacy.
Building trust and empathy in social service administration and program design must fully embrace not only the mainstream human condition but also the ‘outliers’ that AI standardisation excludes.Footnote 123 At the program design level this at a minimum calls for rejection of any AI or ADM that removes or restricts inclusion of otherwise appropriate elements of personalisation, subjective human judgement, or exercise of discretion relevant to advancing agreed social policy goals. This extends to AI outside the program itself, including being sensitive to indirect exclusion from discriminatory impacts of poorly designed technological tools such as smartphones.Footnote 124
Half a century ago in the pre-ADM 1970s, the ‘new administrative law’ of merits review and oversight bodies was touted as the way to cultivate citizens’ trust in government administration and provide access to administrative justice for the ordinary citizen, though even then the shortfall of preventive avenues was recognised.Footnote 125 Overcoming the ability of government to game first-tier AAT by keeping adverse rulings secret, and arming it with ways of raising systemic issues (such as a form of ‘administrative class action’) might go a small way to restoring trust and access to justice. But much more creative thinking and work is still to be done at the level of dealing with individual grievances as well.Footnote 126
In short, this chapter suggests that the conversation about the ADM implications for the socioeconomic rights of marginalised citizens in the social services has barely begun. Few remedies and answers currently exist either for program design or for individual welfare administration.
Open access