Policy Significance Statement

Data Protection Impact Assessments (DPIAs) are a relatively new development in the governance of data processing. The recommendations in this article are a tool for policy-makers and civil servants developing data processing systems for government and undertaking DPIAs, to assist government to meet rule of law standards when implementing new data processing systems. The article also adds to analysis of the governance of data processing systems for all stakeholders, including civil society.

Overview

The focus of this article is on data processing by government in the United Kingdom and rule of law principles because the law defines what government can do—the exercise of public power by government must be within the scope defined by law. Whereas for the private sector, it could be said that law defines what private actors may not do; for the public sector, there is no authority or lawful ability to act without a basis in law. Government use of data processing when exercising public power can give rise to concerns in rule of law terms, and risk failing to comply with existing principles and standards for public power. The aim of this article is not to argue against the use of automated data processing systems by government, but to highlight rule of law principles and standards that such systems should meet. This article explores the potential to use Data Protection Impact Assessments (DPIAs) as a mechanism through which to put into operation abstract rule of law principles such as equality before the law, to help ensure that government data processing systems meet rule of law principles and standards that apply to all government decision-making and activities.

The EU General Data Protection Regulation (GDPR) introduced a new requirement in Article 35 for DPIAs to be conducted when data processing is likely to pose a “high risk to the rights and freedoms of natural persons”. The UK Information Commissioner’s Office (ICO), the regulator responsible for information rights, explains that DPIAs are “a process designed to help you systematically analyse, identify and minimise the data protection risks of a project or plan” and are “a key part of … accountability obligations under the GDPR.”Footnote ¹ The ICO previously encouraged the use of Privacy Impact Assessments as a means of ensuring compliance with the Data Protection Act 1998, but those assessments were limited in scope to privacy. By contrast, DPIAs have broader scope encompassing all rights and freedoms.

The analysis in this article is framed with reference to rule of law principles. That public power must be exercised in accordance with the law is one of the rule of law principles identified by the former Lord Chief Justice Lord Bingham (Bingham, Reference Bingham2010). There are different views on the scope and content of the rule of law, but for the purposes of the UK context, Lord Bingham’s approach has significant authority. The eight rule of law principles identified by Lord Bingham can be summarized as:

These principles are essential for a fair and just society, and apply to government activities regardless of whether those activities are undertaken by a human or automated data processing.

The rule of law analysis in this article has grown out of the author’s work at The Legal Education Foundation and the Bingham Centre for the Rule of Law.Footnote ² The Legal Education Foundation aims to help people understand and use the law, and the implementation of these rule of law principles is a necessary precondition to that mission. When the law is implemented by government using data processing systems, there is a risk of reduced transparency and accountability in the operation of the law. For people to understand and use the law, government systems implementing the law must be transparent so that people can understand and navigate those government processes. Organizations to which The Foundation has made grants are increasingly addressing preventable legal problems that are due to government data processing systems causing unlawful wrong government decisions. There is thus a need to look for systematic ways in which these data processing systems could be improved in rule of law terms.

The focus of this article is law and governance of data in the United Kingdom in relation to government use of data processing in the exercise of public power. This article sets out two case studies of automated data processing used by government in the exercise of public power, and the rule of law concerns that arise in relation to those systems. Applying rule of law principles to these case studies provides a sketch of the issues and concerns that this article’s proposals for DPIAs seek to address. The following section of the article outlines the law on DPIAs in the GDPR, and provides international context in terms of proposals for impact assessments of data processing in other jurisdictions. Although DPIAs are also required under EU Directive 2016/680 concerning protection of personal data in the context of criminal law enforcement, the scope of this article is limited to civil and administrative law and does not include DPIAs in the criminal law context.

The article then explores the possibility of using DPIAs to improve the compliance of data processing systems used by government with rule of law principles. These sections offer some guidance on how DPIAs could be used to strengthen the governance of data processing by government in rule of law terms, drawing on comparative analysis of other governance schemes to identify specific recommendations for DPIAs. The main areas analyzed to develop the governance recommendations for DPIAs are human rights, administrative law, and environmental law. Broadly speaking, the different aspects of governance are categorized in terms of process versus substance, although that is a false distinction as failure to undertake steps in process such as public participation will affect substance, and failure to ensure that the substance informs the process means that the process will be meaningless. Finally, the article notes the shortcomings of current governance and standards for data processing by government to achieve the principle of transparency in reality.

In light of this legal analysis and comparisons to other governance schemes, this article identifies a number of recommendations for good practice in the substance and process of DPIAs by government in the United Kingdom. The article generally refers to “government” without distinguishing among UK central government, devolved government, or local government—the conclusions and recommendations are relevant to all of these levels of government.

Case Study 1: Housing Benefit and Council Tax Benefit Applications

Housing Benefits and Council Tax Benefits are welfare benefits that are administered by local authorities rather than the central government department responsible for welfare, which is the UK’s Department for Work and Pensions (DWP). Since 2012, the DWP has allowed local authorities to voluntarily adopt the use of so-called “Risk-Based Verification” systems as part of the application processes for these benefits. As explained in a DWP Local Authority Insight Survey:

The pieces of legislation that establish and govern HB are the Social Security Contributions and Benefits Act 1992, Housing Benefit Regulations 2006, and Housing Benefit (persons who have attained the qualifying age for state pension credit) Regulations 2006. CTB was replaced by Council Tax Reduction in 2013, but for consistency, this article refers to CTB because the DWP policy documents refer to CTB. The Welfare Reform Act 2012 repealed CTB, and the Local Government Finance Act 2012 provides for Council Tax Reduction, which is further governed in England by the Council Tax Reduction Schemes (Prescribed Requirements) (England) Regulations 2012 and Council Tax Reduction Schemes (Prescribed Requirements and Default Scheme) (England) Regulations 2012.

DWP gives the following examples of the kind of risk ratings or categories that RBV assigns in the circular setting out guidance on RBV of HB/CTB Claims:

By way of illustration as to how this RBV works in practice, the following section extracted from a table (not the full table) in the Chichester District Council’s Risk-Based Verification Policy 2017 sets out the evidence requirements based on the category of risk assigned to an applicant (Chichester District Council, 2017):

The DWP also explains (citations omitted):

There appears to be no information on what data points these RBV systems use to make their assessments of risk.

Furthermore, the DWP guidance on governance leaves rule of law questions unresolved. Local authorities are required to produce an RBV Policy and a baseline against which to assess the impact of an RBV, and to undertake monthly monitoring of RBV performance. However, rule of law questions remain, for example, there are no criteria for monitoring impact in relation to protected characteristics under equality law. There are no requirements for transparency in the governance arrangements, in fact, DWP advises that the RBV Policy should not be made public “due to the sensitivity of its contents.” The underlying concern is that information on the system would allow applicants to game the system because they would know the risk profiles (DWP, 2011). However, it is not clear how the proper exercise of public power can be verified when people are not told that they have been subject to an RBV system nor the basis for its assessment of them which leads to different process thresholds for applicants. Furthermore, there would not be the same individual fraud risk in relation to the disclosure of aggregate baseline information, nor the aggregate findings of performance monitoring.

Case Study 2: Settled Status Application Process for EEA Nationals

The settled status scheme has been established by the UK Home Office in the context of Brexit to regularize the immigration status of European Economic Area (EEA) nationals and their families living in the United Kingdom. Difficulties with the identity verification aspect of the application process have been relatively high profile because it relies on mobile phone technology, but did not work on Apple devices for the first 6 months of the scheme. The automated checks of welfare and tax data to verify residence have received less attention, but are a significant automated part of the application process, albeit the decision-making process is partially, not fully, automated.

Part of the settled status scheme involves sharing data between government departments and algorithmic assessment of those data. Under the UK’s Immigration Rules Appendix EU, EEA nationals who have lived in the United Kingdom for at least 5 years are entitled to “settled status.” Those who have lived in the United Kingdom for less than 5 years are entitled to “pre-settled status,” and will need to apply for settled status when they reach the 5-year threshold. The Home Office application process uses automated data processing to analyze data from the DWP and Her Majesty’s Revenue and Customs (HMRC) (the UK central government department responsible for taxation) to verify how long applicants have been in the United Kingdom. Where the application process finds a “partial match,” the applicant is granted pre-settled status unless they challenge that decision.

The most recent statistics on the EU Settlement Scheme (2019) published by the Home Office on November 14, 2019 stated that 2,450,500 applications had been received up to the end of October, of which 1,925,300 had been concluded. Of the applications that had been concluded, 40% were granted pre-settled status (approximately 770,120) and 60% were granted settled status (approximately 1,155,180) (Home Office, 2019g).

According to evidence given by then Immigration Minister Caroline Nokes on July 16, 2019, up to May 31, 2019 when 788,200 applications had been received (Home Office, 2019d), there had been 253 administrative review applications challenging outcomes of pre-settled instead of settled status that had been received and processed since the review mechanism was established in November 2018. Administrative review is carried out by the Home Office when an applicant challenges the initial decision. Of the 253 administrative reviews challenging a grant of pre-settled status:

1. 1. 22 of these grants of pre-settled status were upheld following the administrative review; and

2. 2. 231 were overturned and the applicants were instead granted settled status following administrative review. “In those cases, the applicant had generally accepted a grant of pre-settled status when making their application and then given additional evidence of their eligibility for settled status alongside their application for administrative review.”Footnote ³

It is notable that around 91% of the administrative reviews were successful and the initial decision of the Home Office overturned. It is possible that there could be many other applicants who have been granted pre-settled status unlawfully when they were entitled to settled status, but who did not apply for administrative review.

As outlined above, more than one-third of the decisions made in the scheme so far have granted pre-settled status, but there were no measures in the system as initially designed to check whether those applicants had been granted the correct status. It should now be possible to approximately monitor this across the scheme because the government has introduced a question on whether applicants have been in the United Kingdom for less or more than 5 years, although the scheme did not originally include this.Footnote ⁴ This change to the design of the application system would be an improvement for monitoring purposes, but given the figures from administrative reviews there remains a concern as to how the Home Office will avoid and mitigate the risk of unlawful initial decisions that wrongly grant pre-settled status rather than settled status. Furthermore, the monthly monitoring statistics do not include data from this new question, which undermines the ability of external stakeholders to monitor the scheme.

There was a proposal for manual checks to mitigate the risk of errors in the automated checks resulting in applicants being wrongly granted pre-settled status instead of settled status, to which the Minister responded:

Many problems with the automated checks have been reported by applicants, including employed and self-employed applicants, and there are particular concerns for applicants who are low-income or vulnerable (Dunt, Reference Dunt2019). Coram Children’s Legal Centre, an NGO with specialist expert knowledge of children’s rights and immigration, are concerned that:

Although the Home Office has consulted with user groups, there has been a lack of transparency and information on the data processing used in the settled status scheme. The memoranda of understanding for data sharing between the Home Office and DWP, and the Home Office and HMRC were published at the end of March 2019 in response to a proposed amendment from Stuart McDonald MP.Footnote ⁶ DPIA(s) for the settled status application process have not been published, nor has an Equality Impact Assessment.Footnote ⁷

Rule of Law Analysis of These Case Studies

The case studies illustrate some of the rule of law concerns that arise when government uses data processing systems for decision-making. The following rule of law analysis applies the rule of law principles articulated by Lord Bingham (set out in the opening of this article) to the case studies.

It is difficult to be confident that public officers have exercised their powers in accordance with the law when their decision-making is informed by RBV or automated checks of data for residency. This difficulty arises because there is no publicly available analysis of how the RBV systems implement the law nor clear non-technical explanation of how the systems operate. The results of automated checks in the settled status scheme do not inform applicants which categories of data were used in the checks (e.g., which DWP benefits were taken into account in assessing residence). Furthermore, the administrative review results discussed above demonstrate that many applicants entitled to settled status accepted pre-settled status, which the results from the automated checks had wrongly assigned them. This suggests that the application process did not provide those applicants with sufficient information to make an informed and correct decision on whether to accept that status. The lack of information on these data processing systems and how they work reduces the accessibility of the law. Law is made less accessible by the use of opaque systems to implement the law.

Furthermore, the design and operation of data processing systems could produce systematic indirect discrimination. The basis for sorting the “risk categories” by RBV is unknown and could be indirectly discriminatory if it is based on data points that act as proxies for characteristics such as race. Similarly, the design of the automated checks of the settled status scheme means that vulnerable groups such as children and recipients of child tax credit (predominantly women) will have a more difficult process to navigate to secure their immigration status. Can such unequal processes be said to be consistent with the principle of equality before the law?

The consequences of the use of the systems in the case studies produce further rule of law concerns in terms of impact on human rights and access to justice. There are many legal instruments that give rise to the UK’s human rights obligations under international law, including the European Convention on Human Rights (ECHR), International Covenant on Civil and Political Rights (ICCPR), and International Covenant on Economic, Social, and Cultural Rights (ICESCR). Where people are wrongly granted pre-settled status, then there is a risk that they will be denied settled status at the end of their pre-settled status period, which could affect many of their rights such as the right to family life (Article 10 ICESCR; Articles 17 and 23 ICCPR; Article 8 ECHR) and right to work (Article 6 ICESCR). Similarly, the delay or denial of people’s applications for HB or CTB because of the additional checks required by RBV classifying them as “high risk” could affect their right to an adequate standard of living (Article 11 ICESCR). Wrongly denying people such rights could be in breach of the UK’s international human rights obligations. Moreover, the opacity of the systems result in difficulties for people to understand the decisions made about them, which in turn could affect their right to access to justice (Zalnieriute et al., Reference Zalnieriute, Bennett Moses and Williams2019). It will be more difficult for people to challenge an unlawful decision if they are not informed of the basis for that decision.

Given the rule of law concerns illustrated by these case studies, there is a need for laws and processes that could be used to improve the fidelity of government data processing systems to rule of law principles.

When considering the legal governance of automated decision-making, the limits of the provisions on automated decision-making in GDPR Article 22 should be noted (Cobbe, Reference Cobbe2018). Article 22 provides that “The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her.” Both of the case studies probably fall outside of the scope of Article 22, because the RBV for HB or CTB claims would not have legal effects, and the automated checks for settled status applications are followed by an option to upload documents and human decision-makers are involved in the process. In order for human involvement in decision-making to be meaningful, the automated data process must be interpretable by the human decision-maker as discussed further below (Binns and Gallo, Reference Binns and Gallo2019).

DPIAs provide a mechanism to govern data processing used in government decision-making regardless of whether such data processing falls within the scope of Article 22. Article 22 provides only a partial and limited legal framework to regulate automated decision-making that is ill-suited to address the kinds of partially automated decision-making processes discussed in the case studies. The rule of law questions and concerns raised in relation to the data processing in the case studies are not answered by finding that they fall outside of the Article 22 prohibition on automated decision-making. Government use of automated data processing can be lawful in relation to Article 22, and yet fail to comply with the rule of law principles articulated by Lord Bingham.

Data Protection Impact Assessments

Rather than proposing a new set of governance mechanisms to address the rule of law concerns identified above, the following sections focus on DPIAs because they are the current governance process for many data processing systems, including partially and fully automated government decision-making systems.

Article 35 of the GDPR provides (emphasis added):

The scope of impact to be considered in DPIAs under Article 35 encompasses all rights and freedoms, and is not limited to privacy. The ICO’s Code of Practice for Privacy Impact Assessments focused on identifying and managing privacy risks in compliance with the Data Protection Act 1998, but not other human rights (ICO, 2014). As Janssen has argued, the references to “rights and freedoms” in Article 35’s provisions for DPIAs should be read in the context of EU law as referring to the rights set out in the European Union Charter of Fundamental Rights (Janssen, Reference Janssen2018). The Charter includes freedom of expression (Article 11), right to education (Article 14), right to engage in work (Article 15), equality before the law and nondiscrimination (Articles 20 and 21), social security (Article 34), and health care (Article 35). This understanding of the scope of Article 35’s reference to “rights and freedoms” is consistent with the views of the Article 29 Data Protection Working Party, an independent body on data protection for the EU (Article 29 Working Party, 2017).Footnote ⁸

There are no GDPR obligations for transparency or public engagement and consultation on DPIAs as such, although Article 35(9) provides that the views of data subjects on the intended processing be sought where appropriate. In the United Kingdom, the ICO has responsibility for monitoring the application of the GDPR as the relevant national independent public authority required under the GDPR. The ICO’s guidance on DPIAs only states that you “should” seek and document the views of individuals, but does not make clear that there is a legal obligation under Article 35(9) to seek the views of data subjects.Footnote ⁹ The enforceability of consultation under Article 35(9) is hampered by the provision’s being restricted to data subjects, which relies on the data controller identifying affected subjects and seeking their views. Without accompanying rights and obligations of public transparency, it is difficult for stakeholders to identify that they are data subjects whose views should have been sought. There are no legal requirements for public disclosure of DPIAs. There is an obligation of prior consultation with the ICO before action where a DPIA indicates that the proposed data processing “would result in a high risk.”

Although the focus of this article is law and governance of data in the United Kingdom, this article’s consideration of DPIAs is relevant across EU member states by reason of the GDPR, and there are proposals for law and policy to provide for impact assessments of data processing in other jurisdictions. Furthermore, of relevance for all Council of Europe member states (which includes all EU member states), the Council’s Commissioner for Human Rights has issued a 10-point Recommendation on AI and human rights. The first point from the Commissioner recommends that “Member states should establish a legal framework that sets out a procedure for public authorities to carry out human rights impact assessments (HRIAs) on AI systems acquired, developed and/or deployed by those authorities” (Council of Europe Commissioner for Human Rights, 2019).

In the United States, the proposed Algorithmic Accountability Bill would require “entities that use, store, or share personal information to conduct automated decision system impact assessments and data protection impact assessments.” The Bill is sponsored by Democrat Senators Cory Booker and Ron Wyden, with an equivalent in the House of Representatives sponsored by Democrat Yvette Clarke (Robertson, Reference Robertson2019). Putting to one side the political question of whether the Bill might be passed into law, it illustrates the currency of the idea of impact assessments in jurisdictions outside of the EU. In Australia, the federal government’s Department of Industry, Innovation and Science recently consulted on Artificial Intelligence: Australia’s Ethics Framework (n.d.), which includes impact assessments as part of a “toolkit for ethical AI” (Dawson et al., Reference Dawson, Schleiger, Horton, McLaughlin, Robinson, Quezada and Hajkowicz2019).

Of direct relevance to this article’s discussion of data processing by government, the Canadian government now has a Directive on Automated Decision-Making (2019), which took effect on April 1, 2019. The Directive applies to the federal Canadian government’s use of automated decision-making—technology “used to recommend or make an administrative decision” about an individual or business, excluding National Security Systems (per Section 5). The Directive requires that an algorithmic impact assessment be produced prior to the production of an automated decision system. Rather than applying a one size fits all approach to automated decisions, the Directive sets out a risk-based framework whereby the proposed automated decision system is classified as one of four levels depending on the impact of the decision on environmental sustainability or the rights, health, or economic interests of individuals or communities. The Directive’s framework then sets out a sliding scale of requirements for impact assessment depending on the level at which the impact has been classified (per Section 6.1 and Appendices B and C).

For the purposes of discussion of DPIAs in the UK context, it is notable that a key objective of the Directive is that decisions made by federal government departments comply with procedural fairness and due process (per Section 4.2.1), which in UK law are crystallized in administrative law. The Directive does not provide for public participation or consultation on impact assessments prior to production of an automated decision system. However, it does require that results from Algorithmic Impact Assessments be released in a publicly accessible format, and sets transparency requirements for use of systems after production (per Section 6.2 and appendices B and C).

What Does a DPIA Process Include in the United Kingdom?

The ICO identifies the following key elements of a DPIA process:

This process as described by the ICO means that DPIAs should explain the function, purpose, and anticipated consequences of proposed data processing systems for the rights and interests of individuals. Under Step 2, the description of the processing must include the nature, scope, context, and purposes of the processing. Under Step 3, the ICO recommends consulting with individuals, unless there is good reason not to. Under Step 5 to identify and assess risks, the ICO advises: “look at whether the processing could contribute to:

1. 1. inability to exercise rights (including but not limited to privacy rights);

2. 2. inability to access services or opportunities;

3. 3. loss of control over the use of personal data;

4. 4. discrimination;

5. 5. identity theft or fraud;

6. 6. financial loss;

7. 7. reputational damage;

8. 8. physical harm;

9. 9. loss of confidentiality;

10. 10. reidentification of pseudonymized data; or

11. 11. any other significant economic or social disadvantage”Footnote ¹¹

Another relevant aspect of the regulatory landscape in Great Britain is the public sector equality duty. Under Section 149 of the Equality Act, all public authorities must, in the exercise of their functions, have due regard to the need to eliminate discrimination, harassment and victimization in relation to protected characteristics such as age, sex, pregnancy and maternity, and race. To assist compliance with equality duties, public authorities often carry out an Equality Impact Assessment for proposed policies to look at whether the policy would have a disproportionate impact on persons with protected characteristics.

Given that DPIAs should look at the potential for discrimination, there would be value in government integrating DPIAs with measures to comply with the public sector equality duty. Thus, proper equality analysis of the potential for direct and indirect discrimination could be informed by technical information on the data processing, and assessment of the impact of the data processing on rights and freedoms could be informed by expertise in equality.

What Might a “Good” DPIA Look Like in Substance?

This section looks primarily at human rights and administrative law for guidance on what the substance of DPIAs should include.

Janssen suggests a framework of stages in the operation of data processing systems across which DPIAs should consider human rights risks when designing a data processing system: data capture stage, analytics stage, and distribution stage. This framework highlights the different risks that the different stages of the operation of data processing systems can raise (Janssen, Reference Janssen2018). For the data capture stage, attention should be directed to the data used in the process—what is the quality of those data given the likelihood of decision-making errors resulting from errors in data, and what categories of data will be used since special categories of data such as race or sexual orientation have higher levels of protection under the GDPR. At the analytics stage, risk to human rights arises from the design of the automated decision-making system’s weightings and inferences from analytics. Finally, the distribution stage creates risks of sharing data to other entities or for purposes that are not authorized.

If this suggested framework is applied to the case study examples, it helps to structure the risks relating to each system.

1. 1. For HB/CTB RBV systems:

   1. (a) At data capture stage, there are questions of the source of the “statistical information and risk propensity data gathered over many years” (Central Bedfordshire Council, 2018) used to profile claims;

   2. (b) At analytics stage, how is analysis performed on claims, for example what weightings are used and is machine learning analysis applied; and

   3. (c) At distribution stage, how are risk scores used once a claim has been processed—for example, is the risk score kept on the claimant’s file such that it can be seen by local authority staff at a later time?

2. 2. For the settled status scheme:

   1. (d) At data capture stage, there is a risk of errors in DWP or HMRC data;

   2. (e) At analytics stage, the focus is on the “business logic” applied to those data to determine whether there is a pass, partial pass, or fail; and

   3. (f) At distribution stage, for what purposes may information produced through the scheme can be accessed into the future. The memoranda of understanding on data sharing for the scheme state that DWP, HMRC, and Home Office personnel with appropriate security clearance can access information under the memorandum where there is a “genuine business need.”Footnote ¹² In answer to a parliamentary question on the HMRC and Home Office data sharing, the Minister has said that “A genuine business need means only staff at the Home Office and Her Majesty’s Revenue and Customs who require access to the data to carry out their duties will be granted access.”Footnote ¹³

To assess the risk to rights and freedoms potentially affected by data processing, DPIAs need to look at all human rights, not just privacy. For example, data processing in the settled status scheme could affect applicants’ rights to family life, to housing, to work, and to access healthcare because of the immigration law restrictions on the right to work, right to rent, and access to health services in the context of UK immigration policy. The Human Rights, Big Data, and Technology Project (HRBDT) has proposed a human rights based approach to the design, development and implementation of big data and AI projects. HRBDT’s report shows that these technologies can affect all of the rights set out in the Universal Declaration of Human Rights—not only equality and privacy, but others such as the rights to education, healthcare, social care of the elderly, and law enforcement. HRBDT’s proposed human rights based approach would include undertaking full human rights impact assessments against all human rights (McGregor et al., Reference McGregor, Ng and Shaheed2018). The systems described in the case studies potentially engage a wide range of rights including the right to work, right to housing, right to an adequate standard of living, and freedom from discrimination.

There is existing analysis and guidance for human rights impact assessments that has been produced by the business and human rights sector, which has relevance to DPIAs regardless of whether they are conducted by a public or private entity.Footnote ¹⁴ The Human Rights Impact Assessment Guidance and Toolbox by The Danish Institute for Human Rights (DIHR) is a particularly extensive guide (Götzmann et al., Reference Götzmann, Bansal, Wrzoncki, Poulsen-Hansen, Tedaldi and Høvsgaard2016). The Guidance recommends scoping an impact assessment by considering the type of project, human rights context, and who relevant stakeholders are, then collecting data to establish a baseline to better understand human rights identified in the scoping. Where government introduces automated processes to replace existing human decision-making, the baseline could be based on data from the human decision-making process and outcomes. Per the DIHR Guidance, indicators for which data have been collected to establish a baseline could then be measured to assess change and impact from automated data processing. The impact of a system on all human rights could thus be considered and analyzed, including assessing the severity of the impact in terms of the human rights consequences.

The DPIA process is an opportunity not only to identify risks and potential impact from data processing but also to include safeguards to manage and minimize the risks as part of the development of the data processing system. The DIHR Guidance provides for planning effective management of the impact as part of the human rights impact assessment process, including what steps will be taken to avoid and minimize impacts. Partially automated decision-making systems are likely to rely on the involvement of human decision-makers as a key safeguard. This means that the information, training, policies, and guidance given to human decision-makers will be important for the safeguarding of human rights in such data processing systems.

In contrast with the DIHR’s approach to assessing impact on human rights by collecting data and measuring human rights indicators, current monitoring of the settled status scheme as reported does not have regard to some potential human rights impacts. For example, there are data on application results disaggregated by pre-settled status versus settled status, application data disaggregated by nationality and by age groups, but there are no disaggregated data relating to the sex of applicants (Home Office, 2019f). From a rule of law perspective, in relation to the proper implementation of the law through the application process, the important question for the 40% of applicants granted pre-settled status is whether this was the correct status for them or whether they were legally entitled to settled status. Including in the Home Office’s monthly Official Statistics reports the data on the answer to the question of whether applicants have been continuously resident for less or more than 5 years—which is not presently included in published reports—would help to monitor the legal accuracy of the application process if those data were disaggregated against what kind of status (if any) applicants were granted.

Specific to the area of government use of automated decision-making systems, New York University’s AI Now Institute has proposed mandatory Algorithmic Impact Assessments (AIAs). The first step in an AIA would be to define the scope of the automated decision system, then provide public notice of existing and proposed systems with information on the “purpose, reach, internal use policies, and potential impacts” (Reisman et al., Reference Reisman, Schultz, Crawford and Whittaker2018) of the systems and have a public consultation process. Public participation through consultation is discussed further below in relation to environmental governance. For a public agency’s self-assessment, AI Now notes the importance of agencies being “experts on their own automated decision systems… to ensure public trust” and emphasizes that AIAs are an opportunity for agencies to build internal capacity (Reisman et al., Reference Reisman, Schultz, Crawford and Whittaker2018). The DIHR Guidance also emphasizes the importance of the right expertise and capacity to properly carry out a human rights impact assessment. Finally, AI Now recommends allowing researchers and auditors meaningful access to automated decision systems to review the systems once they are deployed.

In line with the DIHR Guidance and AI Now’s recommendations, DPIAs will operate best if they are complemented by follow up measurement of indicators and monitoring, including by external researchers and auditors.

DPIAs should include analysis based on testing of data processing systems. AI now argues for agencies to work with vendors and researchers to conduct testing and research on automated decision-making systems. Janssen has recommended automated decision-making systems should be run in trials, giving them a dry-run “in a controlled setting, prior to release to the public at large” in order to identify and address risks to human rights before a system goes live (Janssen, Reference Janssen2018). Unlike proposed projects for which an environmental impact assessment is produced, it is possible to test data processing. Where a system already exists, for example, the Xantura, Callcredit, Capita RBV systems (Data Justice Lab, 2018), it should be possible to test those systems for a DPIA to assess whether they profile risk in a discriminatory fashion. Unlike this proposal for testing of data processing systems using a dry-run, the settled status scheme was tested on individuals in pilot phases, rather than in a sandbox or with synthetic data, and the cohorts of applicants that participated in pilot stages may not have been representative of the population.Footnote ¹⁵

Finally, assessment in DPIAs against administrative law principles will encourage compliance with rule of law principles since administrative law defines what constitutes the lawful and proper exercise of government power. Administrative law is part of the common law of England and Wales that has been developed by the courts through case law. Administrative law contains a number of principles that public authorities must meet when they exercise public power. For example, in Padfield v Minister of Agriculture, Fisheries and Food [1968] UKHL 1, the UK court of the House of Lords held that a Minister’s decision-making discretion under a piece of legislation may not be exercised to frustrate the overall policy and objects of that legislation. Cobbe has already undertaken insightful analysis of the application of administrative law principles to automated decision processes (Cobbe, Reference Cobbe2018), so there is no need for detailed discussion here. DPIAs for government data processing systems will enhance rule of law governance if they include analysis of the system and administrative law principles. A key administrative law principle that all government data processing systems must adhere to is the principle of legality—government data processing systems must correctly implement the law and not exceed the scope of the power granted under law. Including analysis of the legality of a data processing system in DPIAs will mitigate the risk of the system being found unlawful under judicial review.

In order to make the law more accessible, data processing systems should incorporate explanations of how the law is being applied and implemented, and DPIAs could include assessment of such explanations in administrative law terms. Explanations could be used where there is a duty to provide reasons for a decision, which tends to be the case for more serious decisions such as the refusal to grant a passport (Cobbe, Reference Cobbe2018). Providing explanations for decisions by default would enhance the rule of law by increasing the clarity of the law and how it has been implemented. Explanations for data processing systems used by government will help to demonstrate compliance with another administrative law principle that requires decision-makers take all relevant considerations into account and not take into account irrelevant considerations.

Partially automated systems that are developed with transparency and explanation by design will help human decision-makers to understand the nature of and basis for the result of the automated data processing, and to therefore properly take that automated processing into account in their decision (Zalnieriute et al., Reference Zalnieriute, Bennett Moses and Williams2019). This includes enabling human decision-makers to disregard the result of automated data processing where the explanation for the result indicates that irrelevant factors were taken into account in the system. As Binns and Gallo have explained:

DPIAs should assess how human decision-makers in a partially automated process will interact with the automated data processing, including whether the automated data processing will be interpretable.

Lessons from Environmental Law—Good Governance through Good Process

Having considered what the substance of DPIAs should include with regard to human rights and administrative law, this section turns to environmental law for lessons on good processes that promote good governance.

The efficacy and legitimacy of the governance of data processing would be enhanced by following the example of environmental governance. As noted above, the GDPR lacks effective governance requirements for transparency or public participation in DPIAs, although the GDPR stipulates consultation with affected data subjects. By comparison, the 1998 Convention on Access to Information, Public Participation in Decision-making and Access to Justice in Environmental Matters (Aarhaus Convention) provides a framework of transparency, participation, and access to justice in order to improve environmental governance. As Article 1 of the Aarhaus Convention sets out:

The principle of public participation under the Aarhaus Convention is strengthened and reinforced by information rights that help to overcome information asymmetry between the public and those proposing a project. The public participation principle includes providing the public with information on the proposed project and the approval process for the proposal, including opportunities for public participation in the decision (Article 6(2)). Another aspect of the principle is providing the public with access to relevant information on the proposal (Article 6(6)). Access to such information enables public participation to be informed and therefore more meaningful because, as a meeting of the Aarhaus Convention members has observed, “Access to information is an essential prerequisite for effective public participation.”Footnote ¹⁶

The process for DPIAs provides an opportunity to systematically implement the principles of transparency and public participation in relation to data processing proposed by government. As noted above, AI Now’s analysis highlights the relevance of the principles of transparency and public participation in the context of algorithmic systems. If the kind of meaningful consultation and public participation described in the Aarhaus Convention was applied in the processes of DPIAs, it could improve governance of data processing by government through informed public participation.

By contrast, little information was made available to the public on how data processing in the settled status scheme would operate prior to the scheme’s establishment, although some information was published when the scheme was established (Home Office, 2019b). For example, the memoranda of understanding on data sharing between the Home Office and the DWP and Home Office and HMRC were not disclosed to the public until the scheme was live for the public, following questions in Parliament. Those memoranda refer to DPIAs having been conducted, but the DPIAs have not been published. As such, although there were stakeholder groups established by the Home Office to consult on the settled status scheme, the participation of those groups was limited by the information available to them.

One of the reasons it is useful to compare public participation on data processing as part of a DPIA to public participation on environmental impacts is that both contexts concern public engagement with information produced through technical expertise. UK government guidance and law on environmental impact assessments is that “The Environmental Statement must include at least the information reasonably required to assess the likely significant environmental effects of the development” (Ministry of Housing, Communities, & Local Government, 2014). The specialist technical expertise needed to identify and assess environmental impacts such as environmental science expertise is analogous to the expertise needed to properly conduct a DPIA, which includes data science and human rights expertise. Public participation under Aarhaus requires the translation of technical information into a “non-technical summary” (Article 6(6)(d)), recognizing that without such summation the technical nature of the information will pose a barrier to meaningful public participation.

DPIAs have not yet been published on RBV systems for HB/CTB applications, but some Equality Impact Assessments have been published. Consideration of some examples of publicly available Equality Impact Assessments of RBV systems for HB/CTB applications suggest that there may be a failure to understand how the RBV systems determine risk. In two examples, the assessment was that the proposed RBV policies would not have an impact on people with protected characteristics because all applications would be subject to the RBV. The risk that there might be disproportionate impacts on certain groups and hence indirect discrimination, for example, the risk of all people of non-White ethnicity being classified as high risk, appears not to have been considered (Ealing Council, 2018; Rochdale Borough Council, 2015). Interestingly, another example of an equality assessment undertaken to determine whether an RBV should continue contained some useful analysis of data disaggregated by protected characteristics, with consideration of whether there was disproportionate impact on such groups (Waltham Forest Council, 2015). However, there still appear to be some shortcomings in the analysis, for example in relation to race it is not clear why the impact of RBV on non-White race groups was not considered. DPIAs would need to include much better information than these Equality Impact Assessments on the nature and functioning of RBVs in order to meaningfully assess the risk they pose to rights and interests.

Failure of Existing Policies to Secure Transparency

This section highlights the need to uphold and fulfill the principle of transparency in reality. The principle of transparency, along with other principles discussed in this article, is well recognized by the United Kingdom but not well implemented in relation to government data processing systems. This kind of gap between recognized principles and the actual issues with government data processing systems is one of the reasons for this article’s discussion of DPIAs as mechanisms for governance through which these principles could be implemented.

Transparency and openness are encouraged by UK government policy, yet seem not to be implemented in practice. In addition to the legally binding provisions of equality and data protection law, there are UK government policies relevant to the design and implementation of data processing systems. For example, the Department of Culture, Media, and Sport’s Data Ethics Framework was written to guide the design of data use by government and the public sector, aimed at all those in the public sector working with data. Principle 6 of the Framework is “Make your work transparent and be accountable.” The Guidance for this principle states:

Similarly, the seven principles of public life (the “Nolan principles”) are reflected in the Ministerial Code, and apply to all who work in the civil service, public bodies, and public services. Those principles include:

Notably, the Committee on Standards in Public Life is undertaking a review into “artificial intelligence and its impact on standards across the public sector.”Footnote ¹⁷ This suggests that the Committee recognizes that data processing tools are affecting the established and accepted standards for the public sector.

Despite the emphasis on the principle of transparency and openness in UK government policy, the use of data processing by public authorities in the United Kingdom in their exercise of public power often lacks transparency. The UN Special Rapporteur on Extreme Poverty and Human Rights made the following observations after his visit to the United Kingdom in 2018 (citations omitted):

A recent report by The Bureau of Investigative Journalism identified significant concerns about the transparency of government procurement of data systems and data processing. The authors of the report conducted an extensive investigation of government procurement data and requests under freedom of information to examine government procurement related to data processing. The report concluded that:

It can be concluded therefore that the use and operation of data processing systems by government is not transparent at present. This opacity is illustrated by the systems in this article’s case studies. There is almost no information available on the data used in RBV assessments for HB and CTB, nor how those data are processed. There is more information on the automated checks in the settled status scheme in terms of the data gathered for processing and the way residence is automatically calculated, although there remain questions as to the precise business logic that the Home Office applies. There is also a question as to the purposes for which the data under the memoranda of understanding for data sharing between the Home Office and the DWP and HMRC can be accessed by Home Office, DWP and HMRC officials.Footnote ¹⁸

Requiring that government DPIAs be published would help to address the current lack of transparency on government data processing systems. The DIHR Guidance on human rights impact assessments explains the importance of reporting on the assessment and providing access to the report to stakeholders. The Aarhaus Convention also emphasizes the importance of making information publicly available. This principle of transparency through public information is relevant for the written reports from DPIAs. In the same way that all UK legislation is published on legislation.gov.uk, access to the information in DPIAs would be enhanced by having them published on one online location (not on each department’s website), in a searchable database, that is developed and maintained by an independent public authority.

Conclusion and Recommendations

Rather than arguing against government use of data processing systems, this article translates between data-focused, legal, and policy epistemologies to provide some shared understanding of the rule of law risks posed by government use of data processing systems. There could be advantages in rule of law terms to using such systems for public purposes, such as ensuring that only legally relevant considerations are taken into account in a decision as required by administrative law (Zalnieriute et al., Reference Zalnieriute, Bennett Moses and Williams2019). Interdisciplinary approaches are needed for good design and good governance of such systems, incorporating expertise across a range of areas including data science, digital technology, design, human rights and administrative law, and policy. In order for the recommendations identified in this article to uphold the rule of law effectively, it is essential that government agencies and departments have the necessary resources and expertise across these areas to carry out DPIAs and ongoing monitoring.

The purpose of this article’s recommendations for the substance and process of DPIAs is to begin a practical discussion on how to improve the conformity of government data processing systems with rule of law principles. Although the scope of this article has not included the lex specialis for data processing in the context of criminal law under EU Directive 2016/680, some of the conclusions from this analysis in relation to civil and administrative law may also be relevant to the criminal law context. This article’s analysis of human rights and administrative law identifies the following components for the substance of a DPIA for data processing by government to enhance fidelity to rule of law principles:

1. 1. assessment of whether the data processing is within the scope of the power granted under the law (per administrative law);

2. 2. assessment of whether the data processing system will correctly implement the law (per administrative law);

3. 3. analysis of any risk of discrimination, including through testing and dry-runs to assess indirect discrimination—this analysis should inform or be integrated with measures to comply with the public sector equality duty (per the Equality Act in Great Britain);

4. 4. impact assessment across all human rights, including economic, social, and cultural rights, as well as civil and political rights such as freedom of association and freedom of expression (per the Human Rights Act, ICESCR, and ICCPR);

5. 5. analysis of how decision-makers and individuals subject to the system will interact with the system, including in particular the features of the system that will provide transparency and explanations as to the data that the system uses, the way in which it processes those data, and how the system’s data will be distributed (per administrative law);

6. 6. identification of the indicators or data points that will be measured and collected for ongoing regular monitoring and auditing of the system, including to monitor impact on human rights and whether the system is producing just and lawful outcomes; and

7. 7. explanation of the access to justice mechanisms for those subject to the data processing system with a goal of ensuring accountability and enforcing proper implementation of the law by government—such mechanisms could include enabling individuals subject to the system to know what data were used in the making of a decision and to correct those data if they are inaccurate (per rule of law principles and administrative law).

Learning the lessons from the principles of transparency and public participation in environmental governance, DPIA processes should include the following elements of transparency, public consultation, and testing. Such transparency, public consultation, and testing may not be strictly required by law, but will help to ensure that the DPIA and proposed data processing meet the legal standards and requirements identified in the list above and this article generally.

1. 8. Public notice of the proposed data processing, including a nontechnical summary and explanation of how the processing system would work in terms of its purpose, reach/scope, internal use policies, and potential impacts;

2. 9. public consultation and participation in both the design and development of the data processing system; and

3. 10. testing of the data processing system before it goes live, with information on the results of that testing informing the public participation on the development of the system.

Finally, transparency and accountability would be enhanced if DPIAs by government were published by default, so that there was a publicly accessible database of government DPIAs. DPIAs should be followed by regular monitoring and auditing of government data processing systems, the reports from which should also be publicly available.