II. How risk management and risk acceptability fit into the AI Act’s goals and mechanics

Let us set the scene for the discussion in this paper. What is AI? What are its risks and benefits? How does the Commission’s Proposal implement a “risk-based” approach? Where does Article 9(4) fit in?

III. Problems with the Commission’s “AFAP” approach to risk acceptability: a historical perspective from medical devices regulation

The Commission’s Proposal requires risks to be reduced “AFAP” through risk management. What does that actually mean? We will try to answer that question in this section, describing and evaluating the Commission’s very narrow historical approach to interpreting AFAP in the context of medical devices regulation. We will also illustrate how the stringency of the criterion perversely creates uncertainty about how much risk management is enough and encourages stakeholders to fudge acceptability judgments.

Let us start with the history of AFAP. The language of Article 9(4) in the Commission’s Proposal, including the AFAP criterion, closely follows language from the General Requirements section of the European medical devices regulation.Footnote ²⁵ However, it leaves out the General Requirements’ definition of AFAP: “the requirement to reduce risks as far as possible means the reduction of risks as far as possible without adversely affecting the benefit–risk ratio”. ^{Footnote 26} References to the benefit–risk ratio are conspicuously absent from Article 9. It is not clear why. The implication is that the standard is intended to be interpreted narrowly, excluding consideration of the opportunity costs of risk management.

The European Commission has also, in the past, explicitly rejected a broad interpretation of AFAP. It objected in 2010 to risk management provisions in EN ISO 14971, the medical device risk management standard, which treated AFAP as analogous to “as low as reasonably practicable” (ALARP), allowing cost–benefit considerations to enter into risk-acceptability judgments (albeit with a heavy weighting towards risk reduction). ^{Footnote 27} Subsequently, ISO, working together with the Commission, clarified that in the European context all medical device risks had to be reduced AFAP “without there being room for economic considerations”.Footnote ²⁸

The problems with a narrow approach to AFAP are evident from the reaction to the changes to the ISO standard. The industry group for “notified bodies” tasked with certifying medical devices against ISO 14971 reported that the change created uncertainty about “where to stop reducing risk before a product can be placed on the market” in Europe. ^{Footnote 29} Other responses highlighted how unlikely businesses are to completely disregard “economic considerations”. ^{Footnote 30} Some risk management consultants simply advised their clients to conceal their cost–benefit analyses: “If you want to consider the economic impact of risk and keep track of business risks, you should create a separate document for that purpose.” ^{Footnote 31}

The very impracticability of AFAP seems to encourage a lack of rigour in the approach to risk acceptability. In medical device risk management, demonstrating “state-of-the-art” risk management measures is now taken as evidence in a conformity assessment of having reduced risks AFAP. ^{Footnote 32} The same is likely to be the case for high-risk AI systems, given the explicit reference to “state of the art” in Article 9(3).Footnote ³³

The meaning of “state of the art” is vague, however, to the point of being circular. The European Commission’s Draft Standardization Request in Support of Safe and Trustworthy Artificial Intelligence defines “state of the art” as follows:

It is unclear whether standards are supposed to conform with the developed state of technical capability or whether they themselves determine the state of the art. It is also not clear whether the state of the art calls for the highest level of precaution feasible given current technology. Its meaning, at least in the medical devices context, seems more akin to current good practice in industry. ^{Footnote 35}

What is the import of all of this? If AFAP is interpreted in the AI context as it is for medical devices, it will pose several problems. Firstly, concerns about indeterminate risk-management costs are as relevant to AI systems as they are to medical devices. AI risks are “emergent” and unpredictable, and it is always possible to implement just one more measure to reduce risk.Footnote ³⁶ Regulatees need more certainty about where to stop. Secondly, it is not clear that the “state of the art” or standards provide principled reasons for where to draw the line for risk management. Finally, the prospect of regulatees pretending to disregard economic considerations while concealing their cost–benefit analyses – in effect, fudging AFAP – is at odds with the Proposal’s goal of trustworthy AI.

IV. “Reasonableness” in Parliament’s proposals regarding risk management and risk acceptability

Changes proposed by both Council and Parliament to Article 9 indicate that they are alive to the problems of indeterminacy of risk management and risk acceptability in the Commission’s Proposal. Both the Council and (to a much greater extent) the Parliament use the language of reasonableness to keep the burden of risk management in proportion to risk. In this section, we set out the key amendments proposed by each that bear on Article 9(4), before exploring jurisprudence from negligence and medical devices regulation that gives a sense of how reasonableness brings risk–benefit and cost–benefit analyses into risk-acceptability judgments.

The overarching obligation to identify, analyse and manage risks comes from Article 9(2) of the Commission’s Proposal. The Council’s amendments limit the application of Article 9(2) (and consequently the whole of Article 9) to only those risks “most likely to occur”, and they state that risk management should “concern only those [risks] which may be reasonably mitigated or eliminated through the development or design of the high-risk AI system, or the provision of adequate technical information”. ^{Footnote 37}

The Parliament’s amendments place even more emphasis on reasonableness and proportionality in risk management, and they engage more directly with Article 9(4) along with the rest of Article 9.Footnote ³⁸ Where the Council proposes to limit risk management only to risks “most likely to occur” (perhaps too narrow a scope), the Parliament proposes more balanced scope limitations for risk management. It envisages that the obligations to identify, estimate and evaluate risks apply to “reasonably foreseeable risks” (Amendment 263) and that the obligation to evaluate risks emerging from post-market monitoring apply only to “significant” risks (Amendment 265).

In addition to limiting the scope of risk management, the Parliament also proposes to limit its burden. A proposed change to Article 9(2)(d) qualifies the requirement to adopt risk management measures so that only “appropriate and targeted” risk management measures are required (Amendment 266). An amendment to Article 9(3) adds wording to the effect that when regulatees consider the effects of risk management, they should have regard to mitigating risks effectively while “ensuring an appropriate and proportionate implementation of the requirements” (Amendment 267). Finally, proposed text in Article 9(5) indicates that risk management measures should be weighed against the potential benefits and intended goals of the system (Amendment 273).

Parliament’s changes to Article 9(4) indicate a similar intent. Rather than requiring risk management to be such that risks are judged “acceptable” and reduced AFAP, the proposed wording is “reasonably judged to be acceptable” and “elimination or reduction of identified risks as far as technically feasible through adequate design and development of the high-risk AI system, involving when relevant, experts and external stakeholders” (Amendment 269, showing amended wording in italics).

What emerges from all of this is a contest between a narrow risk-acceptability criterion – the Commission’s AFAP – and more flexible approaches proposed by the Council and especially the Parliament, which take into account the significance of risks, the burden of risk management and the value-laden nature of risk-acceptability judgments. There will be many matters for the Parliament, Council and Commission to resolve in the trilogue, but the scope and burden of risk management and risk acceptability will be important ones. An understanding of reasonableness is relevant to the negotiations on the final form of Articles 9 and 9(4) and, ultimately, their implementation. It is important to unpack, then, what reasonableness entails.

Reasonableness permits risks to be judged in context rather than according to a rigid, abstract standard. A possible pan-European approach to “reasonableness” in tort law is summarised in Article 4:102(1) of the Principles of European Tort Law:

The essence of reasonableness is not vastly different between European and common law doctrines, even if there are differences in emphasis.

In common law negligence, four factors come into play in assessing reasonableness: (1) likelihood of harm; (2) severity of harm; (3) cost of precautions; and (4) impact of precautions on the utility of the activity. ^{Footnote 39} The economic approach to tort law is also based on this four-factor approach. ^{Footnote 40} The four-factor approach is less prevalent in civil law jurisdictions, but in France and Germany at least the four factors are mentioned in the legal literature,Footnote ⁴¹ as well as in recent commentaries on AI and civil liability in Europe. ^{Footnote 42} In practice, the balancing is rarely conducted with algebraic precision because of the difficulty of ascribing precise probabilities and values to possible outcomes.Footnote ⁴³ Rather, the general objective is to avoid obvious disproportion between risk management and risk (in either direction).

Reasonableness supports proportionality in risk management because it provides a framework for principled, transparent weighing of the trade-offs involved. ^{Footnote 44} Precautions against risk generally involve loss of money, time, convenience or value, including value to the public. ^{Footnote 45} Applying reasonableness principles opens the door to considerations of cost–benefit ratios, risk–benefit ratios and materiality of risk. That is to say, it permits (indeed encourages) judgments about: whether the cost of a given risk management measure is worth the reduction in risk; whether risk management negatively impacts the overall benefit of an AI system; and whether risks are significant enough to warrant expenditure of finite risk management resources.

Given the AI Act’s overarching goal of proportionality, the reason to consider a cost–benefit ratio is that increasing the cost of risk management for an activity or product increases the marginal cost of its production. That cost is likely to be passed on to the public. ^{Footnote 46} ISO’s guidance on the medical device risk management standard is consistent with negligence jurisprudence on this point:

This passage shows that there is a difference between considering economic matters and using them to justify unnecessary risk. It also indicates that economic considerations may have broader significance to matters of public interest (such as the availability of beneficial systems and devices).

AI systems with the greatest potential social value (improving medical triage, reducing bias in loan or employment or judicial decisions, etc.) may pose the highest risks because of the high-stakes domains where they are deployed. If an AI system would be beneficial to society (and especially if it would be beneficial to the enjoyment or protection of fundamental rights or safety relative to the status quo), it would be undesirable unduly to increase the cost of putting such a system into use.

The reason to consider a risk–benefit ratio is that risk management may have utility costs as well as economic costs. ^{Footnote 48} The classic example of risk management reducing the utility of an activity is speed limits. Lower speed limits reduce the risk of car accidents but also the overall usefulness to the public of cars and driving (speed, convenience and the generation of all kinds of public good via rapid movement of people and resources). That is why we are not all driving around at five kilometres per hour.

Similarly, current ISO guidance for medical device risk management (where AFAP still means AFAP without disturbing the risk–benefit ratio) comments on how risk management may reduce the utility of a medical device.Footnote ⁴⁹ It explains, by way of example, that reducing the power of an electrosurgical unit below its effective level in order to reduce risks would have an overall negative effect on the balance between benefit and risk for the device. Because risk–benefit considerations are permitted for medical devices, such an intervention would not be required. ^{Footnote 50}

According to the same logic, negative impacts of risk management on the usefulness of a high-risk AI system (eg decreases in its effectiveness at reducing errors in administrative, medical or financial decision-making) should also factor into whether a risk has been reduced to an acceptable level.

Risk–benefit ratio assessments may also militate in favour of precaution – meaning that systems might in some circumstances not be judged to be “reasonably” acceptable, even if risks are managed as well as they possibly can be. In the medical device context, the final acceptability judgment turns not only on whether risks have been reduced AFAP, but also on whether the overall risk–benefit picture is acceptable. It is no good putting AI systems into use on the basis that their risks have been reduced AFAP or “as far as technically feasible” if this still leaves an intolerably high level of risk relative to the benefits of the system.

Since risk management may both increase the expense (cost–benefit impact) and decrease the utility (risk–benefit impact) of useful activities, a third reasonableness consideration is materiality or significance of risk. As risks become less significant, risk management delivers diminishing returns. ^{Footnote 51} This is presumably why risk management based on “risk scoring”, where risks of low probability and severity did not require further reduction, was the norm for medical devices before the Commission’s intervention in 2010. ^{Footnote 52} It is also why negligence doctrine, at least in some jurisdictions, does not require the implementation of any risk management measures for insignificant risks. ^{Footnote 53}

The language of the proposed amendments to Articles 9 and 9(4), especially Parliament’s, invites these kinds of cost–benefit, risk–benefit and materiality threshold analyses. The jurisprudence we have just described could helpfully inform a number of judgments required under the Parliament’s amendments to Article 9, including: what it means to “reasonably” judge a risk acceptable; judgments about what is “appropriate and proportionate” or “appropriate and targeted” risk management; the weighing of risk management measures against the benefits of a system; and the approach to setting the threshold of materiality for risk management. In that respect, the Parliament’s approach to risk management has the capacity to be not only more proportionate, but also more principled, than the Commission’s (as well as the Council’s, which is more tentative in proposing pro-reasonableness amendments).

V. Which approach to risk acceptability is better for trustworthiness?

So much for the question of which approach to risk acceptability best serves the goal of proportionate regulatory burden. What about the Act’s other goal of trustworthiness? Here we consider the reasons why the Commission’s approach to risk acceptability (AFAP), despite appearances, would not necessarily produce trustworthy outcomes; while the Parliament’s approach, which is more explicit about cost–benefit judgments and who is qualified to make them, is overall more likely to promote trust.

We start, however, with the objections that the Parliament’s approach might have to overcome. Setting the threshold of acceptable risk is one of the most challenging and controversial tasks in risk governance.Footnote ⁵⁴ Questions about acceptability of risks to human rights are not merely technical. They are contextual, social, normative and political. ^{Footnote 55} The disadvantage of reasonableness-style balancing is that it leaves room for disagreement about how to weigh different factors. It involves implicit or explicit judgments about the value of the activity in question and the interests affected by it. ^{Footnote 56}

It is also conceivable that proportionality considerations might undermine trustworthiness. The European Parliament has said that citizens’ trust in AI must be built on an ethics by design, “in line with the precautionary principle that guides Union legislation and should be at the heart of any regulatory framework for AI”, and it has called for the “utmost precaution” in relation to automated decisions in the areas of justice and law enforcement. ^{Footnote 57} In essence, the precautionary principle favours erring on the side of caution, especially in the face of uncertain risks posed by new technologies. ^{Footnote 58}

The European Commission has also noted that the application of the precautionary principle inevitably calls for political, normative judgment:

Risk-acceptability judgments, which are qualitative in nature, rely heavily on skilful socio-cultural judgment. ^{Footnote 60} They will not further the goal of trustworthiness unless those who make them have an imprimatur of legitimacy.

There is force in critiques of courts’ abilities to judge difficult questions of social value and utility – particularly cost–benefit trade-offs involved in the design of complex systems. ^{Footnote 61} For some, the fiction of the reasonable person is simply a way to disguise subjective judicial preferences.Footnote ⁶² Judges are expected to weigh uncertain values and trade off rights and interests, and courts have the constitutional legitimacy to do so – yet the weighing is still fraught.

How, then, can technologists implementing Article 9(4) be expected to make such tricky value judgments before a system has even been put into use? This is a strong objection to permitting reasonableness into the AI risk-acceptability criterion (although we think it can be overcome). The objection is further fortified by the fact that commercial AI providers have an interest in resolving uncertainties in favour of lowering their costs.

One of the apparent advantages of the Commission’s approach to risk acceptability, with the AFAP risk criterion, is that it appears to evade normative, political questions about the relative risks and benefits of any given high-risk system or set of risk management measures. Instead of making difficult value judgments, all that is required is to reduce the risk AFAP (apparently without regard to cost).

AFAP also has symbolic force that may counteract commercial bias in risk management. The choice of the AFAP risk criterion signals that the European regulator will not tolerate cowboys. This might discourage regulatees from resolving uncertainties about costs and benefits in favour of selfish cost-saving. When it comes to the goal of trustworthy AI, this is a point in its favour.

Nonetheless, as we have argued above, risk management, even for high-risk systems, does have to stop somewhere. The AFAP criterion relies heavily on the state of the art and standards to mark that stopping point. The result is not to produce value-neutral risk management, but rather to cede control over important matters of public policy to technical standards bodies and the “state of the art”.

If “state of the art” boils down to “best practice” or “good industry practice”, then AI practitioners are likely to look to large technology companies, who have invested substantial resources into research on “ethical” or “fair, accountable and transparent” AI.Footnote ⁶³ Yet, to take a recent example, Google has announced that it will “recalibrate” the level of AI risk it is willing to accept, and it will accelerate product reviews. ^{Footnote 64} It is at least possible that Google’s level of risk management was, as a matter of public interest, too high. The problem is that Google has openly stated that its motivation for recalibrating its risk practices is purely commercial: to ensure it is not outcompeted by OpenAI. ^{Footnote 65} This is not a principled decisions about AI risk acceptability in the public interest.

It is also not clear that standards bodies have that political legitimacy or the expertise to make the value judgments involved in setting risk acceptability standards in relation to fundamental rights.Footnote ⁶⁶ They are typically constituted by technology industry insiders, have little experience with human rights and make minimal provision for participation by civil society and other stakeholders.Footnote ⁶⁷ Standards generally govern the engineering of technical features and processes rather than trade-offs between competing rights and interests in complex socio-political contexts.

If the Commission’s risk acceptability criterion (AFAP × state of the art) is not likely to promote trustworthiness, is there a way of making the value-laden exercise contemplated by the Parliament more trustworthy? Two things are needed to make it so. Firstly, the approach to risk management and risk acceptability must be compatible with the precautionary principle. Secondly, European legislators and regulators have a duty to ensure that those who decide how risk acceptability for AI works have the capability and the civic, regulatory legitimacy to do so.

The first challenge can be met by stipulating with sufficient clarity that the costs of risk management and the quantum of risk should not be given equal weight in reasonableness assessments. Uncertainties must be resolved in favour of precaution. This message is conveyed to some extent by the choice of risk acceptability criterion in the Parliament’s draft compromise amendments: “as far as technically feasible” – which has the advantage over AFAP of not having a documented history of literalistic interpretation by the Commission.

The message could be equally well (or better) conveyed with the “ALARP” criterion that the European Commission rejected over ten years ago for medical devices.Footnote ⁶⁸ ALARP is well understood by risk managers to require a precautionary approach. Generally, ALARP excuses risk managers only from risk management measures whose burden is grossly disproportionate to risk. ^{Footnote 69} And (at least in some quarters) the approach to disproportion follows an exponential scale: the greater the risk, the higher the disproportion may be.Footnote ⁷⁰ This is well suited to AI, which has a “fat tail” of low-probability, high-gravity risks. ALARP also does not need to be read in light of other reasonableness requirements in Article 9 in order to be practicable and proportionate (in contrast to the Parliament’s current wording, as far as technically feasible, which does need that kind of “reading down”).

The second challenge is one of regulatory capacity and civic legitimacy. The Proposal’s architecture involves “regulated self-regulation”, sometimes described as meta-regulation or co-regulation.Footnote ⁷¹ It is widely acknowledged in regulatory theory that de-centred regulation – a sharing of responsibilities between private and public entities (eg standards bodies and government regulators) – is inevitable in a complex world.Footnote ⁷² But the allocation and structuring of regulatory discretion warrants careful consideration.Footnote ⁷³ It should be attuned to the institutional and cultural characteristics and capabilities of the participants in regulation, and in this case there are doubts about whether standards bodies have the requisite capabilities.Footnote ⁷⁴

Two things would help here. Firstly, there needs to be input from affected groups and stakeholders on value-laden matters such as acceptability of risks to human rights.Footnote ⁷⁵ Secondly, regulators with political and institutional legitimacy need to be more involved in making or at least guiding risk-acceptability judgments.

At the time of writing, there are positive signs regarding both these two things: more provision for stakeholder involvement and more input from regulators on both standards and risk acceptability. The Commission’s standardisation request to European standards organisations calls on them to take steps to ensure the involvement of small and medium enterprises and civil society organisations and to ensure relevant expertise in the field of fundamental rights.Footnote ⁷⁶ While this is promising, more could be done. Galvagna identifies prohibitive barriers to meaningful civil society participation in standardisation. These include a lack of resources for engagement in standardisation, restrictive eligibility requirements, lack of influence and status in relevant standards organisations and lack of awareness and understanding of the standards process.Footnote ⁷⁷ However, she recommends several changes, including: amending the Regulation on European Standardisation to broaden the categories eligible for funding and mandated participation; funding for individuals to participate; and funding for a central hub to support civil society participation in standards.Footnote ⁷⁸

The Parliament’s amendments seem to contemplate both stakeholder input on standards and more guidance from regulators and stakeholders on risk acceptability. In the Parliament’s version of Article 9(4), risk-acceptability judgments would involve “when relevant, experts and external stakeholders”. Likewise, the Parliament’s amendments explicitly provide for participation in standards-making by a range of stakeholders (Amendments 103 and 104). They also make provision for the Commission to develop guidance on the requirements for high-risk systems, with the input of a dedicated AI Office (Amendment 529). Such a body would be well placed to provide guidance on Article 9. Alternatively, there is the option of the European Commission issuing its own common specifications about risk management under Article 41 of the Proposal, should standards fail to satisfy fundamental rights concerns.Footnote ⁷⁹ Either guidance or common specifications would have an imprimatur of legitimacy and could articulate workable principles (ideally in line with the frameworks we have suggested here) to help both standards bodies and regulatees decide how much risk management is enough.