I. Introduction
In April 2021, the European Commission (EC) published a proposal for an Artificial Intelligence Act (AI Act). Footnote 1 As the first comprehensive attempt to regulate Footnote 2 artificial intelligence (AI)Footnote 3 in a major jurisdiction, the AI Act will inevitably serve as a benchmark for other countries like the USA and the UK. Due to the so-called “Brussels Effect”, Footnote 4 it might even have de facto effects in other countries, Footnote 5 similar to the General Data Protection Regulation (GDPR). Footnote 6 It will undoubtedly shape the foreseeable future of AI regulation in the European Union (EU) and worldwide.
Within the AI Act, the requirements on risk management Footnote 7 are particularly important. AI can cause or exacerbate a wide range of risks, including accident, Footnote 8 misuse Footnote 9 and structural risks. Footnote 10 Organisations that develop and deploy AI systems need to manage these risks for economic, legal and ethical reasons. Being able to reliably identify, accurately assess and adequately respond to risks from AI is particularly important in high-stakes situations (eg if AI systems are used in critical infrastructure Footnote 11 ). This will become even more important as AI systems become more capable and more general in the future. Footnote 12
In recent years, attention on AI risk management has increased steadily amongst practitioners. As of 2022, several standard-setting bodies are developing voluntary AI risk management frameworks; the most notable ones are the NIST AI Risk Management Framework Footnote 13 and ISO/IEC 23894. Footnote 14 Existing enterprise risk management (ERM) frameworks like COSO ERM 2017 Footnote 15 have also been applied to an AI context. Footnote 16 Many consulting firms have published reports on AI risk management. Footnote 17 However, there is only limited academic literature on the topic. Footnote 18 In particular, I could only find a single paper that analyses (parts of) the risk management provision in the AI Act. Footnote 19
This article conducts a doctrinal analysis Footnote 20 of Article 9 using the four methods of statutory interpretation: literal, systematic, teleological and historical interpretation. Footnote 21 But as there is not yet a final text, I have to rely on drafts and proposals, Footnote 22 namely the original draft by the EC, Footnote 23 as well as the proposed changes by the Council Footnote 24 and the European Parliament (EP). Footnote 25 It is therefore possible that future changes will make my analysis obsolete. However, there are three main reasons why I am willing to take that risk. First, I do not expect the provision to change significantly. The requirements are fairly vague and do not seem to be particularly controversial. In particular, I am not aware of significant public debates about Article 9 (although the Council Footnote 26 and the EP Footnote 27 have suggested changes). Second, even if the provision is changed, it seems unlikely that the whole analysis would be affected. Most parts would probably remain relevant. Section III, which determines the purpose of the provision, seems particularly robust to future changes. Third, in some cases, changes might even be desirable. In Section VII, I suggest several amendments myself. In short, I would rather publish my analysis too early than too late.
The article proceeds as follows. Section II gives an overview of the regulatory concept behind Article 9. Section III determines its purpose and Section IV its scope of application. Section V contains a comprehensive interpretation of the specific risk management requirements, while Section VI outlines ways in which they can be enforced. Section VII contains a summary and concludes with recommendations for the further legislative process.
II. Regulatory concept
In this section, I give an overview of the regulatory concept behind Article 9. I analyse its role in the AI Act, its internal structure and the role of standards.
The AI Act famously takes a risk-based approach. Footnote 28 It prohibits AI systems with unacceptable risks Footnote 29 and imposes specific requirements on high-risk AI systems, Footnote 30 while leaving AI systems that pose low or minimal risks largely unencumbered. Footnote 31 To reduce the risks from high-risk AI systems, providers of such systems must comply with the requirements set out in Chapter 2, Footnote 32 but the AI Act assumes that this will not be enough to reduce all risks to an acceptable level: even if providers of high-risk AI systems comply with the requirements, some risks will remain. The role of Article 9 is to make sure that providers identify those risks and take additional measures to reduce them to an acceptable level. Footnote 33 In this sense, Article 9 serves an important backup function.
The norm is structured as follows. Paragraph 1 contains the central requirement, according to which providers of high-risk AI systems must implement a risk management system, while paragraphs 2–7 specify the details of that system. The risk management system consists of two parts: the risk management process (paragraphs 2–4) and the testing procedures (paragraphs 5–7). The remainder of Article 9 contains special rules for children and credit institutions (paragraphs 8 and 9).
In the regulatory concept of the AI Act, standards play a key role. Footnote 34 By complying with harmonised standards, Footnote 35 regulatees can demonstrate compliance with the requirements set out in the AI Act. Footnote 36 This effect is called “presumption of conformity”. Footnote 37 In areas where no harmonised standards exist or where they are insufficient, the EC can also develop common specifications. Footnote 38 Harmonised standards and common specifications are explicitly mentioned in Article 9(3), sentence 2. It is worth noting that the Council has suggested deleting the reference to harmonised standards and common specifications. Footnote 39 However, this would not undermine the importance of harmonised standards and common specifications. They would continue to provide guidance and presume conformity. Harmonised standards and common specifications on AI risk management do not yet exist. The recognised European Standards Organisations Footnote 40 have jointly been tasked with creating technical standards for the AI Act, including risk management systems, Footnote 41 but that process may take several years. In the meantime, regulatees could use international standards like the NIST AI Risk Management Framework Footnote 42 or ISO/IEC 23894. Footnote 43 Although this will not presume conformity, these standards can still serve as a rough guideline. In particular, I expect them to be similar to the ones that will be created by the European Standards Organisations, mainly because standard-setting efforts usually strive for some level of compatibility, Footnote 44 but, of course, there is no guarantee for this. With this regulatory concept in mind, let us now take a closer look at the purpose of Article 9.
III. Purpose
In this section, I determine the purpose of Article 9. This is an important step because the purpose has significant influence on the extent to which different interpretations of the provision are permissible.
Pursuant to Recital 1, sentence 1, the purpose of the AI Act is “to improve the functioning of the internal market by laying down a uniform legal framework … in conformity with Union values”. More precisely, the AI Act intends to improve the functioning of the internal market through preventing fragmentation and providing legal certainty. Footnote 45 The legal basis for this is Article 114 of the Treaty on the Functioning of the European Union (TFEU). Footnote 46
At the same time, the AI Act is intended to ensure a “high level of protection of public interests”. Footnote 47 Relevant public interests include “health and safety and the protection of fundamental rights, as recognised and protected by Union law”. Footnote 48 Note that the Council has suggested adding a reference to “health, safety and fundamental rights” in Article 9(2), sentence 2, point (a). Footnote 49 Protecting these public interests is part of the EU’s objective of becoming a leader in “secure, trustworthy and ethical artificial intelligence”. Footnote 50
It is unclear whether Article 9 is also intended to protect individuals. This would be important because, if it does, it would be easier for the protected individuals to assert tort claims in certain Member States. Footnote 51 Recital 42 provides an argument in favour. It states that the requirements for high-risk AI systems are intended to mitigate the risks to users Footnote 52 and affected persons. Footnote 53 However, one could also hold the view that the risk management system is primarily an organisational requirement that only indirectly affects individuals. Footnote 54 As this question is beyond the scope of this article, I will leave it open.
Understanding the purpose of Article 9 helps with interpreting the specific risk management requirements. But before we can turn to that, we must first determine who needs to comply with these requirements.
IV. Scope of application
In this section, I determine the scope of Article 9. This includes the material scope (what is regulated), the personal scope (who is regulated), the regional scope (where the regulation applies) and the temporal scope (when the regulation applies). Footnote 55
Article 9 only applies to “high-risk AI systems”. This can be seen by the formulation in paragraph 1 (“in relation to high-risk AI systems”) and the location of Article 9 in Chapter 2 (“Requirements for high-risk AI systems”). The term “AI system” is defined in Article 3, point 1, Footnote 56 while Article 6 and Annex III specify which AI systems qualify as high risk. This includes, for example, AI systems that screen or filter applications as well as risk assessment tools used by law enforcement authorities. Note that both the Council Footnote 57 and the EP Footnote 58 have suggested changes to the AI definition.
The risk management system does not need to cover AI systems that pose unacceptable risks; these systems are prohibited. Footnote 59 But what about AI systems that pose low or minimal risks? Although there is no legal requirement to include such systems, I would argue that, in many cases, it makes sense to do so on a voluntary basis. There are at least two reasons for this. First, if organisations want to manage risks holistically, Footnote 60 they should not exclude certain risk categories from the beginning. The risk classification in the AI Act does not guarantee that systems below the high-risk threshold do not pose any other risks that are relevant to the organisation, such as litigation and reputation risks. It therefore seems preferable to initially include all risks. After risks have been identified and assessed, organisations can still choose not to respond. Second, most of the costs for implementing the risk management system will likely be fixed costs, which means that including low- and minimal-risk AI systems would only marginally increase the operating costs.
In addition, the Council has suggested extending Article 9 to “general purpose AI systems”. Footnote 61 Meanwhile, the amendments under consideration by the EP range from extending Article 9 to general purpose AI systems to completely excluding them from the scope of the AI Act. Footnote 62 Overall, the best approach to regulating general purpose AI systems is still highly disputed and beyond the scope of this article. Footnote 63
As Article 9 is formulated in the passive voice (“a risk management system shall be established”), it does not specify who needs to comply with the requirements. However, Article 16, point (a) provides clarity: Article 9 only applies to “providers of high-risk AI systems”. The term “provider” is defined in Article 3, point 2. Footnote 64 Note that Article 2(4) excludes certain public authorities and international organisations from the personal scope.
Article 9 has the same regional scope as the rest of the AI Act. According to Article 2(1), the AI Act applies to providers who place on the market Footnote 65 or put into service Footnote 66 AI systems in the EU or where the output produced by AI systems is used in the EU. It does not matter whether the provider of such systems is established within the EU or in a third country.
Providers of high-risk AI systems must have implemented a risk management system twenty-four months after the AI Act enters into force, Footnote 67 though the Council has proposed to extend this period to thirty-six months. Footnote 68 The AI Act will enter into force twenty days after its publication in the Official Journal of the European Union. It is unclear when this will be the case. The EC and the Council are currently waiting for the EP to finalise its position, which is expected to happen in early 2023. Then, the Council and the EP will enter interinstitutional negotiations assisted by the EC – the so-called “trilogue”. Against this background, it seems unlikely that the final regulation will enter into force before mid-2023. Providers of high-risk AI systems therefore have time until early 2025 (or 2026 according to the proposal by the Council Footnote 69 ) to comply with the requirements set out in Article 9. But what exactly do these requirements entail?
V. Requirements
In this section, I offer a comprehensive interpretation of the specific risk management requirements set out in Article 9.
1. Risk management system, Article 9(1)
Pursuant to paragraph 1, “a risk management system shall be established, implemented, documented and maintained in relation to high-risk AI systems”. This is the central requirement of Article 9.
The AI Act does not define the term “risk management system”, Footnote 70 but the formulation in paragraph 8 suggests that it means all measures described in paragraphs 1–7, namely the risk management process (paragraphs 2–4) and testing procedures (paragraphs 5–7). Analogous to the description of the quality management system in Article 17(1), one could hold the view that a “system” consists of policies, procedures and instructions.
The risk management system needs to be “established, implemented, documented and maintained”. As none of these terms are defined in the AI Act, I suggest the following definitions. A risk management system is “established” if risk management policies, procedures and instructions are created Footnote 71 and approved by the responsible decision-makers. Footnote 72 It is “implemented” if it is put into practice (ie the employees concerned understand what is expected of them and act accordingly). Footnote 73 It is “documented” if the system is described in a systematic and orderly manner in the form of written policies, procedures and instructions Footnote 74 and can be demonstrated upon request of a national competent authority. Footnote 75 It is “maintained” if it is reviewed and, if necessary, updated on a regular basis. Footnote 76
2. Risk management process, Article 9(2)
The first component of the risk management system is the risk management process. This process specifies how providers of high-risk AI systems must identify, assess and respond to risks. Paragraph 2 defines the main steps of this process, while paragraphs 3 and 4 contain further details about specific risk management measures. Footnote 77 Note that most terms are not defined in the AI Act. But as the risk management process in the AI Act seems to be inspired by ISO/IEC Guide 51, Footnote 78 I use or adapt many of their definitions.
a. Identification and analysis of known and foreseeable risks, Article 9(2), sentence 2, point (a)
First, risks need to be identified and analysed. Footnote 79 “Risk identification” means the systematic use of available information to identify hazards, Footnote 80 whereas “hazard” can be defined as a “potential source of harm”. Footnote 81 As the AI Act does not specify how providers should identify risks, they have to rely on existing techniques and methods (eg risk taxonomies, Footnote 82 incident databases Footnote 83 or scenario analysis Footnote 84 ). Footnote 85 It is unclear what the AI Act means by “risk analysis”. The term typically refers to both risk identification and risk estimation, Footnote 86 but this does not make sense in this context, as both steps are described separately. To avoid confusion, the legislator should arguably remove the term “analysis” from Article 9, sentence 2, point (a) or adjust point (b), as has been suggested by the Council Footnote 87 (see Section V.2.b).
Risk identification and analysis should be limited to “the known and foreseeable risks associated with each high-risk AI system”. However, the AI Act does not define the term “risk”, nor does it say when risks are “known” or “foreseeable”. I suggest using the following definitions.
“Risk” is the “combination of the probability of occurrence of harm and the severity of that harm”; Footnote 88 “harm” means any adverse effect on health, safety and fundamental rights, Footnote 89 while the “probability of occurrence of harm” is “a function of the exposure to [a] hazard, the occurrence of a hazardous event, [and] the possibilities of avoiding or limiting the harm”. Footnote 90 It is worth noting, however, that these definitions are not generally accepted and that there are competing concepts of risk. Footnote 91 In addition, the Council has suggested a clarification, Footnote 92 according to which the provision only refers to risks “most likely to occur to health, safety and fundamental rights in view of the intended purpose of the high-risk AI system”. Footnote 93
A risk is “known” if the harm has occurred in the past or is certain to occur in the future. To avoid circumventions, “known” refers to what an organisation could know with reasonable effort, not what they actually know. For example, a risk should be considered known if there is a relevant entry in one of the incident databases Footnote 94 or if a public incident report has received significant media attention.
A risk is “foreseeable” if it has not yet occurred but can already be identified. The question of how much effort organisations need to put into identifying new risks involves a difficult trade-off. On the one hand, providers need legal certainty. In particular, they need to know when they are allowed to stop looking for new risks. On the other hand, the AI Act should prevent situations where providers cause significant harm but are able to exculpate themselves by arguing that the risk was not foreseeable. If this were possible, the AI Act would fail to protect health, safety and fundamental rights. A possible way to resolve this trade-off is the following rule of thumb: the greater the potential impact of the risk, the more effort an organisation needs to put into foreseeing it. For example, it should be extremely difficult for a provider to credibly assure that a catastrophic risk was unforeseeable. Footnote 95
b. Estimation and evaluation of risks that may emerge from intended uses or foreseeable misuses, or risks that have been identified during post-market monitoring, Article 9(2), sentence 2, points (b), (c)
Next, risks need to be estimated and evaluated. Footnote 96 “Risk estimation” means the estimation of the probability of occurrence of harm and the severity of that harm. Footnote 97 As the AI Act does not specify how to estimate risks, providers have to rely on existing techniques (eg Bayesian networks and influence diagrams). Footnote 98 “Risk evaluation” means the determination of whether a risk is acceptable. Footnote 99 I discuss this question in more detail below (see Section V.3).
Risk estimation and evaluation should only cover risks “that may emerge when the high-risk AI system is used in accordance with its intended purpose and under conditions of reasonably foreseeable misuse”. Footnote 100 The terms “intended purpose” and “reasonable foreseeable misuse” are both defined in the AI Act. Footnote 101 If the system is not used as intended or is misused in an unforeseeable way, the risks do not have to be included. This ensures that the provider is only responsible for risks they can control, which increases legal certainty. To prepare this step, providers must identify potential users, intended uses and reasonably foreseeable misuses at the beginning of the risk management process. Footnote 102
Providers of high-risk AI systems also need to evaluate risks that they have identified through their post-market monitoring system. Footnote 103 This provision ensures that providers also manage risks from unintended uses or unforeseeable misuses if they have data that such practices exist. While this expands the circle of relevant risks, it does not threaten legal certainty.
Note that the Council has proposed to delete Article 9(2), sentence 2, point (b) and to add a sentence 3 instead: “The risks referred to in [paragraph 2] shall concern only those which may be reasonably mitigated or eliminated through the development or design of the high-risk AI system, or the provision of adequate technical information.” Footnote 104 These changes would limit the types of risks that providers of AI systems are responsible for compared to the original proposal by the EC.
c. Adoption of risk management measures, Article 9(2), sentence 2, point (d)
Finally, suitable risk management measures need to be adopted. Footnote 105 “Risk management measures” (also known as “risk response” or “risk treatment”) are actions that are taken to reduce the identified and evaluated risks. Paragraphs 3 and 4 contain more details about specific measures (see Section V.3).
Although the three steps are presented in a sequential way, they are meant to be “iterative”. Footnote 106 As alluded to in Section II, the risk management process needs to be repeated until all risks have been reduced to an acceptable level. After the first two steps, providers need to decide whether the risk is already acceptable. If this is the case, they can document their decision and complete the process. Otherwise, they need to move on to the third step. After they have adopted suitable risk management measures, they need to reassess the risk and decide whether the residual risk is acceptable. If it is not, they have to take additional risk management measures. If it turns out that it is not possible to reduce residual risks to an acceptable level, the development and deployment process must be stopped. Footnote 107 Although the AI Act does not reference it, the iterative process described in paragraph 2 is very similar to the one described in ISO/IEC Guide 51. Footnote 108 It is illustrated in Figure 1.
Figure 1. Overview of the risk management process described in Article 9(2) of the Artificial Intelligence Act based on the iterative process of risk assessment and risk reduction described in ISO/IEC Guide 51. Footnote 111
The risk management process needs to “run throughout the entire lifecycle of a high-risk AI system”. Footnote 109 The original EC proposal does not define “lifecycle of an AI system”, but the Council has suggested a new definition. Footnote 110 According to the Council, the risk management process also needs to be “planned” throughout the entire lifecycle. Footnote 112 In practice, providers will need to know how often and when in the lifecycle they must complete the risk management process. In the absence of an explicit requirement, providers have to rely on considerations of expediency. A sensible approach would be to perform a first iteration early on in the development process and, based on the findings of that iteration, decide how to proceed. For example, if they only identify a handful of low-probability, low-impact risks, they may decide to run fewer and less thorough iterations later in the lifecycle. However, two iterations, one during the development stage and one before deployment, Footnote 113 seem to be the bare minimum. A single iteration seems incompatible with the wording of the norm (“run throughout the entire lifecycle”).
3. Risk management measures, Article 9(3), (4)
Paragraphs 3 and 4 contain more details about the risk management measures referred to in paragraph 2, sentence 2, point (d). According to paragraph 3, the risk management measures “shall give due consideration to the effects and possible interactions resulting from the combined application of the requirements set out in … Chapter 2”. Footnote 114 Besides that, they “shall take into account the generally acknowledged state of the art, including as reflected in relevant harmonised standards or common specifications”. Footnote 115 It is worth noting that there are not yet any harmonised standards Footnote 116 or common specifications Footnote 117 on AI risk management. It is probably also too early for a “generally acknowledged state of the art”, but emerging AI risk management standards Footnote 118 and ERM frameworks Footnote 119 could serve as a starting point.
Paragraph 4 contains three subparagraphs. The first specifies the purpose of adopting risk management measures, the second lists specific measures and the third is about the socio-technical context.
The purpose of adopting risk management measures is to reduce risks “such that any residual risk … is judged acceptable”. A “residual risk” is any “risk remaining after risk reduction measures have been implemented”. Footnote 120 “Acceptable risk” (or “tolerable risk”) can be defined as the “level of risk that is accepted in a given context based on the current values of society”. Footnote 121 To determine whether a risk is acceptable, providers have to weigh the risks and benefits. Footnote 122 In general, a risk is acceptable if the benefits (clearly) outweigh the risks. However, as the AI Act is intended to protect health, safety and fundamental rights (see Section III), the amount of risk that providers can accept is limited – it is not merely a matter of their own risk appetite. Footnote 123 Weighing the risks and benefits involves many empirical uncertainties and difficult normative judgments. But as the norm does not provide any guidance and harmonised standards are still lacking, Footnote 124 providers are left to their own devices. They also cannot rely on the literature, because defining normative thresholds is still an open problem in AI ethics, Footnote 125 both for individual characteristics (eg how fair is fair enough?) and in terms of trade-offs between different characteristics (eg increasing fairness can reduce privacy). Footnote 126 Against this background, further guidance is urgently needed. Paragraph 4, subparagraph 1 further states that “each hazard as well as the overall residual risk” must be judged acceptable. In other words, providers must consider risks both individually and collectively, but only if the system “is used in accordance with its intended purpose or under conditions of reasonably foreseeable misuse”. Footnote 127 Finally, “those residual risks [that are judged acceptable] shall be communicated to the user”. Footnote 128
Providers of high-risk AI systems must adopt three types of risk management measures. These measures resemble the “three-step-method” in ISO/IEC Guide 51. Footnote 129 First, providers must design and develop the system in a way that eliminates or reduces risks as far as possible. Footnote 130 For example, to reduce the risk that a language model outputs toxic language, Footnote 131 providers could fine-tune the model. Footnote 132 Second, if risks cannot be eliminated, providers must implement adequate mitigations and control measures, where appropriate. Footnote 133 If fine-tuning the language model is not enough, the provider could use safety filters Footnote 134 or other approaches to content detection. Footnote 135 Third, they must provide adequate information and, where appropriate, training to users. Footnote 136 Risks can only be judged acceptable if all steps have been implemented to the requisite standard (eg risks have been eliminated “as far as possible”). Footnote 137 Figure 2 gives an overview of the three types of measures and illustrates how they collectively reduce risk.
Figure 2. Overview of risk management measures described in Article 9(4), subsection 2 of the Artificial Intelligence Act, inspired by ISO/IEC Guide 51. Footnote 138
Finally, when adopting the abovementioned risk management measures to reduce risks related to the use of the system, providers must give “due consideration … to the technical knowledge, experience, education, training to be expected by the user and the environment in which the system is intended to be used”. The provision acknowledges that AI systems are always embedded in their socio-technical context.
4. Testing procedures, Article 9(5)–(7)
The second component of the risk management system consists of testing procedures. Pursuant to paragraph 5, sentence 1, “high-risk AI systems shall be tested”. “Testing” can be defined as a “set of activities conducted to facilitate discovery and evaluation of properties of the test items”. Footnote 139 This typically involves the use of metrics and probabilistic thresholds. Footnote 140 Below, I discuss the “why”, “when”, “how” and “who” of testing.
Pursuant to paragraph 5, testing has three purposes. First, it is aimed at “identifying the most appropriate risk management measures”. Footnote 141 Let us revisit our example of a language model that outputs toxic language. While providers could take many different measures to reduce that risk, testing (eg using toxicity classifiers Footnote 142 ) can give them a better understanding of the risk and thereby help them adopt more appropriate measures. Second, testing shall “ensure that high-risk AI systems perform consistently for their intended purpose”. Footnote 143 AI systems often perform worse when the environment in which they are actually used differs from their training environment. This problem is known as “distributional shift”. Footnote 144 Testing can help providers detect when it is particularly likely that the system will perform poorly in the environment it is intended for (so-called “out-of-distribution detection”). Third, testing shall ensure that high-risk AI systems “are in compliance with the requirements set out in [Chapter 2]”. Footnote 145 Some of these provisions require the system to have certain properties like being “sufficiently transparent” Footnote 146 or having “an appropriate level of accuracy, robustness and cybersecurity”. Footnote 147 Testing can evaluate how well the system performs on these dimensions relative to certain benchmarks, helping providers interpret whether the current level is in fact “sufficient” or “appropriate”. Footnote 148
Paragraph 6 only refers to “AI systems”, not “high-risk AI systems”, but this seems to be the result of a mistake in the drafting of the text. The provision states that testing procedures “shall be suitable to achieve the intended purpose” and not “go beyond what is necessary to achieve that purpose”. This is essentially a restatement of the principle of proportionality. Besides that, the paragraph does not seem to have a discrete regulatory content. Presumably in light of this, the Council has proposed to substitute the provision with a reference to a new Article 54a that lays out rules on testing in real-world conditions. Footnote 149
Paragraph 7, sentence 1 specifies when providers must test their high-risk AI systems, namely “as appropriate, at any point in time throughout the development process, and, in any event, prior to the placing on the market or the putting into service”. Note that this is different from the risk management process (see Section V.2). While the risk management process needs to “run through the entire lifecycle”, Footnote 150 testing only needs to be performed “throughout the development process”. Although the formulation “as appropriate” indicates that providers have discretion as to when and how often to test their systems, testing must be performed “prior to the placing on the market or the putting into service”. Footnote 151
Paragraph 7, sentence 2 specifies how providers must test their high-risk AI systems, namely “against preliminarily defined metrics and probabilistic thresholds that are appropriate to the intended purpose of the high-risk AI system”. “Metric” includes assessment criteria, benchmarks and key performance indicators. “Probabilistic thresholds” represent a special kind of metric evaluating a property on a probabilistic scale with one or more predefined thresholds. It is not possible to make any general statements as to which metric or probabilistic threshold to use, mainly because their appropriateness is very context-specific and because there are not yet any best practices. Providers will therefore have to operate under uncertainty and under the assumption that metrics they have used in the past might not be appropriate in the future. Presumably, this is the reason why the norm speaks of “preliminarily defined metrics”.
The norm does not specify who must perform the testing. As discussed in Section IV, it applies to providers of high-risk AI systems. But do providers need to perform the testing themselves or can they outsource it? I expect that many providers want to outsource the testing or parts thereof (eg the final testing before placing the system on the market). In my view, this seems to be unproblematic, as long as the provider remains responsible for meeting the requirements. Footnote 152
5. Special rules for children and credit institutions, Article 9(8), (9)
Paragraph 8 contains special rules for children. The Council has specified this as “persons under the age of 18”. Footnote 153 When implementing the risk management system, “specific consideration shall be given to whether the high-risk AI system is likely to be accessed by or have an impact on children”. Children take a special role in the AI Act because they are particularly vulnerable and have specific rights. Footnote 154 Providers of high-risk AI systems must therefore take special measures to protect them.
Paragraph 9 contains a collusion rule for credit institutions. As credit institutions are already required to implement a risk management system, Footnote 155 one might ask how the AI-specific requirements relate to the credit institution-specific ones. Paragraph 9 clarifies that the AI-specific requirements “shall be part” of the credit institution-specific ones. In other words, Article 9 complements existing risk management systems; it does not replace them. In light of this, the Council has suggested extending paragraph 9 to any provider of high-risk AI systems that is already required to implement a risk management system. Footnote 156
But what happens if providers of high-risk AI systems do not comply with these requirements? The next section gives an overview of possible enforcement mechanisms.
VI. Enforcement
In this section, I describe ways in which Article 9 can be enforced. This might include administrative, civil and criminal enforcement measures.
Providers of high-risk AI systems that do not comply with Article 9 can be subject to administrative fines of up to €20 million or, if the offender is a company, up to 4% of its total worldwide annual turnover for the preceding financial year, whichever is higher. Footnote 157 (The Council has proposed to limit this fine in case of a small and medium-sized enterprise to 2% of its total worldwide annual turnover for the preceding financial year. Footnote 158 ) The AI Act only contains high-level guidelines on penalties (eg how to decide on the amount of administrative fines Footnote 159 ); the details will be specified by each Member State. Footnote 160 In practice, I expect administrative fines to be significantly lower than the upper bound, similar to the GDPR. Footnote 161 Before imposing penalties and administrative fines, national competent authorities Footnote 162 will usually request providers of high-risk AI systems to demonstrate conformity with the requirements set out in Article 9. Footnote 163 Supplying incorrect, incomplete or misleading information can entail further administrative fines. Footnote 164
Providers of high-risk AI systems might also be subject to civil liability. First, the provider might be held contractually liable. If a contracting party of the provider is harmed, then this party might claim compensation from the provider. This will often depend on the question of whether complying with Article 9 is a contractual accessory obligation. Second, there might be a tort law liability. Footnote 165 If a high-risk AI system harms a person, that person may claim compensation from the provider of that system. In some Member States, this will largely depend on the question of whether Article 9 protects individuals (see Section III). Footnote 166 Third, there might be an internal liability. If a company has been fined, it might claim recourse from the responsible manager. Footnote 167 This mainly depends on the question of whether not implementing a risk management system can be seen as a breach of duty of care.
Finally, Article 9 is not directly enforceable by means of criminal law. Although the AI Act does not mention any criminal enforcement measures, violating Article 9 might still be an element of a criminal offence in some Member States. For example, a failure to implement a risk management system might constitute negligent behaviour. Footnote 168
VII. Summary and conclusion
This article has analysed Article 9, the key risk management provision in the AI Act. Section II gave an overview of the regulatory concept behind the norm. I argued that Article 9 shall ensure that providers of high-risk AI systems identify risks that remain even if they comply with the other requirements set out in Chapter 2 and take additional measures to reduce them. Section III determined the purpose of Article 9. It seems uncontroversial that the norm is intended to improve the functioning of the internal market and protect the public interest. But I also raised the question as to whether the norm also protects certain individuals. Section IV determined the norm’s scope of application. Materially and personally, Article 9 applies to providers of high-risk AI systems. Section V offered a comprehensive interpretation of the specific risk management requirements. Paragraph 1 contains the central requirement, according to which providers of high-risk AI systems must implement a risk management system, while paragraphs 2–7 specify the details of that system. The iterative risk management process is illustrated in Figure 1, while Figure 2 shows how different risk management measures can collectively reduce risk. Paragraphs 8 and 9 contain special rules for children and credit institutions. Section VI described ways in which these requirements can be enforced, in particular via penalties and administrative fines as well as civil liability.
Based on my analysis in Section V, I suggest three amendments to Article 9 (or specifications in harmonised standards). First, I suggest adding a passage on the organisational dimension of risk management, similar to the Govern function in the NIST AI Risk Management Framework, Footnote 169 which is compatible with existing best practices like the Three Lines of Defence model. Footnote 170 Second, I suggest adding a requirement to evaluate the effectiveness of the risk management system. The most obvious way to do that would be through an internal audit function. Third, I suggest clarifying that the risk management system is intended to reduce individual, collective and societal risks, Footnote 171 not just risks to the provider of high-risk AI systems.
The article makes three main contributions. First, by offering a comprehensive interpretation of Article 9, it helps providers of high-risk AI systems to comply with the risk management requirements set out in the AI Act. Although it will take several years until compliance is mandatory, such providers may want to know as early as possible what awaits them. Second, the article has suggested ways in which Article 9 can be amended. And third, it informs future efforts to develop harmonised standards on AI risk management in the EU.
Although my analysis focuses on the EU, I expect it to be relevant for policymakers worldwide. In particular, it might inform regulatory efforts in the USA Footnote 172 and UK, Footnote 173 especially as risk management as a governance tool is not inherently tied to EU law Footnote 174 and there is value in compatible regulatory regimes. The UK has already warned against a global fragmentation, Footnote 175 while the USA and the EU have initiated a dialogue on AI risk management intended to support regulatory and standardisation efforts. Footnote 176
Acknowledgments
I am grateful for valuable comments and feedback from Leonie Koessler, Markus Anderljung, Christina Barta, Christoph Winter, Robert Trager, Noemi Dreksler, Eoghan Stafford, Jakob Mökander, Elliot Jones, Andre Barbe, Risto Uuk, Alexandra Belias, Haydn Belfied, Anthony Barrett, James Ginns, Henry Fraser and Emma Bluemke. I also thank the participants of a seminar hosted by the Centre for the Governance of AI in July 2022. All remaining errors are my own.
Competing interests
The author declares none.
Open access
