7.3.1 Material Scope of Application – The Processing of Personal Data

7.3.2 Personal Scope of Application – Controllers and Processors

7.4.1 The Versatility of AI Systems v. the Necessity and Compatibility Tests

7.4.1.1 Lawfulness and Purpose Limitation at the Heart of the GDPR

In order to prevent function creep, Article 5(1)a introduces the principle of “lawfulness,” which requires controllers to justify their processing operations using one of the six lawful grounds listed in Article 6. These include not only the consent of the data subject – often erroneously perceived as the only option – but also the alternatives such as the “performance of a contract” or the “legitimate interests of the controller.” Relying on any of these lawful grounds (except for consent) requires the controller to assess and demonstrate that the processing at stake is “objectively necessary” to achieve the substance of that lawful ground. In other words, there is no other, less-intrusive way to meet that objective. As recently illustrated by the Irish regulator’s decision in the Meta Ireland case,Footnote ⁴⁵ the processing of Facebook and Instagram users’ personal data for the purpose of delivering targeted advertising is not, for instance, objectively necessary to fulfil the essence of the contractual relationship between these platforms and their users.Footnote ⁴⁶ As a result, the processing cannot be based on Article 6(1)b, and it has to rely on another lawful ground. Consent, on the other hand, must be “freely given, specific, informed and unambiguous,” thereby undermining its validity when obtained in a scenario that involves unbalanced power or information asymmetries, such as when given by an employee to its employer.Footnote ⁴⁷

With that same objective in mind, Article 5(1)b lays down the principle of “purpose limitation,” according to which personal data shall be “collected for specified, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes.”Footnote ⁴⁸ In practice, this requires controllers to, first, determine the exact reasons why personal data are collected and, then, assess the compatibility of every subsequent processing activity in light of the purposes that were specified at the collection stage. Doing so requires to take into account various criteria such as, for instance, the context in which the personal data have been collected and the reasonable expectations of the data subjects.Footnote ⁴⁹ While compatible further processing can rely on the same lawful ground used to justify the collection, incompatible processing must specify a new legal basis. Reusing a postal address originally collected to deliver goods purchased online for marketing purposes is a straightforward example of an incompatible further processing. The purposes specified during the collection also serve as the basis to assess the amount of personal data collected (i.e., “data minimization”), the steps that must be taken to ensure their correctness (i.e., “accuracy”) and their retention period (i.e., “storage limitation”).

Lawfulness and purpose limitation are strongly interconnected, as the purposes specified for the collection will influence the outcome of both the necessity test required when selecting the appropriate lawful ground – with the exception of consent, for which the purposes delimit what can and cannot be done with the data – and the compatibility assessment that must be conducted prior to each further processing. Ensuring compliance with these principles therefore calls for a separate analysis of each “personal data – purpose(s) – lawful ground” triad, acting as a single, indissociable whole (see Figure 7.3).

Figure 7.3

Lawfulness and purpose limitation, combined

Severing the link between these three elements would empty Articles 5(1)a and 5(1)b from their substance and render any necessity or compatibility assessment meaningless. Whether a webshop can rely on its legitimate interests (Article 6(1)f) to profile its users and offers targeted recommendations, for instance, heavily depends on the actual personal data used to tailor their experience, and therefore the intrusiveness of the processing.Footnote ⁵⁰

7.4.2 The Complexity of AI Systems v. Transparency and Explainability

7.4.3 The Dynamicity of AI v. the Risk-Based Approach

7.4.3.1 Accountability, Responsibility, Data Protection by Design and DPIAs

Compared to its predecessor,Footnote ⁷³ one of the main objectives of the GDPR was to move away from compliance as a mere ticking-the-box exercise – or window dressingFootnote ⁷⁴ – by incentivizing controllers to take up a more proactive role in the implementation of appropriate measures to protect individuals’ rights and freedoms. This led to the abolition of the antique, paternalistic obligation for controllers to notify their processing operations to national regulators in favor of a more flexible approach articulated around the obligation to maintain a record of processing activities (Article 30), to notify data breaches to competent authorities and the affected data subjects (Articles 33 and 34) and to consult the former in cases where a data protection impact assessment (“DPIA”) indicates that the processing would result in a high risk in the absence of measures taken by the controller to mitigate the risk (Article 36). The underlying idea was to responsibilize controllers by shifting the burden of analyzing and mitigating the risks to data subject’s rights and freedoms onto them. Known as the “risk-based approach,” it ensures both the flexibility and scalability needed for the underlying rules to remain pertinent in a wide variety of scenarios. As noted in legal literature, the risk-based approach “provides a way to carry out the shift to accountability that underlies much of the data protection reform, using the notion of risk as a reference point in light of which we can assess whether the organisational and technical measures taken by the controller offer a sufficient level of protection.”Footnote ⁷⁵

The combined reading of Articles 5(2) (“accountability”), 24(1) (“responsibility”), and 25(1) (“data protection by design”) now requires controllers to take into account the state of the art, the cost of implementation, and the nature, scope, context, and purposes as well as the risks posed by the processing. They should implement, both at the time of determination of the means for processing and at the time of the processing itself, appropriate technical and organizational measures to ensure and demonstrate compliance with the Regulation. In other words, they must act responsibly as of the design stage, and throughout the entire data processing lifecycle. Data protection-specific risks are usually addressed in a DPIA, which should at least provide a detailed description of the relevant processing activities, an assessment of their necessity and proportionality, as well as an inventory of the risks and corresponding mitigation strategies (see Figure 7.4).Footnote ⁷⁶ While Article 35(1) obliges controllers to conduct a DPIA for processing activities that are “likely to result in a high risk for rights and freedoms of natural persons,” such an exercise, even if succinct, is also considered as best practice for all controllers regardless of the level of risk.Footnote ⁷⁷

Figure 7.4

Overview of the main steps of a Data Protection Impact Assessment

8.4.1 Software as a Product?

Article 1 of the Product Liability Directive stipulates that the producer is liable for damage caused by a defect in the product. Technology and industry, however, have evolved drastically over the last decades. The division between products and services is no longer as clear-cut as it was. Producing products and providing services are increasingly intertwined.Footnote ⁵⁹ In this regard, the question arises whether software is a product or instead is provided as a service, and thus falling outside the scope of the PLD.Footnote ⁶⁰ Software and AI systems merit specific attention in respect of product liability. Software is essential to the functioning of a large number of products and affects their safety. It is integrated into products but it can also be supplied separately to enable the use of the product as intended. Neither a computer nor a smartphone would be of particular use without software. The question whether stand-alone software can be qualified as a product within the meaning of the Product Liability Directive or implementing national legislation has already attracted a lot of attention, both in academic scholarshipFootnote ⁶¹ and in policy initiatives.Footnote ⁶² That is because software is a collection of data and instructions that is imperceptible to the human eye.Footnote ⁶³

Unclarity remains as to whether software is (im)movable and/or a (in)tangible good.Footnote ⁶⁴ The Belgian Product Liability Act – implementing the PLD – stipulates that the regime only concerns tangible goods.Footnote ⁶⁵ Although the Belgian Court of Cassation and/or the European Court of Justice have not yet ruled on the matter, the revised PLD specifically qualifies software and digital manufacturing files as products.Footnote ⁶⁶ The inclusion of software is rather surprising, yet essential.Footnote ⁶⁷ Recital (13) of the revised PLD states that it should not apply to “free and open-source software developed or supplied outside the course of a commercial activity” in order not to hamper innovation or research. However, where software is supplied in exchange for a price or personal data is provided in the course of a commercial activity (i.e., for other purposes than exclusively improving the security, compatibility or interoperability of the software), the Directive should apply.Footnote ⁶⁸ Regardless of the qualification of software, the victim of an accident involving an AI system may have a claim against the producer of a product incorporating software such as an autonomous vehicle, a robot used for surgery or a household robot. Software steering the operations of a tangible product could be considered as a part or component of that product.Footnote ⁶⁹ This means that an autonomous vehicle or material robot used for surgery would be considered as a product in the sense of the Product Liability Directive and can be defective if the software system it uses is not functioning properly.Footnote ⁷⁰

8.4.2 “Defective” Product

Liability under the Product Liability Directive requires a “defect” in the product. A product is defective when it does not provide the safety that a person is entitled to expect, taking all circumstances into account (the so-called “consumer expectations test” as opposed to the “risk utility test”).Footnote ⁷¹ This does not refer to the expectations of a particular person but to the expectations of the general publicFootnote ⁷² or the target audience.Footnote ⁷³ Several elements can be used to determine the legitimate expectations regarding the use of AI systems. These include the presentation of the product, the normal or reasonably foreseeable use of it and the moment in time when the product was put into circulation.Footnote ⁷⁴ This enumeration of criteria, however, is not exhaustive as other factors may play a role as well.Footnote ⁷⁵ Especially the criterion of the presentation of the product is important for manufacturers of autonomous vehicles or medical robots. That is because they often tend to market their products explicitly as safer than existing alternatives. The presentation of the product may on the other hand also provide an opportunity for manufacturers of AI systems to reduce their liability risk through appropriate warnings and user information. Nevertheless, it remains uncertain how technically detailed or accessible such information should be.Footnote ⁷⁶ The revised PLD also refers to the legitimate safety expectations.Footnote ⁷⁷ A product is deemed defective if it fails to “provide the safety which the public at large is entitled to expect, taking all circumstances into account.”Footnote ⁷⁸ The non-exhaustive list of such circumstances that allow to assess the product’s defectiveness is expanded and also includes “the effect on the product of any ability to continue to learn after deployment.”Footnote ⁷⁹ It should, however, be noted that the product cannot be considered defective for the sole reason that a better product, including updates or upgrades to a product, is already or subsequently placed on the market or put into service.Footnote ⁸⁰

That being said, the criterion of legitimate expectations remains very vague (and problematicFootnote ⁸¹). It gives judges a wide margin of appreciation.Footnote ⁸² As a consequence, it is difficult to predict how this criterion will and should be applied in the context of AI systems.Footnote ⁸³ The safety expectations will be very high for AI systems used in high-risk contexts such as healthcare or mobility.Footnote ⁸⁴ At the same time, however, the concrete application of this test remains difficult for AI systems because of their novelty, the complexity to compare these systems with human or technological alternatives and the characteristics of autonomy and opacity.Footnote ⁸⁵ The interconnectivity of products and systems also makes it hard to identify the defect. Sophisticated AI systems with self-learning capabilities also raise the question of whether unpredictable deviations in the decision-making process can be treated as defects. Even if they constitute a defect, the state-of-the-art defenseFootnote ⁸⁶ may eventually apply. The complexity and the opacity of emerging digital technologies such as AI systems further complicate the chance for the victim to discover and prove the defect and/or causation.Footnote ⁸⁷ In addition, there is some uncertainty on how and to what extent the Product Liability Directive applies in the case of certain types of defects, for example, those resulting from weaknesses in the cybersecurity of the product.Footnote ⁸⁸ It has already been mentioned that the revised PLD establishes a presumption of defectiveness under certain conditions to remedy these challenges.Footnote ⁸⁹

8.4.3 The Concept of Fault in the AI Liability Directive

In addition to the challenging application of “traditional” existing tort law concepts in an AI context, the introduction of new legislation in this field may also contain notions that are unclear. This unclarity could affect legal certainty, especially considering the remaining importance of national law. We will illustrate this with the requirement of “fault” as proposed in the AI Liability Directive.

It has already been mentioned that Article 4.1 of the AI Liability Directive contains a rebuttable presumption of a “causal link between the fault of the defendant and the output produced by the AI system or the failure of the AI system to produce an output.” The fault of the defendant has to be proven by the claimant according to the applicable EU law or national rules. Such a fault can be established, for example, “for non-compliance with a duty of care pursuant to the AI Act.”Footnote ⁹⁰ The relationship between the notions of “fault” and “duty of care” under the AI Liability Directive, and especially in Article 4, is unclear and raises interpretation issues.Footnote ⁹¹ The AI Liability Directive uses the concept of “duty of care” at several occasions. Considering that tort law is still to a large extent national, the reliance on the concept of “duty of care” in supranational legislation is rather surprising. A “duty of care” is defined as “a required standard of conduct, set by national or Union law, in order to avoid damage to legal interests recognized at national or Union law level, including life, physical integrity, property and the protection of fundamental rights.”Footnote ⁹² It refers to how a reasonable person should act in a specific situation, which also “ensure[s] the safe operation of AI systems in order to prevent damage to recognized legal interests.”Footnote ⁹³ In addition to the fact that the content of a duty of care will ultimately have to be determined by judges, a more conceptual issue arises as well. That is because the existence of a generally applicable positive duty of care has already been contested, for instance, in Belgium. Kruithof concludes that case law and scholarship commonly agree that no breach of a “pre-existing” duty is required for a fault to be established. As noted by Kruithof, what is usually referred to as the generally required level or the duty of care, “is therefore more properly qualified not as a legal duty or obligation, but merely a standard of behavior serving as the yardstick for judging whether an act is negligent or not for purposes of establishing liability.”Footnote ⁹⁴ However, Article 4.1 (a) seems to equate the “fault” with the noncompliance with a duty of care, thereby implicitly endorsing the view that the duty of care consists in a standalone obligation. This does not necessarily fit well in some national tort law frameworks, and may thus cause interpretation issues and fragmentation.Footnote ⁹⁵

Article 1.3 (d) of the AI Liability Directive mentions that the Directive will not affect “how fault is defined, other than in respect of what is provided for in Articles 3 and 4.” A fault under Belgian law (and by extension other jurisdictions) consists of both a subjective component and an objective component. The (currently still applicable) subjective component requires that the fault can be attributed to the free will of the person who has committed it (“imputability”), and that this person generally possesses the capacity to control and to assess the consequences of his or her conduct (“culpability”).Footnote ⁹⁶ This subjective element does, however, not seem to be covered by the AI Liability Directive. This raises the question whether the notion of “fault,” as referred to in the Articles 3 and 4, requires such a subjective element to be present and/or allows for national law to require this. The minimal harmonization provision of Article 1.4 does not answer this question.Footnote ⁹⁷ The objective component of a fault refers to the wrongful behavior in itself. Belgian law traditionally recognizes two types of wrongdoings, namely a violation of a specific legal rule of conductFootnote ⁹⁸ and the breach of a standard of care.Footnote ⁹⁹ Under Belgian law, a violation of a standard of care requires that it was reasonably foreseeable for the defendant that his or her conduct could result in some kind of damage.Footnote ¹⁰⁰ This means that a provider of a high-risk AI system would commit a fault when he or she could reasonable foresee that a violation of a duty of care following provisions of the AI Act would result in damage. However, it is unclear whether the notion of a “duty of care” as relied upon in the AI Liability Directive also includes this requirement of foreseeability or, instead, whether it is left to national (case) law to determine the additional modalities under which a violation of a “duty of care” can be established.Footnote ¹⁰¹

9.3.1 Restrictive Agreements

Restrictive agreements come in two types: they are horizontal when entered into between competitors (“collusion”) and vertical when entered into between firms at different levels of the supply chain (e.g., supplier and distributor). An agreement does not require a contract; more informal types of understanding between parties (“concerted practices”) also fall under Article 101 TFEU.Footnote ³⁰ To be illegal, the common understanding must have the object or effect of restricting competition. According to the case law, “by object” restrictions are those types of coordination that “can be regarded, by their very nature, as being harmful to the proper functioning of normal competition.”Footnote ³¹ Given that such coordination reveals, in itself, a sufficient degree of harm to competition, it is not necessary to assess its effects.Footnote ³² “By effect” restrictions do require such an assessment. In general, horizontal agreements are more likely to fall into the “by object” category (price-fixing being the typical example), while vertical agreements are more likely to be categorized as “by effect” (e.g., recommending retail prices). Let us look at horizontal and vertical agreements in turn.

9.3.2 Abuse of Dominance

Abusive conduct comes in two types: it is exclusionary when it indirectly harms consumers by foreclosing competitors from the market and exploitative when it directly harms consumers, for example, by charging excessive prices. I discuss the main algorithmic concern under each category of abuse, that is, discriminatory ranking and personalized pricing, respectively. While I focus on abusive conduct, remember that such conduct only infringes Article 102 TFEU if the firm in question is also in a dominant position.

10.3.1 Challenges to the Fundamental Underpinnings of Consumer Law

Historically, the emergence of consumer law is linked to the development of a consumer society. In fact, this legal domain has been referred to as a “reflection of the consumer society in the legal sphere.”Footnote ²⁰ The need for legal rules to protect those who consume, was indeed felt more urgently when consumption, above the level of basic needs, became an important aspect of life in society.Footnote ²¹ The trend to attach increasing importance to consumption had been ongoing for several centuries,Footnote ²² but the increasing affluence, the changing nature of the way business was conducted, and the massification of consumption, all contributed to a body of consumer protection rules being adopted, mainly from the 1950s.Footnote ²³ More consumption was thought to equal more consumer welfare and more happiness. Consumer protection law in Europe first emerged at national level.Footnote ²⁴ It was only from the 1970s on that European institutions started to develop an interest in consumer protection and that the first consumer protection programs followed.Footnote ²⁵ The first binding instruments were adopted in the 1980s, and consisted mostly of minimum harmonization instruments. This means that member states are allowed to maintain or adopt more protective provisions, as long as the minimum standards imposed by the harmonization instrument are respected. From 2000 onwards, the shift to maximum harmonization in the European consumer protection instruments reduced the scope for a national consumer (protection) policy.

While originally the protection of a weaker consumer was central in many national regimes, the focus in European consumer law came to be on the rational consumer whose right to self-determination (private autonomy) on a market must be guaranteed.Footnote ²⁶ This right to self-determination can be understood as the right to make choices in the (internal) market according to one’s own preferencesFootnote ²⁷ thereby furthering the realization of the internal market.Footnote ²⁸ This focus on self-determination presupposes a consumer capable of making choices and enjoying the widest possible options to choose from.Footnote ²⁹ EU consumer law could thus be described as the guardian of the economic rights of the nonprofessional player in the (internal) market. Private autonomy and contractual freedom should in principle suffice to protect these economic rights and to guarantee a bargain in accordance with one’s own preferences, but consumer law acknowledges that the preconditions for such a bargain might be absent, especially due to information asymmetries between professional and nonprofessional players.Footnote ³⁰ Information was and is therefore used as the main corrective mechanism in EU consumer law.Footnote ³¹ Further reaching intervention – for example, by regulating the content of contracts – implies a greater intrusion into private autonomy and is therefore only a subsidiary protection mechanism.Footnote ³²

AI and the far-reaching possibilities of personalization and manipulation it entails, especially when used in combination with personal data, now challenges the assumption of the rational consumer with its “own” preferences even more fundamentally. The efficiency of information as a means of protection had already been questioned before the advent of new technologies,Footnote ³³ but the additional complexity of AI leaves no doubt that the mere provision of information will not be a solution to the ever increasing information asymmetry and risk of manipulation. The emergence of an “attention economy” whereby companies strive to retain consumers’ attention in order to generate revenue based on advertising and data gathering, furthermore also makes clear that “more consumption is more consumer welfare” is an illusion.Footnote ³⁴ The traditional underpinnings of consumer law therefore need revisiting.

10.3.2 Challenges to the Basic Concepts of Consumer Law

European consumer law uses the abstract concept of the “average” consumer as a benchmark.Footnote ³⁵ This is a “reasonably well informed and reasonably observant and circumspect” consumer;Footnote ³⁶ a person who is “reasonably critical […], conscious and circumspect in his or her market behaviour.”Footnote ³⁷ This benchmark, as interpreted by the Court of Justice of the European Union, has been criticized for not taking into account cognitive biases and limitations of the consumers and for allowing companies to engage in exploitative behavior.Footnote ³⁸ AI now creates exponential possibilities to exploit these cognitive biases and the need to realign the consumer benchmark with the realities of consumer behavior is therefore even more urgent. There is furthermore some, but only limited, attention to the vulnerable consumer in EU consumer law.Footnote ³⁹ Thus, the Unfair Commercial Practices Directive, for example, allows to assess a practice from the perspective of the average member of a group of vulnerable consumers even if the practice was directed to a wider group, if the trader could reasonably foresee that the practice would distort the behavior of vulnerable consumers.Footnote ⁴⁰ The characteristics the UCPD identifies to define vulnerability (such as mental or physical infirmity, age, or credulity) are however not particularly helpful nor exhaustive in a digital context. Interestingly, however, the Commission Guidance does stress that vulnerability is not a static concept, but a dynamic and situational conceptFootnote ⁴¹ and that the characteristics mentioned in the directive are indicative and non-exhaustive.Footnote ⁴² The literature has however rightly argued that a reinterpretation of the concept of vulnerability will not be sufficient to better protect consumers in a digital context. It is submitted that in digital marketplaces, most, if not all consumers are potentially vulnerable; digitally vulnerable and susceptible “to (the exploitation of) power imbalances that are the result of increasing automation of commerce, datafied consumer-seller relations and the very architecture of digital marketplaces.”Footnote ⁴³ AI and digitalization thus create a structural vulnerability that requires a further reaching intervention than just to reinterpret vulnerability.Footnote ⁴⁴ More attention to tackling the sources of digital vulnerability and to the architecture of digital marketplaces is hence definitely necessary.Footnote ⁴⁵

10.3.3 Challenges to the Silo Approach to Consumer Law

Consumer law has developed in parallel with competition law and data protection law but, certainly in digital markets, it is artificial – also in terms of enforcement – to strictly separate these areas of the law.Footnote ⁴⁶ The use of AI often involves the use of (personal) consumer data and concentration in digital markets creates a risk of abuses of personal data also to the detriment of consumers. Indeed, there are numerous and frequent instances where the same conduct will be covered simultaneously by consumer law, competition law, and data protection law.Footnote ⁴⁷ The German Facebook case of the BundesgerichtshofFootnote ⁴⁸ is just one example where competition law (abuse of dominant position) was successfully invoked also to guarantee consumer’s choice in the data they want to share and in the level of personalization of the services provided.Footnote ⁴⁹ There is certainly a need for more convergence and a complementary application of these legal domains, rather than artificially dividing them, especially when it comes to enforcement. The case law allowing consumer protection organizations to bring representative actions on the basis of consumer law (namely unfair practices or unfair contract terms), also for infringements of data protection legislation, is therefore certainly to be welcomed.Footnote ⁵⁰

10.4.1 The Unfair Commercial Practices Directive

The UCPD is a maximum harmonization instrument that regulates unfair commercial practices occurring before, during and after a B2C transaction. It has a broad scope of application and the combination of open norms and a blacklist of practices that are prohibited in all circumstances allows it to tackle a wide range of unfair business practices, also when these practices result from the use of AI.Footnote ⁵⁴ Practices are unfair, according to the general norm, if they are contrary to the requirements of “professional diligence and are likely to materially distort the economic behaviour of the average consumer.”Footnote ⁵⁵ The UCPD furthermore prohibits misleading and aggressive practices. Misleading practices are actions or omissions that deceive or are likely to deceive and cause the average consumer to make a transactional decision they would not have taken otherwise.Footnote ⁵⁶ Aggressive practices are practices that entail the use of coercion or undue influence which significantly impairs the average consumer’s freedom of choice and causes them to make a transactional decision they would not have taken otherwise.Footnote ⁵⁷

The open norms definitely offer some potential to combat the use of AI to manipulate consumers, either using the general norm or the prohibition of misleading or aggressive practices.Footnote ⁵⁸ However, the exact application and interpretation of these open norms makes the outcome of such cases uncertain.Footnote ⁵⁹ When exactly does the use of AI amount to “undue influence,” how is the concept of the “average consumer” to be used in a digital context; when exactly does personalized advertising become misleading. We make these problems more concrete in our analysis of dark patterns below (Section 10.5). More guidance on the application of these open norms could make the application to AI-based practices easier.Footnote ⁶⁰ Additional blacklisted practices could also provide more legal certainty.

10.4.2 Consumer Rights Directive

The CRD – also a maximum harmonization directiveFootnote ⁶¹ – regulates the information traders must provide to consumers when contracting, both for on premises contracts and for distance and doorstep contracts. In addition, it regulates the right of withdrawal from the contract. The precontractual information requirements are extensive and they include an obligation to provide information about the main characteristics and total price of goods or services; about the functionality and interoperability of digital content and digital services, and the duration and conditions for termination of the contract.Footnote ⁶² However, as Ebers mentions, these obligations are formulated quite generally, making it difficult to concretize their application to AI systems.Footnote ⁶³ The Modernization directiveFootnote ⁶⁴ – adopted to “modernize” a number of EU consumer protection directives in view of the development of digital toolsFootnote ⁶⁵ – introduced a new information obligation for personal pricing.Footnote ⁶⁶ Art. 6 (1) (ea) of the modernized CRD now requires the consumer to be informed that the price was personalized on the basis of automated decision-making. There is however no obligation to reveal the algorithm used nor its methodology; neither is there an obligation to reveal how the price was adjusted for a particular consumer.Footnote ⁶⁷ This additional information obligation has therefore been criticized for being too narrow as it hinders the finding of price discrimination.Footnote ⁶⁸

10.4.3 Unfair Contract Terms Directive

The UCTD in essence requires contract terms to be drafted in plain, intelligible language and the terms must not cause a significant imbalance in the parties’ rights and obligations, to the detriment of the consumerFootnote ⁶⁹ Contract terms that do not comply with these requirements can be declared unfair and therefore nonbinding.Footnote ⁷⁰ The directive has a very broad scope of application and applies to (not individually negotiated) clauses in contracts between sellers/suppliers and consumers “in all sectors of economic activity.”Footnote ⁷¹ It does not require that the consumer provides monetary consideration for a good or service. Contracts whereby the consumer “pays” with personal data or whereby the consideration provided consists in consumer generated content and profiling are also covered.Footnote ⁷² It is furthermore a minimum harmonization directive, so stricter national rules can still apply.Footnote ⁷³

The UCTD can help consumers to combat unfair clauses (e.g., exoneration clauses, terms on conflict resolution, terms on personalization of the service, terms contradicting the GDPR)Footnote ⁷⁴ in contracts with businesses that use AI. It could also be used to combat untransparent personalized pricing whereby AI is used. In principle, the UCTD does not allow for judges to control the unfairness of core contract terms (clauses that determine the main subject matter of the contract), nor does it allow to check the adequacy of price and remuneration.Footnote ⁷⁵ This is however only the case if these clauses are transparent.Footnote ⁷⁶ The UCTD could furthermore also be invoked if AI has been used to personalize contract terms without disclosure to the consumer.Footnote ⁷⁷ Unfair terms do not bind the consumer and may even lead to the whole contract being void if the contract cannot continue to exist without the unfair term.Footnote ⁷⁸

10.4.4 Consumer Sales Directive and Digital Content and Services Directive

When AI is the subject matter of the contract, the new Consumer Sales Directive 2019/771 (“CSD”) and Digital Content and Services Directive 2019/770 (“DCSD”), provide the consumer with remedies in case the AI application fails. The CSD will apply when the digital element – provided under the sales contract – is thus incorporated or connected with the good that the absence of the digital element would prevent the good from performing its function.Footnote ⁷⁹ If this is not the case, the DCSD will apply. Both directives provide for a similar – but not identical – regime that determines the requirements for conformity and the remedies in case of nonconformity. These remedies include specific performance (repair or replacement in case of a good with digital elements), price reduction and termination. Damages caused by a defect in an AI application continue to be governed by national law. The directives also provide for an update obligation (including security updates) for the seller of goods with digital elements and for the trader providing digital content or services.Footnote ⁸⁰

10.4.5 Digital Markets Act and Digital Services Act

The Digital Markets Act (“DMA”), which applies as of May 2, 2023Footnote ⁸¹ aims to maintain an open and fair online environment for businesses users and end users by regulating the behavior of large online platforms, known as “gatekeepers,” which have significant influence in the digital market and act as intermediaries between businesses and customers.Footnote ⁸² Examples of such gatekeepers are Google, Meta, and Amazon. The regulation has only an indirect impact on the use of AI, as it aims to prevent these gatekeepers from engaging in unfair practices, which give them significant power and control over access to content and services.Footnote ⁸³ Such practices may involve the use of biased or discriminatory AI algorithms. The regulation imposes obligations on gatekeepers such as providing the ability for users to uninstall default software applications on the operating system of the gatekeeper,Footnote ⁸⁴ a ban on self-preferencing,Footnote ⁸⁵ and the obligation to provide data on advertising performance and ad pricing.Footnote ⁸⁶ The DMA certainly provides for additional consumer protection, but it does so indirectly, by mainly regulating the relationship between platforms and business users and by creating more transparency. Consumer rights are not central in the DMA and this is also apparent from the lack of involvement of consumers and consumer organizations in the DMA’s enforcement.Footnote ⁸⁷

The Digital Services Act (“DSA”),Footnote ⁸⁸ which applies as of February 17, 2024,Footnote ⁸⁹ establishes a harmonized set of rules on the provision on online intermediary services and aims to ensure a safe, predictable, and trustworthy online environment.Footnote ⁹⁰ The regulation mainly affects online intermediaries (including online platforms), such as online marketplaces, online social networks, online travel and accommodation platforms, content-sharing platforms, and app stores.Footnote ⁹¹ It introduces additional transparency obligations, including advertising transparency requirements for online platformsFootnote ⁹² and a ban on targeted advertisement of minors based on profilingFootnote ⁹³ as well as a ban on targeted advertising based on profiling using special categories of personal data, such as religious belief or sexual orientation.Footnote ⁹⁴ It also introduces recommender system transparency for providers of online platforms.Footnote ⁹⁵ The regulation furthermore obliges very large online platforms to carry out a risk assessment of their services and systems, including their algorithmic systems.Footnote ⁹⁶

10.4.6 Artificial Intelligence Act

The Artificial Intelligence Act (“AI Act”) Act, adopted June 13, 2024, provides harmonized rules for “the placing on the market, the putting into service and the use of AI systems in the Union.”Footnote ⁹⁷ It uses a risk-based methodology to classify certain uses of AI systems as entailing a low, high, or unacceptable risk.Footnote ⁹⁸ AI practices that pose an unacceptable risk are prohibited, including subliminal techniques that distort behavior and cause significant harm.Footnote ⁹⁹ The regulation foresees penalties for noncomplianceFootnote ¹⁰⁰ and establishes a cooperation mechanism at European level (the so-called European Artificial Intelligence Board), composed of representatives from the Member States and the Commission, to ensure enforcement of the provisions of the AI Act across Europe.Footnote ¹⁰¹ Concerns have been expressed whether the AI Act is adequate to also tackle consumer protection concerns. It has been argued that the list of “high-risk” applications and the list of forbidden AI practices does not cover all problematic AI applications or practices for consumers.Footnote ¹⁰² Furthermore, the sole focus on public enforcement and the lack of appropriate individual rights for consumers and collective rights for consumers organization to ensure an effective enforcement has been criticized.Footnote ¹⁰³

10.5.1 The Concept of Dark Patterns

The OECD Committee on Consumer Policy uses the following working definition of dark patterns:

A universally accepted definition is lacking, but dark patterns can be described by their common features involving the use of hidden, subtle, and often manipulative designs or marketing tactics that exploit consumer biases, vulnerabilities, and preferences to benefit the business or provider of intermediary services that presents the information that may not align with the consumer’s own preferences or best interest.Footnote ¹⁰⁵ Examples of such marketing practices include (i) false hierarchy (the button for the business’ desired outcome is more prominent or visually appealing than the others),Footnote ¹⁰⁶ (ii) hidden information,Footnote ¹⁰⁷ (iii) creating a sense of false urgency,Footnote ¹⁰⁸ (iv) forced continuity or roach motel (making it significantly more difficult for consumers to cancel their subscription than it was to sign up or automatically renew the service without the user’s express consent and repeatedly asking consumers to reconsider their choice).Footnote ¹⁰⁹ All of these illustrations are practices closely related to the concept of choice architecture and hyper personalization discussed in Section 10.2 presenting choices in a non-neutral way.

Dark patterns may involve the use of personal data of consumers and the use of AI.Footnote ¹¹⁰ AI is an asset for modifying dark patterns to have a greater impact on consumers behavior in a subtle way. It allows business operators to examine which dark patterns work best, especially when personal data is involved, and dark patterns are adapted accordingly. Examples of the power of the combination of dark patterns and AI can be found in platforms encouraging consumers to become paying members by presenting this option in different ways and over different time periods.Footnote ¹¹¹ Machine learning applications can analyze personal data to optimize dark patterns and find more innovative ways to convince consumers to buy a subscription. They can examine how many hours are spent a day watching videos, how many advertisements are being skipped and whether the app is closed when an ad is shown.Footnote ¹¹² The ad play may be increased if the consumer refuses to become a paying member.Footnote ¹¹³ Such a process can be stretched over quite a long time, making the consumer believe it is its own decision to subscribe, without him feeling tricked.Footnote ¹¹⁴ In essence, the combination of AI, personal data and dark patterns, results in an increased ability to manipulate consumers.

10.5.2 Overview of the Relevant Instruments of Consumer Protection against Dark Patterns

The UCPD is a first instrument that offers a number of possible avenues to combat dark patterns. As mentioned, it covers a wide range of prohibited practices in a business to consumer context.Footnote ¹¹⁵ First, the general prohibition of unfair commercial practices of art. 5 UCPD that functions as a residual control mechanism can be invoked. It prohibits all practices that violate a trader’s professional diligence obligation and that cause the average consumer to make a transactional decision that they would not otherwise have made.Footnote ¹¹⁶ This includes not only the decision to purchase or not purchase a product but also related decisions, such as visiting a website, or viewing content.Footnote ¹¹⁷ As mentioned, the standard of the “average” consumer (of the target group) is a normative standard that has (so far) been applied rather strictly, as rational behavior is the point of departure in the assessment.Footnote ¹¹⁸ The fact that the benchmark can be modulated to the target group does however offer some possibilities for a less strict standard in case of personalization, as the practice could then even be assessed from the perspective of a single targeted person.Footnote ¹¹⁹

Article 5(3) UCPD, furthermore creates some possibilities to assess a practice from the perspective of a vulnerable consumer, but the narrow definition of vulnerability as mental or psychical disability, age or credulity is – as mentioned – not suitable for the digital age. Indeed, any consumer can be temporarily vulnerable due to contextual and psychological factors.Footnote ¹²⁰ According to the European Commission, the UCPD provides a non-exhaustive list of characteristics that make a consumer “particularly susceptible” and therefore states that the concept of vulnerability should include these context-dependent vulnerabilities, such as interests, preferences, psychological profile, and even mood.Footnote ¹²¹ It will indeed be important to adopt such a broader interpretation to take into account the fact that all consumers can be potentially vulnerable in a digital context. The open norms of the UCPD might indeed be sufficiently flexible for such an interpretation,Footnote ¹²² but a clearer text in the directive – and not only (nonbinding) guidance of the Commission guidance – would be useful.

The more specific open norms prohibiting misleading and especially aggressive practices (arts. 6–9 UCPD) can also be invoked. But it is again uncertain how open concepts such as “undue influence” (art. 8 UCPD) must be interpreted in an AI context and to what extent the benchmark of the average consumer can be individualized. At what point does an increased exposure to advertising, tailored on past behavior, in order to convince a consumer to “choose” a paid subscription, amount to undue influence? More guidance on the interpretation of these open norms would be welcome.Footnote ¹²³

The blacklist in Annex I of the UCPD avoids the whole discussion on the interpretation of these benchmarks. That list prohibits specific practices that are considered unfair in all circumstancesFootnote ¹²⁴ and does not require an analysis of the potential effect on the average (or – exceptionally – vulnerable) consumer. The practices also do not require proof that the trader breached his professional diligence duty.Footnote ¹²⁵ The list prohibits several online practices, including disguised ads,Footnote ¹²⁶ false urgency (e.g., fake countdown timers),Footnote ¹²⁷ bait and switch,Footnote ¹²⁸ and direct exhortations to children.Footnote ¹²⁹ However, these practices were not specifically formulated to be applied in an AI context and interpretational problems therefore also occur when applying the current list to dark patterns. Thus, it is for instance mentioned in the Commission guidance that “making repeated intrusions during normal interactions in order to get the consumer to do or accept something (i.e., nagging) could amount to a persistent and unwanted solicitation.”Footnote ¹³⁰ The same interpretational problem then rises: how much intrusion and pressure is exactly needed to make a practice a “persistent and unwanted solicitation”? Additional blacklisted (AI) practices would increase legal certainty and facilitate enforcement.

Finally, the recently added Article 7(4a) UCPD requires traders to provide consumers with general information about the main parameters that determine the ranking of search results and their relative importance. The effectiveness of this article in protecting consumers by informing them can be questioned, as transparency about the practices generated by an AI system collides with the black box problem. Sharing information about the input-phase, such as the data set and learning algorithm that were used, may to some extent mitigate the information asymmetry but it will not suffice as a means of protection.

While the UCPD has broad coverage for most types of unfair commercial practices, the case-by-case approach does not allow to effectively address all forms of deceptive techniques known as “dark patterns.” For example, BEUC’s report of 2022 highlights the lack of consumer protection for practices that use language and emotion to influence consumers to make choices or take specific actions, often through tactics such as shaming, also referred to as confirmshaming.Footnote ¹³¹ In addition, there is uncertainty about the responsibilities of traders under the professional diligence duty and whether certain practices are explicitly prohibited.Footnote ¹³² Insufficient enforcement by both public and private parties further weakens this instrument.Footnote ¹³³

A second piece of legislation that provides some protection against dark patterns is the DSA. The regulation refers to dark patterns as practices “that materially distort or impair, either purposefully or in effect, the ability of recipients of the service to make autonomous and informed choices or decisions.”Footnote ¹³⁴ The DSA prohibits online platforms from designing, organizing, or operating their interfaces in a way that “deceives, manipulates, or otherwise materially distorts or impacts the ability of recipients of their services to make free and informed decisions”Footnote ¹³⁵ in so far as those practices are not covered under the UCPD and GDPR.Footnote ¹³⁶ Note that the important exception largely erodes consumer protection. Where the UCPD applies, and that includes all B2C practices, the vague standards of the UCPD will apply and not the more specific prohibition of dark patterns in the DSA. A cumulative application would have been preferable. The DSA inter alia targets exploitative design choices and practices as “forced continuity,” that make it unreasonably difficult to discontinue purchases or to sign out from services.Footnote ¹³⁷

The AI Act contains two specific prohibitions on manipulation practices carried out through the use of AI systems that may cover dark patterns.Footnote ¹³⁸ These bans prohibit the use of subliminal techniques to materially distort a person’s behavior in a manner that causes or is likely to cause significant harm and the exploitation of vulnerabilities in specific groups of people to materially distort their behavior in a manner that causes or is likely to cause significant harm.Footnote ¹³⁹ These prohibitions are similar to those in the UCPD, except that they are limited to practices carried out through the use of AI systems.Footnote ¹⁴⁰ They furthermore have some limitations. The ban relating to the abuse of vulnerabilities only applies to certain explicitly listed vulnerabilities, such as age, disability or specific social or economic situation, yet the mentioned problem of digital vulnerability is not tackled. A further major limitation was fortunately omitted in the final text of the AI Act. Whereas in the text of the AI proposal, these provisions only applied in case of physical and mental harm – which will often not be present and may be difficult to proveFootnote ¹⁴¹ – the prohibitions of the final AI Act also apply to (significant) economic harm.

The AI Act is complementary to other existing regulations, including data protection, consumer protection, and digital service legislation.Footnote ¹⁴² Finally, taking into account the fact that this Regulation strongly focuses on high-risk AI and that there are not many private services that qualify as high risk, the additional protection for consumers from this regulation seems limited.

The Consumer Rights Directive with its transparency requirement for pre-contractual informationFootnote ¹⁴³ and its prohibition to use pre-ticked boxes implying additional payments might also provide some help.Footnote ¹⁴⁴ However, the prohibition on pre-ticked boxes does not apply to certain sectors that are excluded from the directive, such as financial services.Footnote ¹⁴⁵ The UCPD could however also be invoked to combat charging for additional services through default interface settings and that directive does apply to the financial sector.Footnote ¹⁴⁶ The CRD does not regulate the conditions for contract termination, except for the right of withdrawal. An obligation for traders to insert a “withdrawal function” or “cancellation button” in contracts concluded by means of an online interface has recently been added to the CRD.Footnote ¹⁴⁷ This function is meant to make it easier for consumers to terminate distance contracts, particularly subscriptions during the period of withdrawal. This has could be a useful tool to combat subscription traps.

11.2.1 Protection under Patent Law

Patent law seeks to reward investment into research and development in order to spur future innovation. It does so by providing patentees with a temporary right to exclude others from using a certain “invention,” a technological improvement that takes the form of a product or a process (or both). This monopoly right is limited to 20 years following the patent application, subject to payment of the applicable annual fees.Footnote ² It is also limited in scope: while patentees can bring both direct and indirect infringements of their patent(s) to an end, they must accept certain exceptions as a defense to their claims, including use for experimental purposes and noncommercial use.Footnote ³ In order to be eligible for a patent, the invention must satisfy a number of conditions.

First, certain exclusions apply. The list of excluded subject matter under the European Patent Convention (EPC)Footnote ⁴ includes ideas that are deemed too abstract, such as computer programs as such, methods for performing mental acts and mathematical methods.Footnote ⁵ Pure abstract algorithms, which are essential to AI systems, qualify as a mathematical method, and are thus ineligible for patent protection as such.Footnote ⁶ However, this does not exclude patent protection for computer-implemented inventions such as technology related to AI algorithms, especially given the lenient interpretation of the “as such” proviso in practice. If the invention has a technical effect beyond its implementation on a computer – a connection to a material object in the “real” world – patentability may yet arise.Footnote ⁷ This will for example be the case for a neural network used “in a heart monitoring apparatus for the purpose of identifying irregular heartbeats,” as well as – in certain circumstances – methods for training AI systems.Footnote ⁸

Further, a patentable invention must satisfy a number of substantive conditions: it must be novel and inventive as well as industrially applicable.Footnote ⁹ The novelty requirement implies that the invention may not have been made available to the public at the date of filing of the patent application, indicated as the “state of the art.”Footnote ¹⁰ The condition of inventive step requires the invention to not have been obvious to a theoretical person skilled in the art (PSA) on the basis of this state of the art.Footnote ¹¹ Finally, the invention must be susceptible to use in an industrial context.Footnote ¹² Both the novelty and industrial applicability requirements do not appear to pose any challenges specific to AI-related innovation.Footnote ¹³ However, the inventiveness analysis only takes account of the patent claim features that contribute to the “technical character” of the invention, to the solution of a technical problem. Conversely, nontechnical features (such as the abstract algorithm) are removed from the equation.Footnote ¹⁴

The “patent bargain” between patentee and issuing government may lead to another obstacle. This implies that a prospective patentee must disclose their invention in a way that is sufficiently clear and complete for it to be carried out by a PSA, in return for patent protection.Footnote ¹⁵ This requirement of disclosure may be at odds with the apparent “black box” nature of many forms of AI technology, particularly in a deep learning context. This refers to a situation where we know which data were provided to the system (input A) and which result is reached (output B), but where it is unclear what exactly makes the AI system go from A to B.Footnote ¹⁶ Arguably, certain AI-related inventions cannot be explained in a sufficiently clear and complete manner, excluding the procurement of a patent therefor. However, experts will generally be able to disclose the AI system’s structure, the applicable parameters and the basic principles to which it adheres.Footnote ¹⁷ It is plausible that patent offices will deem this to be sufficient. The risk of being excluded from patent protection constitutes an additional incentive to invest in so-called “explainable” and transparent AI.Footnote ¹⁸ The transparency requirements established by the EU AI Act also play a role in this context.Footnote ¹⁹ Simultaneously, an overly strict assessment of the requirement of disclosure may push innovators toward trade secrets as an alternative way to protect AI-related innovation.Footnote ²⁰

It is often difficult to predict the outcome of the patenting process of AI-related innovation. This uncertainty does not seem to deter prospective patentees, as evidenced by the rising number of AI-related patent applications.Footnote ²¹ Since the 1950s, over 300,000 AI-related patent applications have been filed worldwide, with a sharp increase in the past decade: in 2019, it was already noted that more than half of these applications had been published since 2013.Footnote ²² It is to be expected that more recent numbers will confirm this evolving trend.

11.2.2 Protection under Copyright Law

AI-related innovation may also enjoy copyright protection. Copyright protection is generated automatically upon the creation of a literary and artistic work that constitutes a concrete and original expression by the author(s).Footnote ²³ It offers exclusive exploitation rights as to protected works, such as the right of reproduction and the right of communication to the public (subject to a number of exceptions), as well as certain moral rights.Footnote ²⁴ Copyright protection lasts until a minimum period of 50 years has passed following the death of the longest living author, a period that has been extended to 70 years in, for example, the EU Member States.Footnote ²⁵

The validity conditions for copyright are the requirement of concrete form and the requirement of originality. First, copyright protection is not available to mere abstract ideas and principles; these must be expressed in a concrete way.Footnote ²⁶ Second, the condition of originality implies that the work must be an intellectual creation of the author(s), reflecting their personality and expressing free and creative choices.Footnote ²⁷ Applied to AI-related works in particular, the functional algorithm in its purest sense does not satisfy the first condition and is therefore not susceptible to copyright protection.Footnote ²⁸ However, the object and source code of the computer program expressing this idea are sufficiently concrete, allowing for copyright protection once the condition of originality is fulfilled.Footnote ²⁹ Given the low threshold set for originality in practice, software that implements AI technology is likely to receive automatic protection as a computer program under copyright law upon its creation.Footnote ³⁰

11.3.1 AI Authorship

Can AI systems ever claim authorship? To answer this question, we must first ascertain whether “creative” machines already exist. Second, we discuss whether an AI system can be considered an author and, if not, whether it should be.

Certain AI systems available today can be used as a tool to create works that would satisfy the conditions for copyright protection if they had been solely created by humans. Many examples can be found in the music sector.Footnote ³² You may be reading this chapter with AI-generated music playing, such as piano music by Google’s “DeepMind” AI,Footnote ³³ an album released by the “Auxuman”Footnote ³⁴ algorithm, a soundscape created by the “Endel”Footnote ³⁵ app or one of the unfinished symphonies of Franz Schubert or Ludwig van Beethoven as completed with the aid of an AI system.Footnote ³⁶ If you would rather create music yourself, Sony’s “Flow Machines” project may offer assistance by augmenting your creativity through its AI algorithm.Footnote ³⁷ If you are bored with this text, which was written (solely) by a human author, you may instead start a conversation with “ChatGPT 4,”Footnote ³⁸ read a novelFootnote ³⁹ drafted by an AI algorithm or translate it using “DeepL.”Footnote ⁴⁰ AI-generated artwork is also available.Footnote ⁴¹ Most famously, Rembrandt van Rijn’s paintings were fed to an AI algorithm that went on to create a 3D-printed painting in Rembrandt’s style in 2016.Footnote ⁴² Since then, the use of AI in artwork has skyrocketed, with AI-powered image-generating applications such as “DALL-E 3”Footnote ⁴³ and “Midjourney”Footnote ⁴⁴ gaining exponential popularity.Footnote ⁴⁵

In most cases, there is still some human intervention, be it by a programmer, a person training the AI system through data input or somebody who modifies and/or selects output deemed “worthy” to disclose.Footnote ⁴⁶ If such human(s) were to have created the work(s) without the intervention of an AI system, copyright protection would likely be available.

Copyright law requires the work at issue to show authorship; the personal stamp of the author. The author is considered to be a physical person, especially in the civil law tradition, where copyright protection is viewed as a natural right, granted to the author to protect emanations of their personality.Footnote ⁴⁷ Creativity is viewed as a quintessentially human faculty, whereby a sentient being expresses their personality by making free, deliberate choices.Footnote ⁴⁸ This tenet pervades all aspects of copyright law. First, copyright laws grant initial ownership of copyright in a certain work to its author.Footnote ⁴⁹ Further, the term of protection is calculated from the author’s death. Also, certain provisions expressly seek to protect the author, such as those included in copyright contract law as well as the resale right applicable to original works of art. Moreover, particular copyright exceptions only apply if the author is acknowledged and/or if an equitable remuneration is paid to the author, such as the exception for private copies. The focus on the human author also explains the importance of the author’s moral rights to disclosure, integrity, and attribution.Footnote ⁵⁰ Such a system leaves no room for the authorship of a nonhuman entity.Footnote ⁵¹ If there is insufficient human input in the form of free and creative choices on the part of an author, if the AI crosses a certain threshold of autonomy, copyright protection is unavailable.Footnote ⁵² This anthropocentric view is unsurprising, since IP laws were largely drafted at a time when the concept of nonhuman “creators” belonged squarely in the realm of fiction.

However, the core of the issue is whether the abstract idea of originality should be held to include the creating behavior of an AI system. Account must hereby be taken of the broad range of potential AI activity and the ensuing distinction between AI-assisted and truly AI-generated content. At the one end of the spectrum, we may find AI systems that function as a tool to assist and/or enhance human creativity, where the AI itself acts as a mere executer.Footnote ⁵³ We can compare this to the quill used by William Shakespeare.Footnote ⁵⁴ Further down the line, there are many forms of AI-exhibited creativity that still result from creative choices made by a human, where the output flows directly from previously set parameters.Footnote ⁵⁵ Such AI activity may still be viewed as pure execution. In such cases, copyright should be reserved to the human actor behind the machine.

At the far end of the spectrum, we could find a hypothetical, more autonomous, “creative” AI, having independently created a work that exhibits the requisite creativity, which experts and nonexperts alike cannot distinguish from a work generated by a human. Even in such a case, it may be argued that there is no real act of “conception” in the AI system, given that every piece of AI-generated output is the result of prior human input.Footnote ⁵⁶ Arguably, precisely this act, the process of creation, is the essence of creativity. As long as the human thought process cannot be formulated as an algorithm that may be implemented by a computer, this process will remain human, thus excluding AI authorship. However, the “prior input” argument also applies mutatis mutandis to humans, who create literary and artistic works while “standing on the shoulders of giants.”Footnote ⁵⁷ This could render the “act of conception” argument against AI authorship moot, as could choosing the end result and thus the originality of the output as a (functionalist) focal point instead of the creative process.Footnote ⁵⁸ Additionally, it is argued that granting AI systems authorship may stimulate further creative efforts on the part of AI systems. This appears to be in line with the economic, utilitarian rationale of copyright.Footnote ⁵⁹ However, copyright seeks to incentivize human creators, not AI systems.Footnote ⁶⁰ Moreover, it is difficult to see how AI systems may respond to incentives in the absence of human consciousness.Footnote ⁶¹ Without convincing economic evidence, caution is advised against tearing down one of the fundamental principles of copyright law. The mere fact that we can create certain incentives does not in itself imply that we should. Further, if we were to allow AI authorship, we must be prepared for an upsurge in algorithmic creations, as well as the effects on human artistic freedom that this would entail.Footnote ⁶²

The risk of extending authorship to AI systems could be mitigated by instead establishing a related or sui generis right to AI-generated works and provide a limited degree of exclusivity in order to protect investments and incentivize research in this area. Such a right could be modelled in a similar way to the database right established by the EU in 1996.Footnote ⁶³ This requires a substantial investment for protection to be available.Footnote ⁶⁴

11.3.2 AI Inventorship

We now turn to AI inventorship. By analogy to the previous section, the first question is whether “inventive” machines already exist. Such systems are much scarcer than AI systems engaged in creative endeavors.Footnote ⁶⁵ However, progress on this front is undeniable.Footnote ⁶⁶ The AI sector’s primary allegedly inventive champion is “DABUS,”Footnote ⁶⁷ labelled the “Creativity Machine” by its inventor, physicist Dr Stephen Thaler.Footnote ⁶⁸ DABUS is a neural network-based system meant to generate “useful information” autonomously, thereby “simulating human creativity.”Footnote ⁶⁹ In 2018, a number of patent applications were filed for two of DABUS’ inventions.Footnote ⁷⁰ The prosecution files indicate DABUS as the inventor and clarify that Dr Thaler obtained the right to the inventions as its successor in title.Footnote ⁷¹ These patent applications offer a test case for the topic of AI inventorship.

Patent law requires inventors to be human. While relevant legislative provisions do not contain any explicit requirement in this sense, the inventor’s need for physical personhood is implied in the law.Footnote ⁷² While the focus on the human inventor is much less pronounced than it is on the human author, a number of provisions would make no sense if we were to accept AI inventorship. First, many patent laws stipulate that the “inventor” is the first owner of an invention, except in an employment context, where the employer is deemed to be the first owner under the laws of some countries.Footnote ⁷³ Since AI systems do not have legal personality (as of yet), they cannot have ownership rights, nor can they be an employee as such.Footnote ⁷⁴ Given that those are the only two available options, AI systems cannot be considered “inventors” as the law currently stands, as confirmed in the DABUS case, not only by the Boards of Appeal of the European Patent Office in the DABUS case, but also by the UK Supreme Court and the German Federal Supreme Court.Footnote ⁷⁵ Another argument against AI inventorship may be drawn from the inventor’s right of attribution. Every inventor has the right to be mentioned as such and all patent applications must designate the inventor.Footnote ⁷⁶ This moral right, which is meant to incentivize the inventor to innovate further, may become meaningless upon the extension of the concept of inventorship to AI systems.Footnote ⁷⁷

The second aspect of the discussion is whether there should be room for AI inventorship. The main argument in favor of this is that it would incentivize research and development in the field of AI.Footnote ⁷⁸ However, in the absence of compelling empirical evidence, the incentive argument is not convincing, especially since AI systems as such are not susceptible to incentives and the cost of AI invention will likely decrease over time.Footnote ⁷⁹ Another reason to accept AI inventorship would be to avoid humans incorrectly claiming inventorship. However, the as of yet instrumental nature of AI systems provides a counterargument.Footnote ⁸⁰ Further, there is no AI-generated output without some form of prior human input. The resulting absence of an act of “conception,” of the process of invention, excludes any extension of the scope of inventorship to nonhuman actors such as AI systems.Footnote ⁸¹ Again, however, the “prior input” argument also applies mutatis mutandis to humans. Also as to patent law, therefore, the “act of conception” argument against AI inventorship is susceptible to counterarguments.Footnote ⁸² A final aspect is that allowing AI inventorship would entail an increased risk of both overlapping sets of patents indicated as “patent thickets,” and the so-called “patent trolls,” which are nonpracticing entities that maintain an aggressive patent enforcement strategy while not exploiting the patent(s) at issue themselves.Footnote ⁸³

11.3.3 Ownership

The next question is how ownership rights in AI-powered creations should be allocated.Footnote ⁸⁴ As explained earlier, IP law does not allow AI systems to be recognized as either an author or an inventor. This begs the question whether the intervention of a creative and/or inventive AI excludes any kind of human authorship or inventorship (and thus ownership) as to the output at issue. It is submitted that it does not, as long as there is a physical person who commands the AI system and maintains the requisite level of control over its output.Footnote ⁸⁵ In such a case, IP rights may fulfil their role of protecting the interests of creators as well as provide an indirect incentive for future creation and/or innovation.Footnote ⁸⁶ However, if there is no sufficient causal relationship between the (in)actions of a human and the eventual end result, the argument in favor of a human author and/or inventor becomes untenable. What exactly constitutes “sufficient” control is tough to establish. A further layer of complexity is added by the black box nature of some AI systems: How can we determine whether a sufficient causal link exists between the human and the output, if it is impossible to find out exactly why this output was reached?Footnote ⁸⁷ However, both copyright and patent protection may be available to works and/or inventions that result from coincidence or even dumb luck.Footnote ⁸⁸ If we take a step back, both AI systems and serendipity may be considered as a factor outside the scope of human control. Given that Jackson Pollock may claim protection in his action paintings and given the role that chance plays in Pollock’s creation process, can we really deny such protection to the person(s) behind “the next Rembrandt”?

In copyright jargon, we could say that for a human to be able to claim copyright in a work created through the intervention of AI, their “personal stamp” must be discernible in the end result. If we continue the above analogy, Pollock’s paintings clearly reflect his personal choices as an artist. In patent law terms, human inventorship may arise in case of a contribution that transcends the purely financial, abstract or administrative and that is aimed at conceiving the claimed invention – be it through input or output selection, algorithm design, or otherwise.Footnote ⁸⁹ In an AI context, different categories of people may stake a claim in this regard.

First in line are the programmer(s),Footnote ⁹⁰ designer(s),Footnote ⁹¹ and/or producer(s) of the AI system (hereinafter collectively referred to as “AI creators”). By creating the AI system itself, these actors play a substantive role in the production of AI-generated output.Footnote ⁹² However, the allocation of rights to the creator sits uneasily with the unpredictable nature of AI-generated output.Footnote ⁹³ While the AI creator’s choices define the AI system, they do not define the final form of the output.Footnote ⁹⁴ This argument gains in strength the more autonomous the AI algorithm becomes.Footnote ⁹⁵ Then again, a programmer who is somehow dissatisfied with the AI’s initial output may tweak the AI’s algorithm, thus manipulating and shaping further output, as well as curate the AI output based on their personal choices.Footnote ⁹⁶ However, an economic argument against granting the AI creator rights in AI-generated output is that this may lead to “double-dipping.” This would be the case if the creator also holds rights in patents granted as to the AI system or the copyright therein, or if the AI system is acquired by a third party for a fee and the output at issue postdates this transfer.Footnote ⁹⁷ In both cases, the creator would obtain two separate sources of income for essentially the same thing. Moreover, enforcing the AI creator’s ownership rights would be problematic if the AI system generates the output at issue after a third party has started using it. Indeed, knowing that ownership rights would be allocated to the creator, the user would have strong incentives not to report back on the (modalities of) creation of output.Footnote ⁹⁸

A similar claim to the AI system’s creator may be made by the AI’s trainer who feeds input to the AI system.Footnote ⁹⁹ Alternatively, the user who has contributed substantially to the output at issue may claim ownership.Footnote ¹⁰⁰ The list of stakeholders continues with the investor, the owner of the AI system and/or the data used to train the algorithm, the publisher of the work, the general public, and even the government. Moreover, some form of joint ownership may be envisaged.Footnote ¹⁰¹ However, this would entail other issues, such as an unnecessary fragmentation of ownership rights and difficulties in proving (the extent of) ownership claims.Footnote ¹⁰² It could even be argued that, in view of the ever-rising number of players involved, no individual entity can rightfully claim to have made a significant contribution “worthy” of IP ownership.Footnote ¹⁰³

As of yet, no solution to the ownership conundrum appears to be wholly satisfactory. The void left by this lingering uncertainty will likely be filled with contractual solutions.Footnote ¹⁰⁴ Consequent to unequal bargaining power, instances of unfair ownership and licensing arrangements are to be expected.Footnote ¹⁰⁵ A preferable solution could be to not allocate ownership in AI-generated output to anyone at all and instead allot such output to the public domain. Stakeholders could sufficiently protect their investment in AI-related innovation by relying on patent protection for the AI system itself, first-mover advantage, trade secret law, contractual arrangements, and technological protection measures, as well as general civil liability and the law of unfair competition.Footnote ¹⁰⁶ However, there is a very pragmatic reason not to ban AI-generated output to the public domain, namely that it is increasingly difficult to distinguish output in the creation of which AI played a certain role from creations that were made solely by a human author.Footnote ¹⁰⁷ This could be remedied by requiring aspiring IP owners to disclose the intervention of an AI-powered system in the creation and/or innovation process. However, the practical application of such a requirement remains problematic at present. The prospect of having a work be banished to the public domain would provide stakeholders seeking a return on investment with strong incentives to keep quiet on this point. This could invite misleading statements on authorship and/or inventorship of AI-generated output in the future.Footnote ¹⁰⁸ Transparency obligations, such as the watermarking requirement imposed on providers of certain AI systems (including general-purpose AI models) under the EU AI Act, may bring us closer to a solution in this regard, likely combined with a “General-Purpose AI Code of Practice” that is to be drafted under the auspices of the AI Office at the EU level.Footnote ¹⁰⁹

11.4.1 IP Infringement

First, in order to train an AI algorithm, a significant amount of data is often required. If (part of) the relevant training data is subject to IP protection, the reproduction and/or communication to the public thereof in principle requires authorization by the owner, subject to the applicability of relevant exceptions and limitations to copyright. The question thus arises whether actively scraping the internet for artists’ work to reuse in the context of, for example, generative AI art tools constitutes an infringement. At the time of writing, several legal proceedings are pending on this question across the globe.Footnote ¹¹⁰ Importantly, the EU AI Act (1) confirms the applicability of text and data mining exceptions to the training of general-purpose AI models, subject to a potential opt-out on the part of rightholders; and (2) mandates the drawing up and public availability of “a sufficiently detailed summary about the content used for training of the general-purpose AI model.”Footnote ¹¹¹ Further, in order to ensure that authors, performers and other rightholders receive fair and appropriate remuneration for the use of their content as training data, contractual solutions may be envisaged.Footnote ¹¹²

Also after the training process, AI systems may infringe IP rights. By way of example, an AI program could create a song containing original elements of a preexisting work, thus infringing the reproduction right of the owner of the copyright in the musical work at issue. An inventive machine may develop a process and/or product that infringes a patent, or devise a sign that is confusingly similar to a registered trademark, or a product that falls within the scope of a protected (un)registered design. This in turn leads to further contentious matters, such as whether or not relevant exceptions and/or limitations (should) apply and whether fundamental rights such as freedom of expression may still play a role.Footnote ¹¹³

11.4.2 Impact of AI on Key Concepts of IP Law

Next, the rise of AI may significantly affect a number of key concepts of IP law that are clearly tailored to humans, in addition to the concepts of “authorship” and “inventorship.” First in line in this regard is the inventiveness standard under patent law, which centers around the so-called “person skilled in the art” (PSA).Footnote ¹¹⁴ This is a hypothetical person (or team) whose level of knowledge and skill depend on the field of technology.Footnote ¹¹⁵ If it is found that the PSA would have arrived at the invention, the invention will be deemed obvious and not patentable. If the use of inventive machines becomes commonplace in certain sectors of technology, the PSA standard will evolve into a PSA using such an inventive machine – and maybe even an inventive machine as such.Footnote ¹¹⁶ This would raise the bar for inventive step and ensuing patentability, since such a machine would be able to innovate based on the entirety of available prior art.Footnote ¹¹⁷ Taken to its logical extreme, this argument could shake the foundations of our patent system. Indeed, if the “artificially superintelligent” PSA is capable of an inventive step, everything becomes obvious, leaving no more room for patentable inventions.Footnote ¹¹⁸ We therefore need to start thinking about alternatives and/or supplements to the current nonobviousness analysis – and maybe even to the patent regime as a way to incentivize innovation.Footnote ¹¹⁹

Questions also arise in a trademark law context, such as how the increased intervention of AI in the online product suggestion and purchasing process may be reconciled with the anthropocentric conception of trademark law, as apparent from the use of criteria such as the “average consumer,” “confusion,” “imperfect recollection” – all of which are criteria that have a built-in margin for human error.Footnote ¹²⁰

11.4.3 Use of AI in IP Practice

Finally, the clear hesitancy of the IP community toward catering for additional incentive creation in the AI sphere by amending existing IP laws may be contrasted with apparent enthusiasm as to the use of AI in IP practice. Indeed, the increased (and still increasing) use of AI systems as a tool in the IP sector is striking. The ability of AI systems to process and analyze vast amounts of data quickly and efficiently offers a broad range of opportunities. First, the World Intellectual Property Organization (WIPO) has been mining the possibilities offered by AI with regard to the automatic categorization of patents and trademarks as well as prior art searches, machine translations, and formality checks.Footnote ¹²¹ Other IP offices are following suit.Footnote ¹²² Second, AI technology may be applied to the benefit of registrants. On a formal level, AI technology may be used to suggest relevant classes of goods and services for trademarks and/or designs. On a substantive level, AI technology may be used to aid in patent drafting and to screen registers for existing registrations to minimize risk. AI technology may assist in determining the similarity of trademarks and/or designs and even in evaluating prior art relating to patents.Footnote ¹²³ AI-based IP analytics and management software is also available.Footnote ¹²⁴ Finally, AI-powered applications are used in the fight against counterfeit products.Footnote ¹²⁵

12.2.1 The European AI Strategy

The European Commission published a European strategy for AI in 2018, setting in train Europe’s AI policyFootnote ⁸ to promote and increase AI investment and uptake across Europe in pursuit of its ambition to become a global AI powerhouse.Footnote ⁹ This strategy was formulated against a larger geopolitical backdrop in which the US and China were widely regarded as frontrunners, battling it out for first place in the “AI race” with Europe lagging significantly behind. Yet the growing Tech-Lash made it politically untenable for European policymakers to ignore public concerns. How, then, could they help European firms compete more effectively on the global stage while assuaging growing concerns that more needed to be done to protect democracy and the broader public interest? The response was to turn a perceived weakness into an opportunity by making a virtue of its political ideals and creating a unique “brand” of AI: infused with “European values” – charting a “third way,” distinct from both the Chinese state-driven approach and the US’ laissez-faire approach to AI governance.Footnote ¹⁰

At that time, the Commission resisted calls for the introduction of new laws. In particular, in 2018 the long-awaited General Data Protection Regulation (GDPR) finally took effect,Footnote ¹¹ introducing more stringent legal requirements for collecting and processing personal data. Not only did EU policymakers believe these would guard against AI-generated risks, but it was also politically unacceptable to position this new legal measure as outdated even as it was just starting to bite. By then, the digital tech industry was seizing the initiative, attempting to assuage rising anxieties about AI’s adverse impacts by voluntarily promulgating a wide range of “Ethical Codes of Conduct” proudly proclaiming they would uphold. This coincided with, and concurrently nurtured, a burgeoning academic interest by humanities and social science scholars in the social implications of AI, often proceeding under the broad rubric of “AI Ethics.” In keeping with industry’s stern warning that legal regulation would stifle innovation and push Europe even further behind, the Commission decided to convene a High-Level Expert Group on AI (AI HLEG) to develop a set of harmonized Ethics Guidelines based on European values that would serve as “best practice” in Europe, for which compliance was entirely voluntary.

12.2.2 The High-Level Expert Group on AI

This 52 member group was duly convened, to much fanfare, selected through open competition and comprised of approximately 50% industry representatives, with the remaining 50% from academia and civil society organizations.Footnote ¹² Following a public consultation, the group published its Ethics Guidelines for Trustworthy AI in April 2019,Footnote ¹³ coining “Trustworthy AI” as its overarching objective.Footnote ¹⁴ The Guidelines’ core consists of seven requirements that AI practitioners should take into account throughout an AI system’s lifecycle: (1) human agency and oversight (including the need for a fundamental rights impact assessment); (2) technical robustness and safety (including resilience to attack and security mechanisms, general safety, as well as accuracy, reliability and reproducibility requirements); (3) privacy and data governance (including not only respect for privacy, but also ensuring the quality and integrity of training and testing data); (4) transparency (including traceability, explainability, and clear communication); (5) diversity, nondiscrimination and fairness (including the avoidance of unfair bias, considerations of accessibility and universal design, and stakeholder participation); (6) societal and environmental wellbeing (including sustainability and fostering the “environmental friendliness” of AI systems, and considering their impact on society and democracy); and finally (7) accountability (including auditability, minimization, and reporting of negative impact, trade-offs, and redress mechanisms).Footnote ¹⁵

The group was also mandated to deliver Policy Recommendations which were published in June 2019,Footnote ¹⁶ oriented toward Member States and EU Institutions.Footnote ¹⁷ While attracting considerably less attention than the Ethics Guidelines, the Recommendations called for the adoption of new legal safeguards, recommending “a risk-based approach to AI policy-making,” taking into account “both individual and societal risks,”Footnote ¹⁸ to be complemented by “a precautionary principle-based approach” for “AI applications that generate ‘unacceptable’ risks or pose threats of harm that are substantial.”Footnote ¹⁹ For the use of AI in the public sector, the group stated that adherence to the Guidelines should be mandatory.Footnote ²⁰ For the private sector, the group asked the Commission to consider introducing obligations to conduct a “trustworthy AI” assessment (including a fundamental rights impact assessment) and stakeholder consultations; to comply with traceability, auditability, and ex-ante oversight requirements; and to ensure effective redress.Footnote ²¹ These Recommendations reflected a belief that nonbinding “ethics” guidelines were insufficient to ensure respect for fundamental rights, democracy, and the rule of law, and that legal reform was needed. Whether a catalyst or not, we will never know, for a few weeks later, the then President-elect of the Commission, Ursula von der Leyen, announced that she would “put forward legislation for a coordinated European approach on the human and ethical implications of Artificial Intelligence.”Footnote ²²

12.2.3 The White Paper on AI

In February 2020, the Commission issued a White Paper on AI,Footnote ²³ setting out a blueprint for new legislation to regulate AI “based on European values,”Footnote ²⁴ identifying several legal gaps that needed to be addressed. Although it sought to adopt a risk-based approach to regulate AI, it identified only two categories of AI systems: high-risk and not-high-risk, with solely the former being subjected to new obligations inspired by the Guidelines’ seven requirements for Trustworthy AI. The AI HLEG’s recommendation to protect fundamental rights as well as democracy and the rule of law were largely overlooked, and its suggestion to adopt a precautionary approach in relation to “unacceptable harm” was ignored altogether.

On enforcement, the White Paper remained rather vague. It did, however, suggest that high-risk systems should be subjected to a prior conformity assessment by providers of AI systems, analogous to existing EU conformity assessment procedures for products governed by the New Legislative Framework (discussed later).Footnote ²⁵ In this way, AI systems were to be regulated in a similar fashion to other stand-alone products including toys, measuring instruments, radio equipment, low-voltage electrical equipment, medical devices, and fertilizers rather than embedded within a complex and inherently socio-technical system that may be infrastructural in nature. Accordingly, the basic thrust of the proposal appeared animated primarily by a light-touch market-based orientation aimed at establishing a harmonized and competitive European AI market in which the protection of fundamental rights, democracy, and the rule of law were secondary concerns.

12.2.4 The Proposal for an AI Act

Despite extensive criticism, this approach formed the foundation of the Commission’s subsequent proposal for an AI Act published in April 2021.Footnote ²⁶ Building on the White Paper, it adopted a “horizontal” approach, regulating “AI systems” in general rather than pursuing a sector-specific approach. The risk-categorization of AI systems was more refined (unacceptable risk, high risk, medium risk, and low risk), although criticisms persisted given that various highly problematic applications were omitted from the list of “high-risk” and “unacceptable” systems, and with unwarranted exceptions.Footnote ²⁷ The conformity (self)assessment scheme was retained, firmly entrenching a product-safety approach to AI regulation, yet failing to confer any rights whatsoever for those subjected to AI systems; it only included obligations imposed on AI providers and (to a lesser extent) deployers.Footnote ²⁸

In December 2022, the Council of the European Union adopted its “general approach” on the Commission’s proposal.Footnote ²⁹ It sought to limit the regulation’s scope by narrowing the definition of AI and introducing more exceptions (for example for national security and research), sought stronger EU coordination for the Act’s enforcement; and proposed that AI systems listed as “high-risk” systems would not be automatically subjected to the Act’s requirements. Instead, providers could self-assess whether their system is truly high-risk based on a number of criteria – thereby further diluting the already limited protection the proposal afforded. Finally, the Council took into account the popularization of Large Language Models (LLMs) and generative AI applications such as ChatGPT, which at that time were drawing considerable public and political attention, and included modest provisions on General-Purpose AI models (GPAI).Footnote ³⁰

By the time the European Parliament formulated its own negotiating position in June 2023, generative AI was booming and called for more demanding restrictions. Additional requirements for the GPAI models that underpin generative AI were thus introduced, including risk-assessments and transparency obligations.Footnote ³¹ Contrary to the Council, the Parliament sought to widen some of the risk-categories; restore a broader definition of AI; strengthen transparency measures; introduce remedies for those subjected to AI systems; include stakeholder participation; and introduce mandatory fundamental rights impact assessments for high-risk systems. Yet it retained the Council’s proposal to allow AI providers to self-assess whether their “high-risk” system could be excluded from that category, and hence from the legal duties that would otherwise apply.Footnote ³² It also sprinkled the Act with references to the “rule of law” and “democracy,” yet these were little more than rhetorical flourishes given that it retained the underlying foundations of the original proposal’s market-oriented product-safety approach.

12.3.1 Scope

The AI Act aims to harmonize Member States’ national legislation, to eliminate potential obstacles to trade on the internal AI market, and to protect citizens and society against AI’s adverse effects, in that order of priority. Its main legal basis is Article 114 of the Treaty of the Functioning of the European Union (TFEU), which enables the establishment and functioning of the internal market. The inherent single-market orientation of this article limits the Act’s scope and justification.Footnote ³³ For this reason, certain provisions on the use of AI-enabled biometric data processing by law enforcement are also based on Article 16 TFEU, which provides a legal basis to regulate matters related to the right to data protection.Footnote ³⁴ Whether these legal bases are sufficient to regulate AI practices within the public sector or to achieve nonmarket-related aims remains uncertain, and could render the Act vulnerable to (partial) challenges for annulment on competence-related grounds.Footnote ³⁵ In terms of scope, the regulation applies to providers who place on the market or put into service AI systems (or general purpose AI models) in the EU, regardless of where they are established; deployers of AI systems that have their place of establishment or location in the EU; and providers and deployers of AI systems that are established or located outside the EU, while the output produced by their AI system is used in the EU.Footnote ³⁶

The definition of AI for the purpose of the regulation has been a significant battleground,Footnote ³⁷ with every EU institution proposing different definitions, each attracting criticism. Ultimately, the Commission’s initial proposal to combine a broad AI definition in the regulation’s main text with an amendable Annex that exhaustively enumerates the AI techniques covered by the Act was rejected. Instead, the legislators opted for a definition of AI which models that of the OECD, to promote international alignment: “a machine-based system designed to operate with varying levels of autonomy, that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments.”Footnote ³⁸

AI systems used exclusively for military or defense purposes are excluded from the Act, as are systems used for “nonprofessional” purposes. So too are AI systems “solely” used for research and innovation, which leaves open a substantive gap in protection given the many problematic research projects that can adversely affect individuals yet do not fall within the remit of university ethics committees. The AI Act also foresees that Member States’ competences in national security remain untouched, thus risking very weak protection of individuals in one of the potentially most intrusive areas for which AI might be used.Footnote ³⁹ Finally, the legislators also included certain exemptions for open-source AI models and systems,Footnote ⁴⁰ and derogations for microenterprises.Footnote ⁴¹

12.3.2 A Risk-based Approach

The AI Act adopts what the Commission describes as a “risk-based” approach: AI systems and/or practices are classified into a series of graded “tiers,” with proportionately more demanding legal obligations that vary in accordance with the EU’s perceptions of the severity of the risks they pose.Footnote ⁴² “Risks” are defined rather narrowly in terms of risks to “health, safety or fundamental rights.” The Act’s final risk categorization consists of five tiers: (1) systems that pose an “unacceptable” risk are prohibited; (2) systems deemed to pose a “high risk” are subjected to requirements akin to those listed in the Ethics Guidelines; (3) GPAI models are subjected to obligations that primarily focus on transparency, intellectual property protection, and the mitigation of “systemic risks”; (4) systems posing a limited risk must meet specified transparency requirements; and (5) systems that are not considered as posing significant risks do not attract new legal requirements.

12.3.3 Supporting Innovation

The White Paper on AI focused not only on the adoption of rules to limit AI-related risks, but also included a range of measures and policies to boost AI innovation in the EU. Clearly, the AI Act is a tool aimed primarily at achieving the former, but the EU still found it important to also emphasize its “pro-innovation” stance. Chapter VI of the AI Act therefore lists “measures in support of innovation,” which fits into the EU’s broader policy narrative which recognizes that regulation can facilitate innovation, and even provide a “competitive advantage” in the AI “race.”Footnote ⁷⁸ These measures mainly concernFootnote ⁷⁹ the introduction of AI regulatory sandboxes, which are intended to offer a safe and controlled environment for AI providers to develop, test, and validate AI systems, including the facilitation of “real-world-testing.” National authorities must oversee these sandboxes and help ensure that appropriate safeguards are in place, and that their experimentation occurs in compliance with the law. The AI Act mandates each Member State to establish at least one regulatory sandbox, which can also be established jointly with other Member States.Footnote ⁸⁰ To avoid fragmentation, the AI Act further provides for the development of common rules for the sandboxes’ implementation and a framework for cooperation between the relevant authorities that supervise them, to ensure their uniform implementation across the EU.Footnote ⁸¹

Sandboxes must be made accessible especially to Small and Medium Enterprises (SMEs), thereby ensuring that they receive additional support and guidance to achieve regulatory compliance while retaining the ability to innovate. In fact, the AI Act explicitly recognizes the need to take into account the interests of “small-scale providers” and deployers of AI systems, particularly costs.Footnote ⁸² National authorities that oversee sandboxes are hence given various tasks, including increasing awareness on the regulation, promoting AI literacy, offering information and communication services to SMEs, start-ups, and deployers, and helping them identify methods that lower their compliance costs. Collectively, these measures are aimed to offset the fact that smaller companies will likely face heavier compliance and implementation burdens, especially compared to large tech companies that can afford an army of lawyers and consultants to implement the AI Act. It is also hoped that the sandboxes will help national authorities to improve their supervisory methods, develop better guidance, and identify possible future improvements of the legal framework.

12.4.1 Legal Rules and Interpretative Discretion

Many of the AI Act’s core provisions are written in broad, open-ended language, leaving the meaning of key terms uncertain and unresolved. It will be here that the rubber will hit the road, for it is through the interpretation and application of the Act’s operative provisions that it will be given meaning and be translated into on-the-ground practice.

For example, when seeking to apply the essential requirements applicable to high-risk systems, three terms used in Chapter III, Section 2 play a crucial role. First, the concept of “risk.” Article 3 defines risk as “the combination of the probability of an occurrence of harm and the severity of that harm,” reflecting conventional statistical risk assessment terminology. Although risks to health and safety is a relatively familiar and established concept in legal parlance and regulatory regimes, the Annex III high-risk systems are more likely to interfere with fundamental rights and may adversely affect democracy and the rule of law. But what, precisely, is meant by “risk to fundamental rights,” and how should those risks be identified, evaluated and assessed? Secondly, even assuming that fundamental rights-related risks can be meaningfully assessed, how then is a software firm to adequately evaluate what constitutes a level of residual risk judged “acceptable”? And thirdly, what constitutes a “risk management system” that meets the requirements of Article 9?

The problem of interpretative discretion is not unique to the AI Act. All rules which take linguistic form, whether legally mandated or otherwise, must be interpreted before they can be applied to specific real-world circumstances. Yet how this discretion is exercised, and by whom, will be a product of the larger regulatory architecture in which those rules are embedded. The GDPR, for instance, contains a number of broadly defined “principles” which those who collect and process personal data must comply with. Both the European Data Protection Board (EDPB) and national level data protection authorities – as public regulators – issue “guidance” documents offering interpretative guidance about what the law requires. Compliance with this guidance (often called “soft law”) does not guarantee compliance – for it does not bind courts when interpreting the law – but it nevertheless offers a valuable, and reasonably authoritative assistance to those seeking to comply with their legal obligations. This kind of guidance is open, published, transparent, and conventionally issued in draft form before-hand so that stakeholders and the public can provide feedback before it is issued in final form.Footnote ⁸³

In the AI Act, similar interpretative decisions will need to be made and, in theory, the Commission has a mandate to issue guidelines on the AI Act’s practical implementation.Footnote ⁸⁴ However, in contrast with the GDPR, the Act’s adoption of the “New Approach” to product-safety means that, in practice, providers of high-risk AI systems will likely adhere to technical standards produced by European Standardization Organizations on request from the Commission and which are expected to acquire the status of “harmonized standards” by publication of their titles in the EU’s Official Journal.Footnote ⁸⁵ As we explain below, the processes through which these standards are developed are difficult to characterize as democratic, transparent or based on open public participation.

12.4.2 The AI Act as a Form of “Meta-Regulation”

At first glance, the AI Act appears to adopt a public enforcement framework with both national and European public authorities playing a significant role. Each EU Member State must designate a national supervisory authorityFootnote ⁸⁶ to act as “market surveillance authority.”Footnote ⁸⁷ These authorities can investigate suspected incidents and infringements of the AI Act’s requirements, and initiate recalls or withdrawals of AI systems from the market for non-compliance.Footnote ⁸⁸ National authorities exchange best practices through a European AI Board comprised of Member States’ representatives. The European Commission has also set up an AI Office to coordinate enforcement at the EU level.Footnote ⁸⁹ Its main task is to monitor and enforce the requirements relating to GPAI models,Footnote ⁹⁰ yet it also undertakes several other roles, including (a) guiding the evaluation and review of the AI Act over time,Footnote ⁹¹ (b) offering coordination support for joint investigations between the Commission and Member States when a high-risk system presents a serious risk across multiple Member States,Footnote ⁹² and (c) facilitating the drawing up of voluntary codes of conduct for systems that are not classified as high-risk.Footnote ⁹³

The AI Office will be advised by a scientific panel of independent experts to help it develop methodologies to evaluate the capabilities of GPAI models, to designate GPAI models as posing a systemic risk, and to monitor material safety risks that such models pose. An advisory forum of stakeholders (to counter earlier criticism that stakeholders were allocated no role whatsoever in the regulation) is also established under the Act, to provide both the Board and the Commission with technical expertise and advice. Finally, the Commission is tasked with establishing a public EU-wide database where providers (and a limited set of deployers) of stand-alone high-risk AI systems must register their systems to enhance transparency.Footnote ⁹⁴

In practice, however, these public authorities are twice-removed from where much of the real-world compliance activity and evaluation takes place. The AI Act’s regulatory enforcement framework delegates many crucial functions (and thus considerable discretionary power) to the very actors whom the regime purports to regulate, and to other tech industry experts. The entire architecture of the AI Act is based on what regulatory governance scholars sometimes refer to as “meta-regulation” or “enforced self-regulation.”Footnote ⁹⁵ This is a regulatory technique in which legally binding obligations are imposed on regulated organizations, requiring them to establish and maintain internal control systems that meet broadly specified, outcome-based, binding legal objectives.

Meta-regulatory strategies rest on the basic idea that one size does not fit all, and that firms themselves are best placed to understand their own operations and systems and take the necessary action to avoid risks and dangers. The primary safeguards through which the AI Act is intended to work rely on the quality and risk management systems within the regulated organizations, in which these organizations retain considerable discretion to establish and maintain their own internal standards of control, provided that the Act’s legally mandated objectives are met. The supervisory authorities oversee adherence to those internal standards, but they only play a secondary and reactionary role, which is triggered if there are grounds to suspect that regulated organizations are failing to discharge their legal obligations. While natural and legal persons have the right to lodge a complaint when they have grounds to consider that the AI Act was infringed,Footnote ⁹⁶ supervisory authorities do not have any proactive role to ensure the requirements are met before high-risk AI systems are placed on the market or deployed.

This compliance architecture flows from the underlying foundations of the Act, which are rooted in the EU’s “New Legislative Framework,” adopted in 2008. Its aim was to improve the internal market for goods and strengthen the conditions for placing a wide range of products on the EU market.Footnote ⁹⁷

The AI Act largely leaves it to Annex III high-risk AI providers and deployers to self-assess their conformity with the AI Act’s requirements (including, as discussed earlier, the judgment of what is deemed an “acceptable” residual risk). There is no routine or regular inspection and approval or licensing by a public authority. Instead, if they declare that they have self-assessed their AI system as compliant and duly lodge a declaration of conformity, providers can put their AI systems into service without any independent party verifying whether their assessment is indeed adequate (except for certain biometric systems).Footnote ⁹⁸ Providers are, however, required to put in place a post-market monitoring system, which is intended to ensure that the possible risks emerging from AI systems that continue to “learn” or evolve once placed on the market or put into service can be better identified and addressed.Footnote ⁹⁹ The role of public regulators is therefore largely that of ex post oversight, unlike the European regulation of pharmaceuticals, reflecting the regulatory regime as permissive rather than precautionary. This embodies the basic regulatory philosophy underpinning the New Legislative Framework, which builds on the “New Approach” to technical standardization. Together, these are concerned first and foremost with strengthening single market integration, and hence with ensuring a single EU market for AI.

12.4.3 The New Approach to Technical Standardization

Under the EU’s “Old Approach” to product safety standards, national authorities drew up detailed technical legislation, which was often unwieldy and usually motivated by a lack of confidence in the rigour of economic operators on issues of public health and safety. However, the “New Approach” framework introduced in 1985 sought instead to restrict the content of legislation to “essential requirements,” leaving technical details to European Harmonized StandardsFootnote ¹⁰⁰ thereby laying the foundation for technical standards produced by European Standardization Organizations (ESOs) in support of Union harmonization legislation.Footnote ¹⁰¹

The animating purpose of the “New Approach” to standardization was to open up European markets in industrial products without threatening the safety of European consumers, by allowing the entry of those products across European markets if and only if they meet the “essential [safety] requirements” set out in sector-specific European rules developed by one of the three ESOs: the European Committee for Standardization (CEN), the European Committee for Electrotechnical Standardization (CENELEC) and the European Telecommunications Standards Institute (ETSI).Footnote ¹⁰²

Under this approach, producers can choose to either interpret the relevant EU Directive themselves or to rely on “harmonized (European) standards” drawn up by one of the ESOs. This meta-regulatory approach combines compulsory regulation (under EU secondary legislation) and “voluntary” standards, made by ESOs. Central to this approach is that conformity of products with “essential safety requirements” is checked and certified by producers themselves who make a declaration of conformity and affix the CE mark to their products to indicate this, thereby allowing the product to be marketed and sold across the whole of the EU. However, for some “sensitive products,” conformity assessments must be carried out by an independent third-party “notified body” to certify conformity and issue a declaration of conformity. This approach was taken by the Commission in its initial AI Act proposal, and neither the Parliament nor the Council has sought to depart from it. By virtue of its reliance on the “New Approach,” the AI Act lays tremendous power in the hands of private, technical bodies who are entrusted with the task of setting technical standards intended to operationalize the “essential requirements” stipulated in the AI Act.Footnote ¹⁰³

In particular, providers of Annex III high-risk AI systems that fall under the AI Act’s requirements have three options. First, they can self-assess the compliance of their AI systems with the essential requirements (which the AI Act refers to as the conformity assessment procedure based on internal control, set out in Annex VI). Under this option, whenever the requirements are vague, organizations need to use their own judgment and discretion to interpret and apply them, which – given considerable uncertainty about what they require in practice – exposes them to potential legal risks (including substantial penalties) if they fail to meet the requirements.

Second, organizations can rely on a conformity assessment by a “notified body,”Footnote ¹⁰⁴ which they can commission to undertake the conformity assessment. These bodies are independent yet nevertheless “private” organizations that verify the conformity of AI systems based on an assessment of the quality management system and the technical documentation (a procedure set out in Annex VII). AI providers pay for these certification services, with a flourishing “market for certification” emerging in response. To carry out the tasks of a notified body, it must meet the requirements of Article 31 of the AI Act, which are mainly concerned with ensuring that they possess the necessary competences, a high degree of professional integrity, and that they are independent from and impartial to the organizations they assess to avoid conflicts of interest. Pursuant to the AI Act, only providers of biometric identification systems must currently undergo an assessment by a notification body. All others can opt for the first option (though in the future, other sensitive systems may also be obliged to obtain approval via third-party conformity assessment).

Third, AI providers can choose to follow voluntary standards currently under development by CEN/CENELEC following acceptance of the Commission’s standardization request which are intended, once drafted, to become “harmonized standards” following citation in the Official Journal of the European Commission. This would mean that AI providers and deployers could choose to follow these harmonized standards and thereby benefit from a legal presumption of conformity with the AI Act’s requirements. Although the presumption of compliance is rebuttable, it places the burden of proving non-compliance on those claiming that the AI Act’s requirements were not met, thus considerably reducing the risk that the AI provider will be found to be in breach of the Act’s essential requirements. If no harmonized standards are forthcoming, the Commission can adopt “common specifications” in respect of the requirements for high-risk systems and GPAI models, which likewise, will confer a presumption of conformity.Footnote ¹⁰⁵

Thus, although harmonized standards produced by ESOs are formally voluntary, providers are strongly incentivized to follow them (or, in their absence, to follow the common specifications) rather than carrying the burden of demonstrating that their own specifications meet the law’s essential requirements. This means that harmonized standards are likely to become binding de facto, and will therefore in practice determine the nature and level of protection provided under the AI Act. The overwhelming majority of providers of Annex III high-risk systems can self-assess their own internal controls, sign and lodge a conformity assessment declaration, affix a CE mark to their software, and then notify the Commission’s public register.

12.4.4 Why Technical Standardization Falls Short in the AI Act’s Context

Importantly, however, several studies have found that products that have been self-certified by producers are considerably more likely to fail to meet the certified standard. For example, Larson and JordanFootnote ¹⁰⁶ compared toy safety recalls in the US, within a toy safety regime requiring independent third-party verification, and the EU’s toy self-certification regime which relies on self-assessment and found stark differences. Over a two-year period, toy safety recalls in the EU were 9 to 20 times more frequent than those in the US. Their findings align with earlier policy studies finding that self-assessment models consistently produce substantially higher rates of worker injury compared with those involving independent third-party evaluation. Based on these studies, Larson and Jordon conclude that transnational product safety regulatory systems that rely on the self-assessment of conformity with safety standards fail to keep products off the market, which do not comply with those standards.

What is more, even third-party certification under the EU’s New Approach has shown itself to be weak and ineffective, as evidenced by the failure of the EU’s Medical Device regime which prevailed before its more recent reform. This was vividly illustrated by the PIP breast implants scandal in which approximately 40,000 women in France, and possibly 10 times more in Europe and worldwide, were implanted with breast implants that were filled with industrial grade silicon, rather than the compulsory medical grade standard required under EU law.Footnote ¹⁰⁷ This occurred despite the fact that the implants had been certified as “CE compliant” by a reputable German notified body, which was possible because, under the relevant directive,Footnote ¹⁰⁸ breast implant producers could choose between different methods of inspection. PIP had chosen the “full quality assurance system,” whereby the certifiers’ job was to audit PIP’s quality management system without having to inspect the breast implants themselves. In short, the New Approach has succeeded in fostering flourishing markets for certification services – but evidence suggests that it cannot be relied on systematically to deliver trustworthy products and services that protect individuals from harm to their health and safety.

Particularly troubling is the New Approach’s reliance on testing the quality of internal document keeping and management systems, rather than an inspection and evaluation of the service or product itself.Footnote ¹⁰⁹ As critical accounting scholar Mike Power has observed, the process of “rendering auditable” through measurable procedures and performance – is a test of “the quality of internal systems rather than the quality of the product or service itself specified in standards.”Footnote ¹¹⁰ As Hopkins emphasizes in his analysis of the core features that a robust “safety case” approach must meet, “without scrutiny by an independent regulator, a safety case may not be worth the paper it is written on.”Footnote ¹¹¹ The AI Act, however, does not impose any external auditing requirements. For Annex III high-risk AI systems, the compliance evaluation remains primarily limited to verification that there is requisite documentation in place. Accordingly, we are skeptical of the effectiveness of the CE marking regime for delivering meaningful and effective protections for those affected by rights-critical products and services regulated under the Act.Footnote ¹¹²

What, then, are the prospects that the technical standards which the Commission has tasked CEN/CENELEC to produce will translate into practice the Act’s noble aspirations to protect fundamental rights, health, safety and uphold the rule of law? We believe there are several reasons to worry. Technical standardization processes may appear “neutral” as they focus on mundane technical tasks, conducted in a highly specialized vernacular, yet these activities are in fact highly political. As Lawrence Busch puts it: “Standards are intimately associated with power.”Footnote ¹¹³ Moreover, these standards will not be publicly available. Rather, they are protected by copyright and thus only available on payment.Footnote ¹¹⁴ If an AI provider self-certifies its compliance with an ESO-produced harmonized standard, that will constitute “deemed compliance” with the Act. But, if, in fact, that provider has made no attempt to comply with the standard, no-one will be any the wiser unless and until action is taken by a market surveillance authority to evaluate that AI system for compliance, which it cannot do unless it has “sufficient reasons to consider an AI system to present a risk.”Footnote ¹¹⁵

In addition, technical standardization bodies have conventionally been dominated by private sector actors who have had both the capacity to develop particular technologies and can leverage their market share to advocate for the standardization of the technology in line with their own products and organizational processes. Standards committees tend to be stacked with people from large corporations with vested interests and extensive resources. As Joanna Bryson has pithily put it, “even when technical standards for software are useful they are ripe for regulatory capture.”Footnote ¹¹⁶ Nor are they subject to democratic mechanisms of public oversight and accountability that apply to conventional law-making bodies. Neither the Parliament nor the Member States have a binding veto over harmonized standards, and even the Commission has only limited powers to influence their content, at the point of determining whether the standard produced in response to its request meets the essential requirements set out in the Act, but otherwise the standard is essentially immune from judicial review.Footnote ¹¹⁷

Criticisms of the lack of the democratic legitimacy of these organizations has led to moves to open up their standard-setting process to “multi-stakeholder” dialogue, with civil society organizations seeking to get more involved.Footnote ¹¹⁸ In practice, however, these moves are deeply inadequate, as civil society struggles to obtain technical parity with their better-resourced counterparts from the business and technology communities. Stakeholder organizations also face various de facto obstacles to use the CEN/CENELEC participatory mechanisms effectively. Most NGOs have no experience in standardization and many lack EU level representation. Moreover, active participation is costly and highly time-consuming.Footnote ¹¹⁹

Equally if not more worrying is the fact that these “technical” standard-setting bodies are populated by experts primarily from engineering and computer science, who typically have little knowledge or expertise in matters related to fundamental rights, democracy, and the rule of law. Nor are they likely to be familiar with the analytical reasoning that is well established in human rights jurisprudence to determine what constitutes an interference with a fundamental right and whether it may be justified as necessary in a democratic society.Footnote ¹²⁰ Without a significant cadre of human rights lawyers to assist them, we are deeply skeptical of the competence and ability of ESOs to translate the notion of “risks to fundamental rights” into tractable technical standards that can be relied upon to facilitate the protection of fundamental rights.Footnote ¹²¹

Furthermore, unlike risks to safety generated by chemicals, machinery, or industrial waste, all of which can be materially observed and measured, fundamental rights are, in effect, political constructs. These rights are accorded special legal protection so that an evaluation of alleged interference requires close attention to the nature and scope of the relevant right and the specific, localized context in which a particular right is allegedly infringed. We therefore seriously doubt whether fundamental rights can ever be translated into generalized technical standards that can be precisely measured in quantitative terms, and in a manner that faithfully reflects what they are and how they have been interpreted under the European Charter on Fundamental Rights and the European Convention on Human Rights.

Moreover, the CENELEC rules nevertheless state that any harmonized standard must contain “objectively verifiable requirements and test methods,”Footnote ¹²² which does not alleviate our difficulties in trying to conceive of how “risks to fundamental rights” can be subject to quantitative “metrics” and translated into technical standards such that the “residual risk” can be assessed as “acceptable.” Taken together, this leaves us rather pessimistic about the capacity and prospects for ESOs (even assuming a well-intentioned technical committee) to produce technical standards that will, if duly followed, provide the high level of protection to European values that the Act claims to aspire to, and which will constitute “deemed compliance” with the regulation. And if, as expected, providers of high-risk AI systems will choose to be guided by the technical standards produced by ESOs, this means that the “real” standard-setting for high-risk systems will take place within those organizations, with little public scrutiny or independent evaluation.