A. Entangled logics of secrecy and transparency

The concepts of transparency and of secrecy are to be considered in relation to one another and as each other’s complement. As we have seen, the very word transparency connotes the ability to see through. Visibility is literally in the name transparency: it implies a subject seeing as well as the object being seen. In this sense, it is a two-way street: a beholder and an object form a complex set of affairs.Footnote ²⁵ The beholder may force the object to react to its ‘gaze’ and in this sense may be controlling. In the European Union, transparency has as a floating signifier carried an almost impossible burden over the years. It would allegedly, according to the Commission, make the Union closer to the people, and stimulate a more informed and involved debate on EU policy.Footnote ²⁶ The evolution of the debate and practice of transparency has been closely connected with and in part seen as a remedy for the so-called ‘democratic deficit’ of the EU. Democracy requires institutional accountability and the preliminary sine qua non for that is the provision of reliable information.

Secrecy, on the other hand, can be defined as all those behaviours whereby one party intentionally conceals information from another.Footnote ²⁷ Secrecy creates a form of opacity, which makes it hard to discover who takes the decisions, what they are, and who gains and who loses – precisely the opposite of what transparency is meant to achieve.

There are arguably two main competing forms or logics within transparency which will be detailed below also as a framing to move beyond the approach taken in the AI Act. The first logic concerns a logic of disclosure or public access and comes closest to the core meaning of transparency – transparency in its unadulterated self, amounting to public release of information through the direct disclosure of documents (such as reports, meeting transcripts, etc). The system is one of passive disclosure by distinct actors once disclosure is requested but the idea is that over time more active disclosure through public registers of documents and special public databases will also occur. At the same time, this logic includes whole areas being removed from the rules on disclosure for various legal and political reasons (including rules on secrecy). The very essence of rules on professional secrecy is to define a system of insiders and outsiders, those who are secluded and those who are excluded.

The second logic is substantively mediated and at no point involves actual visibility or seeing through. This is the logic of communication or explanation and in its very essence it does not employ the medium of transparency but rather of ‘describing’, communicating, or explaining aspects of what is kept hidden or not revealed. The disclosure involves a triad between the subject listening, the object not being seen and the secret keeper revealing the rationale or explanation.

This exposes a fundamental qualitative difference between the two logics: the logic of disclosure is one of little or no mediation, whereas the logic of communication often relies on active curation behind closed doors and subsequent narration around essentially undisclosed information. The limited mediation involved in the logic of disclosure is the act of formally granting access (through a public register or subject to an access to documents request). Information that is hidden or not public necessarily requires technically granting access to the document or information in question. This stands in stark contrast to the substantive mediation involved in the logic of communication where the specific information is not revealed in its granular and authentic original detail (‘raw’) but rather the narrated reasoning behind it in the words only of the mediator.

B. Logic of disclosure: a public magnifying glass

The logic of disclosure thus pertains to transparency as visibility, reflecting its semantic pedigree of ‘light’ and ‘sight’.Footnote ²⁸ It speaks to ‘the ability to look clearly through the windows of an institution’,Footnote ²⁹ with visibility described as ‘a necessary condition for there to be transparency’: ‘Transparency is about information, and if information is not visible then the first and primary meaning of the parent-word, “transparent” – having to do with light and visual properties – loses its relevance’.Footnote ³⁰

The notion of directly (but not necessarily fully) seeing amounts to a form of immediate verification that speaks to transparency’s enduring appeal: ‘So long as we can witness the reality with our own eyes, we do not need verbal explanations, which are, by virtue of transparency, indirectly considered less reliable than direct visual observation’. ^{Footnote 31} This notion of ‘immediate observability’ stands in ‘a tense relationship’ with mediation: ‘the more the object of transparency becomes mediated, the less immediacy there can be’. ^{Footnote 32} Relatedly, the emphasis on ‘granularity’ – the disclosure of raw, detailed, or close-to-the-source data (such as minutes, transcripts, datasets etc) – regarded as an important dimension of transparency, is a means to reduce the risk of such information being strategically processed or gamed.Footnote ³³ Of course, transparency as disclosure is not without trade-offs.Footnote ³⁴ For instance, disclosure of (granular or raw) information can come at the expense of simplification, which can render such information less accessible or user friendly.Footnote ³⁵ Nevertheless, while there are always trade-offs involved, transparency as disclosure serves a crucial function pertaining to informational reliability: ‘Because raw data is usually less mediated, it typically reflects fewer opportunities for officials to “cook” or “game” it out of professional or political motivations’.Footnote ³⁶

There have been some significant legislative changes in practice that have sought to create an ecosystem around transparency as disclosure for the EU. The earliest and by now most developed were the efforts to copper-fasten in law and in practice informational transparency. This dates from 2001, more than two decades ago, and has been incrementally expanded and fought over both by individuals of one type or another – lawyers, experts, academics, journalists, NGOs, etc – as well as by institutions and institutional representatives (eg members of parliament). Yet transparency in this informational sense has remained troubled and troubling, just as is the case in other parts of the world where there is an even longer tradition of freedom of information (for example the United States).Footnote ³⁷ It is inevitably dependent on the practice of the institutions, their willingness to pro-actively provide information as well as passively on request and also to a large extent on the institutions meant to police it.

As we have seen, this logic of public access comes closest to the understanding of transparency as visibility. It does not patronise the citizen but rather values the role that the public and the informed citizen can play in a wider democratic perspective. In this perspective, transparency is seen as a fundamental citizens’ right and as a means towards securing public accountability.Footnote ³⁸ It implies that all arms of government – the executive, the entire public administration as well as parliaments – should be subject to the requirement of openness or public access.Footnote ³⁹ The deeper democratic meaning of why openness and transparency are important is based on the logic that ‘increased openness ( … ) enables citizens to participate more closely in the decision-making process and guarantees ( … ) greater legitimacy and is more effective and more accountable to the citizen in a democratic system’.Footnote ⁴⁰ This classic way of viewing transparency recognizes a relationship between a beholder (seeing) and an object (being seen) and mirrors an accountability relationship between an actor (any actor) and an accountability forum (judicial, parliamentary, audit etc). This is the case even if reality is such that the beholder cannot see straight through and there are significant shutters and blinds in place. In other words, transparency in this logic is not only an end value (full sight) but also plays a critical instrumental role with respect to its potential to further legitimacy, enhance participation rights, reason-giving in the administration and public accountability. At the same time, transparency is different from these concepts – it is often a key pre-requisite, a facilitator, towards the realisation of these other values. And it performs these functions instrumentally even if full sight is not possible.

Following Bovens,Footnote ⁴¹ transparency is a necessary but insufficient condition in and of itself for accountability. Direct citizen accountability through transparencyFootnote ⁴² may often be an illusion. On the other hand, the media and other stakeholders can and do use transparency to force change. Similarly, elected representatives can use transparency to demand and enact accountability of public bodies and authorities on behalf of citizens.Footnote ⁴³ Transparency, in turn, also enables the media to work as fire alarms, triggering vertical (parliamentary) accountability. Importantly, even if the citizen does not directly ‘understand by seeing’ alone, third-party stakeholders (lawyers, academics, experts, NGOs, advocacy networks, etc) can publicly contribute to understanding as well as verify and act as checks on understandings that are being advanced. These other forums (with more capacity, resources, expertise) can and do act as fire alarms and demand accountability on behalf of adversely affected citizens. This becomes much harder however, if not next to impossible, under conditions of secrecy.

C. Logic of communication: mediation and explanation

Yet, some authors – and policy-makers – approach transparency as (only) communication. Footnote ⁴⁴ As such, they accept that transparency is essentially mediated, and ‘excessively simplified and thus are blind to the complexities of the contemporary state, government information and the public’.Footnote ⁴⁵

In a governance context, communication amounts to the intentional release of redacted information on the substance of decisions and of (some of) the facts and reasons on which they are based. Such release of information previously intentionally concealed, is (normally) to an outside audience, which may be affected by the decision but is not involved in the decision-making. It thus makes the outside actor (market, citizen, or parliament) aware of the existence of what is not known, for example that deliberations, negotiations and other elements of constituent decisions have taken place – but does not give transparency of the actual process and content of that decision-making – at least not in unredacted form.

The logic of communication reveals almost unlimited discretion to the object or target of transparency as to what is revealed and what is kept secret. It is not the beholder controlling the object but rather the opposite. In effect, the logic of communication is a logic of secrecy, not disclosing the actual secret but narrating a story around it either openly or (more commonly) to a specified number of stakeholders, respecting certain rules on professional secrecy. Information is pre-digested, recounted and ‘spun’ into an account at the intermediary’s control. By contrast, in a disclosure logic of transparency, the object is less in control and is forced to assume a more reactive and defensive position by virtue of interactions with others (parliaments, the public, ombudsmen or courts).

The turn towards explanation in algorithmic transparency debates signals, as we will see below, a similar move towards a logic of communication through the provision of pre-digested, simplified information. While such an approach is meant to facilitate understanding on the part of the beholder, it simultaneously affords the object (rather than the beholder) control over openness. We become ‘passive recipients of simplified information’ ‘increasingly dependent on translating intermediaries’,Footnote ⁴⁶ which simultaneously opens opportunities for undue influence and distortion: ‘Explanation includes more human influence than sheer transparency’.Footnote ⁴⁷

It is important to recognize that efforts to reframe transparency in this manner – away from its core meaning of disclosure – are not unprecedented, nor unique to AI algorithmic debates. Such arguments have been recurrently and strategically deployed, for instance, by public actors to advance preferred (and often self-serving) versions of transparency. For instance, under pressure to increase its transparency, the European Central Bank (ECB) advanced notions of transparency as mediated information provision, in an effort to control the parameters of its own transparency.Footnote ⁴⁸ It resisted providing access to raw information in favour of the object explaining and translating the rationale. For instance, it argued explicitly in its annual reports: ‘Transparency means not only releasing information, but also structuring that information in such a way that the public can understand it… Transparency requires central banks to clearly explain how they interpret and implement their mandates’.Footnote ⁴⁹ In doing so, it strategically followed a deliberate and autonomous strategy to assert its discretion and ensure a version of transparency to its own liking, affording itself control over what it chooses to reveal.

Understanding communication as a form of transparency in rationale is heavily contested. Scholars rightfully dispute that it is in fact a form of transparency that can really lead to accountability with the secret keeper able to retain absolute control over what is released, when and how.Footnote ⁵⁰ The very word communication (or explanation) implies linearity of the process and passivity on the part of the beholder or recipient. Transparency in rationale enables the secret keeper to enjoy almost unlimited discretion to autonomously decide what to intentionally reveal and what not to reveal and with what slant to ‘communicate’ it. It may generate some legitimacy for the secret keepers, but it is also more vulnerable to manipulation.

A. Hand in glove: explanation and secrecy protections

With its emphasis on explainability, the discourse on transparency in relation to AI has re-interpreted and reshaped transparency in ways similar to what has been described above, towards a logic of communication. In these understandings, transparency is no longer about seeing or visibility but rather a significant re-framing occurs towards explanation, which is to facilitate understanding – essentially, a form of heavily mediated information provision is being advanced. This shift is visible not only recurrently in scholarly debatesFootnote ⁵¹ but also in policy documents. For example, the High-level Expert Group on Artificial Intelligence in its ethical guidelines for trustworthy AI describes transparency in this context as ‘closely linked with the principle of explicability’Footnote ⁵² and defines transparency as ‘[i]ncluding traceability, explainability and communication’.Footnote ⁵³

This recasting of transparency is motivated by the fact that while in its original meaning transparency ‘assumes that when we see by ourselves, we can understand what is happening’,Footnote ⁵⁴ this assumption, to a significant extent, no longer holds true when it comes to complex AI algorithmic systems’ functioning. In a digitalised word, the promise of seeing by knowing seemingly starts to fray, with the potential to render transparency’s claim to knowability more tenuous.Footnote ⁵⁵ Machine learning models such as neural networks (or ‘deep learning’), the most important part of ML these days, can be extremely complex in their operation: hundreds of layers deep, thousands of features and millions (and even billions) of weights that contribute to one predictive outcome.Footnote ⁵⁶ The sheer size of the parameter space and the complexity of feature interactions give rise to unprecedented opacity. By contrast to the traditional contexts we are accustomed to demand transparency of, in the context of AI, visibility or disclosure does not automatically lead to understanding: ‘Seeing inside a system does not necessarily mean understanding its behavior and origins’.Footnote ⁵⁷

This, in turn, has led some authors to altogether disavow the importance of transparency as seeing inside the black box (ie disclosure of underlying training data, source code, and model) as a means to govern algorithmic models: ‘transparency is an inadequate way to govern algorithmic systems’.Footnote ⁵⁸ Various authors have highlighted the limitations of traditional understandings of transparency in this context, advocating for a move away from its core understanding. For instance, de Fine Licht and de Fine Licht speak of the ‘harms’ of ‘full transparency’ arguing that disclosing how systems goals are defined, coded, and implemented would make ‘[f]inding the relevant information … as difficult as finding the proverbial needle in a haystack’,Footnote ⁵⁹ while Wischmeyer claims that information providing measures fail to impart a sense of agency upon those affected by algorithmic systems.Footnote ⁶⁰ Transparency becomes relegated to: ‘unnecessary and always insufficient to simply look inside structures’ in the context of AI.Footnote ⁶¹

At the same time, such conclusions can be contested in important ways. While lay citizens might not understand through disclosure alone, this does not remove its relevance when it comes to important fire-alarm forums, such as the media, regulators, or scholars. We have seen for instance, computational journalistsFootnote ⁶² and scientistsFootnote ⁶³ play critical roles in this respect, with the added obstacle of having to additionally reverse-engineer secret models to obtain some semblance of model disclosure to be able to investigate model behaviour and exercise crucial fire alarm roles. Such forums now must try to second-guess and imperfectly approximate features of secret models, with much higher thresholds to become activated (if at all) in their fire alarm roles, with proprietary protections acting as ‘a key constraint…making it difficult for independent researchers to dissect them’, left to ‘only guess as to the actual mechanisms’ behind disparities.Footnote ⁶⁴

In addition to being qualified as unnecessary in dominant debates, transparency is simultaneously presented as an impossible ideal. Disclosure, the argument goes, would raise unreasonable demands of private actors to forgo proprietary protections, potentially harming innovation and/or creating opportunities of gaming by those targeted though the deployment of AI systems. For instance, Joshua Kroll and co-authors argue that:

advocating for a move away from transparency as visibility (or disclosure). Yet, the effectiveness of secrecy as an antidote for gaming is far from uncontested in the technical literature.Footnote ⁶⁶ Cynthia Rudin points out that well-designed systems ensure attempts to game the system align with overall goal improvement – for instance, attempting to ‘game’ one’s credit rating by going out of red aligns with the ultimate policy goal of improving one’s credit worthiness.Footnote ⁶⁷ What is more, many characteristics – such as past record and family history – cannot actually be gamed but are immutable. It is rather sub-optimal systems based on poor proxies – such as the use of water usage measurements as an indicator of benefits fraudFootnote ⁶⁸ – that are likely to lend themselves to gaming. Such setups raise important questions whether public systems should not rather be aimed towards overall system improvement (through well-designed systems) rather than trying to ‘catch out’ citizens with poorly designed models protected by secrecy.

These critiques notwithstanding, the risk of misuse of disclosed information has had an enduring appeal and is often invoked as part of arguments for the inadequacy of transparency as disclosure. In these contexts, where it is argued opening the black box could prove harmful to private interests and/or the achievement of end goals, explanation techniques of black boxes have been advanced as an alternative to disclosure, as a way to have our transparency cake and eat it. This amounts to understanding without seeing (without disclosure), and with proprietary protections safely in place.

B. XAI to the rescue? Having your transparency cake and eating it?

The move (or swerve, rather) towards transparency-as-explanation is epitomised in technical solutions in the form of the explainable AI (or XAI). This umbrella term refers to a variety of techniques attempting to explain black-box model behaviour.Footnote ⁶⁹ Traditionally developed by engineers to investigate model behaviour and troubleshoot models, such techniques are being advanced as a solution to black-box transparency problems, this time in institutional and governance contexts. These techniques generally involve interfacing a black-box model with a second post-hoc explanation model.Footnote ⁷⁰ Essentially, using a simpler algorithm to explain the black box, to effectively re-digest the black-box model into something we can understand. The post-hoc explanation model is often a different model altogether to the underlying black box, with different input features than the original black-box model.Footnote ⁷¹ Rather than seeing with our own eyes, with explainable AI, providers of AI systems develop explanation models of their black-box models, playing a mediating role between the AI system and those affected by it. At first sight, in light of their justificatory connotations, such approaches seem to rhyme with the demands of reasoned decision-making (and reasoned administration, when public authority is concerned). What is more, the appeal of such approaches simultaneously stems from the fact that they also bypass opening the black box. With such techniques, the underlying model can remain secret or undisclosed safeguarding intellectual property protections – with the explanation model facilitating understanding without actually seeing inside the black box.

While seemingly summoning the promise of transparent and reasoned decision-making, we argue below that the transparency-as-explanation approach ultimately fails to deliver on this promise. What we potentially stand to lose in the process is precisely what is at stake with the logic of communication more broadly: simply put, the emphasis on explanation opens up transparency for influence. Mediated explanations open possibilities for abuse and raise questions about the reliability and truthfulness of what is revealed. There is now an intermediary (often the provider of the AI system) controlling, through technical means, that what is revealed, in a context where the intermediary has high stakes in what is being revealed or kept secret. Effectively, the quality, fidelity and truthfulness of the account provided hangs on the trustworthiness of an intermediary with heavy vested interests in the content of the information presented.

The issue is not a theoretical one: while such techniques can be valuable for designers and system engineers to understand and investigate their models in the design process, we are seeing growing critique of these approaches when it comes to their ability to allow us to really understand specific outcomesFootnote ⁷² and afford meaningful transparency of system functioning. This raises serious questions about their suitability as tools of transparency in a governance and institutional context, which we discuss in Section 5 below. But first we critically explore the provisions of the draft EU AI Act in terms of access, disclosure, communication, and secret-keeping and demonstrate how the re-framing of transparency has bled into legislative efforts in this regard.

A. Transparency towards users

The term ‘transparency’ is used sparingly in the main body of the Act, mostly in the context of the relationship between providers and users of an AI system. High-risk AI systems are subject to specific transparency requirements under Article 13 of the AI Act. Under this article, any high-risk AI system must be transparent enough to allow users – the ones putting an AI system into use under their own authority,Footnote ⁸⁹ which are not necessarily the end users of a system – to interpret its outputs and use them ‘appropriately’. Transparency, so construed, must be ensured throughout the design and development of the AI system,Footnote ⁹⁰ a formulation that suggests providers are required to adopt technical approaches and measures that foster transparency.Footnote ⁹¹ Under a minimalist reading of this transparency requirement, it would be sufficient to communicate to the user instructions on how to interpret the output, such as potential biases or underlying assumptions of the decisional model. A more expansive approach to communication would require providers to explain how the system arrives at its outcomes to users. But, under a logic of disclosure, proper interpretation and use of the outputs requires information about the system itself and the decision logic embedded in it. The current formulation of transparency in the AI Act does not exclude any of these possible interpretations, giving providers ample leeway in the choice of technical means for complying with Article 13(1) AI Act. This risks perpetuating sub-optimal transparency practices and opaque systems accompanied by poor or incomplete transparency measures that provide little insight into actual system functioning.

The AI Act includes a second source of transparency for users: mandatory use instructions. Article 13(2) AI Act requires that any high-risk AI system be accompanied by instructions to users, which must include at least:Footnote ⁹² the contact information of the provider; the characteristics, capabilities, and limitations of the system, such as its level of accuracy;Footnote ⁹³ any changes to the system and its performance which have been pre-determined by the controller at the moment of the initial conformity assessment; the human oversight measures adopted in the system; and the expected life cycle of the system, including necessary updates and maintenance practices.

However, compliance with these requirements requires the provider to give up little information about the inner workings of the system. Instead, all the required information can be provided in an entirely black-box fashion, in which the provider specifies how users should interact with the system and supplies statistics purporting to represent the properties of system outputs.Footnote ⁹⁴ As a result, even a fully compliant provider still retains non-transparent control over information about the technical aspects of the system they originate. A logic of communication paired with secrecy is what is at stake.

Providers are also required to adopt technical measures meant to ensure users have access to information about the everyday operation of AI systems. Any high-risk AI system must include tools that enable automated logging of what the AI system does,Footnote ⁹⁵ thus enabling the traceability of its operations throughout its life cycle.Footnote ⁹⁶ Providers are also required to identify the circumstances in which human oversight is needed to prevent or minimize risks to health, safety, and fundamental rights,Footnote ⁹⁷ and adopt technical measures that provide access to information necessary for overseeing the operation of the AI system.Footnote ⁹⁸ These formulations are not incompatible with a logic of disclosure; however, they, too, again, can also be addressed by purely communicative means such as explanation techniques.Footnote ⁹⁹ Once again, the AI Act leaves providers with considerable latitude to provide ‘appropriate’ transparency without any meaningful disclosure of the system’s inner workings.

Finally, the AI Act also includes provisions regarding transparency of certain classes of AI systems towards users. One such use concerns AI systems designed for interacting with natural persons, such as the chatbots often used to automate customer service. Providers of such systems must design them in such a way that informs the natural persons in question that they are interacting with an AI system and not with a human.Footnote ¹⁰⁰ By mandating disclosure of the artificial character of the system, the AI Act seeks to close opportunities for impersonation and deception,Footnote ¹⁰¹ which can be harmful even if the system itself is not used for a high-risk purpose.Footnote ¹⁰² Nevertheless, disclosure remains limited to the artificial character of the system, leaving providers with free rein regarding what information they will communicate about the system itself.

B. Transparency to the general public

Providers are subject to various, if somewhat shallow, transparency duties to users of certain AI systems. However, the users covered by the AI Act are seldom the only ones put at risk by the operation of an AI system. In fact, in most public sector applications, one could argue there is little risk for users themselves, as the risk resides rather with the citizen(s) affected by the user’s deployment of the AI system. User-facing transparency measures such as those described above might be of little use for many people adversely impacted by AI systems.

In fact, one of the main lines of AI Act criticism among scholars and civil society is that individuals are afforded no right to obtain information about systems.Footnote ¹⁰³ Instead, the AI Act requires communication to the public of certain kinds of information about AI systems. Users of emotion recognition and biometric categorization systems are required to disclose the existence of these systems to the people exposed to them,Footnote ¹⁰⁴ and users must indicate when they use an AI system to generate a deep fake.Footnote ¹⁰⁵ These communication requirements seek to ensure people will not be fooled into thinking they are dealing with a human rather than a machine. Nevertheless, they do not equip the general population with any mechanisms for obtaining information about the AI systems used for such purposes.

High-risk AI systems are subject to more extensive disclosure requirements. Before a high-risk AI system is placed in the market or put into service, its provider or authorized representative must register it in an EU database of high-risk AI systems.Footnote ¹⁰⁶ During the registration process, providers or their representatives must provide various types of information,Footnote ¹⁰⁷ including the Member States in which the system is or has been placed on the market, put into use, or otherwise made available; the intended purpose of the AI system; information about conformity and certification markings; and an electronic version of the instruction for use mentioned above.Footnote ¹⁰⁸ Since the registered information is required to be accessible to the public,Footnote ¹⁰⁹ the database discloses to the public information about the operations of high-risk AI systems.Footnote ¹¹⁰

However, the reach of this disclosure is limited considerably by various aspects of the AI Act. Only high-risk systems that are placed on the market or put into service in the Union market must be registered,Footnote ¹¹¹ and only the provider – or their representative – is obliged to register AI systems into the database.Footnote ¹¹² Since many deployments of AI systems are made by users who are not providers of the AI systems they use,Footnote ¹¹³ the database is likely to be silent about many of the applications of high-risk AI systems in real-world contexts.Footnote ¹¹⁴ Even in the cases in which an application is registered into the database, the actual information present in the database offers little disclosure about the AI system itself.Footnote ¹¹⁵ Such an approach, in fact, may end up legitimizing high-risk AI applications by offering formal compliance with communicative affairs as an alternative to scrutiny over the system and the purposes for its use.Footnote ¹¹⁶ Transparency in the AI database is, therefore – and yet again – a largely communicative affair, in which the general public has access to bits of information selected and curated by the providers of AI systems.

C. Transparency in the governance of AI systems

We are not arguing that there are no deeper requirements for disclosure in the AI Act. Such requirements, however, are formulated in the language of product safety legislation that evolved in a completely different time frame and for very different regulatory objects. Drawing from the New Legislative Framework for harmonized product safety legislation,Footnote ¹¹⁷ the AI Act requires providers of high-risk AI systems to demonstrate ex ante compliance with the technical requirements laid down for such systemsFootnote ¹¹⁸ and adopt a post-market monitoring system to follow the risks posed by the system placed on the market.Footnote ¹¹⁹ Each of these processes is grounded on information about the AI system and its operation, which means their operation depends on various transparency requirements laid down in the corresponding provisions of the AI Act.

Before placing a high-risk AI system on the market, providers must draw up a written EU declaration of conformity for each systemFootnote ¹²⁰ and affix the CE marking to the system or its documentation.Footnote ¹²¹ Such conformity statements can usually be obtained without disclosing information to entities beyond the controller, as the AI Act determines that most systems can be subject to internal controls for conformity.Footnote ¹²² Even if the provider must rely on a third-party assessment for conformity of a given AI system,Footnote ¹²³ this third party is bound by secrecy requirements, which restrict further disclosure of the information obtained as part of a conformity assessment.Footnote ¹²⁴ As a result, the ex ante mechanisms of the AI Act provide a clear example of transparency seen through a logic of communication.

Communication is also an important element in post-market monitoring. Under Article 61 AI Act, providers are required to ‘actively and systematically’ collect, document, and analyse data about the performance of the AI systems they provide, in order to evaluate compliance with the requirements imposed upon high-risk AI systems.Footnote ¹²⁵ In particular, providers are obliged to report serious incidents and malfunctions of a high-risk AI system,Footnote ¹²⁶ as well as situations in which the use of the system may present risksFootnote ¹²⁷; for example, because of its application to a new purpose or safety failures discovered after the system was placed on the market. Providers thus act as an information nexus for AI systems placed on the market or put into service, collecting information from users and other sources,Footnote ¹²⁸ analysing it and reporting potential issues to the relevant authorities.

Post-market enforcement of the AI Act rests upon market surveillance authorities (MSAs).Footnote ¹²⁹ For AI systems used by EU institutions, agencies, and bodies, this role is to be played by the European Data Protection Supervisor.Footnote ¹³⁰ The scenario is somewhat more fragmented at the national level: as a rule, national supervisory authorities also play market surveillance roles,Footnote ¹³¹ but systems subject to other instruments of EU product harmonization legislation or used by financial institutions regulated by EU financial services legislation are subject to the corresponding supervisory authorities.Footnote ¹³² In addition, market surveillance for certain categories of high-risk AI systems used for law enforcement or immigration, asylum, and border control management purposes falls either to the data protection authority or the authority supervising the sector.Footnote ¹³³ MSAs are expected to cooperate to pursue their shared goal of carrying out activities and taking measures to ensure AI systems comply with the harmonization requirements, as many AI systems are likely to be used in more than one Member State.Footnote ¹³⁴

MSAs need considerable amounts of information to adequately carry out their duties. In part, information is supplied by providers as they discharge their reporting duties outlined above. However, MSAs also have access to mechanisms for proactively obtaining information. These authorities can request access to technical documentation about high-risk AI systems,Footnote ¹³⁵ which providers must draw up and keep updated with the information needed to demonstrate compliance with the requirements of the AI Act.Footnote ¹³⁶ In addition, MSAs have the power to organize technical tests of the high-risk AI system to detect potential breaches of obligations under Union law intended to protect fundamental rights.Footnote ¹³⁷ These provisions require providers – and users, where applicable – to supply substantial amounts of information to MSAs, either spontaneously or upon demand by the authority. Nevertheless, these provisions still follow largely a logic of communication, as the access of the MSA to the system is mediated by the content of the reports and technical documentation or by the reported results of software tests.

Some mechanisms available to MSAs offer the potential of unmediated disclosure of AI systems. Upon a reasoned request, these authorities can obtain access to the source code of a high-risk AI system to the extent such access is needed to assess the conformity of the high-risk AI system with AI Act requirements.Footnote ¹³⁸ MSAs can also obtain access to the data used in the training process of a high-risk AI system,Footnote ¹³⁹ a provision that is particularly relevant for the analysis of machine learning systems. These mechanisms force providers – and users, where applicable – to disclose information about the inner workings of an AI system.

Based on these provisions, MSAs can scrutinize high-risk AI systems from angles unavailable to the general population or even to most users of high-risk AI systems, introducing elements of disclosure into a communication-driven approach to transparency. Yet, the actual impact of these disclosure instruments might not be enough to ensure the widespread transparency expected of AI systems. In the absence of independent fire alarms, such as those provided by civil society activists and complaints to authorities from concerned individuals, the MSAs’ attention is likely to be directed only to these issues brought to their attention by provider reports.

Transparency is further limited by the secrecy requirements surrounding the activities of MSAs. This is an important limitation. Under Article 70 AI Act, these authorities must respect the confidentiality of information and data they obtain as they perform their duties. This provision affords special protection to certain interests, such as national and public securityFootnote ¹⁴⁰ and intellectual property rights and trade secrets,Footnote ¹⁴¹ which often clash with the public interests promoted by transparency in AI systems and elsewhere.Footnote ¹⁴² The confidentiality requirement from Article 70 AI Act does not create formal limits to the information-gathering powers conferred to the MSAs. But it nonetheless creates obstacles to these authorities as they seek to obtain and disclose information about AI.

Part of this comes from the introduction of upstream opacity, as providers and users might be subject to confidentiality requirements that restrict the kinds of information they can communicate to third parties.Footnote ¹⁴³ But confidentiality requirements also reduce downstream flows of information, as the MSAs are constrained in their ability to share information with third parties that might otherwise contribute to the governance framework,Footnote ¹⁴⁴ such as civil society watchdogs. In particular, access to information on high-risk AI systems used in law enforcement and asylum, immigration, and border control management is subject to various constraints: direct access to their documentation is only possible to staff members of the MSAs holding the appropriate level of security clearance.Footnote ¹⁴⁵ Furthermore, any mediated disclosure of information by the MSA in such domains of application requires prior consultation of the user of the AI system – that is, the public body deploying the system.Footnote ¹⁴⁶ While such a requirement is in line with the ‘originator control’ principle that is prevalent in these sectors,Footnote ¹⁴⁷ it nevertheless introduces friction in the cooperation between MSAs and considerably narrows down the possibilities for subjecting this information to the scrutiny of actors other than the MSAs directly involved with the system. These upstream and downstream effects of opacity ensure that providers and certain types of users – particularly public sector users of high-risk AI systems for law enforcement and asylum, migration, and border control management – remain in the driver’s seat of transparency under the AI Act. Disclosure, once again, can only go where communication allows it to.

A. The limits of explanation

Overall, the information ecosystem foreseen in the AI Act is tilted towards protecting the interests of providers and users at the expense of the individuals and the overall accountability system. As we saw above, key provisions on transparency target obligations of providers towards users – enshrined in Article 13, under the heading ‘transparency and provision of information to users’ – rather than to individuals affected by these systems.Footnote ¹⁴⁸ This is likely a by-product of the existing models of product liability that informed the approach taken by the Act as well as its legal basis in the internal market. The result is a curious and unsatisfactory halfway house: while the AI Act is meant, inter alia, to govern the use of AI in the public sector, and while it touts protecting public values such as fundamental rights, its transparency measures, geared towards product safety risks, offer little remedy for the unique risks the use of AI by public authorities imposes upon individuals affected by such deployments: Those adversely affected by the system, and the citizenry at large, are mostly forgotten when it comes to information and disclosure rights.Footnote ¹⁴⁹ The shift from a logic of disclosure to a logic of communication thus erodes the role of transparency in the oversight of public sector decision-making, as the public is deprived of access to any sources of information about AI systems other than those heavily mediated by the very actors against which transparency should provide a safeguard.

What is more, the disclosure obligations of providers (to users), too, while the more elaborate set of information obligations in the draft Act, remain overly underspecified. Requirements to ensure that the operation of high-risk AI systems they develop are ‘sufficiently transparent’ to allow for user interpretation of output and appropriate use, ensuring an ‘appropriate type and degree of transparency’Footnote ¹⁵⁰ leave discretion to providers to make key transparency choices. With conformity a matter of provider self-assessment, what exactly ‘sufficiently transparent’ for interpretation or an ‘appropriate type and degree of transparency’ entails, becomes a matter of provider assessment and ultimately, provider choice. Moreover, while providers are mandated to build high-risk AI systems for human oversight,Footnote ¹⁵¹ the draft Act leaves the measures to be put in place to facilitate interpretation at provider discretion, and ‘as appropriate to the circumstances’.

This will likely perpetuate below par transparency practices opening the door to providers to stack the deck in their favour and provide little insight as to actual system functioning. For instance, popular explanation techniques of black boxes that have received a lot of press as potential transparency ‘fixes’ – such as counterfactual explanation modelsFootnote ¹⁵² – allow for multiple explanations: when applied to the same model, under the same circumstances, different techniques for XAI (or in fact, the same explanation technique deployed by a different developer) may highlight different attributes as being crucial for a particular decision. Since any mathematical model is built upon procedures of abstraction, idealization, and approximation, choices on which elements to abstract, idealize, or approximate afford considerable discretion to the builder of the explanation model: ‘[ … ] left to their own devices, decision makers are afforded a remarkable degree of power to pursue their own welfare through these choices’.Footnote ¹⁵³ As a result, the provider of an explanation – which is usually conflated with the provider of the black-box model being explained in the first place – has considerable leeway to decide which features to disclose in the explanation, opening up opportunities for gaming by providers to self-serving ends: ‘while designed to restore power to decision-subjects, partial explanations grant a new kind of power to the decision maker, to use for good or abuse as desired’.Footnote ¹⁵⁴

Relatedly, an often overlooked aspect with respect to explanation models of black boxes more generally is that they represent ‘approximations’ of underlying models rather than faithful renditions of their logic.Footnote ¹⁵⁵ An explanation model is not merely a simplified version of the original model obtained by removing irrelevant factors; instead, it is a different model that approximates the behaviour of the black box under study, and, in doing so, might ascribe different weights to the features used in the black-box decision, or even use different features or attributes altogether.Footnote ¹⁵⁶ In the words of leading computer scientist Cynthia Rudin in her influential Nature article warning against the use of such techniques in high-stakes decision-making:

While post hoc explanation techniques can be valuable for developers and system engineers ‘as part of the knowledge discovery process’Footnote ¹⁵⁸ to investigate model behaviour, explanations are often partial or incomplete, failing to provide sufficient information to fully understand what the underlying black box is doing.Footnote ¹⁵⁹ There is a real risk, therefore, that explanations – even when produced in good faith – amount to partial and as such misleading descriptions of model functioning,Footnote ¹⁶⁰ rendering them of limited use as tools of oversight from a legal perspective. They cannot provide decision subjects the insights needed to subsequently take up and contest specific decisions and demand accountability.

Such concerns are further echoed by work on ‘misleading explanations’ showing how user trust can be manipulated through unreliable explanations that do not accurately reflect the issues and biases in the underlying black box.Footnote ¹⁶¹ What is more, recent computer science work also illustrates how explanation models can be purposefully manipulated by their owners to hide biases from external auditors, further speaking to the vulnerabilities and shortcomings of such methods.Footnote ¹⁶² These insights perfectly mirror concerns with transparency as communication more broadly: its potential for influence, placing the object rather than the beholder in control to shape what is made visible, and in doing so, to reshape the parameters of its own transparency and accountability.

Given these concerns, leaving the technical measures to be put in place to afford interpretation of system’s output and transparency up to the providers of AI systems, as the draft regulation does, is highly problematic. If model functioning is not actually transparent, diagnosing failure and enacting accountability becomes next to impossible. The absence of robust and unambiguous legal standards as to what measures are required to afford meaningful interpretation of system outputs is a serious shortcoming that will detract from effective transparency and accountability in this area.

B. User disclosure duties as public authorities: on the need to incentivise responsible user behaviour

Users of high-risk AI systems, while recipients of information from providers as detailed in Section 4, are themselves in turn subject to perfunctory, token obligations – primarily amounting to complying with a set of instructions of use provided by providers with no meaningful transparency duties of their own under the Act. In the context of public sector applications, these duties are particularly insufficient, as they fail to attend to the specific needs of transparency in the activities of public authorities, again likely the result of the product safety and market focus reflected in the Act’s legal basis. As we saw in the Dutch context and the toeslagenaffaire, uses of AI systems by public authorities can bring about fundamental, life-changing implications for individuals. The lack of disclosure duties of users towards those adversely affected by the systems they deploy is disconcerting given the growing trend visible across public sectors to rely on algorithmic systems managed by external suppliers to implement consequential public tasks, often subject to secrecy provisions as to these systems’ set-up and functioning. As they do so, public bodies are effectively abdicating consequential public tasks to private providers with the prospect of little or no oversight of system functioning.Footnote ¹⁶³ In this context, the failure to include meaningful information duties for public authorities as a particular class of users within the scope of the legal act that is to regulate the EU-wide use and deployment of AI (also in the public sector) for years – if not decades – to come is a glaring omission.

The absence of safeguards on this will lead to the continued adoption of AI models that preclude scrutiny of their functioning. It lets users of high-risk AI systems ‘off the hook’ and encourages them – given that they lack disclosure duties of their own towards those affected by their systems – to continue to purchase secret and proprietary models. This is disconcerting in a context where public authorities are found to rely on overly-complex, proprietary models to implement public tasks even when non-proprietary alternatives of equivalent performance exist or can be developed.Footnote ¹⁶⁴ Reliance on proprietary models entails that algorithm design, methodology and functioning will remain hidden to the public bodies that deploy them, to individuals adversely affected by decisions informed by them and/or to citizens, exacerbating information asymmetries vis-à-vis private providers and removing opportunities for meaningful oversight. For instance, a report by the Netherlands Court of Auditors noted that public bodies have little insight into the quality of algorithms they rely on when these are managed by external providers:

By contrast, requiring strict transparency duties of users such as public authorities will encourage them to be demanding users and consumers of AI and to in turn push high standards upstream, demanding openness and high transparency standards of providers. For instance, rather than purchasing proprietary AI models, public authorities could contractually require providers to forgo proprietary protections to be able to sell high-risk AI systems into the public sector.Footnote ¹⁶⁶ Where transparency cannot be secured in this context, serious consideration should be given to whether the use of such systems in high-stakes public sector is justified. In this vein, some MEPs have already proposed that the deployment of AI systems should be subject to fundamental rights impact assessments.Footnote ¹⁶⁷

C. Gagging accountability: absent or silenced ‘fire alarms’

In terms of enforcement, while the AI Act affords in principle considerable information disclosure to national competent authorities (MSAs) in their enforcement roles, in the absence of independent fire alarms, providers hold the ‘chokehold’ on monitoring and whether public regulators become ‘activated’ in their enforcement roles to begin with. This is because providers themselves are largely responsible for carrying out post-market monitoring. As discussed above, market supervisory authorities rely on provider notification of incidents and malfunctioning and there is no mechanism provided for individuals adversely affected to lodge a complaint with market surveillance authorities.Footnote ¹⁶⁸ This is a significant omission from an accountability perspective, as the enforcement system lacks much-needed ‘fire alarms’ providing independent feedback on system (mal)functioning. Users of AI systems, too, while they are to monitor and play a role in flagging malfunctioning systems, they do so vis-à-vis the provider (rather than directly to regulators). Such a setup, by concentrating post-market monitoring responsibilities with providers, simultaneously creates a closed information circuit with providers as central nodes controlling access to information on system functioning. This exacerbates a heavy dependence on providers doing their due diligence in monitoring and indeed, on them truthfully reporting on their models’ malfunctioning. To know even where to begin to look, regulators need to rely on providers – a dependency which stands to aggravate already significant informational deficits. Such deficits are likely to be further compounded by anticipated resources and expertise shortages of MSAs themselves, which are likely to be significant,Footnote ¹⁶⁹ especially so in light of the complexity, technical nature and the sheer size of the regulatory domain. These constraints are particularly salient in cases in which supervisory authority is ascribed to under-specialized regulators (remember that the AI Act does not foresee the creation of purpose-specific regulators), which might lack the resources to cultivate extensive in-house expertise in AI, already struggling to keep up with existing responsibilities (as plainly seen with some national data protection authorities).Footnote ¹⁷⁰

Even in the cases when an MSA has already obtained access to aspects of the AI system – for example, access to training data is not conditioned upon a reasonable request – rendering the system transparent actually requires considerable technical work,Footnote ¹⁷¹ which resource-constrained authorities are unlikely to perform in the absence of reasons that warrant attention to that particular system. Consequently, even the unmediated disclosure tools available to MSAs depend upon communication practices by providers and users of high-risk AI systems.

Given this informational dependence, MSAs’ enforcement will likely be geared towards issues flagged by the provider, especially given envisaged capacity constraints to independently monitor and police patrol the crowded domains under their purview. This regulator informational dependence is particularly problematic given stakes shaping provider disclosure incentives – dominant private actors in this area have been found to go to great lengths to quash internal dissent and prevent the publication of internal research evidencing harm.Footnote ¹⁷²

Importantly, it is also unclear to what extent providers will even be able to exercise such post-marketing monitoring roles meaningfully vis-à-vis some categories of users such as public authorities in secrecy-dominated areas (such as law enforcement or asylum, as noted above). Adequate monitoring and enforcement will likely be absent precisely in the highly sensitive domains where they are most needed, given the sensitive nature of the data and state secrecy protections characteristic of these fields.Footnote ¹⁷³

What is more, while market supervisory authorities (once activated) have the possibility for extensive access to information, they are simultaneously bound, as we saw above, by confidentiality clauses. As a result, there is little room for this information to feed back into much needed cross-border regulatory coordination in this area (deepening fragmentation), or in fact, into public discussion or debate on this information, preventing MSAs, in turn, from fulfilling their own fire alarm roles towards other public forums (such as parliaments, other national regulators, courts, civil society watchdogs, or the media). As a result, there is little room for public discussion or debate on this information or opportunities for other forums to take it up further. This in turn removes the raw ingredients for accountability – information and deliberation – from public space. Thus, due to both upstream and downstream opacity, accountability is depleted of the very lifeblood that sustains it: transparency.