2. The problems with explanations

The field of XAI has become central in recent years, as it focuses on ways to mitigate one key issue modern AI techniques face: opacity. The concept of explainability stems from the broader goal of making AI systems more transparent and arose in response to critiques of black-box models and the unintelligibility of algorithms (Aradau and Blanke Reference Aradau and Blanke2022, 172). The main aim is to make the inner workings of AI tools more understandable, with the hope that this will help users understand the overall system, including its outputs. Explainability is thus seen as a way to foster accountability and trustworthiness in systems that cannot be fully observed, paused or completely traced (Baron Reference Baron2025). More broadly, it is a crucial element of algorithmic ethics (Ananny Reference Ananny2016; Mittelstadt et al. Reference Mittelstadt, Allo, Taddeo, Wachter and Floridi2016). Due to their characteristics and scale, the internal workings of such techniques remain unintelligible to both users and developers. Consequently, explaining their results becomes a key task. At first glance, the concept of explanation may seem straightforward: producing information – explanans – about a certain phenomenon – explanandum. In other words, explanations are conceptualized as filling a knowledge gap for those who seek them. Many XAI methods align with this basic idea, utilizing techniques such as heat maps or saliency maps, Local Interpretable Model-Agnostic Explanations and others. Their aim is to give users additional insights into how the model, or a proxy, works.

However, this interpretation of explanation is not unchallenged. Many philosophers of science have argued against a static account of explanations and highlighted the importance of the actors involved in the act of explaining: the explainer, who has the additional information, and the explainee, who seeks to know more. For such scholars, the process of transmitting information is just one piece of the process: The goals of the explainee, their background knowledge and skills, and the broader context in which they operate also play an important role in whether an explanation lands and is meaningful to a given individual. After all, explaining how an AI system works to a data scientist or to a layperson looks like two vastly different tasks. Yet, current XAI approaches often overlook these key aspects of explanation as a social and situated practice. While it is undeniable that the process of explanation requires an epistemic content that the explainee must receive from the explainer, this is not the whole story. To go beyond superficial explanations, an explainer’s role is not just to convey facts, but to present them in a way that promotes the explainee’s understanding (Scriven Reference Scriven1962). This is a more profound undertaking than just adjusting the explanation’s tone or level; to promote understanding, the explainee’s goals, background and features need to be considered to empower them to take meaningful actions. This shift may seem subtle, but it is fundamental as it changes the explainee from a passive recipient to an active actor. The process then becomes essentially relational and dynamic, with the actors on the same level, engaged in the co-construction of understanding (Rohlfing et al. Reference Rohlfing, Cimiano, Scharlau, Matzner, Buhl, Buschmeier, Esposito, Grimminger, Hammer, Hab-Umbach, Horwath, Hullermeier, Kern, Kopp, Thommes, Ngonga Ngomo, Schulte, Wachsmuth and Wagner2021).

We therefore propose abandoning the concept of explanation and focusing on understanding as the guiding principle for technical development. This approach aligns with other scholars who have argued that explanations should be social and not just technical, taking into account the epistemic needs and contexts of different stakeholders (Barman et al. Reference Barman, Caron, Claassen and De Regt2024; De Regt Reference De Regt2017; Sullivan Reference Sullivan2022; van Eck and Barman Reference van Eck, Barman, Illari and Russo2024; Zednik Reference Zednik2021). It emerges that, for genuine understanding, explanations are needed but constructed as a socially situated, action-oriented practice rather than a purely technical output. Moreover, it is not always clear that explaining how AI works is essential or even relevant to understanding its outputs. While knowing the reasoning behind specific outputs certainly increases trust and confidence in the quality and reliability of those results (e.g., Lai et al. Reference Lai, Kim, Kunievsky, Potter and Evans2025), it is less clear how this supports the ability of actors to meaningfully interpret and act on these results – a crucial aspect of utilizing these systems. We thus believe that empowering the explainee to take action should be the focus of transparency in machine learning techniques.

3. AI in medicine and health

For illustrative purposes, two brief examples in the fields of medicine and health can illustrate how XAI has become a particularly important element in AI research and development, albeit sometimes with potential trade-offs between the need to increase a model’s predictive power or accuracy and its enhanced transparency. As Raz et al. (Reference Raz, Heinrichs, Avnoon, Eyal and Inbar2024, 3) claim, to explain powerful medical imaging techniques that rely on large datasets, XAI has become increasingly “tweaked around predictability.” For example, the predictions of image detection algorithms, powered by convolutional neural networks, are often explained by showing cluster features or generating heat maps to highlight key regions of an image that would lead to the final result. In cancer detection, as reported by Cabrita et al. (Reference Cabrita, Dantas, Op Den Akker, Chouvarda, Colantonio, Tsakou and Yang2025, 73), clinicians find visualizations such as heat maps and SHapley Additive exPlanations plots valuable for interpreting AI model outputs. The explainability provided by these tools can therefore be linked to greater acceptance and higher effectiveness of AI in clinical practice. Similarly, researchers and clinicians have sought to use saliency maps from deep learning classifiers to explain rare disease diagnoses based on facial recognition technology (Duong et al. Reference Duong, Johny, Ledgister Hanchard, Fortney, Flaharty, Hellmann, Ping, Javanmardi, Moosa, Patel, Persky, Sümer, Tekendo-Ngongang, Lesmann, Hsieh, Waikel, André, Krawitz and Solomon2024). In this context, their goal was to make the predictions of image detection intuitively “explainable” to clinicians, radiologists and medical professionals such as geneticists, thereby translating a key set of assumptions and variables in the models into a visualization that speaks to the existing skills of those professionals.

Explainability is here not so much a characteristic of an algorithm as an approximated property or quality based on human intuition (Freiesleben and König Reference Freiesleben and König2023; Rudin Reference Rudin2018), where developers have selected specific features of the algorithm (its semantic structure or underpinning data clusters), experimented with what they view as “legible” by medical practitioners and useful to their professional reasoning and proposed visualization specifically geared toward responding to those requirements. Saliency maps and plots provide useful technical justifications for specific algorithmic outcomes by clarifying one of the predictive or evaluative steps through which a given assessment is reached. It remains questionable, however, whether tools can enhance understanding of how the algorithm works, why it produces the outputs it does and what underlying causal patterns it is building upon or uncovering. As others have noted, intuitive-looking explanations often confuse the causal role of an explanation with its justificatory role; in other words, explanations do not necessarily justify outputs, and justifications do not explain outputs (Freiesleben and König Reference Freiesleben and König2023; Krishnan Reference Krishnan2020). Justifications also do not necessarily increase the ability to act upon a given result and use it as evidence toward decision-making. Indeed, while XAI tools such as heat maps may play an important role in justifying an algorithmic outcome, they do not constitute an explanation for why the result was obtained, nor do they necessarily foster an understanding of what the result indicates in diagnostic or therapeutic terms. This raises a range of questions: above all, how humane and ethical decision-making can be made over purely technical choices, but also why, how and to whom explanations need to be provided in these medical interactions, and when explainability of an AI system becomes a placeholder or substitute for genuine understanding of a (diagnostic) decision coproduced with that system.

4. From explanation to understanding

Many instances of medical diagnostics employing AI focus on the comparative analysis of large datasets produced by imaging technology. These examples are often described by practitioners as “revealing” or “uncovering” anomalies as a starting point for diagnosis and treatment, e.g., in cases where MRI scans reveal anomalous tissue growths compared to “reference” scans taken to be healthy. AI is discussed as more accurate and less error-prone than human analysts, improving the precision and speed with which diagnoses can be produced (Resühr and Garnett Reference Resühr and Garnett2025; Van Hespen et al. Reference Van Hespen, Zwanenburg, Dankbaar, Geerlings, Hendrikse and Kuijf2021). This picture, however, focuses only on the identification of potential anomalies, rather than the broader process of producing a diagnosis tailored to the specific patient on the basis of such anomalies. Understanding how anomalies are spotted involves the ability to grasp how AI systems label, locate and single out anomalies, which in turn requires a mixture of data science – to understand what input is provided to the algorithms, computer science – to understand what algorithms are employed and how, and biomedicine – to understand how models employed by the algorithms are set up and what forms of biomedical and biological understanding are incorporated within them. Understanding what such anomalies may mean in the context of a specific diagnostic interaction, by contrast, requires situating those elements within a patient’s own history. This, in turn, necessitates different forms of expertise, particularly doctors’ familiarity with how symptoms manifest in patients and how they should be evaluated given patients’ medical context, in order to be interpreted meaningfully and correctly. Understanding how the system works is undoubtedly helpful in such an interpretative process, but it is arguably secondary to the ability to situate AI outcomes in this way. In other words, the AI output here can only lead to medical understanding and thus provide a satisfactory explanation if complemented by medical know-how and tailored to the specific skills and knowledge of the medical practitioners and the patients involved. For doctors to (intuitively) understand how the system may operate is helpful, but it is not the whole story of what makes the system useful in medical practice, and is perhaps also not crucial to understanding outputs at all – especially since understanding the system requires specialist knowledge of forms that differ enormously from what doctors can actually handle from their own training. The critical issue is supporting a debate between AI developers and medical practitioners centered on the extent and quality of explanations needed to enable doctors to interpret the results of AI. This does not necessarily mean explaining every part of the system itself, but rather focusing on the assumptions that such a system makes about patients and related parameters, as well as ensuring that doctors can take those assumptions into account when calibrating the system outputs to their individual cases. Notably, these assumptions are not fully captured by the data input into the system (and its inherent biases and representativeness), but are also introduced by algorithmic choices made around how models are developed, modified and maintained to be able to produce a given result in the desired format.

Expressed another way and building on key insights from the philosophy of science, explanation is meant to foster understanding as the ability to interpret information in order to take action (De Regt, Leonelli, and Eigner Reference De Regt, Leonelli and Eigner2009). Increasing understanding means acquiring the skills to intervene in specific situations to achieve given goals or solve a given problem. In the case of medical diagnostics, such understanding is crucially targeted to the outcomes of AI systems, rather than the system as a whole: learning to intervene is, for doctors and patients alike, to acquire minimal skills in AI to evaluate the relation between assumptions made by AI models and the specific situation of the patient.