Background

Should artificial intelligence (AI)-generated predictions be deployed in medical decision-making? Should clinicians align their medical verdicts with the output of black-box AI? Is it justified to use opaque AI output in medical decision-making? Consideration of these questions is of central importance for the responsible use of machine learning (ML) models, which have been shown to produce accurate and reliable diagnoses, prognoses, and treatment suggestions in medicine.Footnote ¹

The impressive technical achievements of ML models have been proclaimed to show that medical decision-making is on the verge of a revolution. Articles and editorials in medical journals promise a future where medical decision-making is increasingly deferred to AI devices. Some even go as far as suggesting that medical reasoning can be reduced to the application of a computer program: “if all past, present, and future predictors and processes that contribute to future events were known and quantifiable, algorithms could be constructed that produce perfect risk estimates for individuals—that is, they would predict with perfect accuracy whether an event would occur or not in every individual.”Footnote ²

The most accurate ML models such as deep neural networks are opaque or black boxes in the sense that they are so complex that it is impossible for humans to comprehend why an output is produced from an input. This opacity is a concern for the use of AI systems in medical decision-making, and it has been remarked that there is “a growing chorus of clinicians, lawmakers, and researchers calling for explainable AI models.”Footnote ³ Moreover, it has become clear that it is not straightforward to ensure the deployment of AI systems in clinical settings and that clinicians hesitate to use accurate and reliable AI support in decision-making. Footnote ⁴ Thus, the expected improvements in patient care are not achieved.

Research exploring why clinicians are reluctant to deploy AI systems suggests that clinicians find that in order to justify AI-supported decisions, they must have access to an explanation of why an output was produced from an input.Footnote ⁵ Ensuring explainability is expected to make clinicians more willing to trust a black-box AI output and thus use AI more frequently in decision-making. I will call the view that explanations are required for the justified use of AI systems the Explainability View.

As a result of the need for explanations, there has been a surge in developing methods that can generate post hoc explanations of why a black-box AI produced an individual output. Examples of such explanations are heat maps that provide clinicians with images where, for example, the red colors indicate areas that the AI model considers high importance and other colors, for example, blue, indicate areas with lower importance.Footnote ⁶ Because they allegedly can provide explanations of individual AI output, explainable AI (XAI) methods are expected to make these outputs more trustworthy to clinicians and thereby make clinicians more willing to use them to make decisions.

The use of the notion of trust in the debate about opaque AI is confusing. According to a standard distinction in philosophy, there are two forms of trust: trust as mere reliability and genuine trust. Footnote ⁷ According to this distinction, an opaque AI cannot be genuinely trustworthy and trusted because it cannot entertain the right motives. However, it can be trustworthy in the sense of merely reliable if its accuracy and reliability have been carefully validated. The central question often articulated as a problem of trust is really a question about whether clinicians are justified in using the output of an opaque but reliable and accurate AI when making medical decisions. On the assumption that the performance of the AI has been validated, clinicians would, as a minimum, seem to be justified and perhaps even obligated to use such output as premises when reasoning about what to do.Footnote ⁸

While recognizing the value of post hoc explanations for the development of fair and reliable AI systems, some scholars caution that XAI methods should not be deployed to explain and justify the use of individual output.Footnote ⁹ The proper way to justify black-box AI output is to ensure the “thorough and rigorous validation of these systems across as many diverse and distinct populations as possible, showing that patient and health-care outcomes are improved and that marginalised groups are not disproportionately affected by any given system.”Footnote ¹⁰ Clinicians should rely on the output of validated AI systems and not require explanations. I call this the Validation View of AI justification.

In support of the Validation View, its proponents argue that many drugs and devices used routinely in medicine are in fact black boxes.Footnote ¹¹ And in the case of drugs, the mechanism that generates the desired effect might not be scientifically accounted for despite decades of research. Still, such drugs are considered acceptable for use because they have been validated using “gold standard” randomized controlled trials. If the output of black-box AI systems has been validated using such standards, then they should be considered justified for clinical use, too.

The debate about the justified use of opaque AI output in medical decision-making can thus be divided into two views. On the Validation View, validation is sufficient for justified use. On the Explanation View, explanations are necessary for justified use. Here is a statement of the two views:

Against the Explanation View

Critics of the Explanation View point out that post hoc explanations suffer from at least two problems. One problem is the interpretability gap. Consider a heat map of lung images produced by XAI methods.Footnote ¹³ The heat map will indicate the relative importance of areas of the image for its output. The interpretability gap arises because it is not clear from the heat map what it is about the hot areas that the AI considers important for its output. Thus, it will be up to the clinician to “fill out the blanks” in the explanation. However, as is well documented, humans tend to confirm their own perception, and thus it is very likely that the clinician will assume that what mattered to the model is the same features of the highlighted area that the clinician herself finds important.Footnote ¹⁴

In response to this problem, heat maps may be supplemented with natural language text stating what features it takes to be significant. This will enhance the clinician’s ability to interpret the heat map. Thus, multimodal models comprising, for example, both visual cues and natural language will mitigate the interpretation gap.Footnote ¹⁵ In this context, it is important to highlight that how fine-grained the explanations should be will depend on the purpose of the audience.Footnote ¹⁶ There is not a one-size-fits-all level of granularity that an explanation must achieve to explain an output.

The second problem is that post hoc explanations merely generate “rationales of black-box predictions, which are not necessarily the actual reasons behind those predictions or related causally to them.”Footnote ¹⁷ Accordingly, post hoc explanations are unlikely to contribute to our understanding of the inner workings of the model. Instead, we are left with the false impression that we understand it better.”Footnote ¹⁸ Rudin seconds this criticism claiming that post hoc explanations “must be wrong” because they misrepresent the way in which the opaque model arrives at its output.Footnote ¹⁹

Proponents of the Explanation View may respond to the criticism in different ways. Against the claim that post hoc explanations cannot be genuine explanations, it can be argued by analogy that if idealized scientific models such as the ideal gas law can provide genuine explanations that enable people to better understand complex natural phenomena, then XAI methods can provide genuine explanations too.Footnote ²⁰

Like scientific models, the simplified models deployed to explain the black box will not be completely faithful to the target they are modeling, but that does not exclude them from providing genuine explanations in the sense of enabling users to make causal inferences relevant for some purpose.Footnote ²¹ The real challenge for XAI is to produce explanation models that meet the standards of validated idealized scientific models. However, there is much work going on to develop XAI methods, so it would be premature to assume that they will not become available.

The Insufficiency of Validation

I have presented how proponents of the Explanation View may respond to two central objections to requiring explanations of AI output to be used when making medical decisions. In this section, I consider the case for the Explanation View within the framework of evidence-based medicine (EBM) and shared decision-making (SDM).

Some scholars find it “hard to imagine a person who would feel comfortable in blindly agreeing with a system’s decision in … highly consequential and ethical situations without a deep understanding of the decision making rationale of the system.”Footnote ²² This worry is succinctly expressed by Lipton when he imagines a model developer exclaiming that “we can train a model, and it can even give us the right answer. But we can’t just tell the doctor ‘My neural network says this patient has cancer!’ The doctor just won’t accept that! They want to know why the neural network says what it says. They want an explanation.”Footnote ²³ Moreover, as already noted, much of the discussion of trustworthy AI, in fact, centers on the observation that clinicians hesitate to accept AI output when explanations are unavailable.Footnote ²⁴

To require explanations is to claim that the fact that predictions are produced by an accurate and reliable source may not be sufficient for being justified in using them when making medical decisions. Call this the Insufficiency Claim. Proponents of the Explanation View are committed to the Insufficiency Claim. However, we should ask whether the Insufficiency Claim is justified.

The central aim of EBM is “to ensure that decision making in health care incorporates the best available evidence.”Footnote ²⁵ Importantly, the incorporation of the best evidence is supposed to be judicious “taking into account both clinical expertise and the needs and wishes of individual patients.”Footnote ²⁶ As outlined, validated black-box AI provides decision-makers with predictions that may be useful for making informed decisions. And while EBM is in large part associated with a focus on “cold hard facts,” properly understood it is “a systematic approach to clinical problem solving which allows the integration of the best available research evidence with clinical expertise and patient values”Footnote ²⁷ Importantly, clinical expertise involves both interpretation and appropriate application of the evidence in the circumstances.Footnote ²⁸ A central clinical skill is thus to be able to “interpret the evidence and apply it appropriately to the circumstances—doing the right things.”Footnote ²⁹ Finally, EBM involves communication to patients of “the information they need to make an informed choice.”Footnote ³⁰

Respect for patient values and autonomy is typically understood in terms of the ideal of SDM. In SDM, “clinicians and patients share the best available evidence when faced with the task of making decisions, and (…) patients are supported to consider options, to achieve informed preferences.”Footnote ³¹ Thus, SDM emphasizes patient autonomy or self-determination, and “clinicians need to support patients to achieve this goal, wherever feasible.” Still, as Sandman and Munthe point out, aiming for patient autonomy need not result in “abandoning the patient or giving up the possibility to influence how the patient is benefited.”Footnote ³² It is thus too simplistic to think of SDM as an exchange between clinician and patient where clinicians simply provide the patients with “facts about the diagnosis and about the prognoses without treatment and with alternative treatments” and leave patients to make decisions on their own.Footnote ³³ To support autonomous and informed decision-making, patient and clinician must engage in shared deliberation on the basis of shared information.Footnote ³⁴ Thus, if “the patient is not provided with sufficient information concerning the confidence of a given diagnosis or the rationale of a treatment prediction, she might not be well equipped to give her consent to treatment decisions.”Footnote ³⁵

To illustrate how SDM requires more from clinicians and AI systems than merely being offered the best available evidence, Bjerring and Busch ask us to imagine that an opaque AI is deployed to rank treatment options for a patient with breast cancer.Footnote ³⁶ It recommends that both breasts be surgically removed and informs about what side effects can be expected. When deliberating with the patient about what treatment to choose, the clinician will arguably not be in a position to provide the sort of information required to honor the principles of SDM and, in turn, EBM. A central piece of information that is missing concerns why the AI system finds a treatment involving the removal of both breasts to be more likely to have good results than alternatives. Simply being told that the AI has been validated to be very accurate and reliable does not seem to suffice for justifying deferring to its output when making this decision.

This consideration highlights that it is the ideal of SDM that introduces explainability as a requirement for justification. The clinician may be said to be justified in thinking that there is good reason to think that the AI system gets it right because it has been adequately validated. However, a significant class of medical decisions—those governed by the principles of SDM—require explainability too. Hence, my claim here is not that a clinician may not, in some cases, be justified in deferring to an AI system’s output simply due to its validated accuracy and reliability, when making a medical decision. The point is that insofar as SDM principles are in play, more will seem to be required.

I have argued that on recent accounts of EBM, it seems plausible to require that for clinicians to be justified in using black-box AI output in their deliberations with patients, such output should be explained, and that it is not in principle impossible for XAI methods to provide such explanations.

Implementation and Responsibility

Coming back to the initial issue concerning how to address the low uptake and unwillingness to use AI output, an important insight from the current debate is that when managers want to implement AI devices in the clinic, they must recognize that it is the medical decision-makers that are responsible for curating and communicating evidence relevant for shared decision-making. Hence medical decision-makers must be able to justify and deliberate about the relevance and quality of the evidence that they present in favor of a diagnosis or treatment suggestion regarding an individual in a particular context. Responsible decision-makers must be able to understand the evidence that they rely on in their reasoning.Footnote ³⁷

Thinking about the responsible implementation of opaque AI output forces reflection on the notions of clinical reasoning and judgment. The introduction of accurate and reliable AI tends to emphasize that such tools produce valuable information about the condition of the patient. In that way, clinical reasoning and judgment are more complex tasks than translating an AI output into a decision. Medical decision-making is characterized by uncertainty along several dimensions,Footnote ³⁸ and AI output does not in itself mitigate all these uncertainties. It must still be negotiated by the clinical reasoner.Footnote ³⁹

Given that clinical reasoning involves a range of tasks that cannot be deferred to AI systems, but must be undertaken by clinicians in collaboration with patients, it seems important to recognize clinicians as epistemologically responsible. Footnote ⁴⁰ They are the ones who will have to fit the output of AI systems into a more comprehensive picture of the patient that incorporates and balances other information than the statistical evidence used by the AI system to arrive at its output.

Importantly, given the epistemic responsibility of the clinicians curating the available information and best evidence about the patient, it only seems appropriate that they have the means to take on this responsibility. Being epistemically responsible, they must also be able to justify their reasoning and judgments. Requiring explanations of opaque AI output seems to support this.

These considerations lead to a final important observation. AI is often presented as “better than experts” at some tasks such as diagnosing skin cancer. However, given the argument of this article, this way of presenting AI output is misleading in a clinical context. Making decisions about diagnosis, prognosis, and treatment is not a merely statistical procedure. The practical reasoning behind a medical decision will typically include premises referring to scientific evidence and statistical information. However, being practical reasoning, it also involves evaluative premises.

Practical reasoning, as I understand it here, is reasoning about what to do. Footnote ⁴¹ This is a normative question posed from the perspective of the individual patient. Which of the available options is the best treatment for patient Hannah? As such, it involves reference to evaluative premises, that is, premises based on Hannah’s personal values. For example, Hannah might find a certain kind of treatment very bad because it will make her unable to pursue an activity, which she finds very important to her quality of life. Or she might have certain religious beliefs that makes her exclude a certain type of treatment. These value considerations form part of the practical reasoning that a clinician will perform, in addition to available scientific evidence and statistical information about what to expect given a certain treatment. Hence, there is no way for the statistical information provided by an AI output to directly determine the right decision. And the further information to be added is (also) the responsibility of the clinician. Hence, the information afforded by the AI output cannot in itself determine medical decisions about individual patients regardless of how accurate and reliable it is.