3.1. Accepting data as adequate for the target phenomena

As mentioned in the preceding text, Biddle (Reference Biddle2020) discusses how data choices that go into model construction are value-laden and reflective of nonepistemic values. Biddle often frames the discussion in terms of how (implicit) researcher interests and goals can influence data choices. However, researcher interest concerns identifying the purpose or target of the model (Parker Reference Parker2020; Potochnik Reference Potochnik2015).Footnote ³ Although the way nonepistemic values influence the direction of scientific inquiry is important, nonepistemic values still seem relevant once researchers identify a model target or purpose. Adopting the framework of inductive risk, I argue that nonepistemic values can be relevant for assessing whether, given a specific target or purpose, a particular dataset is adequate for establishing a true counterfactual dependence.

Consider the target of whether there is racial bias in police stops in the United States. Pierson et al. (Reference Pierson, Camelia Simoiu, Sam Corbett-Davies, Amy Shoemaker, Phoebe Barghouty, Shroff and Goel2020) took to examining racial disparities in police stops using a data approach. They discuss the importance of having the right data to evaluate whether there is racial bias in policing practices (i.e., assessing model adequacy based on [a]). One important data point—if not the most important for this purpose—is the race of the person stopped by police. In the data archives Pierson et al. accessed, race was recorded based on the stopping officer’s perception. They argue that this method may introduce errors, suggesting alternative methods that potentially reduce these errors. In addition to self-reporting of the stopped individual, they suggest using a third-party’s perception based on a driver’s license photo (Reference Pierson, Camelia Simoiu, Sam Corbett-Davies, Amy Shoemaker, Phoebe Barghouty, Shroff and Goel2020, 742).

In this example, we see that, in the opinion of the researchers, the model could more accurately find counterfactual dependencies in the target if the dataset were constructed in a different manner. The interests of the researchers do not change if they prescribe data methods that increase accuracy and potentially reduce error. The alternative data classification method is thought to capture the phenomena of racial bias more accurately, where one data method may (incorrectly) uncover racial bias that an alternative method would not. Inductive risk becomes relevant here. First, if the model suggests that police practices are not biased—and the model is wrong—this will prevent necessary public policy interventions. More related to (a), defining the epistemic concepts of “error” and “accuracy” and deciding which data establishes “‘ground truth”—in the police bias case—involves the interplay with social and political values concerning racial identity. The practice of race labeling involves social/political considerations and has a history of marginalization and injustice (Browne Reference Browne2015; Hanna et al. Reference Hanna, Denton, Smart and Smith-Loud2020; James and Burgos Reference James and Burgos2022). As Crawford and Paglen (Reference Crawford and Paglen2019) say, in struggles for justice, people have sought to “define the meaning of their own representations” and that “representations aren’t simply confined to the spheres of language and culture, but have real implications in terms of rights, liberties and forms of self-determination.”

Accepting that a particular dataset provides insight into racial bias requires adopting a particular definition of race that turns an epistemic question concerning model representation into one that is entangled with nonepistemic considerations that bear on how to define accuracy in a dataset and what kinds of error are acceptable. It is not simply a case of value trade-offs or the balancing of false negatives and false positives; it is fundamentally about what data best captures the target. Thus, assessing a model based on (a)—the data is adequate for establishing a true counterfactual dependence—involves inductive risk and weighing nonepistemic values.

3.2 Accepting dependencies as representative of the target phenomena

Even if data classification issues are resolved there is still the question of (b): whether the model, using adequate data, represents or finds counterfactual dependences true of the target. Consider two more ML models: a melanoma detection and sexual-orientation classification model. First, Esteva et al. (Reference Esteva, Brett Kuprel, Novoa, Swetter, Blau and Thrun2017) developed a deep neural network (DNN) that identifies cases of melanoma from healthy moles. The model was trained with semisupervision of human-labeled images of melanoma and healthy moles. When applied to a novel set of images, the model outperforms dermatologists at classification. Second, Wang and Kosinski (Reference Wang and Kosinski2018) developed a facial recognition model that seeks to identify the sexual orientation of individuals using DNNs. Briefly, this model uses roughly the same method as the melanoma model. The input data consisted of human labeled images of purportedly heterosexual men and women along with images of openly self-identifying gay men and lesbians. The model was said to be accurate at identifying sexual orientation when the model had five images of the same person. The researchers sought to add evidential support for the parental hormone theory (PHT), an origin theory for sexual orientation.

The targets in these cases are the visual appearance of a mole versus melanoma and how visual appearance could lend support to PHT theory, respectively. The high-level counterfactuals the models capture are: “If x and y visual patterns had not been present, then mole M would not have been a melanoma” and “if gender atypical facial topologies had been present, then sexual-orientation O would not have been heterosexual.” Are the counterfactual dependencies found by the models true of their targets? Notice that the task of deciding whether a counterfactual dependence should be accepted as truly representative of a target becomes a traditional problem of inductive risk about how much evidence is necessary to accept or reject a hypothesis, and whether nonepistemic considerations impact the amount or kind of evidence required.

The visual appearance of a mole is the leading factor for medical intervention. The general types of counterfactuals the model provides already have significant medical support. There are, of course, risks involved with medical diagnosis that need to be considered when accepting whether the model should be used in practice. Traditional issues regarding the trade-offs between type I and type II errors are relevant here. Moreover, because of bias in the training data, the model is only accurate on light skin and thus has generalization problems and should not be applied globally.

However, using facial appearance as getting at the truth of someone’s sexual identity involves considerably higher demands for model acceptance beyond type I and type II errors. First, the dependencies the model finds between facial topologies and sexual orientation are scientifically controversial. Furthermore, as the proponents of inductive risk argue, social consequences that would result from accepting a hypothesis in error should be weighed. However, it can be difficult to weigh social consequences because the consequences or the severity of them can be unknown. But in the case of phrenology and physiognomy we have considerable historical evidence of the kinds of consequences that have resulted from accepting such hypotheses by looking at its use in justifying racism, sexism, and eugenics. Given this track record, the amount of evidence needed to adopt a physiognomy-based theory is higher compared to accepting a competing theory. When considering Wang and Kosinski’s model in particular, widespread discrimination throughout the world against gay individuals cannot be overlooked. Thus, the presence of such social implications creates a higher demand for evidence that the counterfactuals the model captures is true of the target. We better be reasonably sure that such a hypothesis is true before adopting it.

4.1. Opacity as an external problem

In previous work, I argued that the problem of model opacity is often not resolved by looking inside the model (Sullivan Reference Sullivan2022). Instead, model opacity is an external problem connecting the model to the target; the problem of opacity is a function of how much “link uncertainty” the model has. The less evidence connecting the model to the target phenomena, the less understanding is possible from an opaque ML model. Among other examples, I used the preceding cases of the melanoma and sexual-orientation classification models. The argument rests on the claim that these models seem to provide us with varying degrees of understanding while having similar amounts of model opacity. A better explanation for the varying understanding has to do with the level of independent empirical support connecting the model to the intended phenomena, instead of model opacity qua opacity. I did not consider how nonepistemic values might impact the problem of model opacity. However, if we accept there is inductive risk, in the link uncertainty framework it follows that nonepistemic values are entangled with the (external) problem of opacity.

Resolving the external problem of model opacity requires connecting the model’s data and counterfactual inferences to the target phenomena, thereby reducing link uncertainty and defusing worries of opacity. Judgments about when there is enough evidence connecting a model to its target, such that model’s links are strong enough, is target and domain specific. In cases in which researchers are interested in whether counterfactual dependencies the model relies on are causally representative of phenomena, reducing link uncertainty will involve more traditional empirical research over and above the ML model.Footnote ⁴ In cases in which the target of interest is establishing strong statistical correlations, reducing link uncertainty could alternatively require robustness checks with multiple models.

The greater the inductive risk, the higher demand there is for connecting the model’s data and the model’s inferences to the target, which amounts to a higher burden for reducing link uncertainty. Reducing link uncertainty can again range from improving ground truth methods for data collection to increasing statistical significance before accepting a counterfactual-inference as true of the target (Douglas Reference Douglas2000). For example, the higher threshold of evidence required to connect the sexual-orientation model to its target phenomena because of the risks involved, as discussed in section 3.2, directly entails that a higher threshold of independent evidence is necessary to overcome the model’s opacity problem. If I am right that the extent to which model opacity poses a problem for explanation and understanding depends on the degree to which there is an external connection between the model and target, then the problem of opacity in ML is entangled with nonepistemic values because the process of accepting whether there is sufficient connection between the model and its target is entangled with nonepistemic values. Thus, exploring methods that externally validate models helps with overcoming inductive risk and opacity.

4.2. Opacity as an internal problem

Besides an external problem of opacity, inductive risk also exposes an internal problem for model opacity. The internal problem of model opacity requires verifying that extracted counterfactuals from the model are faithful to how the model works. To accept a model, there must be a set of counterfactual dependencies that we can extract from that model to measure against the target. In more traditional modeling methods—what Knüsel and Baumberger (Reference Knüsel and Baumberger2020) call process-based models—particular counterfactual dependences are deliberately built into the model. However, in data-driven ML models, counterfactual dependencies are not built in, but must be extracted post hoc. This is where ML model opacity strikes us as a problem. If the model is so complex that it is unclear how the model makes its decisions, then how can we be sure that the counterfactual dependencies extracted are faithful to how the model works?

Various ML interpretability methods can provide some insight into the way models make decisions, such as feature importance methods that seek to identify the most salient features that determine a classification. Saliency maps, for example, highlight areas of an image that contribute most to the model’s decision. In the melanoma detection model, a saliency map can highlight areas of the image with various pigmentation differences that the model relies on most for classification. Saliency maps can aid counterfactual reasoning about the target, along the lines of “If x and y visual patterns had not been present, then mole M would not have been a melanoma.” The internal opacity problem of inductive risk requires a higher standard of verification that the counterfactuals extracted from the model are faithful to how the model makes its classifications, and that they are genuine correlations and not mere artifacts. A higher degree of model transparency is necessary.Footnote ⁵ Given the inductive risk present in the sexual-orientation model, more fine-grained counterfactual dependences about how the model made its decisions are required compared to the melanoma model or compared to a very benign model that, say, classifies handwritten numbers. And as it turns out, upon closer inspection of the sexual-orientation model, the researchers found—using saliency maps—the model places a high emphasis on superficial features, like makeup or hairstyle. Thus, the counterfactual inference the model made more often tracked “if x grooming pattern was y, then sexual-orientation O would not have been heterosexual.” This suggests the model does not strongly rely on facial topologies, which was the primary target of interest. Thus, there is not only a higher burden of proof verifying that a dataset and a given counterfactual are representative of a target (the external problem) but also there is a higher burden of proof for extracting the counterfactuals in the first place.

This is not to say that ML models with less inductive risk do not require verification. The claim is that models that have a higher level of inductive risk have a higher bar for verification. Moreover, we need to be aware of the limitations on current interpretability methods. For example, Wachter et al. (Reference Wachter, Mittelstadt and Russell2018) provides an interpretability method that involves counterfactual extraction specifically, providing a counterfactual explanation along the lines of a “what if things had been different?” explanation. The method shows through hypothetical scenarios how small changes to certain feature values would result in changes to the outcome. However, this method has drawbacks, with the possibility to generate contradictory counterfactuals (Molnar Reference Molnar2019). Most, if not all, interpretability methods can be misleading to some degree. Saliency maps can be the same for multiple classes, calling their usefulness into question (Rudin Reference Rudin2019). Such limitations are the reason Rudin (Reference Rudin2019) argues that opaque ML models should not be used in high-stakes cases. Other models that have similar predictive accuracy—that are “in principle” interpretable—should be used instead.Footnote ⁶

To put the point differently, if a highly reliable ML interpretability method would require costly computational power compared to a less reliable method, models that have a high level of inductive risk would require researchers to opt for the costly method over the less costly method. So, while the current limitations on interpretability methods are not a barrier for explaining and understanding certain low stakes phenomena using ML models, the limitations can be a barrier for ML models that involve significant social implications. Thus, the presence of inductive risk not only has an impact on the external problem of ML opacity but it also has an impact on the internal problem of model opacity. Accepting that a ML model is capable of explaining or providing understanding of phenomena requires a greater threshold for model transparency from models that face high stakes or face far reaching social consequences.