1. Two Kinds of Opacity

Deep neural networks (DNNs) are, by now, familiar to most scientists. They are used across all fields of science, for example, to screen for anthrax spores (Jo et al., Reference Jo, Park, Jung, Yoon, Joo, Kim, Kang, Choi, Lee and Park2017), to calculate phenotypic distances from butterfly photographs (Cuthill et al., Reference Cuthill, Guttenberg, Ledger, Crowther and Huertas2019), to improve breast cancer diagnostics (Witowski et al., Reference Witowski, Heacock, Reig, Kang, Lewin, Pysarenko, Patel, Samreen, Rudnicki, Łuczyńska, Popiela, Moy and Geras2022), to monitor urban earthquakes (Yang et al., Reference Yang, Liu, Zhu, Zhao and Beroza2022), in materials science (Zhong et al., Reference Zhong, Gallagher, Liu, Kailkhura, Hiszpanski and Han2022), and so on. DNNs come in many different flavors. Our discussion will not depend on the specifics of the different architectures employed in the wild, but on five common, almost-across-the-board properties of DNNs: DNNs are big (have overwhelmingly many parameters), continuous (propagate continuous values), nonlinear (have nonlinear decision boundaries), instrumental (rely on parameters whose semantics is undefined), and trained in an incremental fashion, that is, on ordered sequences of randomly sampled batches of data. Conventional machine learning models—for example, customer risk estimation models or spam filters—had hundreds or thousands of parameters. Today, neural language models or computer vision models have millions or billions of parameters. Earlier generations of language models were symbolic, with manually defined rules over discrete vocabularies, combining these in mostly linear ways, whereas modern DNNs operate in continuous space and employ nonlinear activation functions. DNNs are instrumental in that they rely on the induction of latent variables that do not track input variables directly, and they are incremental in that parameters are induced by updating the weights for small batches of data at a time, presented in a particular order, and while often doing multiple passes over the data before arriving in a final model state, training is often extremely sensitive to the order in which data happened to be presented.Footnote ¹

DNNs are also often said to be opaque. Footnote ² Opacity of DNNs causes moral problems, depriving users of full agency (Vaassen, Reference Vaassen2022) and obfuscating developer responsibility (Goetze, Reference Goetze2022), but philosophers of science have also discussed the implications of opacity of DNNs. Here, the focus has been whether opacity challenges the application of DNNs to science (Babic et al., Reference Babic, Gerke, Evgeniou and Cohen2021; Gunning et al., Reference Gunning, Stefik, Choi, Miller, Stumpf and Yang2019; Price, Reference Price2018; Rudin, Reference Rudin2019; Sullivan, Reference Sullivan2022b).

Few have considered why DNNs are opaque, or where their opacity stems from. That is:

This is an incredibly important problem, however. If we want to find good trade-offs between transparency and performance (Dziugaite et al., Reference Dziugaite, Ben-David and Roy2020; Johansson et al., Reference Johansson, Sönströd, Norinder and Boström2011), or between transparency and fairness or differential privacy (Rust & Søgaard, Reference Rust and Søgaard2023), we need to know what causes opacity in DNNs. If, for example, continuity is not a source of opacity, binarizing DNNs (Tang et al., Reference Tang, Hua and Wang2017) will not make them transparent.

In the literature, each of the five properties mentioned above has been accused of being the source of opacity in DNNs—size, continuity, nonlinearity, instrumentality, or incrementality—but are they really all sufficient conditions of opacity? This is the question addressed here, shedding new light on the challenges presented by the opacity of DNNs, as well as on what can be done to mitigate the opacity of DNNs.

For this, it is helpful to distinguish between two kinds of opacity:

The two notions of opacity correspond to two ways in which the predictions of DNNs on a data point $ {x}_0 $ can be explained: either in terms of features of the input data $ {x}_0 $ that the DNN is making inferences about—or in terms of past training data $ {x}_1 $ , $ {x}_2 $ , and $ {x}_3 $ that influenced predictions on $ {x}_0 $ . This corresponds to how a doctor can provide two types of rationales for a cancer diagnosis: The first is of the form: “Look! The patch in the upper corner of this x-ray scan looks like a tumor, doesn’t it?” The second is of the form: “Your symptoms are identical to past patients who had cancer.” In the DNN literature, explanations of the second form are often referred to as training data influence attributions.

Training data influence attributions seem to intuitively answer important questions about why DNNs predict what they do, but they are generally unreliable and hard to evaluate (Karthikeyan & Søgaard, Reference Karthikeyan and Søgaard2021): If I train two randomly initialized networks on the same data (in the same order), they will be influenced by different data points. Why? The answer is simple: What data points incur loss, will depend on the network’s current state. Similarly, two exact copies of an initialized network trained on the same data, but in different order, will also be influenced by different data points.

We will, in other words, be interested in the following question:

2. Size

One possible source of both inference-opacity and training-opacity is model size, that is, the number of model parameters. DNNs have overwhelmingly many parameters, for example, modern large language models have tens or hundreds of billions of parameters. Many researchers have highlighted this source of opacity (e.g., Shrestha et al., Reference Shrestha, Krishna and von Krogh2021, §5.2), who point out how DNNs use “millions of neurons, which makes them opaque.” The intuition is this: Humans cannot be expected to familiarize themselves with $ n $ parameters for sufficiently high $ n $ , and they obviously cannot hold this many parameters in short-term memory in an attempt to reconstruct inference dynamics. Even deciding whether two DNNs have induced the same function can be hard. For a single layer with $ n $ parameters, there is $ n! $ equivalent ways of permuting its weights—a number higher than 3.5 million for $ n=10 $ . Humans cannot, upon inspection, say why models with overwhelmingly many parameters predict $ y $ as a result of an input $ x $ . Large DNNs are therefore inference-opaque by Definition 1.1.

To see whether the number of parameters qualify as a source of inference-opacity, let us apply our test from above. Is a decision tree or a Naive Bayes model with a million or a billion parameters inference-opaque? The knee jerk response of many machine learning researchers will be to say “no.” In Naive Bayes, for example, all conditional probabilities are statistically independent, so unlike in DNNs, we can consider each input dimension in isolation. So, the argument would run, can we not just iterate through the parameters, one by one, and identify the predictive variables this way? Clearly, the interdependencies in dense DNNs add to the complexity of these models, but Naive Bayes models with millions or billions of parameters are also opaque. To see this, we need to remember how not all problems have sparse solutions. If sparse solutions exist, they can be computed in polynomial time (Marques-Silva et al., Reference Marques-Silva, Gerspacher, Cooper, Ignatiev, Narodytska, Larochelle, Ranzato, Hadsell, Balcan and Lin2020), but often they simply do not exist. Consider, for example, a classifier used to distinguish positive from negative movie reviews and let the input movie reviews be of increasing length. A 50-word positive review may, for example, include 4 positive words and 2 negative ones. A 100-word positive review may include 8 positive words and 4 negative ones. And so on. While we can (very crudely) explain such decisions by saying “these eight words outweigh these four words,” this becomes impossible in the limit (as the length grows to infinity), since the number of predictive variables will quickly surpass the number of predictive variables that our short-term memory capacity admits. Imagine a decision tree with thousands, or even millions, of nested if-else statements. Such nested statements would be lost on most of us. This is in line with Humphreys, for whom “a process is said to be epistemically opaque when it is impossible for a scientist to know all of the factors that are epistemically relevant to licensing claims on the basis of that process” (Humphreys, Reference Humphreys2009).

Size is, in other words, sufficient for inference-opacity, but how about training-opacity? Is a decision tree or a Naive Bayes model with a million or a billion parameters also training-opaque?

This does not necessarily follow from its inference-opacity. By Definition 1.2, a model is opaque if it is generally not feasible to explain the individual parameters as a result of the training data. Even for large decision trees and Naive Bayes, this is fairly easy, since individual parameters reflect global statistics that are easy to compute. It may not be easy to predict the entire model if sufficiently big, or its posterior distribution, but the value of a single parameter, for example, a likelihood $ P\left({x}_i|y\right) $ , is easy to compute.

3. Continuity

Humans are arguably better at reasoning with discrete variables than with continuous ones. If this is true, symbolic representations and logical reasoning are more intuitive to us than vector representations and probabilities. Many philosophers and computer scientists have pointed to the continuous nature of DNNs as a source of both inference-opacity and training-opacity, for example, claiming how “discrete representations have the advantage of being readily interpretable” (Cartuyvels et al., Reference Cartuyvels, Spinks and Moens2021, §2.1). Many approaches to “explaining” DNNs have discretized their inference dynamics—by uptraining rule-based models to simulate DNNs or by jointly providing discrete explanations, for example, by text generation or concept activation. Other approaches, however, do not, including uptraining of linear models and saliency maps, suggesting that continuity itself is not a source of inference-opacity. Note also how many scientific explanations in the natural sciences are continuous.

I will apply our opacity test, but obviously, we cannot add continuity to Naive Bayes, since the inference dynamics of Naive Bayes classifiers are inherently continuous. Naive Bayes classifiers are, in their own right, an argument for why continuity is not sufficient for inference-opacity or training-opacity. That we consider small Naive Bayes models inference-transparent and training-transparent, is proof that continuity is insufficient for both types of opacity.

That said, maybe probabilities are an exception to this rule? If the weights did not have a probabilistic interpretation, would Naive Bayes still be transparent (on a small scale)? Our reply is as follows: Our interpretation of Naive Bayes models may built on our every-day intuitions about probabilities, but so may our interpretation of other discriminative, linear models. Perceptrons are single-layered neural networks and discriminative Naive Bayes models, for example, but if we add sigmoids, we obtain logistic regression models, which admit for probabilistic interpretations Rosenblatt (Reference Rosenblatt1958). So even if $ \phi $ is said to mean “being continuous and non-probabilistic,” continuity still seems insufficient to generate opacity. Our final example would be a simple decision tree over continuous input features, for example, “if you are taller than 6 five feet and older than 15 years, then $ {y}_1 $ , otherwise $ {y}_2 $ .” Such a tree seems very easy to understand, and if so, this is further evidence that continuity is insufficient for inference-opacity and training-opacity. One way to add continuity to decision trees will give you decision tree regressors. Such regression models are also widely considered inference-transparent and training-transparent.

4. Nonlinearity

Many authors have seen the reliance on nonlinear activation functions as a source of both of inference-opacity and training-opacity, referencing DNN’s “nonlinear structure which makes them opaque” (Joneidy & Ayadurai, Reference Joneidy, Ayadurai, Petrillo, Felice, Lambert-Torres and Bonaldi2021; Sachan et al., Reference Sachan, Yang, Xu, Benavides and Li2020). Why are nonlinearities supposedly hard to grasp? Clearly, nonlinear functions are generally harder to grasp than linear functions, as evidenced by mathematics textbooks and online lectures saying things such as “in order to understand what a nonlinear function is, it is essential to understand what a linear function is.”Footnote ⁷ But are nonlinearities sufficient for opacity? I do not think so. The most common nonlinear functions in DNNs seem easy to understand: ReLU functions, for example, are thresholded, linear activation functions, for example, “ $ f(x)= ax+b $ unless $ x<c $ .” We are familiar with such functions in our daily lives. This is how simple taxation systems work, for example. Sigmoid functions are arguably somewhat harder to compute with by hand, but they express natural relationships, for example, that between crop yield and soil salinity, or those between titrant volume and pH. A second reason to question that nonlinearity is the source of opacity in DNNs, is that DNNs often induce near-linear functions in practice and can be approximated very closely by shallow, linear networks (Ba & Caruana, Reference Ba, Caruana, Ghahramani, Welling, Cortes, Lawrence and Weinberger2014). They nevertheless come across as both inference-opaque and training-opaque.

Let us apply our test from Section 1. We cannot add nonlinearities to decision trees, for decision trees are already nonlinear. They are as such an argument for why nonlinearity is insufficient for (both types of) opacity. Small nonlinear small decision trees are considered transparent; see Section 1 for details. But they cannot be nonlinear and transparent, if nonlinearity is a source of (both types of) opacity.

If we instead add $ \phi $ (nonlinearity) to a Naive Bayes classifier, we may, for example, get classifiers that associate low- and high-probability instances to the same class. Consider a model trained to predict positive product reviews, but used to detect opinionated reviews. Opinionated reviews will tend to be the ones for which P(positive $ \mid \mathbf{x} $ ) is either very high or very low. Such a compound model, albeit nonlinear, seems perfectly intelligible (fathomable) to humans, even if the nonlinearity adds a bit an extra step to the inference procedure. It is likely that the cognitive load of nonlinear activations is higher than for linear activations, but what our test shows is that nonlinearity alone is insufficient for inference-opacity or training-opacity.

One may of course reply that just because I can understand a single ReLU activation does not mean I can understand a sequence of them, especially with high-dimensional inputs. I would reply that high dimensionality is a reference to size and itself a source of opacity. If a sequence of ReLUs with low-dimensional input would lead to inference-opacity, this would show that nonlinearity is a sufficient source of inference-opacity, but I would argue that this is only the case for sufficiently long sequences of ReLUs—a recourse, again, to model size. To see this, consider the following sequence of ReLUs:

It is fairly easy to see that this function for small enough values of $ x $ ( $ x\le 15 $ ) will return $ x $ , and that for large enough values of $ x $ will return $ 12x+19 $ . For mid-range values of $ x $ , it will return $ 4x+3 $ . A sequence of three or more ReLUs will of course be more complex to trace by hand, but note how the complexity is comparable to that seen in decision trees, typically considered transparent machine learning models.

5. Instrumentality

The fourth contender for a sufficient condition for inference-opacity or training-opacity is instrumentality: DNNs are opaque because they are instrumental and their elements are ungrounded. This implication has been floated by several philosophers, complaining, for example, that DNNs “are opaque because there is no inherent meaning to the hidden nodes,” or that they do not “work in accordance with the ways humans themselves assign meaning to the reality that surrounds them” (Landgrebe & Smith, Reference Landgrebe and Smith2021, p. 2070). This is also the intuition behind Boge’s what-opacity, it seems.

The argument here is not just that DNNs have derived intentionality in Searle’s sense (Searle, Reference Searle1983). Decision trees, Naive Bayes models, and ball frames all have derived intentionality. The argument is that the hidden layers of DNNs have less than that. In a decision tree, a node is said to correspond to an input feature, which in turn corresponds to a property of an object or an event that the input example represents. In Naive Bayes, the conditional probabilities represent relations between such properties. The balls on a ball frame correspond to numbers, which may in turn represent objects or events. What about the parameters in the hidden layers of DNNs?

One way to think about hidden nodes is as if they were latent variables or weighted mixtures of other input variables.Footnote ⁸ Latent variables are common in science. Scientists use latent variables to prove the existence of—or determine the causal effects of—everything from black holes in the universe to bias in clinical diagnosis. And lumping several variables together as one is something we do all the time. When computing momentum, for example, we multiply mass and velocity. Mass, however, is itself the product of two input variables, namely volume and density. In a DNN with a hidden layer and two input variables, the hidden nodes are no more complex than using mass as a shorthand for the product of volume and density.

Let us apply our test. Is a small, binary decision tree still transparent if one or more of its nodes correspond to a compound input variable? Does a transparent Naive Bayes model remain transparent after adding a small set of latent variables? I think the answer is “yes” in both cases. A decision tree, in which a node corresponds to the compound input variable “mass,” does not in itself seem opaque. Sufficiently small Bayesian models with latent variables, for example, Gaussian mixtures or topic models, also seem transparent. Note also that clustering methods, such as $ k $ -means or agglomerative clustering, induce categories that do not have direct real-world correspondences. These are also traditionally classified as transparent machine learning methods.

A possible reply, again, is that just because we can describe some compound variables as the product of two simple ones, does not mean I can understand more complex combinations of more input variables. However, this is a reference to model size, not to instrumentality itself. Instrumentality in itself is not a problem, but if a variable $ x $ is a (sufficiently complex) function of a sufficiently large number of input variables, the meaning of $ x $ can be hard to understand. True, but this is exactly the argument made here: We can only wrap our heads around sufficiently small variable combinations.Footnote ⁹

Some methods for inducing the values of latent variables, for example, Monte Carlo or Gibbs procedures,Footnote ¹⁰ may not be training-opaque, but instrumentality is not on its own a sufficient source of training-opacity. Clustering methods that are both instrumental and incremental, for example, incremental $ k $ -means (Halkidi et al., Reference Halkidi, Spiliopoulou and Pavlou2012), are also training-opaque.Footnote ¹¹ See the next section for a discussion of incrementality.

6. Incrementality

Seeing incrementality as a source of opacity generally seems to have gone somewhat under the radar. Some authors have alluded to training history as a general source of opacity, for example, “learning algorithms are even more opaque because they do not rely on pre-specified instructions, but on evolving weights and networks of connections that get refined with each additional data point” (Faraj et al., Reference Faraj, Pachidi and Sayegh2018, §2.1), but such examples are rare.

Why is training history important? Clearly, how a model ended up in its final state is orthogonal to its inference-opacity in that state. Some neural network learning algorithms only update parameters when networks make incorrect classifications, but are randomly initialized. If a network ends up in state A after seeing $ N $ training data points, it is possible to arrive at an identical network B without seeing any data at all, simply by drawing B from the process of random initialization. The probability of this happening is low, but nonzero. Since A and B must be equally inference-opaque, inference-opacity is independent of the training history of A and B. However, how a model ended up in its final state is crucial to training-opacity. Since DNNs converge to local (not global) optima during training, training histories are important for understanding how the DNN came about. This includes what training data were influential in this process. Two identically initialized DNNs trained on different sequences of training data will have very different (and often uncorrelated) influence functions.

The non-incrementality of (standard) decision trees and Naive Bayes models means that they are training-transparent, that is, not training-opaque. Conversely, consider negligible DNNs with two parameters. Such models remain training-opaque because they are trained on random sequences of training data, the order of which is not stored—or deleted once training is complete. Even if we only have two parameters, say, to keep track of, each parameter is affected by hundreds, thousands, or millions of training instances, some of which may be seen multiple times during training, and the final value of each parameter is potentially sensitive to the order in which training data are seen.

Consider what Naive Bayes would look like if it were sensitive to its training history. Or simply think of Perceptrons, a linear model that is sensitive to the ordering of training data. On non-separable problems,Footnote ¹² the final model can be very sensitive to training history, but even on separable problems, margins can be quite different across models trained on different sequences of randomly sampled batches of data. Perceptrons (and by extension, DNNs) are therefore training-opaque; Naive Bayes models are not. Decision tree algorithms, for example, C4.5, are mostly non-incremental. Some incremental algorithms do exist, however, and some of these, such as ID4 and ID5, lead to training-opacity. However, the family of decision tree induction algorithms also reminds us of the subtle difference between incrementality and training-opacity: Some algorithms are incremental, but guaranteed to produce the same decision tree regardless of how the training instances are ordered. ID5R is an example of such an algorithm (Utgoff, Reference Utgoff1989). Here, incrementality does not lead to relevant epistemic opacity with respect to how the model was induced as a result of the training data.

7. Interactions

A general objection to the discussion above would be that I have treated each potential opacity inducer in isolation. This is arguably far too generous. The problem with DNNs, some might say, is not that they are large, continuous, nonlinear, and so on, but that they are all of these things at once. My reply to this objection is twofold:

First of all, it is simply not true that all DNNs satisfy the conjunction of the five properties above. Many DNNs omit nonlinear activation functions; others rely on binary-valued parameters (Hubara et al., Reference Hubara, Courbariaux, Soudry, El-Yaniv, Bengio, Lee, Sugiyama, Luxburg, Guyon and Garnett2016). It is true that all (relevant) DNNs are large, instrumental, and incremental, but the same holds true for many other learning algorithms, some of which are usually considered inference-transparent, for example, online random forests (Lakshminarayanan et al., Reference Lakshminarayanan, Roy, Teh, Ghahramani, Welling, Cortes, Lawrence and Weinberger2014). Here is a sample list of learning algorithms satisfying all three of the above properties, size, instrumentality (Inst), and incrementality (Incr):

These algorithms not only present a challenge to philosophers who claim that DNNs present us with unique kinds of opacity, for example, Boge (Reference Boge2022), but they also show that the conjunction of instrumentality and incrementality does not cause inference-opacity. Small online random forests are clearly inference-transparent, and random forests are often said to be inherently inference-transparent.

Generally—and this is my second reply to the objection above—I have not found any examples in the literature of small and inference-opaque models, regardless of how instrumental they were, or how incrementally they were trained. A bold empirical hypothesis seems to emerge: No small models are inference-opaque. The discussion above, in my view, presents strong evidence that model size, in general, is sufficient for opacity. I have also presented arguments against seeing continuity, nonlinearity, instrumentality, or incrementality as sufficient for opacity. But I have no proof for size being the only sufficient source of opacity, of course. Such a proof would rely on an iteration over all possible sources of opacity, showing that none of them, except for size, are sufficient.

8. Mitigation

My main observation is that the number of parameters in a model is a sufficient source of its inference-opacity, whereas incrementality—training on small batches in ways that are sensitive to the order in which the batches are presented—is a sufficient source of training-opacity (in the absence of convergence guarantees). Other candidates, such as continuity, nonlinearity, and instrumentality, are insufficient to establish opacity.

This observation comes with the promise of making research on DNN transparency more focused, highlighting the need for knowledge distillation or summarization in terms of models with fewer parameters. It also indicates that the opacity of DNNs is related to the opacity of many other learning algorithms. Nothing about the DNN model opacity is unique to DNNs. I will use this section to show that it also follows from this observation that inference-opacity mitigation amounts to computing something akin to an abstractiveness–faithfulness trade-off, known from traditional summarization problems, and that training-opacity amounts to finding a similar trade-off—but on a different scale.

9. Human Baselines

DNNs are said to be opaque machines, and I have discussed the sources of their alleged opacity, and how to mitigate it. Here, in the final section, I ask: Opaque compared to what? Opacity is only of practical importance, if there is a transparent alternative. Where DNNs replace manual processes, the alternative is often a human being. Clearly, humans are not exactly see-through.

10. Concluding Remarks

I have presented five common properties of DNNs and two different kinds of (internal, algorithmic) opacity. I have explored how these five properties and two kinds of opacity interact, to decide which of these properties are sufficient for what type of opacity. Both kinds of opacity stem from the individual properties of learning algorithms: Inference-opacity stems from size; training-opacity stems from incrementality. This has serious consequences for transparency techniques, such as discretization and linearization, which will be insufficient to obtain transparency. I discussed how the two kinds of opacity can be mitigated, and showed how optimal transparency is a trade-off between abstractiveness and faithfulness. Identifying opacity sources and presenting a framework for formalizing transparency, push the horizon for the research program of explainable artificial intelligence.