2. The logic of stable measurement

Consider a classic tube thermometer where the scale—the lines on the outside of the tube that allow us to convert the observed height of a column of fluid to temperature—was designed using the ideal gas law. We might say that measurements of temperature using this thermometer are theory mediated in the sense that the measurements can be expected to be accurate only if the ideal gas law is as well; if the ideal gas law is sufficiently inaccurate, the thermometer should record the wrong temperature. If we have some way of independently measuring temperature, we can therefore use the behavior of the thermometer to test the accuracy of the theory that it presupposes.

Of course, we are often in situations where we have no way of independently measuring the quantities that interest us and so cannot evaluate the accuracy of our assumptions by simply checking the success of the measurement. But even in these cases, we can often check the stability—sometimes robustness or agreement—of the measurement across various permutations of background conditions and assumptions. We can, for instance, design thermometers using different fluids and compare the results. If the assumptions are accurate and the measurements are successful, they should be stable. If—in addition—an inaccuracy in the assumptions would lead to unstable results, then observing stability is potentially powerful evidence for the accuracy of the assumptions.

This second condition is not guaranteed: Whether a particular variation should be expected to generate instability if the assumptions in question are inaccurate depends on the details. Consider an example from George E. Smith (Reference Smith, Beiner and Schliesser2014)—namely, the measurement of the mass of the sun by way of the observed acceleration of another body.Footnote ² Historically, this measurement relied on various elements of Newtonian theory, such as the law of inertia. If the law of inertia is sufficiently inaccurate, then a body can accelerate without being acted on by a force, and thus the magnitude of acceleration is not a good proxy for either the magnitude of the accelerating force or the magnitude of the gravitational mass generating that force. So, if the law of inertia is inaccurate, then any measurement of the mass of the sun by way of the observed acceleration of another body cannot be expected to be accurate either.

Stability is a different question. If the law of inertia is inaccurate, for example, we might expect that repeated measurements of the mass of the sun by way of the observed acceleration of (say) Mars would be stable. If the elliptical orbits of the planets are just a kind of brute fact, to take one way that the law of inertia could be wrong, we should expect that Mars will remain in the same orbit indefinitely, meaning that we should get the same (wrong) result for the mass of the sun across repeated measurements because the acceleration will be the same. By contrast, measuring the mass of the sun by way of the acceleration it generates in different bodies should yield unstable results when the law of inertia is inaccurate. If the orbits are just brute facts, it would be a remarkable coincidence for the acceleration of each planet to have the same relationship with (the inverse square of) its distance from the sun.

As Smith stresses, the story does not end here because astronomical work on celestial bodies did not end once we had an estimate for the mass of the sun. There were many more masses to be estimated, and the methods adopted in these latter cases depended not just on Newtonian theory but on the earlier results as well: To determine the mass of Jupiter, for example, we look at how much the acceleration of other bodies (e.g., Mars and Saturn) deviates from the new baseline defined by a model that includes only the effects of the sun’s mass. This means that the stability of measurements of Jupiter’s mass provides evidence for the accuracy of the measurement of the sun’s mass, which in turn provides further evidence for the accuracy of assumptions—like the law of inertia—that underwrote the latter. Thus do successive measurements serve to (repeatedly) close “loops” of reasoning that constrain our assumptions more and more tightly.

The key lesson of this section: In evaluating the implications of stable measurement results for the theoretical assumptions that those measurements rest on, the crucial question is whether the failure of those assumptions would lead to instability in the measurements. If so, then the observed stability is good evidence for the accuracy of the assumptions. If not, it isn’t.

3. Stability and internal variability

In this section, I’ll examine the stability of attribution results and discuss the implications for internal variability. In the next, I’ll consider the implications for the trustworthiness of the attribution results themselves.

First, though, how do attribution studies actually work? Speaking broadly, attribution studies are regressions: They take the signals of different possible causes of climate change and assign each of them a weight based on how well the resulting combination fits the observed data. Typically—although see the later discussion—an estimate for internal variability is employed as a filter on the data before the regression step, with the goal of isolating that part of the data that is in fact the result of warming. The results of an attribution study are the weights—that is, the weight assigned to the CO₂ signal is what tells us how much CO₂ has contributed to climate change over the relevant period. (For discussion of the details, see Dethier [Reference Dethier2022a] or Hammerling et al. [Reference Hammerling, Katzfuss, Smith, Gelfand, Montse, Hoeting and Smith2019]).

It is difficult to assess the stability of attribution results. For one thing, there’s no fixed period of evaluation for attribution studies, and there’s no principled way of quantitatively comparing results covering (say) 1951–2010 and 1906–2005. Similarly, some attribution studies decouple the effects of aerosols and greenhouse gases in their analyses, whereas others don’t and combine the two into a single anthropogenic factor. Because the estimates for the contribution of greenhouse gases and aerosols are not independent, we cannot compare the two studies by (say) summing the two factors together.Footnote ³

Of course, climate scientists are sensitive to the potential value of stable results in attribution. Virtually every study on the subject examines the degree of stability across variation in the models employed in regressing climate data. These examinations do not provide the kind of evidence that we’re looking for here. For one thing, attribution is not stable across individual models: Although analyses based on individual models never provide evidence that climate change isn’t attributable to humans, they often fail consistency checks or don’t discriminate between anthropogenic and natural causes. For another, there’s an important sense in which the results generated by individual models are not properly seen as the outcome of a measurement process: Individual models are more analogous to individual data points, potentially indicative but not a sufficient basis for analysis (Dethier Reference Dethier2022b). Examining stability across individual models is thus more like examining the statistical properties of a data set to confirm that the method or experiment is behaving as expected—and indeed, this is how (in)stability across individual models is normally used in the science (see, e.g., Ribes and Terray Reference Ribes and Terray2013).

More promising for present purposes are cross-study comparisons. Unfortunately, these are rare. The only noncursory example I’m aware of is found in the most recent IPCC report, which offers a like-for-like comparison of three cutting-edge studies: Gillett et al. (Reference Gillett, Kirchmeier-Young, Ribes, Shiogama, Hegerl, Knutti, Gastineau, John, Li, Nazarenko, Rosenbloom, Seland, Wu, Yukimoto and Ziehn2021), Haustein et al. (Reference Haustein, Allen, Forster, Otto, Mitchell, Matthews and Frame2017), and Ribes et al. (Reference Ribes, Qasmi and Gillett2021). The results are displayed in table 1 and are stable in at least two important senses. First, they’re stable in the sense given by Smith and Seth (Reference Smith and Seth2020, 138): The relevant error bounds from the different studies consistently overlap (indeed, the best estimates given by each study fall within the error bounds of each of the others). Overlapping error bounds allow each study to be accurate within its stated margin of error; insofar as we have good reason to think that the margins of error are accurate, results that are stable in this respect are much more powerful than results that are not and that thus require us to assume that at least one of the error bounds is inaccurate.

Second, they’re stable in the sense outlined by Dethier (Reference Dethier2021): They deliver what is effectively the same answer to the major theoretical questions in the area. In attribution, the major theoretical question is whether humans are the primary driver of observed climate change. The estimates for observed climate change delivered by the most recent Intergovernmental Panel on Climate Change report are given in the first row of table 1; they are indistinguishable in both estimate and error bars from the estimates for attributable warming delivered by the different studies. Regardless of which study we choose to rely on, humans are responsible for essentially all of the observed warming since the Industrial Revolution.

Although these results are stable, this stability does not provide good evidence in favor of the accuracy of estimates of internal variability, precisely because we would still expect stability were the estimates inaccurate. It’s true that the three studies estimate internal variability in different ways: Haustein et al. (Reference Haustein, Allen, Forster, Otto, Mitchell, Matthews and Frame2017) use CMIP5 models, Gillett et al. (Reference Gillett, Kirchmeier-Young, Ribes, Shiogama, Hegerl, Knutti, Gastineau, John, Li, Nazarenko, Rosenbloom, Seland, Wu, Yukimoto and Ziehn2021) use CMIP6 models, and Ribes et al. (Reference Ribes, Qasmi and Gillett2021) fit a model that mixes processes with “short” and “long” memories to the difference between empirically observed temperatures and the CMIP6-generated trend line. It’s also true that they make use of estimates of internal variability in dramatically different ways. Gillett et al. (Reference Gillett, Kirchmeier-Young, Ribes, Shiogama, Hegerl, Knutti, Gastineau, John, Li, Nazarenko, Rosenbloom, Seland, Wu, Yukimoto and Ziehn2021) employ the traditional approach to attribution developed by Hasselmann (Reference Hasselmann1993) and use internal variability as a filter on the data to isolate the signal. Ribes et al. (Reference Gillett, Kirchmeier-Young, Ribes, Shiogama, Hegerl, Knutti, Gastineau, John, Li, Nazarenko, Rosenbloom, Seland, Wu, Yukimoto and Ziehn2021) adopt a Bayesian methodology in which model simulations are used as priors and internal variability plays a role only in the updating step. And so far as I can tell, Haustein et al. (Reference Haustein, Allen, Forster, Otto, Mitchell, Matthews and Frame2017) employ internal variability only in the context of generating the uncertainty bands around their best estimate.Footnote ⁴ Thus, there’s substantial variation in these three studies with respect to internal variability. Nevertheless, we shouldn’t expect this kind of variation to result in unstable estimates for the human contribution to warming because empirical studies such as Imbers et al. (Reference Imbers, Lopez, Huntingford and Allen2013, Reference Imbers2014) and Sippel et al. (Reference Sippel, Meinshausen, Székely, Fischer, Pendergrass, Lehner and Knutti2021) have shown that attribution results are in fact relatively insensitive to different representations of internal variability.

We’ll focus on Sippel et al. (Reference Sippel, Meinshausen, Székely, Fischer, Pendergrass, Lehner and Knutti2021), who ask the simple question, “What if internal variability were larger than we thought?” They operationalize this question in two main ways: (1) by simply doubling (the primary empirical orthogonal functions of) the internal variability and (2) using estimates for internal variability generated by the highest-variability simulations. They report their results in terms of the lower bound of 1980–2019 warming attributable to humans. Current studies typically put this lower bound somewhere in the neighborhood of 85% of observed warming. The effects of increasing internal variability depend on the technique employed, but in the context of the technique most similar to those employed in the studies cited earlier, neither operationalization has a substantive effect: In both cases, the best estimate for the lower bound is around 85% (see the “ridge” element in Sippel et al. Reference Sippel, Meinshausen, Székely, Fischer, Pendergrass, Lehner and Knutti2021, fig. 6). Combining the two—doubling the internal variability in the highest-variability simulations—has a more substantial effect, but even in this extreme case, the method yields a best estimate for the lower bound of around 70%.

Note that these scenarios are not supposed to be realistic; doubling internal variability is essentially an exercise in stress-testing different attribution methods, not a way of evaluating a scenario that might actually obtain. And for our purposes, the main upshot of these studies is that even if our representation of internal variability is inaccurate, we should expect the attribution results to be relatively stable. After all, these studies indicate that we would have to be very wrong about internal variability before we started to see major errors in the attribution results. It turns out that attribution isn’t that sensitive to the details of the representation of internal variability—at least not when we restrict our attention to realistic or relatively probable errors.

As a consequence, we should expect that attribution results will be (relatively) stable across the kinds of differences examined in this section even if they rest on inaccurate assumptions about the nature of internal variability. Reasoning that parallels that outlined by Smith in the astronomical context is thus untenable here: We cannot reason from the stability of the measurement of the human contribution to climate change to the accuracy of assumptions about internal variability.

4. Stability and attribution

The conclusion at the end of the last section is a negative one, but there are two elements of the analogy to the astronomical case that remain unexamined. First, we have not considered the implications of stable attribution estimates for the measurements themselves: Does stability give us a reason to trust these measurements even if it doesn’t give us a reason to trust the assumptions that they rest on? Second, we have not yet considered the possibility of results that depend on estimates for the human contribution to warming in the same way that measurements of the mass of Jupiter depend on estimates for the mass of the sun. If such measurements exist and are stable or independently confirmable, they may provide evidence for the accuracy of attribution results in the same way that the stability of the measurement of Jupiter’s mass provides evidence for the accuracy of the measurement of the sun’s mass.

Both of these additional elements do in fact provide some evidence in favor of the accuracy of attribution results. Take the stability of attribution results themselves first. The studies discussed in the last section provide us with evidence that even relatively large errors in the representation of internal variability will not lead to substantively different measurements of the human contribution to climate change. The reasoning here is simple. Previously, we were worried that inaccuracies in the estimate of internal variability would lead to inaccuracies in the estimate of the human contribution to warming. The studies discussed earlier survey the realistic ways that our estimates about internal variability could be wrong in a relatively thorough way—if the attribution results were inaccurate because of a misrepresentation of internal variability, we would expect to see more instability in these studies. Stability thus gives us reason to think that the attribution results are in fact accurate.

Now consider results that depend on attribution results. At first pass, there is nothing analogous to the case of the mass of Jupiter in climate change attribution. Attribution studies do not proceed in the piecemeal fashion of Newtonian astronomy; prior attribution results are not directly implicated in later studies.Footnote ⁵ At least so far as I am aware, no attribution study has presupposed a prior estimate of, for example, the anthropogenic contribution to warming in estimating the solar contribution, and we would need stability across varied studies of this sort to motivate an analogy to the astronomical case.

Things are more complicated on closer examination, however. Since the early days of attribution, it has been common for climate scientists to use attribution results to estimate other key climate variables, particularly the transient climate response (TCR), which measures how the climate reacts to increases in CO₂ concentration. These estimates presuppose the accuracy of attribution results; roughly, they estimate TCR by dividing their estimate for the contribution of CO₂ to warming by an estimate for the amount of CO₂ that’s been emitted since the beginning of the industrial period. Crucially, there are other means of estimating TCR; we can compare the attribution-based estimates to these alternative estimates with the hope of finding the kind of stability that would indicate that our estimates are correct.

The Intergovernmental Panel on Climate Change (2021) gives two examples of studies that adopt this method—Schurer et al. (Reference Schurer, Hegerl, Ribes, Polson, Morice and Tett2018) and Ribes et al. (Reference Ribes, Qasmi and Gillett2021)—which report estimates of 1.8 C (5–95% band of 1.2–2.4) and 1.84 C (±0.51) respectively. The Intergovernmental Panel on Climate Change (2021, ch. 7) surveys a variety of other estimates, such as those delivered by theory (2.0 C [1.6–2.7]), instrumental records (1.9 C [1.3–2.7]), and the observed response to volcanic eruptions (1.9 C [1.5–2.3]). Once again, there is a noteworthy level of agreement to be found among these different estimates.

To reiterate the lessons from earlier, insofar as our aim is to use the stability of TCR estimates as evidence for the accuracy of not just the attribution-based estimates of TCR but the attribution results themselves, we need to show that we would expect less stability were attribution results are inaccurate. Schurer et al. (Reference Schurer, Hegerl, Ribes, Polson, Morice and Tett2018, 8658) give us some reason expect less stability: Examining the variability in TCR results when the attribution step relies on a single model rather than the multimodel mean indicates that different values for the human contribution to warming will result in moderately different estimates for TCR. These results are less definitive than we might like, however. They indicate that we should expect somewhat worse agreement between attribution-based estimates for TCR and other lines of evidence if the assumptions of the former were inaccurate, but both the strength of this expectation and how much worse remain up in the air.

Still, when we survey all of the evidence considered in this section, the picture is fairly clear: Errors in the representation of internal variability are unlikely to cause serious errors in the measurement of the human contribution to climate change. There are (of course) other ways that we could be wrong, but our measurements are not so sensitive to the representation of internal variability that it’s reasonable to distrust them on that basis. On the contrary, the evidence available indicates that our representations of internal variability could be quite inaccurate without seriously undermining the measurements.

5. The many faces of robustness

My focus in this article has been stability. I could easily have spoken of “robustness” instead, which is largely used as a cognate term. To parrot I. J. Good (Reference Good1983), however, there are more views about the nature of robustness than there are philosophers who work on the subject. Indeed, many philosophers think that there is more than one kind of robustness—a position most famously argued for by Woodward (Reference Woodward2006).

One of the lessons of our case study in attribution science is that this view, at least so flatly stated, is wrong.Footnote ⁶ There is only one kind of robustness or—better—there is only one logic of stable results. What Woodward and others get right is that the details matter: Not every case is similar, and the mere existence of stability tells us nothing. On the contrary, stable results confirm a hypothesis when (and to the extent that) the hypothesis predicts stability and its negation predicts the opposite.

Consider again the examples of stability that we examined earlier:

• • Measurements of the sun’s mass by way of its effects on other bodies. Newtonian theory predicts stability; were the theory (sufficiently) inaccurate, we’d expect instability. Stability confirms the theory.

• • Measurements of Jupiter’s mass by way of its effects on other bodies. We expect stability if the measurement for the sun’s mass is accurate, instability if it’s not. Stability confirms the assumption.

• • Measurements of the human contribution to warming that use internal variability in different ways. We expect stability regardless of whether estimates of internal variability are accurate. Stability doesn’t confirm.

• • Measurements of the human contribution to warming using different values for internal variability. If the measurements are in error because the estimate for internal variability is inaccurate, we’d expect instability; if the mis-estimation of internal variability is not a problem, we’d expect stability. Stability confirms that the measurements are not inaccurate as a result of internal variability.

• • Measurements of TCR by way of methods that both do and don’t rely on attribution results. If the attribution results are accurate, we’d expect stability; if they’re not, we might see instability. This is inconclusive but somewhat positive.

And three we didn’t explicitly discuss:

• • Measurements of Jupiter’s mass by way of its effects on other bodies. Newtonian theory predicts stability; were the theory (sufficiently) inaccurate, we’d expect instability. Stability confirms the theory.

• • Measurements of TCR by way of methods that both do and don’t rely on attribution results. If the TCR results are accurate, we’d expect stability; if they’re not, ???. This is inconclusive.

• • Measurements of TCR by way of methods that both do and don’t rely on attribution results. If estimates of internal variability are accurate, we’d expect stability; if they’re not, we’d still expect stability. Stability doesn’t confirm.

The reasoning for each of these latter conclusions follows from one of the earlier examples.

This list illustrates some, but certainly not all, of the many faces of robustness. As we can see, stable results may or may not confirm. Indeed, a particular set of stable results may be confirmatory with respect to one question but not confirmatory with respect to another. And while we’ve treated confirmation as a simple binary, the real world is much more complicated: Stability across measurements of Jupiter’s mass may offer powerful evidence in favor of the truth of Newtonian theory but only weak evidence in favor of the accuracy of the estimate of the sun’s mass. Nevertheless, in each of these cases, the logic is exactly the same. The question is always whether (and to what extent) the hypothesis predicts stability and its negation predicts the opposite. The point generalizes beyond these cases as well. Indeed, it’s a simple consequence of basic Bayesian principles identified by Myrvold (Reference Myrvold1996): Where there is stability to be found, what matters confirmation-wise is just what the different hypotheses have to say about it.