‘We simply do not know’

In some contexts, risk-related problems involve hazards of ascertainable probability. It might be possible to say that the risk of death from a certain activity is 1/100,000. Or it might be possible to say that the risk of catastrophic harm from some activity is 2%. Alternatively, we might be dealing with a problem where the probability of harm cannot be specified but has a known range – say, from 1/20,000 to 1/40,000, with an exposed population of 10 million. Or we might be able to say that the risk of catastrophic harm is under 10% but above 1%.

At the same time, it is possible to imagine situations in which analysts, agents or observers cannot assign probabilities to potential outcomes – a topic that has received considerable attention (Knight, Reference Knight1921; Caballero and Krishnamurthy, Reference Caballero and Krishnamurthy2008; Nishimura and Ozaki, Reference Nishimura and Ozaki2017; Kay and King, Reference Kay and King2020). They might think or know that outcomes A, B, C, D and E are possible, but they might not know the probability of each. They might not be able to identify a specific probability or even a known range of probabilities. They might be frequentists who do not know and cannot find relevant frequencies. They might be Bayesians who lack necessary information.

In 1921, Frank Knight wrote:

Knight’s own account, influenced by William James, was quite radical (Rizzo and Dold, Reference Rizzo and Dold2021). Knight believed that uncertainty is real, including in markets, and that the problem could not be overcome or brushed aside by modeling subjective probabilities (id.). Jon Elster offers a vivid example: ‘One could certainly elicit from a political scientist the subjective probability that he attaches to the prediction that Norway in the year 3000 will be a democracy rather than a dictatorship, but would anyone even contemplate acting on the basis of this numerical magnitude?’ (Elster, Reference Elster1983).

Regulators, entrepreneurs (Rizzo and Dold, Reference Rizzo and Dold2021) and ordinary people are sometimes acting in situations of Knightian uncertainty, which I will understand here as those in which outcomes can be identified but no probabilities can be assigned. So understood, Knightian uncertainty is easy to distinguish from risk, where it is possible to identify outcome and to assign probabilities to them (Elster, Reference Elster1983; Bewley, Reference Bewley, Jacobs, Kalai, Kamien and Scwartz1988; Davidson, Reference Davidson1991). We can also imagine cases like those above, involving what I shall call bounded uncertainty, or Knightian uncertainty within some kind of band: we know that the probability of certain outcomes is above (say) 10% but below (say) 20%, but we do not know anything else. If we keep in mind the idea of bounded uncertainty, we need to distinguish it from unbounded uncertainty; let us call the latter ‘pure’ Knightian uncertainty. There is a continuum here: If uncertainty is narrowly bounded (say, the probability is at most 2% and at least 1.99%), we are close to a situation of risk; if uncertainty exists within large bounds (say, the probability is at most 99% and at least 1%), we are close to a situation of pure Knightian uncertainty.

Knight emphasized the difficulty or impossibility of assigning probabilities to outcomes, but he also signaled the problem of ignorance, in which we are unable to specify either the probability of bad outcomes or their nature – where we do not even know the kinds or magnitudes of the harms that we are facing (Arrow, Reference Arrow1984; Smithson, Reference Smithson1989; Harremoes, Reference Harremoes2003; Aven and Steen, Reference Aven and Steen2010; Giang, Reference Giang2015; Kay and King, Reference Kay and King2020; Rizzo and Dold, Reference Rizzo and Dold2021). Though Knightian uncertainty is sometimes understood to include ignorance, so understood, my emphasis here is on cases in which outcomes can be defined but probabilities cannot. Some people appear to think that artificial intelligence creates an uncertain risk of catastrophe, including extinction of the human race (Center for AI Safety, 2023). Other people think the same thing about climate change. If so, what is the right response?

Consider in this regard a passage from John Maynard Keynes, also writing in 1921:

Like Knight, Keynes urged that some of the time, we cannot assign probabilities to imaginable outcomes.Footnote ¹ Keynes immediately added, with some bemusement, that

How on earth, he wondered, do we manage to do that? Keynes listed three techniques (and it is worth considering their role in consideration of various hazards in the century since he wrote):

1. (1) We assume that the present is a much more serviceable guide to the future than a candid examination of past experience would show it to have been hitherto. In other words, we largely ignore the prospect of future changes about the actual character of which we know nothing.

2. (2) We assume that the existing state of opinion as expressed in prices and the character of existing output is based on a correct summing up of future prospects, so that we can accept it as such unless and until something new and relevant comes into the picture.

3. (3) Knowing that our own individual judgment is worthless, we endeavor to fall back on the judgment of the rest of the world which is perhaps better informed. That is, we endeavor to conform with the behavior of the majority or the average. The psychology of a society of individuals each of whom is endeavoring to copy the others leads to what we may strictly term a conventional judgment (Keynes, Reference Keynes1937).

Keynes did not mean to celebrate those techniques. Actually he thought that they were ridiculous. We might know, for example, that technological innovations have not produced horrific harm in the past, and so we might think that artificial intelligence will not produce such harm in the future (strategy (1)). As good Hayekians, we might look at the price signal, such as flood insurance premiums and the value of coastline real estate, to assess the risks associated with climate change (strategy (2)). We might follow the wisdom of crowds to assess the likelihood of a pandemic (strategy (3)). But under circumstances of uncertainty, should we trust any of these? ‘All these pretty, polite techniques, made for a well-panelled Board Room and a nicely regulated market, are liable to collapse’, Keynes urged, because and when ‘we know very little about the future’ (Keynes, Reference Keynes1937). (For a creative institutional response, suggesting the importance of cognitive diversity, see Dold and Rizzo, Reference Dold and Rizzo2021.)

One reason for uncertainty or ignorance might be that we are dealing with a novel, unique or nonrepeatable event. Another reason might be that we are dealing with a problem involving interacting components of a system, and we cannot know how those components will interact with each other, which means that ex ante predictions are highly unreliable (Taleb et al., Reference Taleb, Read, Douady, Norman and Bar-Yam2014).Footnote ² Yet another reason is that the factors that will produce one result, or another result, are so numerous, and so hard to identify in advance, that any assignment of probability has no grounding (Kahneman et al., Reference Kahneman, Sibony and Sunstein2021).

Any port in a storm

Keynes pointed to three strategies that he thought intuitive but ‘liable to collapse’. Are there better ones? It is important to say that over time, some problems that involve ignorance might be transformed into problems of uncertainty, and some problems of uncertainty might turn into problems of risk – a point that may counsel in favor of delay while new information is received. (In important ways, that is the arc of human history.) With respect to regulation, consider OMB Circular A-4, the Economic Constitution of the United States, as it was in effect until 2023: ‘For example, when the uncertainty is due to a lack of data, you might consider deferring the decision, as an explicit regulatory alternative, pending further study to obtain sufficient data’. But as the circular also noted, ‘Delaying a decision will also have costs, as will further efforts at data gathering and analysis’ (OMB, 2003).

Delay of regulation may produce serious harm (including large numbers of deaths; consider the coronavirus pandemic). In principle, agencies would calculate the costs and benefits of delay. But because of the very problem that counsels in favor of delay (lack of information), that calculation is unlikely to be possible.

If uncertainty is genuinely bounded, agencies might use breakeven analysis (Sunstein, Reference Sunstein2014). Suppose, for example, that the costs of regulation (involving, say, cybersecurity) are $100 million, that the benefits range from $150 million to $5 billion, and that technical analysts state that at the present time, they cannot assign probabilities to the lower or upper bound, or to points along the range. Even so, it is clear that the regulation should go forward. Or suppose that the monetized costs of some new technology (say, fracking, or certain uses of artificial intelligence) are $500 million, but that the monetized benefits range from $600 million to $500 billion (and we cannot assign probabilities to the various possible outcomes). A regulatory ban would not be a good idea. We could easily imagine variations on these numbers. Breakeven analysis can enable regulators to identify reasonable paths forward even in the midst of uncertainty, so long as it is bounded.

In some cases, however, it will be inadequate. Suppose that we have pure case of Knightian uncertainty: The benefits of some technology are $900 million, and no probabilities can be assigned to a wide spectrum of exceptionally serious harms. Or suppose that we know that the monetized costs of some new technology range from zero to $900 billion (or much more, e.g., extinction of the human race), and that the monetized benefits range from zero to $900 billion (or much more, e.g., $2 trillion) – and that with respect to both benefits and costs, it is impossible to assign probabilities to the various outcomes. If we are really operating under Knightian uncertainty, breakeven analysis cannot solve our problem.

The Principle of Insufficient Reason says that when people lack information about probabilities, they should act as if each probability is equally likely (Dubs, Reference Dubs1942; Luce and Raiffa, Reference Luce and Raiffa1957; Sinn, Reference Sinn1980; Rawls, Reference Rawls1999). There is some evidence that people follow that principle, at least in surveys (Sunstein, Reference Sunstein2006). But why is it rational to do that? By hypothesis, there is no reason to believe that each probability is equally likely. Making that assumption is no better than making some other, very different assumption. The Principle of Insufficient Reason is essentially arbitrary (Kay and King, Reference Kay and King2020).

Maximin

When strategies of avoidance are unappealing or unsuccessful, regulators might be drawn to the maximin rule: Eliminate the worst-case scenario (Arrow and Hurwicz, Reference Arrow, Hurwicz, Carter and Ford1972; Elster, Reference Elster1983). In the context of regulation of new technologies such as artificial intelligence, for example, strong precautions might be justified by reference to the maximin rule, asking officials to identify the worst case among the various options, and to select that option whose worst-case is least bad. Perhaps the maximin rule would lead to what we might call a Catastrophic Harm Precautionary Principle, by, for example, urging elaborate steps to combat potentially horrific risks. It follows that if aggressive measures are justified to reduce the risks associated with artificial intelligence, one reason is that those risks are potentially catastrophic and existing knowledge does not enable us to assign probabilities to the worst-case scenarios. The same analysis might be applied to many problems, including the risks associated with genetically modified food (Taleb et al., Reference Taleb, Read, Douady, Norman and Bar-Yam2014), nuclear energy (Elster, Reference Elster1983), pandemics and terrorism.

To understand these claims, we need to back up a bit. Maximin has sometimes been recommended under circumstances of uncertainty rather than risk (Elster, Reference Elster1983). In an influential discussion, John Rawls, focusing on justice, offers a justification for a rule that ‘directs our attention to the worst that can happen’ (Rawls, Reference Rawls1999).Footnote ³ As it puts it, ‘this unusual rule’ is plausible in light of ‘three chief features of situations’ (Rawls, Reference Rawls1999). The first is that we cannot assign probabilities to outcomes, or at least we are extremely uncertain of them. The second is that the chooser ‘has a conception of the good such that he cares very little, if anything, for what he might gain above the minimum stipend that he can, in fact, be sure of by following the maximin rule’. For that reason, it ‘is not worthwhile for him to take a chance for the sake of further advantage’. The third is that ‘the rejected alternatives have outcomes that one can hardly accept’. In other words, they involve ‘grave risks’. Under the stated conditions, the gains are quite limited from running a catastrophic risk, which means that choosers do not much value them, and it is worthwhile giving them up to protect against a downside outcome that choosers deplore. This is an effort to produce a framework, pragmatic in character, for handling Knightian uncertainty.

Rawls emphasizes that the three ‘features work most effectively in combination’, which means that the ‘paradigm situation for following the maximin rule is when all three features are realized to the highest degree’ (Rawls, Reference Rawls1999). That means that the rule does not ‘generally apply, nor of course is it self-evident’ (Rawls, Reference Rawls1971).Footnote ⁴ It is ‘a maxim, a rule of thumb, that comes in its own in special circumstances’, and ‘its application depends upon the qualitative structure of the possible gains and losses in its relation to one’s conception of the good, all this against a background in which it is reasonable to discount conjectural estimates of likelihoods’ (Rawls, Reference Rawls1971).

Rawls’ own argument is that for purposes of justice, the original position, as he understands it, is ‘defined so that it is a situation in which the maximin rule applies’ (Rawls, Reference Rawls1971) – which helps to justify his principles of justice. The original position, so understood, is one of Knightian uncertainty. We can think of Rawlsian cases as involving something akin to a ‘negative freeroll’: a situation in which one can incur losses but obtain no (real) gains. Who wants that? In such cases, applying maximin seems quite rational.

Knightian uncertainty and precautions

These points bear on regulatory policy, where Rawls’ defense of maximin has inspired a defense and reconstruction of the Precautionary Principle in an important essay by Stephen Gardiner (Gardiner, Reference Gardiner2006). To make the underlying intuition clear, Gardiner begins with the problem of choosing between two options, A and B:

It should be immediately apparent that if we can assign probabilities to outcomes, A might turn out to be the better choice. Suppose that if you choose A, there is a 99.99999% chance of (A1), and that if you choose B, there is a 99.99999% chance of (B2). If so, A might seem better. But let us stipulate that assignment of probabilities is not possible. In Gardiner’s view, this conclusion helps support what he calls the Rawlsian Core Precautionary Principle in the regulatory setting: When Rawls’ three conditions are met, precautions, understood as efforts to avoid the worst-case scenario, should be adopted. (We could consider efforts to create resilience to be precautions, as in the case of climate change.) As he puts it:

Gardiner adds, importantly, that to justify use the maximin rule, the threat posed by the worst-case scenario must satisfy some minimal threshold of plausibility. In his view, ‘the range of outcomes considered are in some appropriate sense “realistic,” so that, for example, only credible threats are considered’. (Gardiner, Reference Gardiner2006, page 51). If they can be dismissed as unrealistic, then the maximin rule should not be followed.Footnote ⁵ Gardiner believes that the problem of climate change, and also that of genetically modified organisms, can be usefully analyzed in these terms and that they present good cases for application of the maximin rule:

In a similar vein, Jon Elster, speaking of nuclear power, contends that maximin is the appropriate choice when it is possible to identify the worst-case scenario and when the alternatives have the same best consequences (Elster, Reference Elster1983). Elster urges that ‘to the extent that we are in a state of uncertainty, it is rational to act as if the worst that can happen is bound to happen’ (again assuming the same best-consequences) (see also Taleb et al., Reference Taleb, Read, Douady, Norman and Bar-Yam2014). Revealingly, the examples of genetically-modified crops and nuclear power look badly out of date, which can be taken as a cautionary note about applications of such arguments to specific problems. But what matters is the general ideas, not the particular applications.

Objections

I now turn to four objections to the argument on behalf of using the maximin rule under circumstances of Knightian uncertainty, in ascending order of force.

Knightian uncertainty is pretty rare

Notwithstanding what I have said here, it is an understatement to insist that regulatory problems do not typically involve uncertainty, certainly (!) in its pure form, or even within wide bands. Using frequentist strategies, regulators are often able to assign probabilities to outcomes, and Bayesian approaches can also be used (Sunstein, Reference Sunstein2020b; OMB, 2024). When they cannot, perhaps regulators can instead assign probabilities to probabilities (or where this proves impossible, probabilities to probabilities of probabilities). There are many techniques to attempt to do that. In many cases, regulators might be able to specify a range of probabilities – saying, for example, that the probability of catastrophic outcomes from a pandemic or climate change is above 2% but below 30%.Footnote ¹⁰

Whatever we think of any particular example, we might be able to agree that pure Knightian uncertainty is pretty rare, at least over significant time horizons. Perhaps we can agree that at worst, regulatory problems typically involve problems of (manageably) bounded uncertainty, in which we cannot assign probabilities within specified and not-so-wide bands. It might be possible to think, for example, that the risk of a catastrophic outcome is above 1% but below 5%, without being able to assign probabilities within that band. The pervasiveness and nature of Knightian uncertainty depend of course on what is actually known. If pure uncertainty is pretty rare, then Gardiner’s argument, or variations on it, do not apply outside of exotic cases. Fair enough. But even if this is so, exotic cases do arise; in fact we are living with some of them. That is important.

On sleeping well at night

A great deal of work explores the question whether people should follow the maximin rule under circumstances of Knightian uncertainty (Arrow and Hurwicz, Reference Arrow, Hurwicz, Carter and Ford1972). Some of this work draws on people’s practices or intuitions, in a way that illuminates actual beliefs but may tell us little about what rationality requires (Harsanyi, Reference Harsanyi1975). Other work is highly formal, adopting certain axioms and seeing whether the maximin rule violates them (Luce and Raiffa, Reference Luce and Raiffa1957). The results of this work are not conclusive (Luce and Raiffa, Reference Luce and Raiffa1957; Arrow and Hurwicz, Reference Arrow, Hurwicz, Carter and Ford1972). Maximin has not been ruled out as a candidate for rational choice under uncertainty.

In deciding whether to follow the maximin rule under circumstances of Knightian uncertainty, or something close to it (such as bounded uncertainty), a great deal should turn on two questions (Elster, Reference Elster1983): (a) How bad is the worst-case scenario, compared to other bad outcomes? (b) What, exactly, is lost by choosing the maximin rule? Of course, it is possible that choosers, including regulators, will lack the information that would enable them to answer these questions. But (and this is the central point) in the regulatory context, answers to both (a) and (b) may well be possible even if it is not possible to assign probabilities to the various outcomes with any confidence. By emphasizing the relative badness of the worst-case scenario, and the extent of the loss from attending to it, I am attempting to build on the Rawls/Gardiner suggestion that maximin is the preferred decision rule when little is lost from following it.

To see the relevance of the two questions, suppose that you are choosing between two options. The first has a best-case outcome of 10 and a worst-case outcome of −5. The second has a best-case outcome of 15 and a worst-case outcome of −6. It is impossible to assign probabilities to the various outcomes. Maximin would favor the first option, to avoid the worse worst-case (which is −6); but to justify that choice, we have to know something about the meaning of the differences between 10 and 15 on the one hand and −5 and −6 on the other. If 15 is much better than 10, and if the difference between −5 and −6 is a matter of relative indifference, then the choice of the first option is hardly mandated. But if the difference between −5 and −6 greatly matters – if it is a matter of life and death – then the maximin rule is much more attractive.

Consider a regulatory analogue. Suppose that as compared with a ban, allowing some new technology would have a best-case outcome of $2 billion in annual net benefits and a worst-case outcome of −$10 million in net benefits. Suppose that we cannot assign probabilities to the various outcomes. Under the maximin rule, we should ban the new technology. But if the net loss of $10 million is not a big deal, we might reject the maximin rule by reference to something like the Rawls/Gardiner theory. Importantly, we might in cases of this kind favor the maximax rule: If Option A and Option B have roughly equivalent downsides, and if Option B has an immeasurably better upside, we should choose Option B. Of course we could vary the numbers in such a way as to make the maximin rule much more attractive.

These points have the important implication of suggesting the possibility of a (rough) cost–benefit analysis of whether to follow the maximin rule under conditions of both risk and uncertainty. Sometimes the worst-case is the worst by far, and sometimes we lose relatively little by choosing the maximin rule. It is typically thought necessary to assign probabilities in order to engage in cost–benefit balancing; without an understanding of probabilities, such balancing might not seem able to get off the ground. But a crude version of cost–benefit balancing is possible even without reliable information about probabilities. For the balancing exercise to work, of course, it must be possible to produce cardinal rankings among the outcomes – that is, it must be possible to rank them not merely in terms of their badness but also in at least rough terms of how much worse each is than the less-bad others. That approach will not work if cardinal rankings are not feasible – as might be the case if (for example) it is not easy to compare the catastrophic loss from a pandemic with the loss from huge expenditures on efforts to control a pandemic. Much of the time, however, cardinal rankings are possible in the regulatory context.

Here is a simpler way to put the point. It is often assumed that in order to undertake cost–benefit analysis, it is necessary to assign probabilities, with the understanding that point estimates represent the average or most probable case. But in some cases, a sensible rule-of-thumb can be adopted without assigning probabilities. An understanding of the magnitude of the relevant payoffs can help regulators to navigate difficult situations. If one option has a large downside but no substantial upside, it can be rejected in favor of one that lacks that downside but that has a roughly equivalent upside. And recall cases for maximax: where one option has a terrific upside and a bad downside, it should be favored over another option that has a merely decent upside and a bad downside.

To appreciate the need for some kind of analysis of the effects of following the maximin rule, imagine an individual or society lacking the information that would permit the assignment of probabilities to a series of hazards with catastrophic outcomes; suppose that the number of hazards is 10, 20 or 1000. Suppose too that such an individual or society is able to assign probabilities (ranging from 1% to 90%) to an equivalent number of other hazards, with outcomes that range from bad to extremely bad, but never catastrophic. Suppose, finally, that every one of these hazards can be eliminated at a cost – a cost that is high, but that does not, once incurred in individual cases, inflict harms that count as extremely bad or catastrophic. The maximin rule suggests that our individual or society should spend a great deal to eliminate each of the 10, 20 or 100 potentially catastrophic hazards. But once that amount is spent on even one of those hazards, there might be nothing left to combat the extremely bad hazards, even those with a 90% chance of occurring. We could easily imagine that a poorly informed individual or society would be condemned to real poverty and distress, or even worse, merely by virtue of following maximin. In these circumstances, the maximin rule does not have a lot of appeal.

This suggestion derives indirect support from the empirical finding that when asked to decide on the distribution of goods and services, most people reject the two most widely discussed principles in the philosophical literature: average utility, favored by Harsanyi, and Rawls’ difference principle (allowing inequalities only if they work to the advantage to the least well-off) (Frohlich and Oppenheimer, Reference Frohlich and Oppenheimer1992). Instead, people choose average utility with a floor constraint – that is, they favor an approach that maximizes overall well-being, but subject to the constraint that no member of society may fall below a decent minimum (Frohlich and Oppenheimer, Reference Frohlich and Oppenheimer1992). Insisting on an absolute welfare minimum to all, they maximize over that floor. Their aversion to especially bad outcomes leads them to a pragmatic threshold in the form of the floor. So too, very plausibly, in the context of precautions against risks. A sensible individual, or society, would not always choose maximin under circumstances of either risk or uncertainty. A great deal depends on what is lost, and what is gained, by eliminating the worst-case scenario (including by increasing resilience); and much of the time, available information makes it possible to answer those questions at least in general terms (on relevant institutional considerations, relevant to government as well as to entrepreneurship, see Dold and Rizzo, Reference Dold and Rizzo2021).

Nothing here is meant as some kind of proof that maximin is forbidden, or even not required, by rationality (Luce and Raiffa, Reference Luce and Raiffa1957). My claim is instead that for prudent regulators, attempting to proceed in the midst of (pure) Knightian uncertainty, the maximin rule makes most sense when the worst-case scenario, under one course of action, is much worse than the worst-case scenario under the alternative course of action, when there are no huge disparities in gains from either option, and when the choice of maximin does not result in extremely significant losses. Variations on this case will present harder challenges, but in some situations, they too will allow room for the maximin rule. At the same time, it is important for prudent regulators to focus as well on the best-case scenarios, which may promise miracles (Rowell, Reference Rowell2020); that possibility may provide an important cautionary note about efforts to eliminate risks, including those posed by new technologies.

In the hardest and most intriguing cases, it is not possible to defend any simple rule. Some kind of judgment must be made. Nothing in decision theory can specify that judgment. But in the face of unknown probabilities of genuine catastrophe for which ‘wait and learn’ is imprudent, it is reasonable to take strong protective measures, whether the problem involves a pandemic, climate change, artificial intelligence, or the kinds of dangers that each of us faces in ordinary life. Those measures will enable us to sleep better at night. And if we end up taking too many precautions, well, so be it.