1. Introduction
In the United States, cost–benefit analyses (CBAs) or cost-effectiveness analyses (CEAs) for policy-making are mandated in certain contexts by a series of regulatory actions (Carey, Reference Carey2014, Reference Carey2022). In the United Kingdom, healthcare technologies are approved largely based on recommendations from the National Institute for Health and Care Excellence (NICE) derived from CEAs and supporting information (NICE, 2013). Institutions such as NICE make decisions by committee, and which factors affect recommendations is an ongoing area of study (Devlin & Parkin, Reference Devlin and Parkin2004; Tappenden, Brazier, Ratcliffe & Chilcott, Reference Tappenden, Brazier, Ratcliffe and Chilcott2007; Dakin et al., Reference Dakin, Devlin, Feng, Rice, O’Neill and Parkin2015).Footnote 1 While the impact of uncertainty in incremental cost-effectiveness ratios (ICERs) has been studied, how “uncertainty information is presented to NICE committees might also affect decisions” and conceptual work is required (Dakin et al., Reference Dakin, Devlin, Feng, Rice, O’Neill and Parkin2015, p. 1269). This paper provides such a conceptual foundation for the decision-making of policy-makers under ambiguity and structural uncertainties. It then studies the implications of presenting ambiguous information to decision-makers (DMs) in ICER or CEA format versus net benefit (NB) or CBA format.
There is a large literature on the economic and welfare foundations of CEA relative to CBA (Weinstein & Zeckhauser, Reference Weinstein and Zeckhauser1973; Birch & Gafni, Reference Birch and Gafni1992; Johannesson & Weinstein, Reference Johannesson and Weinstein1993; Garber & Phelps, Reference Garber and Phelps1997) and the implications of uncertainty for CEA (Stinnett & Paltiel, Reference Stinnett and Paltiel1997; Drummond, Sculpher, Claxton, Stoddart & Torrance, Reference Drummond, Sculpher, Claxton, Stoddart and Torrance2015; Sculpher, Basu, Kuntz & Meltzer, Reference Sculpher, Basu, Kuntz, Meltzer, Neumann, Sanders, Russell, Siegel and Ganiats2017).Footnote 2 CBA and CEA approaches can sometimes be equivalent (Phelps & Mushlin, Reference Phelps and Mushlin1991; Garber & Phelps, Reference Garber and Phelps1997; Garber, Reference Garber, Culyer and Newhouse2000)Footnote 3 but there are clear advantages to a CBA-type approach calculating NB relative to a CEA-type approach calculating ICERs for economic evaluation (Paulden, Reference Paulden2020). O’Mahony (Reference O’Mahony2020) states that using ICERs along with NB is likely not detrimental. However, such assertions may be misplaced if the presentation of information to DMs impacts their recommendations.
Expert DMs are immune neither to bias nor error. Cognitive biases affect professionals ranging from physicians (Redelmeier & Shafir, Reference Redelmeier and Shafir1995; Perneger & Agoritsas, Reference Perneger and Agoritsas2011; Saposnik, Redelmeier, Ruff & Tobler, Reference Saposnik, Redelmeier, Ruff and Tobler2016) to public workers and representatives (Redelmeier & Shafir, Reference Redelmeier and Shafir1995; Bellé, Belardinelli & Cantarelli, Reference Bellé, Belardinelli and Cantarelli2018; Sheffer, Loewen, Soroka, Walgrave & Sheafer, Reference Sheffer, Loewen, Soroka, Walgrave and Sheafer2018). Also, members of the International Society on Priorities in Health are influenced by reminders of opportunity cost in survey experiments of health-related public policy choice (Persson & Tinghög, Reference Persson and Tinghög2020). As the framing of choice problems is known to affect decision-making (Tversky & Kahneman, Reference Tversky and Kahneman1981; Perneger & Agoritsas, Reference Perneger and Agoritsas2011; Bellé et al., Reference Bellé, Belardinelli and Cantarelli2018; Sheffer et al., Reference Sheffer, Loewen, Soroka, Walgrave and Sheafer2018), these results motivate my interpretation of CBA and CEA as potential frames for economic evaluation. The notion that framing economic evaluations can affect decision-making has been advanced by Siverskog and Henriksson (Reference Siverskog and Henriksson2021), who propose a measure of healthcare benefits forgone per those gained in healthcare technology evaluation. This focuses attention on health-valued opportunity costs, framing decision-making around the relative equity weights assigned to those treated. I also suggest ratios for economic evaluation, but I abstract from equity, instead designing these ratios as alternative methods that induce the NB decision.
Moreover, my ratios are derived from the main thrust and advancement of this paper: I do not assume that frames have an impact, but rather provide a theoretical analysis of a representative DM’s behavior under alternative decision rules used to overcome ambiguity. Decision rules are heuristics – plausibly suboptimal processes used to simplify complicated analysis (Tversky & Kahneman, Reference Tversky and Kahneman1974; Hjeij & Vilks, Reference Hjeij and Vilks2023). Conditional on the data presented to DMs, I limit attention to rules denoted by Simon (Reference Simon1955) as classically rational, but more humanistic rules requiring less computational capacity could be studied in principle. The motivating fact is that the implications of CBA versus CEA framing combined with some decision-making heuristic can be studied analytically to obtain specific predictions for a representative DM’s policy recommendations. I now briefly describe this analysis.
In CBA, a policy should be adopted if it yields the greatest net monetary incremental benefits (NMB) across alternatives. In CEA, a policy should be implemented if the ratio of incremental costs (ICs) to incremental benefits (IBs), known as an ICER, is below some cost-effectiveness (CE) threshold
$ g $
. DMs reviewing economic evaluations can be characterized as attempting to maximize these social objective functions for CBA and CEA: (a) NMB; or (b)
$ g $
minus the estimated ICER for the alternative being considered, respectively. I specify a highly stylized setting that highlights the contribution of ambiguity to differences in decision-making under the two frameworks. I restrict attention to (i) a single alternative compared against a status quo policy, (ii) benefits expressed by a single variable, (iii) where the final estimates of IBs and ICs are strictly positive, (iv) where the monetary valuation of benefits is equal to
$ g $
, and (v) a representative DM. Section 2.1 shows that maximizing (a) and (b) yield equivalent policy choices when there is no ambiguity over IBs, ICs, or
$ g $
.
The DM does not ordinarily know these values but instead reviews data from several models and scenario analyses. Risk arises when there are known probabilities by which such unknowns take specific values, while ambiguity arises when no intuitive priors are available. Given structural uncertainties when developing a model for economic evaluation, the latter may routinely arise.Footnote 4 My formalization and differentiation of the decision problems faced by a representative DM under CBA versus CEA extends to the setting with ambiguity over these unknowns, and my main results follow from analysis of the representative DM’s recommendations under well-known decision rules used to overcome ambiguities.
I assume that a simple CBA with aggregated costs and a single category of benefits perfectly captures social welfare. This is a strong restriction, but it is not an unreasonable choice for the analytic purposes of my study.Footnote 5 The assumption permits the interpretation of disagreement with CBA recommendations as errors in judgment when the DM uses CEA under ambiguity. For example, I show that presenting ICERs instead of NMB can induce additional decision errors when there is ambiguity in IBs – even when the approaches are equivalent in unambiguous settings – due to the objective functions implied by each framework combined with the necessary use of simplifying decision rules to overcome ambiguity. If no assumption on the true social welfare function is maintained, normative statements concerning “decision errors” or social optimality are no longer valid but the main results concerning relative comparisons between the two approaches remain sound.
My contributions can be grouped into four categories. First, I formalize and differentiate the decision problems and implied objective functions of a representative DM reviewing the results from an economic evaluation framed under CBA versus CEA. This answers Dakin et al. (Reference Dakin, Devlin, Feng, Rice, O’Neill and Parkin2015)’s call for a conceptual understanding of how the presentation of “uncertainty information” may affect decision-making. My model conceptualizes a specific type of framing bias (Tversky & Kahneman, Reference Tversky and Kahneman1981; Perneger & Agoritsas, Reference Perneger and Agoritsas2011) for economic evaluation, with the presentation of NMB versus ICERs shifting the DM’s social objective function.
Next, I add to the literature evaluating three decision rules in ambiguous settings: the Bayesian, Maximin (MM), and Minimax Regret (MMR) rules (Manski, Reference Manski2009, Reference Manski2010, Reference Manski2013, Reference Manski2017, Reference Manski2018, Reference Manski2019). Textbook treatments of these rules applied to a simple CBA context are available – such as Mishan and Quah (Reference Mishan and Quah2021) – but Section 2.2 discusses these rules for both CBA and CEA objective functions with ambiguity over any of
$ g $
, IBs, and ICs. Section 2.3 then advances a novel visualization of them for both CBA and CEA on the northeastern quadrant of the CE plane, and Section 3 solves for the DM’s decision using each rule under both framings, given the
$ g $
, ICs, and IBs considered possible.
Third, my results generate specific implications of these frames on a representative DM’s behavior under ambiguity: predictions that may be examined empirically through adapted stated and revealed preference studies of committee decision-making. For example, Theorem 7 demonstrates that ambiguity in IBs, but not in ICs or
$ g $
, causes divergence between CBA and CEA solutions. This suggests that appropriately accounting for uncertainty in IBs versus uncertainty in ICs in such studies would be a fruitful avenue for future research.
The relevant intuition is that ratios introduce a nonlinear distortion in the value of benefits under CEA. This qualitative result is related to Stinnett and Paltiel (Reference Stinnett and Paltiel1997)’s demonstration that ICERs based on the ratio of means (ROM) are more aligned with constrained optimization and utility maximization than those based on the mean of ratios (MOR) under second-order uncertainty. My analysis of a DM’s behavior under CBA versus CEA framings when employing a Bayesian rule is directly comparable: the DM employs ROM under CBA but MOR under CEA.Footnote
6 While Stinnett and Paltiel (Reference Stinnett and Paltiel1997) focus on the “correct” method to combine different estimates of ICs and IBs into a single ICER, I accept that many estimates – not one “correct” ratio – will be presented to DMs. My results show that using an objective function implied by CEA rather than CBA may inadvertently induce decision-making based on MOR rather than ROM. My visual representation for aggregating estimates of
$ g $
, IBs, and ICs is also related to but differs somewhat from the vector algebra interpretations for their MOR and ROM methods.Footnote
7 By focusing on the decision rule, I add new intuition and simplify the analysis for a Bayesian DM under CBA (or ROM) by not requiring calculations with respect to the IB-axis. As well, this paper is the first to provide corresponding visualizations for MMR (and for my theorems) on the CE-plane to the best of my knowledge.
Fourth, contrary to O’Mahony (Reference O’Mahony2020), I find that reporting ICERs should matter. Providing only IBs and ICs separately would allow DMs to combine data in a variety of ways, instead of framing the problem away from CBA objectives. However, I prove in Section 3 that the DM approves fewer technologies under CEA relative to CBA in certain reasonable contexts. As economic evaluations are subject to potential bias (Kassirer & Angell, Reference Kassirer and Angell1994) and there is already evidence of excess approval by NICE (Claxton, Sculpher, Palmer & Culyer, Reference Claxton, Sculpher, Palmer and Culyer2015), this may be an underappreciated property of ICERs. I discuss this issue further in Section 4.2. Finally, given preferences to avoid explicit choice of
$ g $
(Phelps & Mushlin, Reference Phelps and Mushlin1991; Garber, Reference Garber, Culyer and Newhouse2000; Paulden, Reference Paulden2020), I provide adjusted ICER analogs that align with CBA decision-making.Footnote
8
This paper is laid out as follows. Section 2 formalizes the setting and defines the ambiguity faced by a DM. I also describe well-known decision rules for decision-making under ambiguity. Decision solutions for these rules are derived for a representative DM in Section 3, with emphasis on when these decisions differ between CBA and CEA frameworks. Section 4.1 discusses where such disagreement may occur in practice using artificial example data detailed in Online Appendix II, Part A. The final sections explore remaining issues, discuss implications, and highlight main conclusions.
2. Methods
I formalize the choice problem of a representative DM recommending either an alternative or a status quo (SQ) policy. My approach isolates the effect of ambiguity over IBs, ICs, and the CE threshold on this recommendation, especially its differential effect when the DM considers CBA-type versus CEA-type information. To do this, I first establish a stylized setting in Section 2.1 where their solutions are equivalent in unambiguous situations. Section 2.2 then defines the ambiguities faced by the DM and the decision rules used in my analyses.
2.1. Setting
Assume all benefits are measured in the same monetary units and consider a simple CBA: a DM wants to approve a project if the benefits exceed the costs to society, that is
$ {B}_a-{C}_a>0 $
. Suppose there exists an SQ policy with benefits
$ {B}_0 $
and costs
$ {C}_0 $
. The DM would then switch to the alternative if
$ {B}_a-{C}_a>{B}_0-{C}_0 $
, or
$ \Delta B-\Delta C>0 $
where
$ \Delta B\equiv {B}_a-{B}_0 $
and
$ \Delta C\equiv {C}_a-{C}_0 $
.
Now assume that all benefits are measured in the same natural units
$ Q $
and let
$ g $
represent the monetary value of a unit benefit. Assuming
$ \Delta Q>0 $
, the DM selects the alternative if
$$ g\left(\Delta Q\right)-\Delta C>0\hskip0.3em \iff \hskip0.3em g>\left(\Delta C/\Delta Q\right), $$
that is, as long as the increase in costs per increase in benefits are below the threshold
$ g $
.Footnote
9
Inequalities 1 provide the underlying justification for CEA when the social welfare function is of the form
$ U= gQ-C $
. Assume that social welfare takes this form and define the CBA welfare (objective) function for the alternative by
$ {W}_a=g\left(\Delta {Q}_a\right)-\Delta {C}_a $
, with SQ welfare defined by
$ {W}_0=0 $
. The equivalence between CBA and social welfare is immediate. As well, define the CEA welfare (objective) function by
$ {V}_a=g-\left(\Delta {C}_a/\Delta {Q}_a\right) $
, with SQ welfare defined by
$ {V}_0=0 $
or equivalently
$ \Delta {C}_0/\Delta {Q}_0\equiv g $
. CBA maximizing
$ W $
and CEA maximizing
$ V $
are clearly equivalent in this one-alternative context where
$ \Delta {Q}_a>0 $
.
$ {W}_a $
and
$ {V}_a $
calculate the value of the alternative in the CBA and CEA frameworks, respectively. When considering a CBA, policymakers evaluate net benefits given by the first expression in Inequalities 1. When inspecting a CEA ratio, they compare the cost-efficiency of the alternative relative to the threshold
$ g $
, as formalized by the second expression. Evaluating the alternative by
$ {W}_a $
or
$ {V}_a $
yields the same policy recommendation in unambiguous environments, but not necessarily when
$ {W}_a $
or
$ {V}_a $
are not well-defined due to ambiguity.
2.2. Ambiguity in economic evaluation
There are three sources of ambiguity:
$ g $
,
$ \Delta Q $
, and
$ \Delta C $
. Assume that there is a “state space” of possible values
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
that defines all possible “states of the world”
$ (g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
that the DM deems feasible. That is,
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
is the collection of cost-effectiveness threshold, benefit increment, and cost increment triplets that the planner believes might represent the actual values. When there is no ambiguity,
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
is the singleton
$ \left\{\left(g,\Delta Q,\Delta C\right)\right\} $
. In this case, the welfare functions are well-defined and given by
$ {W}_a $
for CBA and
$ {V}_a $
for CEA.
Under ambiguity,
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
is non-singleton. It is possible to define the welfare for each possible point in the state space by the welfare functions
$ {W}_a $
and
$ {V}_a $
, but the DM is left with a collection of welfare functions – one for each point in the state space. That is, for CBA:
$$ {W}_a(g,\Delta {Q}_a,\Delta {C}_a)=g(\Delta {Q}_a)-\Delta {C}_a\ \mathrm{\forall}(g,\Delta {Q}_a,\Delta {C}_a)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $$
with SQ welfare defined by
$ {W}_0\left(g,\Delta {Q}_0,\Delta {C}_0\right)=0 $
. And for CEA:
$$ {V}_a(g,\Delta {Q}_a,\Delta {C}_a)=g-\frac{\Delta {C}_a}{\Delta {Q}_a}\ \mathrm{\forall}(g,\Delta {Q}_a,\Delta {C}_a)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $$
with SQ welfare defined by
$ {V}_0\left(g,\Delta {Q}_0,\Delta {C}_0\right)=0 $
. The planner must maximize a collection of functions – a problem which is not well-defined. How should the DM proceed?
Most people would agree that it is reasonable to select an option which is weakly better than all others at every point in the state space. If such a dominant strategy exists, the planner can optimize. In the case of CBA with one alternative, the optimal solution is
$$ {a}_{CBA}^{\ast }=\{\begin{array}{cc}1& \mathrm{i}\mathrm{f}\ {\min}_{(g,\Delta {Q}_a,\Delta {C}_a)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}{W}_a(g,\Delta {Q}_a,\Delta {C}_a)>0\\ {}0& \mathrm{i}\mathrm{f}\ {\max}_{(g,\Delta {Q}_a,\Delta {C}_a)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}{W}_a(g,\Delta {Q}_a,\Delta {C}_a)\le 0\end{array}\operatorname{}. $$
If
$ {\min}_{(g,\Delta {Q}_a,\Delta {C}_a)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}{W}_a(g,\Delta {Q}_a,\Delta {C}_a)<0<{\max}_{(g,\Delta {Q}_a,\Delta {C}_a)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}{W}_a(g,\Delta {Q}_a,\Delta {C}_a) $
then there is no optimal solution. For CEA with one alternative, the optimal solution and the condition for no optimal solution are given by replacing
$ {W}_a\left(g,\Delta {Q}_a,\Delta {C}_a\right) $
with
$ {V}_a\left(g,\Delta {Q}_a,\Delta {C}_a\right) $
.
When there is no optimal solution, the DM cannot optimize and so must employ a decision rule. There are no clear “best” rules, instead only those “reasonable” (Ferguson, Reference Ferguson1967, pp. 28–29).Footnote 10 Following Charles Manski, I evaluate a trinity of rules: Bayesian, MM, and MMR (Manski, Reference Manski2004, Reference Manski2009, Reference Manski2010, Reference Manski2013, Reference Manski2017, Reference Manski2018, Reference Manski2019).
To use a Bayesian decision rule, the DM must be able to formulate a subjective probability distribution
$ \pi $
on
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
which represents their prior as to the likelihood each state is the true state. They then evaluate
$ {W}_a $
and
$ {V}_a $
by their subjective averages
$ {W}_a\equiv {E}_{\pi}\left[{W}_a\left(g,\Delta {Q}_a,\Delta {C}_a\right)\right] $
and
$ {V}_a\equiv {E}_{\pi}\left[{V}_a\left(g,\Delta {Q}_a,\Delta {C}_a\right)\right] $
, respectively. If and only if
$ {W}_a>0 $
(
$ {V}_a>0 $
) does the DM select the alternative using CBA (CEA). It is, however, sometimes unreasonable to expect a DM to formulate such a
$ \pi $
, as the information burden can be large.
The MM rule developed by Wald (Reference Wald1945) selects in this setting the action that maximizes welfare in the state yielding minimum welfare, given that action. That is,
$ {W}_a\equiv {\min}_{(g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}{W}_a(g,\Delta Q,\Delta C) $
and
$ {V}_a $
is defined analogously for CEA. MM is very pessimistic (Savage, Reference Savage1951, p. 63), and Section 3 shows it selects the SQ if no strategy dominates.
Savage (Reference Savage1951) described the use of loss functions that capture maximum regret (MR), corresponding to the MMR rule. Regret is the difference between maximum welfare achievable in the realized state of the world and realized welfare in that state. MMR selects the policy with the lesser MR over states:
$ {W}_a=-{\max}_{(g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}[\max \{0,{W}_a(g,\Delta Q,\Delta C)\}-{W}_a(g,\Delta Q,\Delta C)] $
; or
$ {W}_s=-{\max}_{(g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}[\max \{0,{W}_a(g,\Delta Q,\Delta C)\}-0] $
where
$ s $
denotes the SQ, and with
$ {V}_a $
and
$ {V}_s $
defined analogously for CEA.
Notation: In Section 3, I derive solutions for the CBA and CEA objective functions for each of these decision rules. I often suppress subscript
$ a $
to reduce clutter. Also, note that
$ \unicode{x1D53E} $
,
$ \mathrm{\mathbb{C}} $
, and
$ \mathrm{\mathbb{Q}} $
are each subsets of the real line
$ \mathrm{\mathbb{R}} $
by definition. For convenience, define
$ {g}_L={\min}_{g\in \unicode{x1D53E}}[g] $
,
$ {g}_H={\max}_{g\in \unicode{x1D53E}}[g] $
,
$ \underline{Q}={\min}_{\Delta Q\in \mathrm{\mathbb{Q}}}\left[\Delta Q\right] $
,
$ \overline{Q}={\max}_{\Delta Q\in \mathrm{\mathbb{Q}}}\left[\Delta Q\right] $
,
$ \underline{C}={\min}_{\Delta C\in \mathrm{\mathbb{C}}}\left[\Delta C\right] $
, and
$ \overline{C}={\max}_{\Delta C\in \mathrm{\mathbb{C}}}\left[\Delta C\right] $
. Assume throughout that
$ {g}_L\ge 0 $
and
$ \underline{Q}>0 $
. The first assumption is innocuous as the value of a benefit ought to be positive. The latter assumption is more restrictive. For example, it precludes the possibility that a new drug, the alternative, reduces quality-adjusted life years (QALYs) relative to an SQ treatment. Future work should incorporate adjustments for this possibility. I focus on the many applications in practice where this restriction is valid, instead highlighting attention on the specific role of ambiguity.Footnote
11
One situation where this assumption may be valid is the evaluation of sensitivity analyses. For practical reasons, a researcher may present policymakers a set of sensitivity or scenario analyses
$ (\overset{\sim }{\unicode{x1D53E}},\overset{\sim }{\unicode{x211A}},\overset{\sim }{\unicode{x2102}})\subseteq (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
rather than attempting to communicate the full extent of structural uncertainty and ambiguity. Throughout this paper, I write
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
for simplicity in settings where the DM may actually be using
$ (\overset{\sim }{\unicode{x1D53E}},\overset{\sim }{\unicode{x211A}},\overset{\sim }{\unicode{x2102}}) $
. Certain assumptions may be more reasonable under
$ (\overset{\sim }{\unicode{x1D53E}},\overset{\sim }{\unicode{x211A}},\overset{\sim }{\unicode{x2102}}) $
than
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
. Section 3.2 expands on this distinction.
2.3. Visual representation
My results derive from the nonlinear distortion in the value of benefits introduced by the CEA objective function. Intuition can be developed by inspection of alternative states of the world
$ (g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
plotted on the northeastern quadrant of the CE plane.
Figure 1, Panel A, depicts a single state
$ \left({g}_1,\Delta {Q}_1,\Delta {C}_1\right) $
alongside CBA and CEA objective function calculations for that specific point. In particular, the cost-effectiveness threshold
$ {g}_1 $
is represented by a line from the origin on the plane with slope
$ {g}_1 $
. The point
$ \left(\Delta {Q}_1,\Delta {C}_1\right) $
is plotted with a blue dot, and the line which connects this dot to the origin has slope
$ \Delta {C}_1/\Delta {Q}_1 $
. The CEA calculation for this point is given by
$ {g}_1-\Delta {C}_1/\Delta {Q}_1 $
and is therefore equal to the difference between two slopes. By contrast, the CBA calculation is given by
$ {g}_1\Delta {Q}_1-\Delta {C}_1 $
, and
$ {g}_1\Delta {Q}_1 $
is the point which lies on the line that represents the cost-effectiveness threshold at
$ \Delta {Q}_1 $
. Consequently, the CBA calculation is represented by the vertical gap between this line and the point
$ \left(\Delta {Q}_1,\Delta {C}_1\right) $
. As a small difference in slopes creates a large vertical gap when
$ \Delta {Q}_1 $
is large, the CBA and CEA objective functions defined above provide distinct information.
Ambiguity on the cost-effectiveness plane.
Notes:
$ \Delta C $
represents incremental costs,
$ \Delta Q $
represents incremental benefits, and
$ g $
represents the cost-effectiveness threshold. Each blue dot represents a combination of
$ \left(\Delta Q,\Delta C\right) $
considered possible by the DM, and each grey line represents a
$ g $
considered possible by the DM.
By definition, ambiguity requires that
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
be non-singleton. Panel B provides two examples of a state space
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
. First, consider a setting where the cost-effectiveness threshold is known to be some value
$ {g}_H $
but the values
$ \left(\Delta Q,\Delta C\right) $
are ambiguous and given by the set of blue points. In this case, there are a finite number of possible states of the world considered by the policymaker. Neither policy is a dominant strategy provided that some points are on each side of
$ {g}_H $
. If the DM considers
$ g $
to be ambiguous such that
$ g\in \left[{g}_L,{g}_H\right] $
then this can be represented by an infinite state space including all combinations of a blue point with some threshold
$ g $
contained within the grey-shaded region between
$ {g}_L $
and
$ {g}_H $
.
Finally, I plot the values of
$ \underline{C} $
,
$ \overline{C} $
,
$ \underline{Q} $
, and
$ \overline{Q} $
implied by the set of blue points included in Panel B. I shade the rectangle between these points blue to highlight the role of this region in defining “rectangular” state spaces.Footnote
12 The notion of a rectangular state space is used occasionally in the analyses of this paper, such as within Theorem 5. Strictly speaking, I do not require that
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
include all the shaded area, but instead only that the corners
$ \{({g}_H,\overline{Q},\underset{\_}{C}),({g}_L,\underset{\_}{Q},\overline{C})\}\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
are required. The connection is as follows: if we “rectangularize” the state space defined by the seven blue points to include the entire shaded region, then these corners will be included in
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
if the threshold
$ g $
does not depend on ICs and IBs. On the other hand, it is clear that both these corners are not included in
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
in this particular example without rectangularization.
Figure 2 depicts implementation of the MMR rule under the CBA (Panel A) and the CEA (Panel B) frameworks given unambiguous
$ g $
. Under CBA, if the DM selects the SQ, then the MR is the largest vertical gap between
$ g $
and all the points below
$ g $
. These points are the possible combinations of IBs and ICs that produce positive NMB when combined with
$ g $
. The DM thus regrets not having chosen the alternative and realizing these NMB if the true IBs and ICs are represented by one of these points. Their MR is then given by the maximum vertical distance between these points and
$ g $
. Analogously, MR under the alternative is given by the maximum vertical distance between
$ g $
and the set of points above the line
$ g $
.Footnote
13 In Panel A, it is clear that the MR under the alternative is less than under the SQ, and thus the DM implements the alternative to minimize MR.
Minimax Regret on the cost-effectiveness plane.
Notes:
$ \Delta C $
represents incremental costs,
$ \Delta Q $
represents incremental benefits, and
$ g $
represents the cost-effectiveness threshold. Each blue dot represents a combination of
$ \left(\Delta Q,\Delta C\right) $
considered possible by the DM, and the grey line represents the sole
$ g $
considered possible by the DM.
Panel B depicts MMR under CEA. If the DM selects the SQ, then the MR is the difference in slopes between
$ g $
and all the slopes (with respect to the origin [WRTO]) generated by the points below
$ g $
. These points are the possible (
$ \Delta Q $
,
$ \Delta C $
) that produce ICERs less than
$ g $
. The DM thus regrets not recommending the alternative and realizing this degree of cost-effectiveness if the true IBs and ICs are represented by one of these points. Their MR is given by the maximum difference in the slopes WRTO generated by these points and
$ g $
. Analogously, MR under the alternative is the maximum difference between
$ g $
and the slopes WRTO of the points above
$ g $
. The DM selects the option that minimizes MR.
Figure 2 can also describe the MM rule. Under CBA, if the DM picks the SQ they earn
$ 0 $
added welfare in any state and minimum welfare is
$ 0 $
. If they pick the alternative, minimum welfare is given by the negative of the maximum vertical difference between
$ g $
and a point above
$ g $
. Given no dominant strategy, this is strictly negative and the DM picks the SQ to maximize minimum welfare. The CEA analysis is analogous except that minimum welfare under the alternative is given by the negative of the maximum difference between the slopes WRTO generated by the set of points above
$ g $
and the slope represented by
$ g $
.
Figure 3 depicts a Bayesian rule under CBA (Panel A) and CEA (Panel B) where
$ g $
is unambiguous. A Bayesian DM uses a subjective prior
$ \pi $
to calculate weighted means for comparison. Using CBA, they use a weighted mean of all the vertical distances between
$ g $
and the blue points, and will pick the alternative when this mean exceeds 0 (lies below
$ g $
). Panel A presents a case with equal weights on two scenarios under a fixed
$ g $
, and so each vertical difference is multiplied by 1/2 and compared directly. Here, the positive weighted NMB (0.5*
$ \left[\mathrm{NMB}\left(+\right)\right] $
) outweighs negative weighted NMB (0.5*
$ \left[\mathrm{NMB}\left(-\right)\right] $
) and so this DM picks the alternative. Using CEA in Panel B, they use a weighted mean of all the differences between
$ g $
and the slopes WRTO generated by the blue points. They pick the alternative when this mean exceeds 0 (lies below
$ g $
). When
$ g $
is known, the weighted mean of all the differences between
$ g $
and the slopes WRTO generated by the blue points is simply the difference between
$ g $
and the weighted mean of all these slopes WRTO. For equal weights on two such slopes, this is the slope midway between them. In Panel B, the DM picks the SQ.
Bayesian decision-making on the cost-effectiveness plane.
Notes: NMB denotes “net monetary incremental benefits,”
$ \Delta C $
represents incremental costs,
$ \Delta Q $
represents incremental benefits, and
$ g $
represents the cost-effectiveness threshold. Each blue dot represents a combination of
$ \left(\Delta Q,\Delta C\right) $
considered possible by the DM, and the grey line represents the sole
$ g $
considered possible by the DM.
3. Results
In this section, I solve the relevant decision problems and establish when the representative DM will recommend the alternative or SQ. I compare results between CBA and CEA for each decision rule to highlight the role of ambiguity. A main result is that CEA can cause errors in judgment when there exists ambiguity in IBs. Ambiguity in
$ g $
or ICs is less problematic.
DMs should always select a dominant strategy: one that is better than all others in every state of the world
$ (g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
. In the one-alternative context, it suffices to compare the alternative against the SQ.
Theorem 1 The alternative or the status quo is the dominant strategy under CBA if and only if (iff) it is also the dominant strategy under CEA.Footnote 14
Using CBA or CEA will be equivalent when either policy is dominant. There is otherwise no clear solution using either approach, and there may be ambiguity in
$ g $
,
$ \Delta Q $
, or
$ \Delta C $
. Consider first a setting where the only ambiguity is
$ g\in \left[{g}_L,{g}_H\right] $
and no dominant strategy.
Theorem 2 If there is no dominant strategy,
$ \underline{Q}=\overline{Q}=\Delta Q $
,
$ \underline{C}=\overline{C}=\Delta C $
, and
$ g\in \left[{g}_L,{g}_H\right] $
, then for both CBA and CEA welfare functions, (i) a Bayesian planner puts a prior
$ \pi $
on
$ g $
and selects the alternative iff
$ {E}_{\pi}\left[g\right]>\frac{\Delta C}{\Delta Q} $
; (ii) an MM planner selects the alternative iff
$ {g}_L>\frac{\Delta C}{\Delta Q} $
; and (iii) an MMR planner picks the alternative iff
$ {g}_H\Delta Q-\Delta C>-\left[{g}_L\Delta Q-\Delta C\right] $
.
CBA and CEA are equivalent when the only ambiguity is in
$ g $
and so errors in judgment will not occur in this context. Thus, the discordance between CEA and CBA is not necessarily due to unwillingness to specify
$ g $
. To demonstrate where issues arise, the next subsection considers when there can be ambiguity in the threshold
$ g $
, costs
$ \Delta C $
, and benefits
$ \Delta Q $
.
3.1. Multidimensional ambiguity
In the general case where
$ (g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
, I find that only ambiguity in benefits
$ \Delta Q\in \mathrm{\mathbb{Q}} $
matter for discordance between CBA and CEA in this setting. I first analyze Bayes, MM, and MMR decisions under CBA and CEA welfare functions. Under certain conditions, a DM using CEA will be conservative relative to CBA, but only if there is ambiguity in
$ \mathrm{\mathbb{Q}} $
.
Theorem 3 If there is no dominant strategy,
$ g\in \unicode{x1D53E} $
,
$ \Delta Q\in \mathrm{\mathbb{Q}} $
, and
$ \Delta C\in \mathrm{\mathbb{C}} $
, then the solutions used by a Bayesian DM with distribution
$ \pi $
on
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
are not generally equivalent under CBA and CEA: (i) under CBA, a Bayesian DM selects the alternative iff
$ {E}_{\pi}\left[g\Delta Q\right]>{E}_{\pi}\left[\Delta C\right] $
; and (ii) under CEA, a Bayesian DM selects the alternative iff
$ {E}_{\pi}\left[g\right]>{E}_{\pi}\left[\frac{\Delta C}{\Delta Q}\right] $
. Moreover, if (a) the marginal distributions on
$ \unicode{x1D53E} $
and
$ \unicode{x211A} $
are independent under
$ \pi $
, the Bayesian DM using CBA selects the alternative iff
$ {E}_{\pi}\left[g\right]>\frac{E_{\pi}\left[\Delta C\right]}{E_{\pi}\left[\Delta Q\right]} $
; (b) the marginal distributions on
$ \mathrm{\mathbb{Q}} $
and
$ \mathrm{\mathbb{C}} $
are independent under
$ \pi $
, the Bayesian DM using CEA selects the alternative iff
$ {E}_{\pi}\left[g\right]>{E}_{\pi}\left[\Delta C\right]{E}_{\pi}\left[\frac{1}{\Delta Q}\right] $
; and (c) the marginal distributions on
$ \unicode{x1D53E} $
and
$ \mathrm{\mathbb{Q}} $
are independent and the marginal distributions on
$ \unicode{x211A} $
and
$ \unicode{x2102} $
are also independent under
$ \pi $
, then the Bayesian DM selects the alternative under CBA if they select the alternative under CEA.
Theorem 3 demonstrates that in the presence of ambiguity, a planner using a Bayesian decision rule will not generally arrive at the same solution using CEA as they would had they used a CBA welfare function. This could lead to errors in judgment. In general, a planner using CEA could accidentally approve an alternative they ought to have rejected using their prior
$ \pi $
or they might reject an alternative they ought to have approved. When the marginal distributions on
$ \mathrm{\mathbb{Q}} $
and
$ \mathrm{\mathbb{C}} $
as well as those on
$ \mathrm{\mathbb{Q}} $
and
$ \unicode{x1D53E} $
are independent under
$ \pi $
, it may be the case that the DM would approve the alternative under CBA but reject it under CEA; 3(c) precludes the converse error. CEA is conservative in this context.Footnote
15
Figure 4 provides a visualization for result (c) of Theorem 3. In Panel A, a Bayesian DM reviews data from
$ \left\{\left(\Delta {Q}_1,\Delta {C}_1\right),\left(\Delta {Q}_2,\Delta {C}_2\right)\right\} $
with
. For
$ j\in \left\{1,2\right\} $
, we have (i)
$ \Delta {Q}_j $
can take any value between 0.1 and 3 in increments of 0.1; (ii)
$ \Delta {C}_j $
can take any value between £5,000 and £120,000 in increments of £5,000; and (iii) the marginal distributions on
$ \mathrm{\mathbb{Q}} $
and
$ \mathrm{\mathbb{C}} $
are independent under
$ \pi $
with the probability of
$ \Delta {Q}_1 $
and
$ \Delta {C}_1 $
each taking any value between 0.1 and 0.9 in increments of 0.1 only, with
$ \mathrm{Prob}\left(\Delta {Q}_2\right)=1-\mathrm{Prob}\left(\Delta {Q}_1\right) $
and
$ \mathrm{Prob}\left(\Delta {C}_2\right)=1-\mathrm{Prob}\left(\Delta {C}_1\right) $
. There are then
$ ({30}^2)({24}^2)({9}^2)=41,990,400 $
possible combinations of data and
$ \pi $
in this example. Each grey point plots the expected values of the CBA versus the CEA objective functions, denoted by
$ W $
and
$ V $
respectively for convenience, for one such combination (in thousands).Footnote
16 Dropping if
$ |W|<1 $
x
$ {10}^{-12} $
or
$ |V|<1 $
x
$ {10}^{-12} $
to avoid issues with rounding error, it is never the case that
$ V>0 $
with
$ W<0 $
. That is, this DM selects the alternative under CBA if they do so under CEA. There also clearly exist combinations where they reject the alternative under CEA but approve it under CBA.
Visualization of Theorem 3c.
Notes: The expected values of the CBA and CEA objective functions are denoted by
$ W $
and
$ V $
, respectively. The horizontal (vertical) red lines denote
$ W=0 $
(
$ V=0 $
). Panel A depicts a random sample of 1% of
$ W $
and
$ V $
(in thousands) for all 41,990,400 possible combinations of data and
$ \pi $
in a simple example where a Bayesian DM reviews data from
$ \left\{\left(\Delta {Q}_1,\Delta {C}_1\right),\left(\Delta {Q}_2,\Delta {C}_2\right)\right\} $
with
, and where the marginal distributions on
$ \mathrm{\mathbb{Q}} $
and
$ \mathrm{\mathbb{C}} $
are independent. Panel B depicts
$ W $
and
$ V $
(in thousands) from 10,000 randomized draws in a simple example where a Bayesian DM reviews data from 10 total pairs of
$ \left(\Delta Q,\Delta C\right) $
with
, and where the marginal distributions on
$ \mathrm{\mathbb{Q}} $
and
$ \mathrm{\mathbb{C}} $
are independent. Arbitrary data generated and analysis performed in Stata MP, Version 18.5.
In Panel B, a Bayesian DM reviews data instead from 10 total pairs of
$ \left(\Delta Q,\Delta C\right) $
with
. For
$ j=1,\dots, 10 $
, we have (i) each
$ \Delta {Q}_j $
is drawn uniformly from
$ \left[\mathrm{0.1,3}\right] $
, (ii) each
$ \Delta {C}_j $
is drawn uniformly from
, and (iii) the marginal distributions on
$ \mathrm{\mathbb{Q}} $
and
$ \mathrm{\mathbb{C}} $
are independent under
$ \pi $
with the probability of
$ \Delta {Q}_j $
given by
$ {w}_j/\left({\sum}_{j=1}^{10}{w}_j\right) $
where each weight
$ {w}_j $
is drawn independently and uniformly from
$ \left[0,1\right] $
. The probabilities of
$ \Delta {C}_j $
are generated analogously. I repeat this analysis 10,000 times and each grey point in Panel B plots
$ W $
and
$ V $
(as defined above) for each combination. Once again, it is clear that the Bayesian DM selects the alternative under CBA if they select the alternative under CEA.
Theorem 4 If there is no dominant strategy,
$ g\in \unicode{x1D53E} $
,
$ \Delta Q\in \mathrm{\mathbb{Q}} $
, and
$ \Delta C\in \mathrm{\mathbb{C}} $
, then, (i) under CBA, an MM planner selects the alternative iff
$ {\min}_{(g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}[g(\Delta Q)-\Delta C]>0 $
; and (ii) under CEA, an MM planner selects the alternative iff we have
$ {\min}_{(g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}[g-\frac{\Delta C}{\Delta Q}]>0 $
. The MM planner thus rejects the alternative under both CBA and CEA. Moreover, an MM planner approves the alternative iff it is the dominant strategy.
Theorem 4 says that under ambiguity, a DM using an MM rule will generally arrive at the same solutions using CEA and CBA objective functions. This occurs because MM is very conservative and picks the point in the state-space that is “worst” in
$ {W}_a $
or
$ {V}_a $
if the DM picks the alternative. Given no dominant strategy, the minimum of both must be weakly negative. The DM thus always picks the SQ under both CBA and CEA framings given no dominant strategy, and so a less conservative DM would select a different decision rule.
Theorem 5 If there is no dominant strategy,
$ g\in \unicode{x1D53E} $
,
$ \Delta Q\in \mathrm{\mathbb{Q}} $
, and
$ \Delta C\in \mathrm{\mathbb{C}} $
, then MMR recommendations are not generally equivalent under CBA and CEA. An MMR planner, (i) under CBA, picks the alternative iff
$ {\max}_{(g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}\{g\Delta Q-\Delta C\}+{\min}_{(g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}\{g\Delta Q-\Delta C\}>0 $
; and (ii) under CEA, picks the alternative iff
$ {\max}_{(g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}[g-\frac{\Delta C}{\Delta Q}]+{\min}_{(g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}[g-\frac{\Delta C}{\Delta Q}]>0 $
. If the state space is "rectangular" such that
$ \{({g}_H,\overline{Q},\underset{\_}{C}),({g}_L,\underset{\_}{Q},\overline{C})\}\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
, then an MMR planner, (a) under CBA, selects the alternative iff
$ \left(\overline{C}+\underline{C}\right)<\left({g}_H\overline{Q}+{g}_L\underline{Q}\right) $
; and (b) under CEA, selects the alternative iff
$ \frac{\overline{C}\overline{Q}+\underline{C}\underline {Q}}{2\underline{Q}\overline{Q}}<\frac{g_L+{g}_H}{2} $
.
Theorem 5 states that under ambiguity, a DM using an MMR rule will not generally arrive at the same solutions using CEA and CBA. When the state space is “rectangular” as defined in the theorem, the DM might approve the alternative under CBA but reject it under CEA. I show below that the converse error is precluded in this context [(ii)] or when all
$ g\in \unicode{x1D53E} $
are considered possible at any
$ \left(\Delta Q,\Delta C\right) $
and the data satisfy a condition on
$ \overline{Q} $
[(i)].
Theorem 6 Suppose there is no dominant strategy,
$ g\in \unicode{x1D53E} $
,
$ \Delta Q\in \mathrm{\mathbb{Q}} $
,
$ \Delta C\in \mathrm{\mathbb{C}} $
, and also (i)
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})\equiv \unicode{x1D53E}\times (\unicode{x211A},\unicode{x2102}) $
where
$ \times $
denotes the Cartesian product, and
$ \mathrm{\exists}(g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}):\Delta Q=\overline{Q}\wedge (g,\Delta Q,\Delta C)\in {\mathrm{argmax}}_{(g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}[g-\frac{\Delta C}{\Delta Q}] $
; or (ii) the state space is “rectangular” as defined in the statement of Theorem 5. Then an MMR planner using CBA will select the alternative if an MMR planner under CEA will select the alternative.
Figure 5 provides a sketch of this result. Panel A depicts 6(i) using arbitrary data. Here, the blue (black) dot represents the maximum (minimum) slope generated by any of the dots
$ \left(\Delta Q,\Delta C\right) $
WRTO. All the remaining grey dots representing other possible
$ \left(\Delta Q,\Delta C\right) $
must thus lie in the cone between these maximum and minimum slopes. The fact that
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})\equiv \unicode{x1D53E}\times (\unicode{x211A},\unicode{x2102}) $
simply means that all
$ g\in \unicode{x1D53E} $
are considered possible at any
$ \left(\Delta Q,\Delta C\right) $
. That
$ \mathrm{\exists}(g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}):\Delta Q=\overline{Q}\wedge (g,\Delta Q,\Delta C)\in {\mathrm{argmax}}_{(g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})}[g-\frac{\Delta C}{\Delta Q}] $
means the black dot lies on
$ \overline{Q} $
in this setting. The basic intuition is that if the difference between
$ {g}_H $
and the slope generated by this point is larger than the difference between
$ {g}_L $
and the slope generated by the blue dot (DM selects alternative under CEA), then the vertical gap between
$ {g}_H $
and the black dot must be larger than the vertical gap between
$ {g}_L $
and any possible point on or to the left of
$ \overline{Q} $
. Because the relevant difference in slopes is smaller for the blue dot than the black dot,
$ \Delta Q $
must be larger than
$ \overline{Q} $
for the vertical gap between
$ {g}_L $
and a point with the same slope as the blue dot to be larger than the vertical gap between
$ {g}_H $
and the black dot. As this is not possible and these vertical gaps represent regret, it is easy to see that the MR from selecting the alternative would occur if there was some
$ \left(\Delta Q,\Delta C\right) $
on the line generated by the blue dot WRTO and
$ \Delta Q=\overline{Q} $
. But this hypothetical MR must be smaller than the MR from the SQ, which is given by the vertical gap between
$ {g}_H $
and the black dot. The DM thus minimizes MR by selecting the alternative under CBA.
Sketch of Theorem 6.
Notes: MR denotes “Maximum Regret,” Alt. denotes “Alternative,” and SQ denotes “Status Quo.”
$ \Delta C $
represents incremental costs,
$ \Delta Q $
represents incremental benefits, and
$ g $
represents the cost-effectiveness threshold. Each dot represents a combination of
$ \left(\Delta Q,\Delta C\right) $
considered possible by the DM, and each solid line from the origin represents a
$ g $
considered possible by the DM. The minimum (maximum) value of
$ \Delta Q $
considered possible by the DM is given by
$ \underline{Q} $
(
$ \overline{Q} $
). The minimum (maximum) value of
$ \Delta C $
considered possible by the DM is given by
$ \underline{C} $
(
$ \overline{C} $
).
Panel B depicts 6(ii). Here, a “rectangular” state space means
$ \left({g}_L,\underline{Q},\overline{C}\right) $
(blue) and
$ \left({g}_H,\overline{Q},\underline{C}\right) $
(black) are possible, and other possible
$ \left(g,\Delta Q,\Delta C\right) $
must lie in the shaded rectangle with
$ g\in \unicode{x1D53E} $
some slope of a line between
$ {g}_H $
and
$ {g}_L $
. The largest difference in slopes between
$ {g}_L $
and the slopes generated by points above
$ {g}_L $
WRTO will occur at
$ \left({g}_L,\underline{Q},\overline{C}\right) $
and the largest difference between
$ {g}_H $
and the slopes generated by points below
$ {g}_H $
WRTO will occur at
$ \left({g}_H,\overline{Q},\underline{C}\right) $
. Then clearly the blue point has the largest vertical gap above any
$ g\in \unicode{x1D53E} $
(MR from alternative) and the black point has the largest vertical gap below any
$ g\in \unicode{x1D53E} $
(MR from SQ). If the difference between
$ {g}_H $
and the slope generated by the black point WRTO is larger than the difference between
$ {g}_L $
and the slope generated by the blue point WRTO (DM picks alternative under CEA), then by the same logic as for Panel A it must be that the vertical gap between
$ {g}_H $
and the black point is larger than the vertical gap between
$ {g}_L $
and any point on the line generated by the blue point WRTO, provided
$ \Delta Q\le \overline{Q} $
. The MR of the alternative is thus less than that of the SQ: the DM picks the alternative under CBA.
CEA is again conservative under ambiguity in this context, and can lead to errors in judgment. But this “conservatism” of CEA over CBA under ambiguity for Bayesian and MMR planners only occurs when there is ambiguity in benefits. Ambiguity in costs and the cost-effectiveness threshold
$ g $
might cause the solutions to change under different decision rules, but the solutions for a given decision rule will be the same under CBA and CEA.
Theorem 7 If there is no dominant strategy and
$ \underline{Q}=\overline{Q}=\Delta Q $
, CBA and CEA are equivalent for each of (a) Bayes, (b) MM, and (c) MMR planners.
Figure 6 depicts intuition for Theorem 7, focusing on MMR planners. In Panel A, there is only ambiguity in
$ g\in \unicode{x1D53E} $
. That is,
$ \left(\mathrm{\mathbb{Q}},\mathrm{\mathbb{C}}\right)=\left\{\Delta Q,\Delta C\right\} $
: there is one possible point on the CE plane but the DM is unsure about
$ g $
. Any point in I (II) has a dominant strategy of rejecting (approving) the alternative for all
$ g\in \unicode{x1D53E} $
and so CBA and CEA are equivalent by Theorem 1. Any point in III–IV is above some
$ g $
and below others in
$ \unicode{x1D53E} $
. However, the blue line that divides these regions has the same absolute value difference in slopes with
$ {g}_H $
and
$ {g}_L $
at any fixed
$ \Delta Q $
. The blue line thus also represents the vertical midpoint between
$ {g}_H $
and
$ {g}_L $
at any fixed
$ \Delta Q $
. Any point below (above) the blue line in III (IV) will thus both generate a slope WRTO with a larger absolute difference to
$ {g}_H $
(
$ {g}_L $
) than
$ {g}_L $
(
$ {g}_H $
) and will have a larger vertical distance to
$ {g}_H $
(
$ {g}_L $
) than
$ {g}_L $
(
$ {g}_H $
): an MMR planner should select the alternative (SQ) under both CBA and CEA. In all areas, the MMR planner makes the same decision under CEA and CBA.
Intuition for Theorem 7 for Minimax Regret.
Notes: Panel A depicts when there is only ambiguity in
$ g $
, with
$ \left(\unicode{x211A},\unicode{x2102}\right)=\left\{\Delta Q,\Delta C\right\} $
. Panel B depicts when there is only ambiguity in
$ \Delta C $
, with
$ (\unicode{x1D53E},\unicode{x211A})=\{g,\Delta Q\} $
. NMB denotes “net monetary incremental benefits,”
$ \Delta C $
represents incremental costs,
$ \Delta Q $
represents incremental benefits, and
$ g $
represents the cost-effectiveness threshold. Each blue dot represents a combination of
$ \left(\Delta Q,\Delta C\right) $
considered possible by the DM, and each solid line from the origin represents a
$ g $
considered possible by the DM. The minimum (maximum) value of
$ \Delta Q $
considered possible by the DM is given by
$ \underline{Q} $
(
$ \overline{Q} $
). The minimum (maximum) value of
$ \Delta C $
considered possible by the DM is given by
$ \underline{C} $
(
$ \overline{C} $
).
In Panel B, there is only ambiguity in
$ \Delta C\in \unicode{x2102} $
. That is,
$ (\unicode{x1D53E},\unicode{x211A})=\{g,\Delta Q\} $
: there is a single
$ g $
and all points must be on the line
$ \Delta Q=\underline{Q}=\overline{Q} $
. It is clear that the point at
$ \overline{C} $
(
$ \underline{C} $
) will be the point above (below)
$ g $
with the largest distance to
$ g $
and the largest absolute difference between
$ g $
and the slope generated by the point WRTO. An MMR planner under CBA selects the alternative iff the difference in the vertical distances is positive:
$ \left(g\overline{Q}-\underline{C}\right)-\left(g\overline{Q}-\overline{C}\right)=\overline{C}-\underline{C}>0 $
. An MMR planner under CEA selects the alternative iff the difference of the difference in the slopes generated by the points WRTO and
$ g $
is positive:
$ \left[g-\left(\underline{C}/\overline{Q}\right)\right]-\left[g-\left(\overline{C}/\overline{Q}\right)\right]={\overline{Q}}^{-1}\left[\overline{C}-\underline{C}\right]>0 $
. Clearly,
$ \overline{C}-\underline{C}\gtreqless 0\hskip0.3em \iff \hskip0.3em {\overline{Q}}^{-1}\left[\overline{C}-\underline{C}\right]\gtreqqless 0 $
for
$ \overline{Q}>0 $
and so the MMR planner makes the same decision under CEA and CBA.
We can also allow ambiguity in both
$ g $
and
$ \Delta C $
, but CEA and CBA decisions will remain equivalent. If IBs are unambiguous, then CEA and CBA decisions are equivalent for Bayes, MM, and MMR planners. If they are ambiguous, the solutions are generally not equivalent except for MM planners. In practice, there will often be significant ambiguity in benefits. Section 4.1 demonstrates this reality with example data.
3.2. Sensitivity analyses and conservativism
Ambiguity in benefits can lead DMs to be conservative when using CEA instead of CBA. Bayesian DMs make this mistake when the marginal distributions on
$ \unicode{x1D53E} $
and
$ \mathrm{\mathbb{Q}} $
as well as those on
$ \mathrm{\mathbb{Q}} $
and
$ \mathrm{\mathbb{C}} $
are independent under
$ \pi $
. MMR planners do so when the state space considered is “rectangular” or when both: (a)
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102})\equiv \unicode{x1D53E}\times (\unicode{x211A},\unicode{x2102}) $
; and (b) some
$ \left(g,\overline{Q},\Delta C\right) $
yields the largest difference between
$ g $
and
$ \left(\Delta C/\Delta Q\right) $
among
$ (g,\Delta Q,\Delta C)\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
. These assumptions are strong but may often hold for DMs evaluating a set of sensitivity analyses.
In practice, a researcher presents sensitivity analyses
$ (\overset{\sim }{\unicode{x1D53E}},\overset{\sim }{\unicode{x211A}},\overset{\sim }{\unicode{x2102}})\hskip2pt \subseteq \hskip2pt (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
to a DM. The DM considers
$ (\overset{\sim }{\unicode{x1D53E}},\overset{\sim }{\unicode{x211A}},\overset{\sim }{\unicode{x2102}}) $
and makes a decision without always knowing or considering all of the correlations between
$ g $
,
$ \Delta C $
, and
$ \Delta Q $
within the models used by the researcher. Thus, the Bayesian distribution
$ \pi $
does not necessarily reflect underlying statistical associations. Instead,
$ \pi $
reflects how the decision-maker combines data in
$ (\overset{\sim }{\unicode{x1D53E}},\overset{\sim }{\unicode{x211A}},\overset{\sim }{\unicode{x2102}}) $
, and they may consider information independently even when costs and benefits are related, for example.
Consider next conditions (a) and (b) above. In practice, (a) could often hold when
$ g $
is chosen by the DM or determined separately from the relevant economic evaluations. It is unclear how often (b) holds, but it is not an unreasonable condition. As
$ \Delta Q $
is found in the denominator of
$ \left(\Delta C/\Delta Q\right) $
, this condition will hold, for example, when
$ \overline{Q} $
is sufficiently large relative to other estimates of IBs and its corresponding estimate of
$ \Delta C $
is not as dissimilar to other estimates of ICs. Finally, consider “rectangular” spaces such that
$ \{({g}_H,\overline{Q},\underset{\_}{C}),({g}_L,\underset{\_}{Q},\overline{C})\}\in (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
. This amounts to the DM considering both the best and worst possible combinations of values feasible. This seems reasonable and so the set
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
implicitly considered by DMs may sometimes satisfy such a condition.Footnote
17
4. Discussion
I next explore where decision errors might occur in practice based on my results, conditional on the decision rule used by a DM evaluating sensitivity analyses
$ (\overset{\sim }{\unicode{x1D53E}},\overset{\sim }{\unicode{x211A}},\overset{\sim }{\unicode{x2102}})\hskip2pt \subseteq \hskip2pt (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
provided by an analyst. I then discuss how my results inform the review of economic evaluations.
4.1. Practical example
In Online Appendix II, Part A, I show discordance between decision-making under CBA and CEA objective functions in a practical example. I use artificial example data for the final estimates of incremental costs and incremental benefits (e.g. QALYs) from a “preferred” model and nine sensitivity analyses. Allowing
$ g $
to take NICE’s standard £20,000 and £30,000 thresholds (NICE, 2013, p. 38), I find that using CEA instead of CBA can lead to errors. In particular, I demonstrate that there are subjective
$ \pi $
for which CEA and CBA provide differing recommendations. I also find discordance between CEA and CBA decision-making for an MMR planner when the threshold is ambiguous or known to be £30,000.Footnote
18 Employing the CBA objective function is therefore critical when evaluating alternatives under ambiguity, and DMs should pay keen attention to the decision rules which they implicitly apply.
4.2. Policy implications
O’Mahony (Reference O’Mahony2020) suggested that continuing to use ICERs in tandem with net benefits is likely not detrimental. My analysis instead shows that any framing of results from economic evaluations as ICERs may inadvertently lead to decision errors. While empirical evidence will be required to support this prediction, presenting only information such as IBs and ICs in appraisal submissions rather than ICERs may allow DMs to combine ambiguous information into final ICER estimates if desired while otherwise mitigating framing-induced error.
However, CEAs are often performed by industry, involve many discretionary decisions, and are therefore subject to potential bias (Kassirer & Angell, Reference Kassirer and Angell1994). Hillman et al. (Reference Hillman, Eisenberg, Pauly, Bloom, Glick, Kinosian and Schwartz1991) recommend conservative assumptions and the publication of sensitivity analyses to help avoid bias in favor of evaluated technologies. This translates to honesty about ambiguity in decisions made by invigilators and the need for conservatism due to potential bias. As well, there is already evidence of excess approval by NICE (Claxton et al., Reference Claxton, Sculpher, Palmer and Culyer2015). As I show in Section 3 that the DM under CEA will approve fewer technologies relative to CBA in certain reasonable contexts, the heavy reliance on ICERs in economic evaluation may sometimes have an underappreciated “pro” when there is ambiguity in IBs. Reducing the use of ICERs in favor of NMB may inadvertently increase approval rates and policy-makers must therefore weigh the potential theoretical benefits and costs of such a change in guidelines.
Also, CEA is often used to avoid explicit specification of
$ g $
, which is required in welfare analyses and CBA (Phelps & Mushlin, Reference Phelps and Mushlin1991; Garber, Reference Garber, Culyer and Newhouse2000; Paulden, Reference Paulden2020). My results suggest a compromise in this paper’s stylized setting. Theorems 3 and 5 show that under certain conditions, a DM should select the alternative under CBA iff
$ \theta (\unicode{x211A},\unicode{x2102})<f(\unicode{x1D53E}) $
, where
$ \theta \left(\cdot, \cdot \right) $
and
$ f\left(\cdot \right) $
are simple functions: (i) If there is no dominant strategy,
$ g\in \unicode{x1D53E} $
,
$ \Delta Q\in \mathrm{\mathbb{Q}} $
, and
$ \Delta C\in \mathrm{\mathbb{C}} $
, and the Bayesian DM has a subjective distribution
$ \pi $
on
$ (\unicode{x1D53E},\unicode{x211A},\unicode{x2102}) $
such that the marginal distributions on
$ \unicode{x1D53E} $
and
$ \mathrm{\mathbb{Q}} $
are independent, then the Bayesian DM using CBA selects the alternative iff
$ {E}_{\pi}\left[\Delta C\right]/{E}_{\pi}\left[\Delta Q\right]<{E}_{\pi}\left[g\right] $
. (ii) If there is no dominant strategy,
$ \unicode{x1D53E}=\{g\} $
,
$ \Delta Q\in \mathrm{\mathbb{Q}} $
,
$ \Delta C\in \mathrm{\mathbb{C}} $
, and
$ \left\{\left(\overline{Q},\underline{C}\right),\left(\underline{Q},\overline{C}\right)\right\}\in \left(\mathrm{\mathbb{Q}},\mathrm{\mathbb{C}}\right) $
then under CBA an MMR planner selects the alternative iff
$ \left(\overline{C}+\underline{C}\right)/\left(\overline{Q}+\underline{Q}\right)<g $
.Footnote
19 The left-hand side ratio of either can be computed without specifying
$ g $
. DMs wishing to avoid explicit choice of
$ g $
may use these ambiguity-adapted ICERs derived from reasonable decision rules; however, this requires the explicit selection of a decision rule and for (i), of
$ \pi $
. This may be a more challenging task.
In any case, implicit choice of
$ g $
cannot be fully avoided when making a decision (Phelps & Mushlin, Reference Phelps and Mushlin1991; Paulden, Reference Paulden2020). Similarly, decision-making based upon the inspection of various ICER estimates and other information obscures the formulation of an implicit decision rule. I thus urge embracing ambiguity: set a range of possible values for
$ g $
, ICs, and IBs, and make decision rules explicit.
4.3. Research implications
This paper focused on (i) one alternative compared to an SQ, (ii) a single category of benefits, (iii) where the final estimates of IBs and ICs are strictly positive, (iv) where the monetary valuation of benefits equals the cost-effectiveness threshold, and (v) a representative DM. These restrictions permitted crisp intuition and predictions concerning the effects of ambiguity on decision-making when the results from economic evaluations are framed in CBA (or NMB) versus CEA (or ICER) format. Also, the objective functions used in my formalized decision problems generalize to non-healthcare contexts. I leave to future work the exploration of ambiguity on decision-making with fewer restrictions, when the source of ambiguity can be modeled more precisely in tailored applications, or using alternative decision rules.
My results also suggest paths for empirical research on the decision-making process in regulator appraisal of technologies. Theorems 3 and 6 imply that CBA in place of CEA framing results in more approvals in reasonable settings, all else equal. These are testable hypotheses, though it may be difficult to measure the specific decision rules used by DMs to overcome ambiguity in practice. Theorem 7 predicts that DMs should treat ambiguity in ICs versus IBs differently, and specifically that ambiguity in IBs is most relevant for distortions of CEA decision-making relative to CBA. Separately accounting for ambiguities in IBs versus ICs in revealed and stated preference studies would thus be a productive research agenda.
5. Conclusion
Ambiguity is commonplace in economic evaluation and should be understood by analysts, DMs interpreting their results, and researchers of this process. To promote this understanding, this paper provided a conceptual foundation for the analysis of regulator decision-making under ambiguity in the information obtained from evaluations. How this information is presented are frames that shift the social objective functions used by a representative DM, and decision rules are heuristics that simplify the problem of determining policy.
My results suggest that presenting information to regulators in an ICER or CEA format instead of an NMB or CBA framework may induce distortions in decision-making when there exists ambiguity in incremental benefits, under the assumption that a DM uses certain well-known decision rules. By contrast, ambiguity in incremental costs or the value of the cost-effectiveness threshold does not yield distortions in recommendations on whether to implement an alternative policy such as reimbursing a novel healthcare technology. In specific contexts, I showed that framing information as ICERs can result in the approval of fewer technologies relative to when information is evaluated in an NMB format. This may be an underappreciated benefit of CEA when regulators are concerned about pro-technology bias.
All results extend to non-healthcare contexts and provide predictions on how presenting information from economic evaluations to regulators may frame and distort recommendations under ambiguity. One takeaway is that presenting incremental benefits and costs separately only, and not as ICERs, may mitigate potential framing-induced distortions. Empirical examination and theoretical extensions will be required to fully appreciate the impact of ambiguity on regulator appraisal of healthcare technologies. My work particularly counsels that appropriately accounting for uncertainties in incremental benefits versus incremental costs in revealed and stated preference studies would be a productive research agenda.
Acknowledgments
I thank Charles Manski, Rick Szostak, session participants at the Annual Conference of the Canadian Economics Association in Toronto (2024), as well as seminar participants at Kellogg School of Management and Alberta School of Business. All errors are my own. I was supported in part by funding from the Social Sciences and Humanities Research Council of Canada (Doctoral Fellowship No. 752-2016-0273), by a Northwestern University Fellowship, and by a departmental research grant and a Research Start-Up Grant from the University of Alberta. This paper supplants an earlier version titled “Ambiguity in Cost-Effectiveness Analysis.”
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/bca.2024.39.
Data availability statement
Code available upon request.
Acknowledgments
I thank Charles Manski, Rick Szostak, session participants at the Annual Conference of the Canadian Economics Association in Toronto (2024), as well as seminar participants at Kellogg School of Management and Alberta School of Business. All errors are my own. I was supported in part by funding from the Social Sciences and Humanities Research Council of Canada (Doctoral Fellowship No. 752-2016-0273), by a Northwestern University Fellowship, and by a departmental research grant and a Research Start-Up Grant from the University of Alberta. This paper supplants an earlier version titled “Ambiguity in Cost-Effectiveness Analysis.”
Competing interest
I declare no competing interests. I was supported in part by funding from the Social Sciences and Humanities Research Council of Canada (Doctoral Fellowship No. 752-2016-0273), by a Northwestern University Fellowship, and by a departmental research grant and a Research Start-Up Grant from the University of Alberta.
Open access
