Regulatory impact analyses (RIAs) weigh the benefits of regulatory proposals against the burdens they impose. They are invaluable tools for informing decision makers about the effects of regulatory choices; even regulatory decisions that are ultimately made on political, legal, ethical, or other grounds will benefit from the structured evaluation of trade-offs and alternatives that a good RIA provides.
Although RIAs are a core feature of regulatory practice in the United States and other countries (Kirkpatrick & Parker, Reference Kirkpatrick and Parker2007; OECD, 2016) they can be difficult to interpret. Key methodologies, assumptions, limitations, and uncertainties can be obscured by dense, technical writing, and RIAs sometimes appear to be “used to justify decisions already made, rather than to inform those decisions” (Carrigan & Shapiro, Reference Carrigan and Shapiro2016; see also Harrington, Heinzerling & Morgenstern, Reference Harrington, Heinzerling and Morgenstern2009). RIAs often serve as legal documents, running to hundreds or even thousands of pages, prepared by agencies in anticipation of litigation.
U.S. regulatory agencies develop RIAs before issuing significant new regulations, and nongovernmental interests may also present their own analyses of how different policies will affect outcomes. Dense or complex RIAs can be challenging for policy officials and interested parties to comprehend and interpret, making it difficult to distinguish facts from conjecture and to understand the likely consequences of alternative policy choices.
While numerous technical guidelines exist to aid development of RIAs (OMB, 2003, 2010; OECD, 2008), none are geared toward nonspecialist policymakers and interested stakeholders who will be reading RIAs as consumers. This guide attempts to fill that gap. It first reviews the purpose of an RIA, and then offers policy makers and other consumers of RIAs 10 tips for asking informed questions when reviewing and interpreting them.
The guide is designed to (1) explain why RIAs are valuable, review their key elements, and describe best practices; (2) point out the ways in which RIAs might fall short of achieving these best practices; (3) help readers to better judge the quality of information provided in RIAs, and make discerning assessments about the methods employed; and (4) improve readers’ capacity to critically evaluate the justification offered to support regulatory actions.
What are RIAs and when are they used?
Before issuing new regulations, governments across the world conduct RIAs to understand possible consequences of alternative decisions (OECD, 2015). Executive branch agencies in the United States have for more than 35 years been required to conduct RIAs before issuing economically significant regulations (those with impacts of $100 million or more in a year), and to rely on those analyses in designing regulations. Other developed nations also rely on impact assessments to support regulatory actions, although they are often less quantitative than those conducted in the United States.
In the simplest terms, the goal of an RIA is to present information to decision makers to help them to ensure that proposed regulations do more good than harm. In the United States, presidential Executive Orders 12866 (Clinton, Reference Clinton1993) and 13563 (Obama, Reference Obama2011) set forth principles of good regulatory decision-making; the Office of Management and Budget’s (OMB’s) Circular A-4 (2003) provides detailed guidance for developing RIAs, and its RIA Checklist (2010), FAQs (2011a ) and Primer (2011b ) highlight key elements and principles. Agencies have also developed detailed guidance specific to their regulatory mandates (for example, EPA, 2000).
In general, the level of analysis should be pertinent to the decision in question. The OMB reviews hundreds of draft regulations each year, most of which are accompanied by some form of analysis, and, on average, 80 or more per year are deemed to be economically significant and thus require a quantitative RIA (RegInfo.gov).
Per OMB’s RIA Primer (2011b ), the three basic elements that each RIA should include are (1) “a statement of the need for the regulatory action,” (2) “a clear identification of a range of regulatory approaches,” and (3) “an estimate of the benefits and costs – both quantitative and qualitative – of the proposed regulatory action and its alternatives.” Thus, RIAs often involve considerations and analysis that go beyond benefit-cost analysis (BCA, also called cost-benefit analysis or CBA). Nevertheless, BCA, which examines economic welfare differences among alternative policies, is an important component of the RIA framework (Arrow et al., Reference Arrow, Cropper, Eads, Hahn, Lave, Noll, Portney, Russell, Schmalensee, Kerry Smith and Stavins1996). For rules aimed at protecting public health and safety, risk assessment is another key component, attempting to evaluate the risks posed under certain conditions, and the potential changes in risk achievable due to different policy options (OIRA & OSTP, 2007). Cost-effectiveness analyses may also be relevant for some policy questions (OMB, 2003).
President Trump has recently required agencies to offset the costs of new regulations by removing or modifying existing regulations (E.O. 13771). The OMB’s guidance on this order states that it does not negate longstanding requirements to consider benefits or to conduct RIAs (OMB, 2017). It may lead to an increased use of RIAs to examine the effects of modifying existing regulations, as well as prospective regulations.
In the United States, administrative regulatory agencies develop RIAs both to inform decision makers within the executive branch and to inform the public and Congress. There may be tension between these goals, especially in instances where decision makers have a politically driven reason to prefer a regulatory option with lower net benefits than an alternative, or when authorizing legislation constrains policy options (Lutter, Reference Lutter1999; Graham, Reference Graham2008).
The 10 tips presented below are the result of a consensus effort of a diverse group of experts in RIA, organized by the George Washington University Regulatory Studies Center.Footnote 1 We recognize that not all of these tips will apply to every RIA, and RIAs need not include all of the elements discussed here to provide valuable information for evaluating policy options. We hope that these 10 tips can help policy makers and other interested readers to appreciate the value of RIAs, ask appropriate questions of an RIA, understand what the analysis really means, and judge its implications for regulatory policy.
1 Core problem: Determine whether the RIA identifies the core problem (compelling public need) the regulation is intended to address
Regulatory impact analyses should clearly articulate the “need for government action” (OMB, 2011b ), the core problem that requires federal regulatory intervention. Generally, this should be a description of the “material failures of private markets” (E.O. 12866, Section 1), although, as discussed below, regulations may be justified by other goals.
1.1 Appreciate the role of markets when assessing regulatory policies
The concept of “market failure” is an important one in regulation. In a market economy disciplined by competition, the exchange of goods and services between willing buyers and sellers uses price signals to allocate scarce resources to their most valued uses, to encourage innovation, and to meet consumer needs. Regulation and other forms of government intervention can disrupt those signals, making the market less efficient and harming social welfare. Thus, the first question when reviewing an RIA will often be whether it explains why market forces will not achieve an outcome at least as efficient as what government reasonably would be expected to accomplish through regulation.
1.2 Recognize when market forces may be inadequate
Markets may not efficiently allocate resources for several reasons. First, efficient markets need an adequate infrastructure, including the rule of law, well-defined property rights, and a system of exchange. Open-access fisheries, for example, may be vulnerable to collapse without either an effective system of property rights or a suitable regulatory alternative. Second, existing policies that were poorly designed may impede the functioning of markets. For example, economic regulation of private sector prices, entry, and exit tends to distort market signals and has historically kept prices of some goods and services higher than necessary, harming consumers rather than protecting them (OMB, 2003, p. 6). These problems might fit into the category of “failures of … public institutions,” described in E.O. 12866 (Section 1.b.1).
Third, markets may not perform efficiently due to classic “market failures” or deficiencies inherent in the market itself. OMB Circular A-4 breaks these into three categories: (1) externalities, public goods, and common property resources; (2) market power; and (3) inadequate or asymmetric information. (To learn more about market failures, see OMB, 2003, pp. 4–5, and Viscusi, Vernon & Harrington, Reference Viscusi, Vernon and Harrington2005.)
In some cases, a regulation may be initiated not in response to a failure of private markets but to improve the efficiency of government programs, or to implement a legislative mandate of one kind or another where the underlying objective is something other than improvement of efficiency, such as to redistribute incomes (OMB, 2003, p. 5), protect civil rights, or provide for “universal” access to services deemed to be important. In these cases, the RIA can identify which alternative would achieve the regulatory goal in the most cost-effective way.
1.3 Question anecdotal or unrealistic justifications
In evaluating the RIA’s expressed need for regulatory action, anecdotal observations that may illustrate symptoms of a problem without articulating the underlying cause of those symptoms may deserve scrutiny. Regulatory actions that do not explicitly point to a failure of private markets or public institutions underlying the need for action are likely to produce lower net benefits than those that correctly identify and seek to remedy the fundamental problem.
Regulations that derive most of their benefits from providing private monetary gains that individuals can achieve without further government intervention, such as fuel savings from driving energy-efficient cars, require a particularly demanding burden of proof (Gayer & Viscusi, Reference Gayer and Viscusi2013). Does the RIA provide evidence that individuals behave irrationally (and do not learn) in the specific situation covered by the proposed regulation? Some researchers have found that carefully formed “nudges” can help individuals to overcome heuristics and biases to make choices that improve their well-being (Thaler & Sunstein, Reference Thaler and Sunstein2008). What insights do regulators have that make them better able to judge other people’s preferences, or be more faithful agents of their interests, than the people themselves (Mannix & Dudley, Reference Mannix and Dudley2015)?
2 Alternatives: Look for an objective, policy-neutral evaluation of the relative merits of reasonable alternatives
Regulatory impact analyses should examine human welfare differences among alternative policies (OMB, 2003). Thus, it is important to look at whether the RIA considers plausible alternatives or whether it only presents the preferred regulatory approach (perhaps with some straw-man alternatives). Do alternatives vary in their stringency? Are different regulatory instruments considered? Is evidence presented that allows for evaluation of alternatives and their relative effect on human welfare? Or, instead, does the RIA seem to focus on justifying a particular regulatory action?
Are the alternatives likely to target the identified failure of private markets or public institutions? For example, if the problem is localized, are nonfederal solutions considered? Or, if the identified market failure is asymmetric information, do the alternatives serve to address those asymmetries and improve the information available to those apparently lacking it? If consumer decision-making errors support the need for regulation, do alternatives attempt to address the identified bias while respecting the importance to consumers’ individual welfare of being able to make decisions reflecting their own preferences and resources (OMB, 2011c )? There should be a well-defined documentation of the economic basis for attributing benefits to regulatory options that diminish individual choice without considering alternatives that provide information or establish different defaults to improve information processing (OMB, 2011b ). The set of people granted standing – i.e., those whose benefits and costs count in the RIA – should generally be U.S. citizens and residents, although RIAs may analyze and report separately effects on others (OMB, 2003).
The RIA should “consider the alternative of not regulating” (E.O. 12866, Section 1.a). A large body of regulations already addresses market failures. Because regulatory proposals are, by nature, incremental extensions of the existing regulatory framework, they must provide benefits that exceed costs for small changes at the margin. Theory and empirical evidence show that incremental (marginal) costs tend to rise and benefits to fall with intensifying regulation. Given this reality, the presumption should be against additional regulation without a plausible case that existing regulations are not adequately addressing the identified market failure.
3 Baseline: Check whether the RIA presents a reasonable “counterfactual” against which benefits and costs are measured
One key component of regulatory impact analysis is identification of the assumed state of the world in the absence of the regulation (the “counterfactual,” or “baseline”). The difference between this baseline and the state of the world with the regulation is the incremental change that the regulation makes, and measurement of the benefits and costs of this incremental change is what an RIA must do. Thus, the specification of a realistic baseline is a critical part of determining the incremental benefits and costs of a regulatory proposal.
In reviewing an RIA, one should evaluate whether the baseline is a reasonable reflection of the way the world would look in the absence of the proposed action. Does it take into account the effect of other regulations, the evolution of the market in the absence of regulation, and other external factors (OMB, 2003, p. 15)? Because the future state of the world, with or without the regulation, is uncertain, an RIA might use sensitivity analysis with alternative baselines to determine the degree to which baseline assumptions affect the overall conclusions of the analysis. (See also tip 5.) In all cases, one must be sure that the baseline from which costs and benefits are measured is consistently applied; if not, the resulting estimates cannot be reliably compared.
4 Increments: Evaluate whether totals and averages obscure relevant distinctions and trade-offs
For a rule with multiple components (for example, one that both sets permissible exposure levels and requires technology controls), an RIA that estimates the benefits and costs of the rule as a whole, without presenting the marginal impacts of the key elements, will not reveal the merits of individual requirements. For example, the RIA may suggest that a proposed regulation would yield net benefits, but most of those benefits may derive from one requirement, while most of the costs derive from another. This would be revealed if the estimated benefits and costs of each component were estimated separately. In such a case, this incremental analysis would show that one component had much greater net benefits, and might point to a different preferred policy.
Marginal (or incremental) analysis is also important if different degrees of stringency are considered. Knowledge of the estimated incremental benefits and costs of successively more stringent alternatives (in addition to the total benefits and costs of each option) can be informative. For example, when considering alternative emissions limitations, the RIA should measure the benefits and costs of each alternative from the next most stringent alternative, as well as from the baseline.
The “law of diminishing returns” is the common generalization that marginal benefits tend to decrease, and marginal costs tend to increase, as alternatives get more stringent. The most cost-effective actions are generally taken first, so the unit compliance costs presented in the RIA should increase as incremental regulation becomes harder and harder to achieve. Analysis that predicts that the incremental benefits of reducing emissions by an additional unit will be greater than those of the previous unit, for example, might indicate that there are economies of scale to warrant reducing emissions further (or at least to examine further reductions), or that there is a problem with the underlying assumptions.
5 Uncertainty: Recognize that all estimates involve uncertainty, and ask what effect key assumptions, data, and models have on estimates
All estimates involve uncertainty, so one should be skeptical of overly precise estimates of costs or benefits. The OMB directs agencies to “provide expected-value estimates as well as distributions about the estimates, where such information exists” (OMB, 2003, p. 45). Expected value is the most important summary statistic when the government seeks to act in a risk-neutral fashion (as opposed to a risk-averse or risk-seeking manner) in the face of uncertainty. Thus, one should look to see whether the RIA presents unbiased “expected values,” as well as ranges for costs and benefits.
Further, one should realize that assumptions that claim to be “‘public health protective,’ which err on the side of overstating risk when data are lacking,” (Gray & Cohen, Reference Gray and Cohen2012) can inflate estimates of certain risks, benefits, or costs relative to others, and lead to misaligned priorities because the degree of precaution differs across risks (Nichols & Zeckhauser, Reference Nichols and Zeckhauser1988; Hamilton & Viscusi, Reference Hamilton and Viscusi1999). Moreover, distortions in different parts of the analysis can interact and multiply, rendering the conclusions unreliable (Hamilton & Viscusi, Reference Hamilton and Viscusi1999; Fraas & Lutter, Reference Fraas and Lutter2012).
Analyses that do not provide information on how sensitive the primary estimate is to assumptions, data, and models, and the range of outcomes possible under reasonable alternative analytic assumptions should raise questions. Sensitivity analysis examines different “what if” scenarios to see how changes in key assumptions (or combinations of assumptions) influence estimated outcomes. Because many uncertain factors determine the impact of any regulation, one should look for a convincing justification regarding which uncertain parameters have the most consequential effects on outcomes, and a sensitivity analysis that varies these factors over a reasonable range to gauge their effects on the rule’s net benefits.
The OMB requires a quantitative uncertainty analysis for regulations with likely impacts greater than $1 billion (OMB, 2003, pp. 38–42). One valuable way in which agencies inform readers about uncertainty is through Monte Carlo simulations that use probability distributions for important analytic inputs and repeated random sampling to generate a probability distribution for decision-relevant outputs (such as net benefits or lives saved) (Krutilla, Good & Graham, Reference Krutilla, Good and Graham2015). Formal elicitation of experts’ judgments can be used to develop probability distributions for analytic inputs when adequate data are unavailable (OMB, 2003, p. 41). The computation of a break-even level of effectiveness at which the rule’s benefits would justify the costs might also add useful perspective.
The problems that regulations target often have multiple interacting causes, making it difficult to cleanly attribute the effect of the regulation in isolation. For example, commercial vehicle crashes are usually caused by a combination of weather, technology, human behavior, and infrastructure conditions. An hours-of-service regulation designed to reduce crash risks by mitigating driver fatigue will reduce one risk factor in this larger cluster. In such a case, where the fatigue–accident relationship is uncertain, does the RIA use sensitivity analysis to explore alternative assumptions about the relationship of hours-of-service rules to fatigue, and the impact of fatigue on crash risks?
Uncertainty analysis can also highlight what additional information would be most valuable for making a decision. If expected outcomes hinge on the value assumed for a particular uncertain variable, it might be appropriate to gather more information regarding that variable prior to making a decision (OMB, 2003, p. 39), or to ask what policies would generate the information necessary to reduce that uncertainty (Greenstone, Reference Greenstone, Moss and Cisternino2009).
6 Transparency: Look for transparency and objectivity of analytical inputs
OMB guidance requires that significant information disseminated to the public be “capable of being substantially reproduced, subject to an acceptable degree of imprecision” (OMB, 2002, p. 8460). This standard can be difficult for a nonexpert reviewer to assess. A precondition of reproducibility, however, is transparent presentation of the underlying data, assumptions, and models relied on to draw conclusions. Disclosure of underlying data and computer code has become standard among the more prestigious scientific and technical journals, which allow for data sharing agreements when individually identifiable information prevents public disclosure. These disclosure policies appear to improve the reproducibility of the results of published papers (Lutter & Zorn, Reference Lutter and Zorn2016). Without transparent presentation of inputs used and alternative inputs considered, reviewers will find it difficult to judge the objectivity or accuracy of the resulting estimates.
Often, less is understood about the effects of regulatory intervention (such as reductions in health risks, for example) than the valuation of those effects, so one should look particularly for a clear presentation of alternative plausible models and assumptions used to predict regulatory outcomes. Precise-sounding predictions not only can hide considerable uncertainty about risks, benefits, and costs, as noted in tip 5, but also can mask a reliance on biased inferences and assumptions.Footnote 2 While some judgments are necessary to translate scientific evidence into risk assessment, if they are not policy-neutral, they can lead to distorted risk estimates and false precision in the presentation of scientific information (Dudley & Gray, Reference Dudley, Gray and Johnston2012). Furthermore, if they are not transparent, reviewers will be unable to judge their objectivity.
If an RIA for a significant regulation aimed at reducing health, safety, or environmental risks does not provide a discussion of alternative interpretations of the scientific basis, it may not be providing a full and accurate picture of potential risks (Sutherland, Spiegelhalter & Burgman, Reference Sutherland, Spiegelhalter and Burgman2013). Moreover, if it does not provide the information necessary to assess how different inputs (assumptions, data, models, etc.) would change estimated outcomes, it will not provide the information needed to evaluate the consequences of the regulatory action.
7 Benefits: Examine how projected benefits relate to stated objectives
Does the RIA clearly explain how regulatory objectives will be achieved, or is the preferred regulatory outcome simply presumed to work as intended? The analysis should lay out causal linkages between regulatory requirements and desired outcomes, discuss the evidence supporting these linkages, and show how they differ across alternatives (Newcomer, Hatry & Wholey, Reference Newcomer, Hatry and Wholey2015). If regulatory outcomes depend on factors beyond the regulation’s control (e.g., energy prices), one should look for a presentation of how variations in those factors would affect outcomes.
7.1 Look for evidence for and against a causal relationship
It should be remembered that correlation does not always imply causation, so one should look for a persuasive demonstration of causation rather than relying on indicators of correlation or association to predict changes in health risks, costs, or benefits (Cox & Popken, Reference Cox and Popken2008). One should be alert for words such as “linked to” or “associated with.” Does the RIA present information indicating whether a change in the variable to be regulated (e.g., emissions) precedes and causes a change in the targeted outcome (e.g., public health) as well as the effect other possible factors might have on the outcome (Cox, Reference Cox2016)? Do the underlying studies linking harm (e.g., mortality or morbidity) to exposure to some hazard use statistical strategies appropriate to the identification of causal relationships, such as regression discontinuity designs, or quasi-experiments (Dominici, Greenstone & Sunstein, Reference Dominici, Greenstone and Sunstein2014)?
7.2 Ask whether the analysis accurately characterizes indirect benefits and costs
Some RIAs present large “co-benefits” (or “ancillary benefits”). These may need closer inspection, particularly if the co-benefits are much larger than the direct benefits, if the direct benefits on their own are significantly less than the estimated costs, or if the co-benefits appear to materialize “for free.” The presence of co-benefits almost always signals that the agency is counting costs and benefits that arise outside of the specific statutory authority that the regulation operationalizes. That is not a problem per se; indeed, all significant benefits and costs should be counted. However, in such cases, one might ask why this regulation is the best way to achieve those co-benefits. Generally, one would expect that regulation targeted directly at a particular outcome can achieve it more cost-effectively than one that achieves it circuitously as a side effect (co-benefit) of an unrelated regulation, and a sound analysis must make a thorough inventory of both the harmful and the beneficial consequences of each alternative.
8 Costs: Understand what “costs” are included
Ideally, the RIA will estimate the “opportunity cost” of the regulatory action (the lost value of the best alternative forgone). However, opportunity costs can be difficult to measure, so RIAs often rely instead on the costs of compliance, measured by the expenditures businesses make on technology or methods used to fulfill the regulatory requirements. While business compliance costs are not welfare changes, they are often a reasonable proxy for welfare changes that are passed through to consumers, to employees, and to business owners (which, for publicly traded companies, often include large mutual funds and pension funds owned by many individuals) (Mannix, Reference Mannix, Coglianese, Finkel and Carrigan2014). If compliance costs are used, they should exclude taxes, which are mostly transfers to other members of society.
The costs of some types of regulatory actions, such as prohibitions on certain actions or products, cannot be approximated by compliance costs, however. For example, a rule prohibiting a specific product may not involve compliance costs, but it would have an opportunity cost because users would not able to enjoy the product’s benefits.
9 Distribution: Consider how benefits and costs are distributed
Those who bear the costs of a regulation and those who enjoy its benefits often are not the same people. Does the RIA present evidence on the incidence of benefits and costs so that one can understand how they affect different people?
While some government programs are designed to redistribute wealth (e.g., food stamps), others do so inadvertently (e.g., regulations that raise food prices might have disproportionate impacts on low-income Americans, or regulatory compliance might burden small businesses more than large ones). The OMB calls on agencies to “distinguish … between real costs and transfer payments, [which are] monetary payments from one group to another that do not affect total resources available to society” (OMB, 2003, p. 38).
Does the RIA reveal whether a regulation will have different impacts on different subpopulations, including those living in different regions of the country, businesses of different sizes, individuals of different ages, and people with different incomes or ethnic and socioeconomic characteristics? It might be interesting to know whether the benefits are “global,” accruing to foreign countries, but the costs are borne domestically. If so, is that the program’s purpose, and what is the net effect on the United States (Fraas et al., Reference Fraas, Lutter, Dudley, Gayer, Graham, Shogren and Viscusi2016)?
The OMB advises agencies, “Where distributive effects are thought to be important, the effects of various regulatory alternatives should be described quantitatively to the extent possible, including the magnitude, likelihood, and severity of impacts on particular groups” (OMB, 2003, p. 14). If an RIA ignores distributional effects – implying that they are not “thought to be important” by the promulgating agency – one should look for a compelling explanation, based on logic and evidence, that costs and benefits generally fall on the same groups of people.
10 Symmetrical treatment: Ensure that benefits and costs are presented symmetrically
One should pay attention to how benefits and costs are measured. For example, both should be measured from the same baseline and over the same time frame. Similarly, the discount rate used to convert the future streams of benefits and costs to present values should generally be the same (OMB, 2003). The choice of discount rate can have a very large impact on the present value of estimates, so an RIA should clearly defend the use of different discount rates applied for costs and benefits (and present the effects of alternative choices in sensitivity analysis).
Are the boundaries of the analysis framed symmetrically? No analysis will ever be complete, of course. However, major elements should not be missing on one side of the equation, or overemphasized on another. For example, if the analysis presents evidence of co-benefits, are ancillary costs or countervailing risks examined to the same extent?
Final thoughts
Regulatory impact analysis can be an invaluable method for transparently evaluating contentious policy choices before they are put in effect. The OECD states that its “most important contribution to the quality of decisions is not the precision of the calculations used, but the action of analyzing – questioning, understanding real-world impacts and exploring assumptions” (OECD, 2002, p. 47). Thus, an RIA’s purpose is not to compel decisions, but rather to provide policy makers responsible for making decisions with the information needed to think through the possible consequences of different regulatory actions. Despite this informational purpose, RIAs can be opaque, complex, and even intimidating. Wittingly or unwittingly, they may be written in a way that obfuscates important information or skews the analysis to support a particular outcome.
Savvy policymakers and other consumers need to be aware of the basic elements involved in a good RIA and the kinds of questions to ask to get the most from reading one. The preceding 10 tips should help readers of RIAs to interpret what they read and ask appropriate questions.
Open access
