Hostname: page-component-586b7cd67f-tf8b9 Total loading time: 0 Render date: 2024-12-04T21:04:38.693Z Has data issue: false hasContentIssue false

Retrospective Analyses Are Hard: A Cautionary Tale from EPA’s Air Toxics Regulations

Published online by Cambridge University Press:  17 August 2017

Art Fraas*
Affiliation:
Resources for the Future, 1616 P St, NW, Washington, DC 20036-1400, USA, e-mail: fraas@rff.org
Alex Egorenkov
Affiliation:
Resources for the Future, 1616 P St, NW, Washington, DC 20036-1400, USA
*
*e-mail: fraas@rff.org
Rights & Permissions [Opens in a new window]

Abstract

Under the 1990 Clean Air Act Amendments, the U.S. Environmental Protection Agency (EPA) was required to establish standards limiting air toxics emissions from industrial plants. This paper examines the effects of five of the largest-cost rules issued by EPA in the initial round of air toxics rulemaking over the 1995 to 2000 period. Our estimates suggest that plants in the printing and publishing and pulp and paper industries realized important reductions in their air toxics emissions in the period between publication of the final rule and the effective date for compliance with the rule – although the reduction in air toxics emissions by pulp and paper mills fell short of EPA’s ex ante projections. However, our estimates also suggest that plants in three other industries – petroleum refining, pharmaceuticals, and wood furniture – achieved little or no additional reduction in air toxics emissions over the compliance period in response to EPA’s rules. Finally, the paper explores steps that EPA should take in setting up future retrospective analyses.

Type
Article
Copyright
© Society for Benefit-Cost Analysis 2017 

The 1990 Clean Air Act (CAA) Amendments revised the provisions addressing air toxics emissions by requiring the U.S. Environmental Protection Agency (EPA) to set technology-based standards for major sources of air toxics. Congress turned to a technology-based approach because EPA had managed to regulate only a few air toxics and their sources in 20 years of regulation under the 1970 CAA. Technology-based standards were a core piece of the 1977 Clean Water Act, and their implementation over the 1980s was widely viewed as achieving substantial reductions in the industrial discharge of toxics in water. Through the 1990 CAA Amendments to Section 112, Congress hoped to replicate that experience with a widespread initiative to reduce toxic air emissions.

Relatively little retrospective research has evaluated EPA’s industry-specific, technology-based emissions standards for toxic pollutants issued under Section 112. One of the key barriers to conducting retrospective analyses is the absence of data at a sufficiently fine-grained scale to yield meaningful results. Because of the limitations on the availability of data, the retrospective studies of environmental regulations in most cases have been accidents of convenience enabled by the existence of a relevant data set rather than part of a systematic plan.

This study of the 1990s round of air toxics rules issued by EPA pursuant to the 1990 CAA Amendments was undertaken as a part of a broader Resources for the Future (RFF) Regulatory Performance Initiative to develop rigorous retrospective analyses of federal regulations using nonconvenience samples of firm-level data.Footnote 2 The objective of these studies is to expand the universe of rigorous retrospective analyses, demonstrate a range of approaches to some key methodological issues – for example, baseline construction – that arise in conducting retrospective analyses, and advance the conduct and public discussion of the performance of both federal regulatory programs and individual regulations. In addition, by including EPA’s 1998 pulp and paper Cluster Rule, this study complements EPA’s recent retrospective cost study of the Cluster Rule.

As discussed below, this retrospective study originally sought to investigate an entire “class” of rules to mitigate concerns with selection bias. Unfortunately, it serves more as a cautionary tale on the difficulty of doing retrospective analysis because data limitations severely constrained the scope of the study. We were able to study only 5 of 13 manufacturing industries covered by the technology-based rules in this period, and even for these 5, data issues restricted the quality of the analysis. Section 1 provides some background and summarizes the literature. Section 2 outlines our methodology, and Section 3 presents the results. Section 4 discusses lessons learned and offers recommendations.

1 Background

1.1 Regulatory context

Prior to adoption of the 1990 CAA Amendments, EPA had authority to regulate individual air toxics based on their specific health risks. However, over two decades, EPA regulated only seven air toxics emitted by a few sources. To speed up EPA regulation of air toxics, Congress amended the CAA in 1990 to focus regulation on the variety of air toxics emitted by a source category rather than set standards one chemical at a time. Section 112, as amended, lists 188 air toxics subject to regulation and charges EPA with identifying major source categories and establishing a schedule for regulation.Footnote 3 In 1992, EPA identified 174 categories of sources subject to air toxics emissions standards (57 FR 31576). Section 112 also establishes specific requirements for setting technology-based emissions limits, known as maximum achievable control technology (MACT) standards, which are to be based on the average level of control achieved by the best-performing 12% of sources in the relevant industry – the so-called MACT floor.Footnote 4 These standards are intended to raise all the plants in the industry to the level of control achieved by the best performers rather than force the adoption of exotic and unproven technologies.Footnote 5

1.2 Literature review

There is a limited retrospective literature examining ex ante and ex post estimates of the benefits and costs of major environmental rules.Footnote 6 In discussing RFF’s recent Regulatory Performance Initiative, Morgenstern (Reference Morgenstern2015) identifies a set of barriers to conducting such studies, among them the availability of data (particularly microlevel data needed to evaluate individual rules) and the difficulty of constructing a separate, credible counterfactual baseline. Similarly, Kopits et al. (Reference Kopits, McGartland, Morgan, Pasurka, Shadbegian, Simon, Simpson and Wolverton2014), in a recent EPA retrospective study of the costs of five rules – including EPA’s 1998 pulp and paper Cluster Rule – report that the paucity of available cost data and the difficulty of defining a counterfactual baseline were significant barriers to carrying out retrospective analysis.Footnote 7 Relatively little retrospective research has evaluated EPA’s industry-specific, technology-based emissions standards for toxic pollutants issued under Section 112. One exception is that EPA’s recently completed retrospective cost study includes a study of the cost of its 1998 pulp and paper Cluster Rule (EPA, 2014b ; Morgan Pasurka & Shadbegian, Reference Morgan, Pasurka and Shadbegian2014). The EPA study reports:

Our findings suggest EPA’s ex ante cost estimates overstated the costs of both the Cluster Rule and the MACT II rule. Using publicly available data from [the National Council for Air and Stream Improvement], we found that EPA overestimated the capital cost of the Cluster Rule by 30%–100%, depending on the choice of baseline year from which we derived the incremental cost. Among the reasons for EPA’s overestimates of these capital costs are the mills’ use of the clean condensate alternative (CCA), flexible compliance options, extended compliance schedules, site-specific rules, use of equivalent-by-permit, and equipment/mill shutdowns and consolidations. Morgan et al. (Reference Morgan, Pasurka and Shadbegian2014, 219).

As part of the Regulatory Performance Initiative, Gray and Shadbegian (Reference Gray and Shadbegian2015) examined the effect of EPA’s 1998 Cluster Rule on the toxic releases from pulp and paper mills. They report that chloroform releases fell dramatically throughout the sample period, with much of the reduction happening in the 1990s, before the effective date of the Cluster Rule. They find that pulp and paper mills achieved cumulative reductions (including prerule reductions) in chloroform releases of 99%. They also report some reductions in other air toxics, although the reductions are smaller than EPA’s ex ante projections. Reductions in volatile organic compounds (VOCs) are smaller in their fixed effects models for VOC than EPA projected but similar in magnitude to EPA’s ex ante estimate in an alternative model.Footnote 8 Finally, they report no significant reduction in emissions of fine particulate matter (PM).

In addition, several studies have considered other EPA programs aimed at reducing releases of toxic pollutants into the environment, including the Toxics Release Inventory (TRI) and EPA’s voluntary programs, such as 33/50 and the Common Sense Initiative.

1.2.1 EPA’s Toxic Release Inventory

Beginning in 1988, the TRI program required plants in major manufacturing industries to report annual data on toxic releases. Reported TRI releases (encompassing air emissions, water discharges, and land disposal) decreased by 37% in the first years of the program, from 1988 to 1993, but decreases in reported releases slowed to only a 10% reduction between 1993 and 1998 (Hamilton, Reference Hamilton2005).

Overall, studies have yielded mixed results on TRI’s effectiveness in reducing toxic releases, although some studies suggest that the TRI program achieved important reductions in toxic releases in its first years.Footnote 9 Hamilton (Reference Hamilton2005, 250) concludes that “one cannot say what fraction of reported reductions in TRI arose from the provision of information rather than from other factors, such as command-and-control regulation or market-related fluctuations in production.”

1.2.2 EPA’s voluntary programs

EPA established the voluntary 33/50 Program in the early 1990s to promote reductions in the releases of 17 target chemicals, with the goal of reducing the total amount of the target chemicals released into the environment (or transferred off-site) by 33% by the end of 1992 and 50% by the end of 1995.Footnote 10 Some studies of the 33/50 Program report that participating firms achieved the expected emissions reductions over the period during which the program was in effect, but other studies have suggested a more limited outcome – or even that the program had little overall effect on toxic emissions.Footnote 11 Coglianese and Allen (Reference Coglianese and Allen2003) and Coglianese and Nash (Reference Coglianese and Nash2014) examine other voluntary EPA programs – the Common Sense Initiative (1994–98), National Environmental Performance Track (2000–2009), and Project XL (1995–2003) – that followed 33/50 and conclude that they did not yield significant environmental improvements.Footnote 12

1.2.3 State programs

To manage toxic chemicals, the states have also pursued both voluntary approaches and more stringent toxics regulation. California and New Jersey, for example, were early movers in setting up programs to reduce air toxics emissions in the 1980s. However, state programs differ substantially in terms of their stringency.Footnote 13

Unfortunately, there is also a limited literature on the effectiveness of state programs on toxic releases. Bui and Kapon (2012, 43) find “strong evidence that both Federal and state pollution prevention (P2) programs have led to significant reductions in average facility-level toxic releases.” However, these programs were already in place before the period covered by this study.Footnote 14 Bui (Reference Bui2005) finds that refineries in states with more stringent toxics regulation in the form of pollution prevention programs had significantly lower levels of emissions than refineries in states with weaker or no regulation. Bui also reports, though, that reductions in toxic emissions intensity were closely related to traditional command-and-control regulation of nontoxic conventional pollutants.Footnote 15 Finally, Shadbegian and Gray (Reference Shadbegian and Gray2006) also have found that over their sample period (1996–2005), states with stronger political support for stringent regulation had lower toxic emissions.Footnote 16

1.3 Available emissions data

Beginning in 1987, the TRI program has required industrial facilities to report releases of toxic pollutants into the air and water and the disposal of toxics as waste in land-base facilities.Footnote 17 We focused on TRI release data for 1993 to 2003. The data set contains annual reporting of air toxics emissions at the plant level. We identified individual facilities under a MACT standard using the Air Facility System (AFS) data retrieved from EPA through the Envirofacts Data Service API (EPA, 2014a ). The AFS lists basic summary information about each facility, including regulations that the plant is expected to meet. We then selected regulated facilities from the TRI using the Facility Registry System (FRS). Both the TRI data files and the FRS are available for download on the EPA website.Footnote 18

The original inventory contained roughly 300 individually listed chemicals, many of which are VOCs.Footnote 19 We focused on the air emissions data for those chemicals subject to the MACT rules included in the original TRI chemical list.

TRI data are limited by reporting thresholds – plants below the thresholds are not required to report. For example, the reporting threshold was 25,000 pounds per year for chemicals used in manufacturing and processing.Footnote 20 In addition, TRI also limits reporting to facilities with 10 or more full-time employees (EPA, 1998a ). EPA also adopted a short reporting form in 1995 known as the Alternate Threshold Certification Statement, or Form A, for chemicals where the total release is less than 500 pounds and the total manufacture, process, or “otherwise use” of the chemical is less than 1 million pounds (EPA, 1998a ). In using Form A, a plant certifies that its total annual release of the chemical does not exceed 500 pounds. Firms using Form A do not provide quantitative information, such as a specific amount of the release, or apportion the release across media.Footnote 21

TRI data have been used in a number of studies (e.g., Bui, Reference Bui2005; de Marchi & Hamilton, Reference de Marchi and Hamilton2006; Kraft Stephan & Abel, Reference Kraft, Stephan and Abel2011) to evaluate the effects of the TRI initiative and other programs on toxic emissions behavior. However, even while relying on TRI data, these studies have recognized certain issues:

  • TRI release data are self-reported, and generally the data are based on engineering calculations rather than monitored emissions.

  • Changes to reporting in 1991 (triggered by the Pollution Prevention Act) resulted in a substantial increase in reported releases in subsequent years.

  • Both additions and deletions to the list of chemicals and substances occurred over the period; more than 200 chemicals were added in 1995 alone.

  • TRI reporting thresholds limit coverage to facilities with 10 or more full-time employees and to facilities with releases above specified thresholds for manufacture, process, or “otherwise use”.

  • The use of Form A beginning in 1995 reduced the availability of quantitative release reports for toxics below the Form A thresholds.Footnote 22

  • Chemical releases are reported in pounds, without accompanying information on the toxicity of the chemicals.

Several studies have also raised concerns about the underreporting of releases in the TRI database. Tietenberg and Wheeler (Reference Tietenberg and Wheeler1998, 10) point out that “firms have incentives to mislead the public, either by overstating their environmental accomplishments or by selective omission (noting the positive outcomes and ignoring or burying the negative ones).” Surveys of reporting plants in the early years of TRI indicate that a significant fraction of reported reductions mixed real and paper changes (Poje & Horwitz, Reference Poje and Horwitz1990; EPA, 1993). Dudley (Reference Dudley1999) suggests that individual facility reports may contain such large errors that the data may be unreliable for site-specific analysis. On the other hand, EPA (1998d ) reports that by the mid-1990s, more than 80% of surveyed facilities used an appropriate method to estimate releases.

Our study focuses on TRI release data for 1993 to 2003. We selected this period in part in the hope of skirting the problems identified above with the firm-specific release data from the early years of the TRI program. Nevertheless, our study is limited to coverage of a subset of plants – those plants consistently reporting their releases to TRI – in each of the covered industries.

1.4 Air toxics rules for manufacturing

We started with 21 MACT rules issued between December 1994 and December 1998. We excluded the rules for nonmanufacturing source categories established during this period – for example, gasoline distribution, marine vessel loading operations, shipbuilding and ship repair, and off-site waste recovery operations – because they are not manufacturing operations and are generally likely to have dispersed, fugitive emissions that are difficult to monitor.Footnote 23 We also excluded three source categories – flexible polyurethane foam production, basic liquid epoxy resins, and elastomer production – that apply to specific chemical manufacturing processes that would likely be part of a much larger multiproduct chemical plant.Footnote 24 After excluding these eight categories, we were left with 13 MACT rules addressing manufacturing industry categories.

Because of data limitations, we were forced to narrow the focus of our study to rules for five manufacturing industries emitting hazardous air pollutants (HAPs):Footnote 25 petroleum refining, pharmaceuticals, printing and publishing, pulp and paper, and wood furniture.Footnote 26 We found that small sample size effectively ruled out analysis for 8 of the 13 manufacturing rules.Footnote 27 In terms of cost, the 5 rules included in our analysis were projected to incur the largest annualized costs for the MACT rules issued by EPA in the study period (Table 1).Footnote 28

Table 1 EPA estimates of HAP emissions reductions and costs for five industry categories.

Sources: EPA (1995a , b , c , 1996, 1998b , c , 1999a , 2000).

EPA (2000) provided estimates of the projected HAP emissions reductions and rule costs for the five industries covered by this review (Table 1). In a separate 1999 report to Congress, EPA provided higher cost estimates for three of these rules. EPA projected VOC reductions of 252,000 tons per year (or 60%) for refineries (60 FR 43248) and of 450,000 tons per year (or 45%) for pulp and paper mills (63 FR 18575).Footnote 29

Three of the rules covered by this review – those for petroleum refineries, pharmaceuticals, and pulp and paper – were designated as major, with a Regulatory Impact Analysis containing benefits estimates (60 FR 43245). For petroleum refineries, EPA estimated benefits of $109 million per year arising from the projected reduction in VOC emissions, using benefit transfer values developed by the Office of Technology Assessment (60 FR 43245).Footnote 30 For pharmaceuticals, EPA estimated that the annual benefits from the air standards would range from $3.9 million to $67 million per year (63 FR 50410).Footnote 31 For the pulp and paper Cluster Rule, EPA estimated annual benefits ranging from a negative $1040 million to a positive $1054 million per year. The negative benefits estimate arose because EPA also projected a small increase in PM emissions and an increase in sulfur dioxide ( $\text{SO}_{2}$ ) emissions of 105,000 tons per year. With the final Cluster Rule, however, EPA also issued an additional proposed rule, designated MACT II, to limit PM and $\text{SO}_{2}$ emissions (as cobenefits) from boilers located at these mills, largely eliminating any negative benefits attached to the 1998 Cluster Rule.

2 Methodology

We used a modified event study approach because we are looking for a marked change in the targeted emissions over the period of EPA rulemaking. Based on our review of the literature, we believe this is a reasonable approach because the effects of TRI and the voluntary 33/50 Program on toxic releases from manufacturing plants had largely run their course by 1995, when EPA began issuing the MACT round of air toxics rules.

We used two OLS estimation methods – a difference-in-difference model (Model 1) and a first-difference estimate in toxic discharges (Model 2) – to explore the change in air toxics emissions. For the pharmaceuticals and pulp and paper industry categories, we aggregated emissions across all organic HAPs on the original TRI list. For the other three industry categories, we aggregated a shorter list of organic HAPs specifically identified in the rule’s preamble as the targeted focus of the MACT rule because plants release different mixes of toxics.Footnote 32

For each industry, we used a balanced panel, restricting coverage to firms reporting for each of the years (Model 1) or periods (Model 2).Footnote 33 We found greater consistency in reporting (and a larger number of reporting plants) using TRI release data for odd-numbered years (1993, 1995, 1997, 1999, 2001, and 2003).

We considered two alternative approaches to provide a counterfactual baseline of HAP emissions. First, we compared the emissions behavior of regulated plants in two of the MACT-regulated industries with the organic HAP emissions of plants in similar but unregulated industry categories. We paired printing and publishing with the unregulated paper and other web (surface-coating) industry category (67 FR 72329). These two industry categories cover very similar coating activities and emit the same kinds of organic HAPs.Footnote 34 We paired pulp and paper with another wood products manufacturing category, plywood and composite products (69 FR 45943). Plywood manufacturing is also part of the wood products industry; many of these plants are owned by firms that also operate pulp and paper mills. Plants in these two wood products industry categories emit similar organic HAPs and are generally located in the same geographic regions. EPA subsequently issued its final rules for the web surface- coating and plywood industries in a later round of MACT rulemaking (EPA, 2002a , 2004a ).Footnote 35 For petroleum refining, pharmaceuticals, and wood furniture, we were not able to identify a closely related unregulated category.

We also compared emissions behavior for the plants in the five MACT industries in our study with the emissions behavior of plants in six additional unregulated industries – referred to as the “potpourri group” – that emit organic HAPs. The six selected industries are metal can, metal coil, metal furniture, miscellaneous coating manufacturing, miscellaneous metal parts and products, and plastic parts (EPA, 2002b , 2003a , b , c , 2004b , c ). EPA subsequently established MACT standards limiting the organic HAP emissions from the surface-coating operations of these six industries in a later round of MACT rules issued from 2002 to 2004, with compliance dates of 2005 to 2007. Since both the five MACT and the six unregulated industries emit VOC HAPs, any effects of EPA and state regulation to obtain reductions in VOCs and any residual effects of the TRI program would be captured by the potpourri control group.

Appendix B presents the average plant-level emissions (by year) for each of the five industries and for the plants in the control groups.Footnote 36

2.1 Difference-in-difference model (Model 1)

For the difference-in-difference model (Model 1), we used the level of emissions as the dependent variable and the level of production (as measured by our TRI production index), the state’s League of Conservation Voters score, and the local ozone nonattainment status as control variables. The model also includes year dummies for both the control and the regulated industries:

(1) $$\begin{eqnarray}Z=\unicode[STIX]{x1D6FD}_{0}+\unicode[STIX]{x1D6FD}_{1}P+\unicode[STIX]{x1D6FD}_{2}NA+\unicode[STIX]{x1D6FD}_{3}LCV+\unicode[STIX]{x1D6FD}_{4}D+e\end{eqnarray}$$

where $Z$ is the level of emissions for each plant in the odd years from 1993 to 2003.

The independent variables are as follows:

$P=$ the TRI production index over each of the periods;

$\mathit{NA}=$ the 1-hour ozone nonattainment status of the area where the plant was located;

$\mathit{LCV}=$ the League of Conservation Voters score for the state where the plant was located; and

$D=$ a set of dummy variables for each of the years in the database, with separate dummy variables for plants in the unregulated control group versus plants in the treatment group subject to the MACT rule.

2.1.1 TRI reported production index (P)

Firms reporting to TRI also included information on the change in production activity for the reporting year relative to the previous year. We used this information to construct a production activity index over the relevant period. The production index controls for changes in production activity over several periods.Footnote 37

2.1.2 Nonattainment area (NA)

Plants located in ozone nonattainment areas (NAs) may face continuing pressure to reduce their emissions of conventional pollutants (especially VOC emissions to meet ozone standards). In a study of toxic releases from petroleum refineries, Bui (2005, 1) finds that “TRI air releases are affected by being in more stringent regulatory regions for the criteria pollutants”.Footnote 38 We used the 1992 nonattainment status for the 1-hour ozone standard for the ex ante period and the 1998 nonattainment status for the compliance and ex post periods.

2.1.3 State regulation (LCV)

Several studies have reported that state voluntary and regulatory programs achieved additional reductions in toxic releases beyond those required by EPA (see Bennear, Reference Bennear2007; Bui & Kapon, Reference Bui and Kapon2012). Gray and Shadbegian (Reference Gray and Shadbegian2007) have also found that more stringent local regulatory requirements can result in lower emissions. They use an index for states based on environmental voting behavior as reported by the League of Conservation Voters (LCV). We would expect the sign to be negative if states with higher LCV scores required additional reductions in toxic emissions over the relevant period. On the other hand, if states with high LCV scores had already required substantial reductions in emissions, plants within their jurisdictions would not have needed to make substantial additional reductions, and we would expect a positive sign for the LCV coefficient. We used the LCV scores from 1992 for the ex ante period, 1996 for the early compliance period, 1999 for the compliance period, and 2001 for the ex post period.

2.2 First-difference estimate (Model 2)

We also considered an ordinary least squares (OLS) first-difference model to identify the effects of the air toxics rules, as follows:

(2) $$\begin{eqnarray}Y=\unicode[STIX]{x1D6FD}_{0}+\unicode[STIX]{x1D6FD}_{1}DP+\unicode[STIX]{x1D6FD}_{2}NA+\unicode[STIX]{x1D6FD}3LCV+\unicode[STIX]{x1D6FD}_{4}C+\unicode[STIX]{x1D6FD}_{4}C2+\unicode[STIX]{x1D6FD}_{5}XP+e\end{eqnarray}$$

where $Y$ equals the change in emissions over each 2-year period in the analysis.

The independent variables are as follows:

$\mathit{DP}=$ the change in the TRI production index over each of the periods;

$\mathit{NA}=$ the 1-hour ozone nonattainment status of the area where the plant was located;

$\mathit{LCV}=$ the League of Conservation Voters score for the state where the plant was located;

$C=$ dummy variable for the early compliance period, with a separate dummy variable for control versus treatment plants;

$C2=$ dummy variable for the last 2 years of the compliance period, with a separate dummy variable for control versus treatment plants; and

Table 2 Ex ante, ex post, and compliance periods for covered industries.

$\mathit{XP}=$ dummy variable for the ex post period, with separate dummy variable for control versus treatment plants.

2.2.1 Production, nonattainment, and state regulation variable

We used the same information described above for the production index, nonattainment, and state regulation variables to construct corresponding independent variables for Model 2.

2.2.2 Compliance periods for the first-difference estimates

We considered emissions behavior over four periods: a 2-year ex ante period before the rule was issued, an early compliance period including the first year in which the industry moved to comply with the rule (“Early transition” in Table 2), the last 2 years of the compliance period (Compliance in Table 2), and a 2-year ex post period after the required date for compliance. Table 2 presents the periods for each industry category.

3 Model results for covered industries

3.1 Model 1: Difference-in-difference results for level of emissions

The difference-in-difference model results for printing and publishing and pulp and paper suggest a sustained reduction in emissions from peak 1995 levels (Table 3). Overall, the difference-in-difference model explains 10% to 20% of the variation in emissions (see Appendix C, Tables C1 and C2).Footnote 39 Using the potpourri group of industries as a control, the results are statistically significant for both industries for the years from promulgation of the rule to the MACT compliance date. In the cases using a paired industry as a control, the coefficients for the transition and compliance years are negative both for printing and publishing plants and for mills in the pulp and paper industry, but they are statistically significant only for pulp and paper mills and only for the transition period and the first ex post period after the effective date for compliance. For the other three industry categories, the difference-in-difference results are not statistically significant.Footnote 40

Table 3 Difference-in-difference results for MACT-regulated plants.

$^{\ast }p<0.1$ , $^{\ast \ast }p<0.05$ , $^{\ast \ast \ast }p<0.01$ .

Looking at the control variables, the coefficient for the production index is positive and generally statistically significant; however, the magnitude of the effect of a change in production on emissions is relatively modest. The sign for the nonattainment variable is positive and statistically significant for the pulp and paper industry; the sign for this variable for the printing and publishing category is negative and statistically significant. The LCV variable carries a statistically significant negative sign for the pulp and paper industry. The nonattainment and LCV variables are generally not statistically significant for the other industry categories.

3.2 Model 2: First-difference results for changes in emissions

For the printing and publishing industry category, the results suggest a statistically significant reduction in emissions over the course of the compliance period as plants moved to comply with the MACT standards (Table 4; see Appendix C, Tables C3 and C4). For the pulp and paper industry, the compliance variables are negative – as expected – but they are not statistically significant. The results for the other three industry categories over the two compliance periods yield largely positive coefficients that are not statistically significant. Overall, the first-difference model explains less than 10% of the variation in the change in emissions over these several periods.Footnote 41

Table 4 First-difference results for MACT-regulated plants.

$^{\ast }p<0.1$ , $^{\ast \ast }p<0.05$ , $^{\ast \ast \ast }p<0.01$ .

Looking at the control variables, we find that the change in the coefficient for the production index is positive and generally statistically significant; however, the magnitude of the effect of a change in production on emissions is relatively modest. The sign for the nonattainment variable for NAs outside California and the Northeast is positive and statistically significant in the potpourri cases for printing and publishing and wood furniture. Our interpretation of this result is that the regulation of these plants – likely located in NAs in the Southeast and in Gulf Coast states – was relatively less stringent than for other NAs. The LCV variable generally carries a positive sign for the pulp and paper category and a negative sign for the other industry categories, but in all cases the coefficient is small and not statistically significant.Footnote 42

3.3 Estimated emissions reductions for MACT rules

We developed estimates of the emissions reductions achieved (from baseline emissions) by the printing and publishing and pulp and paper plants in coming into compliance with the MACT rules. We focused on these two categories because the OLS regression results were reasonably robust.

For the other industry categories, the results were mixed. For the difference-in-difference results, the coefficients over the compliance periods were negative for pharmaceutical and wood furniture and positive for petroleum refining; however, they were generally not statistically significant. For the first-difference results, the coefficients were positive – a contrary outcome suggesting an increase in emissions – and not statistically significant.Footnote 43

3.3.1 Printing and Publishing

The estimated reduction in organic HAP emissions for the average printing and publishing plant ranges from 100 to 150 tons per year, compared with the emissions behavior of the paired paper and web surface-coating category and the group of six potpourri industries. This represents a reduction of 60% to more than 90%. EPA estimated ex ante that the rule would reduce HAP emissions by 27% from the publication rotogravure printing industry (27 plants) and by 46% from the product and packaging and wide-web flexographic printing industry (1200 facilities). Because of the TRI reporting thresholds, these results likely reflect the emissions behavior of the publication rotogravure printing industry – 27 plants with the larger average baseline emissions – and suggest HAP emissions reductions that substantially exceed EPA’s ex ante projections.

3.3.2 EPA’s 1998 pulp and paper Cluster Rule

As noted above, EPA issued joint rules limiting air toxics emissions under the MACT provisions of the Clean Air Act and effluent guideline limits for toxics for two subcategories of the pulp and paper industry (63 FR 18504). This rulemaking followed almost a decade of substantial public and regulatory pressure, beginning in the late 1980s, to reduce releases of toxic pollutants – especially discharges of dioxin to water. In June 1989, EPA issued final guidance implementing Section 304(l); it spelled out a timetable and process for pulp and paper mill compliance with dioxin limits.Footnote 44 The available evidence suggests that as a result, the industry made substantial reductions in its discharge of toxics to water in the years prior to 1995.Footnote 45 To the extent that the selected dioxin control – typically, the replacement of chlorine as a bleaching agent – reduced other VOC HAPs in water discharges, the resulting control would also likely achieve reductions in air emissions (63 FR 18576).Footnote 46 This continuing regulatory pressure on the industry through the 1990s complicates the identification of a suitable baseline for evaluating the effects of the MACT rule.Footnote 47

Our results for the pulp and paper industry suggest that EPA overestimated the HAP air emissions reductions associated with the MACT rule. The difference-in-difference results for the pulp and paper industry suggest a reduction in organic HAP emissions ranging from 20% to 33% for the average plant, compared with the emissions behavior of the paired plywood and composite products category and the group of six potpourri industries. This is well below EPA’s ex ante projection of a HAP reduction of 60%.Footnote 48 Similarly, our results in a separate paper suggests that the pulp and paper Cluster Rule yielded little or no additional reduction in toxic water discharges (Fraas & Egorenkov, Reference Fraas and Egorenkov2015). It is our view that the Cluster Rule largely ratified changes the industry was already making in the early 1990s in response to EPA’s Section 304(l) initiative and served primarily to force the laggards to catch up.Footnote 49

Several factors may have contributed to EPA’s overestimate of emissions reductions from the MACT rule. First, the baseline estimates for mills were based on calculations using model plants rather than monitoring data on actual discharges (63 FR 18545). Second, these estimates are relative to a declining business-as-usual baseline, since the control groups also realized a modest reduction in emissions over the 1998 to 2001 period.Footnote 50 Finally, EPA made changes to its final rule by basing effluent limits on a less stringent set of control technologies and by allowing averaging across the various emissions points within a pulp and paper mill (rather than requiring each emissions point to meet a specific limit) (63 FR 18549). These changes – providing pulp and paper mills with less stringent discharge standards and greater flexibility in meeting air and water standards – may have operated to lower actual reductions in toxic releases.

This result – smaller reductions in toxic air emissions and water discharges – is consistent with and complements EPA’s recently completed retrospective cost study finding that the agency significantly overestimated the capital costs of the Cluster Rule (EPA, 2014b ; Morgan et al., Reference Morgan, Pasurka and Shadbegian2014). EPA’s retrospective study offers several possible reasons for an ex ante overestimate of the capital costs, including the availability of flexible compliance options, the use of site-specific limits, and mill shutdowns and consolidations. In addition to these factors, we believe EPA may have overstated baseline emissions.

4 Discussion and recommendations

We set out with the objective of adding to our understanding of what technology-based standards actually accomplished. In addition, we believed that a retrospective study could offer lessons on ways to improve ex ante analysis of regulations and identify how retrospective analyses might be done – and done better – in the future.

The ambition of the project was to cover a large number of the air toxics rules issued during the 1990s in the first phase of the MACT program. However, we found that data on releases of toxics are surprisingly limited. We were able to conduct an analysis for only five of the air toxics rules issued under this program over the mid- to late 1990s. Data limitations and sample size precluded analyzing the effect of the MACT rules issued during this period for the eight other industries.Footnote 51

The results were mixed. The TRI data for the printing and publishing industry suggest a substantial reduction in HAP emissions – a reduction exceeding EPA’s ex ante projection – over the several years from promulgation of a final rule to the final compliance date. For pulp and paper, the TRI emissions data suggest some reduction in HAP emissions over the relevant period, but the reduction falls short of EPA’s ex ante projection. Gray and Shadbegian (Reference Gray and Shadbegian2015) report similar results – that is, some reduction in air toxics but “smaller than the ex ante prediction and not always significant.” Our results and those of Gray and Shadbegian complement EPA’s conclusion in its recent retrospective cost study that it overestimated the capital cost of the pulp and paper Cluster Rule by 30% to 100% (EPA, 2014b ). Finally, our results suggest that the MACT rules for the other three industries yielded little or no additional reduction in air toxics emissions. However, these results should be viewed as preliminary and deserving of further study. The limited number of plants reporting consistently to TRI over the period of interest – not only in the early years of the program but also for some later years – underscores the difficulty of carrying out a retrospective study. Unfortunately, there is no other readily accessible source of data for toxic air emissions. We obtained emissions data covering roughly half of the plants subject to MACT in the petroleum refining and pulp and paper industries. Data problems restricted our analysis to roughly one-third of the plants subject to MACT in the pharmaceutical industry and less than one-fifth of the plants in the other two industry categories (Table 5).Footnote 52 Although there is no obvious bias, the absence of data for a substantial fraction of the plants in these three industries raises some concern about the representativeness of the results.

Table 5 Plants reported by EPA versus sample size in Model 1.

$^{a}$ EIA (1999) reports 159 operating refineries.

$^{b}$ EPA (2006) identifies 155 mills subject to the final Cluster Rule when it was issued.

In addition, the explanatory power of the two models is marginal.Footnote 53 Other factors not accounted for in the models may, in fact, help explain the reductions in emissions during the compliance period. For example, additional, complementary information on these plants, such as firm profitability, may also be important in explaining plant emissions characteristics and improving model performance. It would be helpful to have a data set that combines emissions data with data on plant operations; for example, emissions data could be combined with plant-level data from the Census Bureau.

To facilitate future retrospective studies, EPA should include as a part of its final rule a specific plan for conducting a retrospective analysis. The agency should also provide for the collection of data – on emissions, plant production characteristics, and control measures adopted and costs incurred to comply with the rule – for at least a representative sample of plants plus a control group of unregulated plants. To some extent, EPA may be able to reduce the burden of retrospective studies by coordinating with existing data collection by the Census Bureau.

However, even if restricted to a representative sample of covered plants, the information required for these studies would be extensive and costly to collect.Footnote 54 In addition, the cost of conducting retrospective studies and the competition from other EPA initiatives in a period of tight budgets will limit the number of studies EPA can undertake. As a result, EPA will need to be strategic in its selection of retrospective studies.Footnote 55

In its review of agency rules under Executive Order 13563, the Office of Information and Regulatory Affairs (OIRA) should ensure that agencies establish in their rules a process for ex post evaluation of the effectiveness of important rulemakings.Footnote 56 To implement this recommendation, OIRA could issue guidance identifying factors that agencies should consider in selecting rules for regulatory review, the kinds of measurable outcomes targeted in the analysis, the associated data requirements, the type of analysis that will be used, and the time period to be evaluated. OIRA has already provided some general guidance along these lines. In a memorandum titled “Executive Order 13563: Improving Regulation and Regulatory Review,” OIRA (2011a ) identified several areas that agencies should address in conducting retrospective reviews, including analysis of costs and benefits and coordination with other forms of mandated retrospective analysis and review.Footnote 57 OIRA’s guidance should elaborate on these elements.Footnote 58

In addition, EPA will need to obtain OIRA approval under the Paperwork Reduction Act for its data collection. The agency will have to show that the collection has “practical utility” and is the least burdensome way of obtaining the information. The act also requires the agency to go through a public comment process.

OIRA (2010b ) has moved to streamline its paperwork review process by establishing a generic clearance process for specific types of information collection focused on scientific research. In a memo titled “Facilitating Scientific Research by Streamlining the Paperwork Reduction Act Process,” OIRA (2010a ) has outlined options and strategies for agencies to use to streamline the process of getting Office of Management and Budget approval for information collections related to scientific research. We recommend that OIRA explicitly provide this streamlined process for research that collects data for retrospective studies.

Appendix A. Excluded MACT rules

Table A1 EPA projected emissions reductions and costs for eight manufacturing MACT rules excluded from study because of data limitations.

Appendix B. Average level of HAP emissions, by year (pounds per year)

Table B1 Pulp and paper: Plywood.

Table B2 Pulp and paper: Potpourri control.

Table B3 Pharmaceutical manufacturing: Potpourri control.

Table B4 Petroleum refining: Potpourri control.

Table B5 Printing and publishing: Paper and other web surface-coating control.

Table B6 Printing and publishing: Potpourri control.

Table B7 Wood furniture: Potpourri control.

Appendix C. Regression result

Table C1 Difference-in-difference with paired industries.

Table C2 Difference-in-difference with potpourri industries.

Table C3 First-difference with paired industries.

Table C4 First-difference with potpourri industries.

Footnotes

Visiting fellow and research assistant, respectively, Resources for the Future (RFF). We received financial support for the research from the Sloan and Smith Richardson Foundations. We thank Wayne Gray and Ron Shadbegian for sharing their data for pulp and paper mills and for their helpful comments on an earlier draft of the paper. We also thank Ann Wolverton and others at the Regulatory Performance Initiative workshop held at RFF for helpful comments. In addition, we received helpful comments from Randy Lutter and Richard Morgenstern and from the editors and reviewers of the JBCA.

Sources: EPA (1999a , 2000).

$^{a}$ Ethylene oxide.

$^{b}$ Chromium.

$^{c}$ Reduction in organic HAP; rule will also reduce metal HAP emissions by 58 tons per year.

$^{d}$ Reduction in organic HAP.

$^{e}$ Reductions in total fluoride (3680 tons/yr) and polycyclic organic matter (2000 tons/yr).

2 See Morgenstern (Reference Morgenstern2015) for a description of the broader research program on regulatory performance undertaken by RFF.

4 If there are fewer than 30 sources in a source category, the MACT floor must reflect the average level of control achieved by the best-performing 5 sources.

5 EPA can adopt more stringent standards beyond the MACT requirements, taking into account a variety of factors, including technological and economic feasibility, cost and effectiveness, and the expected additional risk reduction achieved. In practice, however, EPA has generally adopted emissions standards based on the MACT floor. For convenience, we refer to the air toxics standards as MACT standards, even though in some cases the standards were set at levels more stringent than the MACT floor.

6 Morgenstern (Reference Morgenstern2015), in reporting on a recent Resources for the Future retrospective effort to evaluate regulatory performance, including ex ante and ex post comparisons, notes that “quality studies are generally few and far between.” In addition, Kopits et al. (Reference Kopits, McGartland, Morgan, Pasurka, Shadbegian, Simon, Simpson and Wolverton2014), based on their identified set of 10 survey articles of U.S. regulations, note that these articles summarize the same sets of underlying studies, so there is substantial overlap among the studies.

7 EPA launched its study in response to concerns over the limited available literature. This EPA initiative sought to apply a common conceptual framework to an ex post evaluation of the costs of five of its regulations (Kopits et al., Reference Kopits, McGartland, Morgan, Pasurka, Shadbegian, Simon, Simpson and Wolverton2014).

8 Volatile organic compounds include any compound of carbon, excluding carbon monoxide, carbon dioxide, carbonic acid, metallic carbides or carbonates, and ammonium carbonate, that participates in atmospheric photochemical reactions, except those designated by EPA as having negligible photochemical reactivity.

9 Konar and Cohen (Reference Konar and Cohen1997) and Khanna, Quimio and Bojilova (Reference Khanna, Quimio and Bojilova1998) state that in the initial years, firms reporting relatively higher TRI releases incurred abnormal losses in stock value, apparently prompting subsequent reductions in on-site toxic releases from 1990 to 1994. However, Konar and Cohen were unable to find any evidence that firms receiving significant negative media attention about their TRI releases reduced their emissions more than other firms of similar size. Similarly, Hamilton (Reference Hamilton2005, 217–18) reports that a 1991 GAO study and a later paper by Atlas, Vasu and Dimock (Reference Atlas, Vasu and Dimock2001) both find that most of the population “remains rationally ignorant about the TRI data” and does not seek out the data. Kraft et al. (Reference Kraft, Stephan and Abel2011, 55) conclude that “the evidence indicates that community pressure does not seem to be a driving force behind chemical management decisions. Rather regulation and concern about potential financial liability more strongly affect corporate decisions about chemical management.”

10 These 17 chemicals were selected because they were deemed high-volume toxic chemicals with feasible control costs. EPA adopted the program because it was seeking quick reductions through a voluntary effort without relying on regulatory requirements (EPA, 1991).

11 Khanna and Damon (Reference Khanna and Damon1999), EPA (1999b ), and Sam Khanna and Innes (Reference Sam, Khanna and Innes2009) find that participating chemical firms achieved the expected reduction in the program’s first 3 years (1991–93). Innes and Sam (Reference Innes and Sam2008) and Bi and Khanna (Reference Bi and Khanna2012) also find that the 33/50 Program was effective in achieving additional reductions in emissions in its early years, but that participating firms did not achieve significant additional reductions after the program ended. Gamper-Rabindran (Reference Gamper-Rabindran2006) reports that the mandatory phaseout of two ozone-depleting chemicals accounted for a significant fraction of the 33/50 Program reductions and that only participants from specific sectors achieved additional reductions over the course of the program. Finally, Khanna and Vidovic (Reference Khanna and Vidovic2001) and Vidovic and Khanna (Reference Vidovic and Khanna2012) reach an even stronger conclusion: that the 33/50 Program had little overall effect on emissions and that participating facilities would have reduced the targeted emissions even in its absence.

12 EPA’s inspector general reports a similar conclusion about EPA’s Performance Track program: “Thus, EPA cannot tell if facilities made overall environmental improvements, or rather improved in one area and faltered in others” (EPA, 2007, 19–22).

13 Kraft et al. (Reference Kraft, Stephan and Abel2011) rank states in terms of the proportion of firms achieving a reduction in toxic chemical releases and the reduction in population exposure risk from 1991 to 1995. On the basis of their ranking, for example, California was sixth in the nation, and “all of the northeastern states fell within the top two tiers of state industrial environmental performance” (Kraft et al., Reference Kraft, Stephan and Abel2011, 96). Of the 14 states identified by Bennear (Reference Bennear2007) as adopting management-based regulations, eight were located in the Northeast or on the West Coast.

14 For example, 27 states had pollution prevention programs in place by 1990, and 48 states had programs in place by mid-1991 (Bui & Kapon, Reference Bui and Kapon2012).

15 Bui (Reference Bui2005) did not address the effect of the 1995 MACT rule limiting toxic air emissions from petroleum refineries.

16 Gray and Shadbegian (Reference Gray and Shadbegian2007) find that state regulatory activity has a significant effect on compliance but not on emissions.

17 We also looked at the National Emissions Inventory (NEI) data set, which provides annual criteria air pollutant emissions such as VOCs and PM at the plant level at 3-year intervals. We found the data to be quite limited in terms of coverage of plants and in the relatively few years with plant reports of emissions. Although the NEI data from before 2000 cover a more complete set of plants, NEI provides reported emissions data only for 1996 (a precompliance year) and 1999 (in the midst of or at the end of the period when plants were coming into compliance). EPA adjusted or interpolated emissions data for the other years using reported data for 1996 and 1999. We found almost no change in reported plant-by-plant emissions for the interpolated years but large changes in 1996 and 1999 from each of the preceding years (i.e., 1995 and 1998). After 1999, EPA cut back on its reporting requirements such that data for 2002 and later years cover a smaller subset of plants. As a result, we used only the TRI release data.

19 The list has been expanded over the past 25 years and now covers 683 individual chemicals.

20 For facilities in the “otherwise use” category, the annual reporting threshold was 10,000 pounds. Persistent, bioaccumulative, and toxic (PBT) chemicals have lower thresholds, but reporting requirements at these lower thresholds for the PBT chemicals did not begin until 2000 (EPA, 1998a ).

21 Total releases are measured by the amount released into the air or water or placed on land, including amounts disposed, treated, recycled, and burned for energy recovery at the facility and amounts transferred from the facility to off-site locations for the purposes of recycling, energy recovery, treatment, or disposal (EPA, 1998a ).

22 Beginning in 1995, EPA allowed facilities to file a short form (Form A) if (1) the chemical being reported is not a PBT chemical; (2) the chemical has not been manufactured, processed, or otherwise used in excess of 1 million pounds; and (3) the total annual waste management (i.e., releases including disposal, recycling, energy recovery, and treatment) of the chemical does not exceed 500 pounds. http://www2.epa.gov/sites/production/files/documents/1998qa.pdf.

23 Gasoline distribution and off-site waste recovery operations were added to TRI reporting in 1998; dry-cleaning establishments are not subject to TRI reporting.

24 In addition, only a relatively small number of facilities were subject to these MACT rules. EPA projected that 11facilities would be subject to the basic liquid epoxy resins MACT, 26 facilities would be subject to the elastomer production MACT, and 45 facilities would be subject to the flexible polyurethane foam production MACT.

25 Hazardous air pollutants, also known as toxic air pollutants or air toxics, are those pollutants that are known or suspected to cause cancer or other serious health effects, such as reproductive effects or birth defects, or adverse environmental effects. Section 112 of the CAA lists 187 toxic air pollutants, including benzene, perchloroethylene, methylene chloride, dioxin, asbestos, toluene, and metals such as cadmium, mercury, and chromium.

26 EPA’s pulp and paper MACT rule was a part of its 1998 pulp and paper Cluster Rule. In addition to the MACT provisions under the Clean Air Act, the Cluster Rule established effluent guideline limits for toxics in water for two subcategories of the pulp and paper industry (63 FR 18504). EPA adopted this joint rulemaking to allow industry to coordinate its air and water pollution control efforts and to provide industry with greater regulatory certainty.

27 EPA projected that MACT rules would cover only a small number of plants in the magnetic tape, primary aluminum, and secondary lead smelters categories. In addition, only a small number of plants consistently reported to TRI in the aerospace, chromium electroplating, commercial sterilizer, halogenated solvent cleaning machine, and polyethylene terephthalate polymer and styrene (Group IV) industry categories. EPA projected relatively small compliance costs for these eight rules. (See Appendix A for EPA’s cost estimates.)

28 Annualized costs typically comprise annualized capital costs plus annual operating, monitoring, recordkeeping, and reporting costs. Table 1 provides a later set of appreciably larger annualized cost estimates for three of the industries from EPA’s 1999 report to Congress (EPA, 1999a ).

29 EPA also anticipated substantial reductions in VOC emissions for the other three industry categories but did not estimate them because of the uncertainty in the extent to which plants would take advantage of the pollution prevention options offered by these rules.

30 These values were developed only for estimated acute health benefits in ozone nonattainment areas.

31 The quantified and monetized benefits estimates were based primarily on estimates of the ozone-related air quality benefits from reducing VOC emissions using benefit transfer values of $602–$2733 per megagram (Mg) (63 FR 50410). The benefit transfer values were taken from the 1997 Ozone National Ambient Air Quality Standards (NAAQS) Regulatory Impact Analysis; $602/Mg does not include ozone mortality, whereas the $2733/Mg does include the mortality risk reduction effects of reductions in ambient ozone concentrations.

32 For example, for petroleum refining, we aggregated the reported TRI air releases for 14 organic HAPs: benzene, cresol, m-cresol, ethylbenzene, N-hexane, methyl ethyl ketone, methyl tert-butyl ether, naphthalene, phenol, toluene, m-xylene, o-xylene, p-xylene, and xylene (mixed isomers) (60 FR 43245). If we focused on any one specific air toxic (e.g., ethylbenzene), our sample of plants for each industry would have been severely limited.

33 A balanced panel provides an observation for each plant for each year in the data set. This approach avoids issues with plants that drop out of (and reenter) the panel in a nonrandom way, raising issues with sample selection and attrition.

34 In fact, EPA issued a direct final rule in 2006 clarifying the scope of coverage for the two rules. https://www.gpo.gov/fdsys/pkg/FR-2006-05-24/pdf/06-4821.pdf.

35 EPA issued its final MACT rule limiting VOC HAP emissions from paper and web surface-coating plants on December 4, 2002, and for plywood plants on July 30, 2004; effective dates for compliance were 2005 and 2007, respectively. Thus, MACT regulation of VOC HAPs for these two “control” industries was imposed a number of years after EPA-imposed MACT limits on the printing and pulp and paper industries.

36 Note that the sample size for the potpourri control group varies across the five MACT rules – for example, 120 plants in the control groups for the pharmaceutical and pulp and paper industries but somewhat smaller sample sizes for the other three industries. As noted above, we used somewhat different aggregate sets of VOC HAPs. A few of the plants in the potpourri industries did not emit any of the VOC HAPs targeted by the MACT rules for these other three industries.

37 We obtained sales data for pulp and paper mills from Gray and Shadbegian, e-mail to author, April 17, 2015. The correlation between our construction of a production index from the TRI database and the sales data was relatively good; the R-squared for the equation relating sales as a function of the TRI production index and paper capacity was 0.6.

38 Petroleum refineries are among the largest sources of VOC and air toxics emissions within their states and receive the continuing attention of the environmental agencies.

39 Based on analysis of variance tests for the printing and publishing and pulp and paper industries, we can reject the hypothesis that the coefficients are zero for the compliance periods in the potpourri cases – for both industries and for the case in which pulp and paper is paired with plywood and composite products.

40 The positive – but not statistically significant – coefficients for the years after promulgation of a final rule for the petroleum refining industry suggest an increase in emissions over the compliance period.

41 Based on analysis of variance tests for the printing and publishing and pulp and paper industries, we can reject the hypothesis that the coefficients are zero for the compliance periods for the potpourri case for both industries and for the paired industry case for the pulp and paper industry.

42 We also considered an alternative measure based on the 14 states with quantitative toxics management programs identified by Bennear (Reference Bennear2007). We believe, though, that the LCV score is conceptually a better measure of state regulatory programs.

43 The standard errors are quite large, though. We can reject the hypothesis that petroleum refining in the difference-in-difference results and the pharmaceutical and wood furniture first-difference results achieved EPA’s projected reductions at the 90% confidence level. However, we are unable to reject the hypothesis at the 90% level for the difference-in-difference results for pharmaceutical and wood furniture or for the first-difference results for the petroleum refining category.

44 Under Section 304(l), states are required to identify water bodies with toxic problems and the sources of those problems. States are then required to establish an independent control strategy in each source’s discharge permit. EPA has the authority to review, independently, the state-established strategy for individual plants. Where the control strategy was approved, the deadline for compliance was June 1992; where disapproved, an EPA-imposed strategy required compliance by June 1993 (Houck, Reference Houck1991).

45 For example, Houck (Reference Houck1991) reports that by 1991 through the Section 304(l) process, “dioxin limits were proposed for 88 of 98 pulp & paper mills and a number of pulp & paper mills had begun to convert to chlorine dioxide and hydrogen peroxide as bleaching agents, instead of using chlorine.” We note that EPA adjustments to its baseline estimates of dioxin discharges in 1993 and 1995, TRI data for the early 1990s, and Gray and Shadbegian (Reference Gray and Shadbegian2015) also provide support for the view that the industry undertook a major pollution control program in the early 1990s.

46 EPA reports that the process changes at the pulp and paper mills subject to best available technology rules, pretreatment standards for existing sources, and best management practices would decrease the emissions of some HAPs – an estimated 7% reduction in total emissions – but have little effect on others (63 FR 18576).

47 Kopits et al. (Reference Kopits, McGartland, Morgan, Pasurka, Shadbegian, Simon, Simpson and Wolverton2014) also note the difficulty of defining a counterfactual baseline for the pulp and paper Cluster Rule.

48 For our sample, the mean emissions level in 2001 (the first year that mills were required to meet the MACT limits) was 67% of 1995 baseline emissions (a 33% reduction). For these mills, the mean emissions level in 2003 was 45% of the 1995 baseline level of emissions (a reduction of 55%), perhaps reflecting the effects of the 2001–3 recession. By 2005, emissions were back up to 2001 levels.

49 Gray and Shadbegian (Reference Gray and Shadbegian2015) also suggest that the Cluster Rule may have focused on the laggards in the industry. Overall, our results are generally consistent with those of Gray and Shadbegian. Both their discussion paper and our study use TRI data but differ in terms of the time period and the pulp and paper mills covered in the sample. In addition, for water discharges, Gray and Shadbegian focus on chloroform, whereas our study uses an aggregation of 27 toxic pollutants discharged by pulp and paper mills.

50 For air emissions, we used two control groups: plants in the plywood industry and plants in the several potpourri industries. For water discharges, the control group comprises pulp and paper mills that were not subject to BAT limits. Some of the decline in the emissions of the control groups may be attributable to EPA requirements for a reduction in VOC emissions in and around ozone NAs and to state actions to reduce air pollution. If EPA or state regulations are responsible for the reductions, pulp and paper mills would also have been targets for similar EPA- and state-mandated reductions in the absence of the MACT rules.

51 As noted above, most of the other MACT manufacturing rules issued over this period imposed only relatively small costs. As a result, it seems unlikely that the MACT manufacturing rules dropped from this analysis would have been selected ex ante as candidates for a retrospective study.

52 We believe the TRI reporting thresholds limited the number of plants reporting to TRI in these industries. In addition, the number of plants reporting is particularly sparse around 2000.

53 The $F$ -statistic is statistically significant and the $R$ -squared ranges from 0.1 to 0.2 for the paired industry runs for the printing and publishing and the pulp and paper industry categories. For the potpourri runs for four of the five industries (excepting petroleum refining), the $F$ -statistics are statistically significant and the $R$ -squared ranges from 0.065 to 0.172. For petroleum refining, although the $F$ -statistic is statistically significant, the $R$ -squared is only 0.034. Gray and Shadbegian (Reference Gray and Shadbegian2015) report $R$ -squared in the range of 0.2 to 0.5: results for the basic ordinary least squares model are more at the lower end of this range, and results for the basic fixed effects model are at the upper end. Although the $F$ -statistic is generally statistically significant, the $R$ -squared ranges from only 0.05 to 0.11 for printing and publishing and for pulp and paper. For the petroleum refining and wood furniture categories (potpourri case only), the $F$ -statistic is statistically significant, but the $R$ -squared ranges from only 0.037 and 0.074. For the pharmaceutical industry, the $R$ -squared is 0.018 to 0.027, and the $F$ -statistic is not significant.

54 Some types of regulations, such as cap and trade or emissions fees, provide information on costs (through prices) and emissions. Collection of emissions data is integral to the enforcement of these programs, and prices (or fees) are an indicator of the cost of control.

55 In the case of the air toxics program, for example, certain rules impose only modest costs and would seem to be unlikely candidates for an intensive retrospective study.

56 OIRA is part of the Office of Management and Budget in the Executive Office of the President.

57 In an additional memorandum, OIRA (2011b ) also recommended that to promote a consistent culture of retrospective review, “future regulations should be designed and written in ways that facilitate evaluation of their consequences and thus promote retrospective analyses. To the extent consistent with law, agencies should give careful consideration to how best to promote empirical testing of the effects of rules both in advance and retrospectively.”

58 Aldy (Reference Aldy2014) reviews the federal government’s experience with retrospective review and offers a set of recommendations to enhance the role of retrospective analysis in improving federal regulation.

References

Aldy, Joseph E.(2014). Learning from Experience: An Assessment of the Retrospective Reviews of Agency Rules & the Evidence for Improving the Design & Implementation of Regulatory Policy, Report prepared for the Administrative Conference of the United States. www.acus.gov/sites/default/files/documents/Aldy%2520Retro%2520Review%2520Draft%252011-17- 2014.pdf.Google Scholar
Atlas, Mark, Vasu, Michael & Dimock, Michael (2001). “TRI to Communicate: An Empirical Evaluation of the Dissemination and Impact of Toxics Release Inventory Data”. Presented at the Association for Public Policy Analysis and Management Conference, Washington, DC, November.Google Scholar
Bennear, Lori Snyder (2007). Are Management-Based Regulations Effective? Evidence from State Pollution Prevention Programs. Journal of Policy Analysis and Management, 26(2), 327348.Google Scholar
Bi, Xiang & Khanna, Madhu (2012). Reassessment of the Impact of the EPA’s Voluntary 33/50 Program on Toxic Releases. Land Economics, 88(May), 3461.Google Scholar
Bui, Linda T. M. (2005). In Public Disclosure of Private Information as a Tool for Regulating Environmental Emissions: Firm-Level Responses by Petroleum Refineries to the Toxics Release Inventory. Washington, DC: Center for Economic Studies, US Census Bureau; Working Paper 05-13.Google Scholar
Bui, Linda T. M. & Kapon, Samuel (2012). The Impact of Voluntary Programs on Polluting Behavior: Evidence from Pollution Prevention Programs and Toxic Releases. Journal of Environmental Economics and Management, 64, 3144.Google Scholar
Coglianese, Cary & Allen, Laurie K. (2003). Building Sector-Based Consensus: A Review of the EPA’s CommonSense Initiative. (Faculty Scholarship Paper 103) , Philadelphia: University of Pennsylvania Law School; http://scholarship.law.upenn.edu/faculty_scholarship/103.Google Scholar
Coglianese, Cary & Nash, Jennifer (2014). Performance Track’s Postmortem: Lessons from the Rise and Fall of EPA’s ‘Flagship’ Voluntary Program. Harvard Environmental Law Review, 38, 186.Google Scholar
de Marchi, Scott & Hamilton, James T. (2006). Assessing the Accuracy of Self-Reported Data: An Evaluation of the Toxics Release Inventory. Journal of Risk and Uncertainty, 32(1), 576.Google Scholar
Dudley, Susan E. (1999). Comments on the EPA’s Proposed Amendments to the TRI Persistent Bioaccumulative Toxic (PBT) Chemicals. Fairfax, VA: Regulatory Studies Program, Mercatus Center, George Mason University.Google Scholar
Energy Information Administration (EIA) (1999). U.S. Number of Operable Refineries as of January 1; http://www.eia.gov/dnav/pet/hist/LeafHandler.ashx?n=PETs=8_NA_8O0_NUS_C&f=A.Google Scholar
Environmental Protection Agency (EPA) (1991). Fact Sheet on EPA’S 33/50 Program. Washington, DC: EPA; http://infohouse.p2ric.org/ref/10/09952.htm.Google Scholar
Environmental Protection Agency (EPA) (1993). Assessment of Changes in Reported TRI Releases & Transfers Between 1989 and 1990. Washington, DC: EPA.Google Scholar
Environmental Protection Agency (EPA) (1995a). National Emission Standards for Hazardous Air Pollutants for Source Category: Proposed Standards for Hazardous Air Pollutant Emissions from the Printing and Publishing Industry. 60, Federal Register, 13664–83.Google Scholar
Environmental Protection Agency (EPA) (1995b). National Emission Standards for Hazardous Air Pollutants for Source Category: Petroleum Refineries. 60, Federal Register, 43244–97.Google Scholar
Environmental Protection Agency (EPA) (1995c). National Emission Standards for Hazardous Air Pollutants for Source Category from Wood Furniture Manufacturing Operations. 60, Federal Register, 62930–62.Google Scholar
Environmental Protection Agency (EPA) (1996). National Emission Standards for Hazardous Air Pollutants for Source Category from the Printing and Publishing Industry. 61, Federal Register, 27132–59.Google Scholar
Environmental Protection Agency (EPA) (1998a). EPCRA Section 313 Questions and Answers. (EPA 745-B-98-004) , Washington, DC: EPA; http//www2.epa.gov/sites/production/files/documents/1998.qa.pdf.Google Scholar
Environmental Protection Agency (EPA) (1998b). National Emission Standards for Hazardous Air Pollutants for Source Category: Pharmaceuticals Production. 63, Federal Register, 50280–386.Google Scholar
Environmental Protection Agency (EPA) (1998c). National Emission Standards for Hazardous Air Pollutants for Source Category: Pulp and Paper Production; Effluent Limitations Guidelines, Pretreatment Standards, and New Source Performance Standards: Pulp, Paper, and Paperboard Category. 63, Federal Register, 18504–751.Google Scholar
Environmental Protection Agency (EPA) (1998d). TRI Data Quality Report. Washington, DC: EPA.Google Scholar
Environmental Protection Agency (EPA) (1999a). Report to Congress on the Benefits & Costs of the Clean Air Act, 1990 to 2010. Washington, DC: EPA.Google Scholar
Environmental Protection Agency (EPA) (1999b). 33/50 Program: The Final Record. EPA-745-R-99. Washington, DC: EPA; http://www.epa.gov/opptintr/3350/33fin01.htm.Google Scholar
Environmental Protection Agency (EPA) (2000). Taking Toxics Out of the Air. Washington, DC: EPA; http//www.epa.gov/oaqps001/takingtoxics/index.html.Google Scholar
Environmental Protection Agency (EPA) (2002a). National Emission Standards for Hazardous Air Pollutants for Source Category: Paper and Other Web Coating. 67, Federal Register, 72329.Google Scholar
Environmental Protection Agency (EPA) (2002b). National Emission Standards for Hazardous Air Pollutants for Source Category: Surface Coating of Metal Coil. 67, Federal Register, 39793.Google Scholar
Environmental Protection Agency (EPA) (2003a). National Emission Standards for Hazardous Air Pollutants for Source Category: Miscellaneous Coating Manufacturing. 68, Federal Register, 69163.Google Scholar
Environmental Protection Agency (EPA) (2003b). National Emission Standards for Hazardous Air Pollutants for Source Category: Surface Coating of Metal Cans. 68, Federal Register, 64431.Google Scholar
Environmental Protection Agency (EPA) (2003c). National Emission Standards for Hazardous Air Pollutants for Source Category: Surface Coating of Metal Furniture. 68, Federal Register, 28605.Google Scholar
Environmental Protection Agency (EPA) (2004a). National Emission Standards for Hazardous Air Pollutants for Source Category: Plywood and Composite Wood Products; Effluent Limitations Guidelines and Standards for the Timber Products Point Source Category; List of Hazardous Air Pollutants, Lesser Quantity Designations, Source Category List. 69, Federal Register, 45943.Google Scholar
Environmental Protection Agency (EPA) (2004b). National Emission Standards for Hazardous Air Pollutants for Source Category: Surface Coating of Miscellaneous Parts and Products. 69, Federal Register, 129.Google Scholar
Environmental Protection Agency (EPA) (2004c). National Emission Standards for Hazardous Air Pollutants for Source Category: Surface Coating of Plastic Parts and Products. 69, Federal Register, 20968.Google Scholar
Environmental Protection Agency (EPA) (2006). Final Report: Pulp, Paper and Paperboard Detailed Study. Washington, DC: EPA; http://water.epa.gov/scitech/wastetech/guide/304m/upload/2006_12_27_guide_304m_2006_pulp-final.pdf.Google Scholar
Environmental Protection Agency (EPA) (2007). Performance Track Could Improve Program Design and Management to Ensure Value. Washington, DC: EPA; http://www.epa.gov/oig/reports/ 2007/20070329-2007-P-00013.pdf.Google Scholar
Environmental Protection Agency (EPA) (2014a). Envirofacts Data Service API. Washington, DC: EPA; http://www.epa.gov/enviro/facts/services.html.Google Scholar
Environmental Protection Agency (EPA) (2014b). Retrospective Study of the Costs of EPA Regulations: A Report of Four Case Studies. Washington, DC: EPA; http://yosemite.epa.gov/EE%5Cepa%5Ceed.nsf/webpages/RetroCost.html.Google Scholar
Fraas, Arthur & Egorenkov, Alexander (2015). “A Retrospective Study of EPA’s Rules Setting Best Available Technology Limits for Toxic Discharges to Water under the Clean Water Act”. Resources for the Future Discussion Paper, DP15-41.Google Scholar
Gamper-Rabindran, Shanti (2006). Did the EPA’s Voluntary Industrial Toxics Program Reduce Emissions? A GIS Analysis of Distributional Impacts and by-Media Analysis of Substitution. Journal of Environmental Economics and Management, 52(1), 391410.Google Scholar
Gray, Wayne B. & Shadbegian, Ronald J. (2007). The Environmental Performance of Polluting Plants: A Spatial Analysis. Journal of Regional Science, 47, 6384.Google Scholar
Gray, Wayne B. & Shadbegian, Ronald J. (2015). “Multimedia Pollution Regulation and Environmental Performance: EPA’s Cluster Rule”. Resources for the Future Discussion Paper, DP15-26.Google Scholar
Hamilton, James T. (2005). Regulation through Revelation: The Origin, Politics, and Impacts of the Toxics Release Inventory Program. New York: Cambridge University Press.Google Scholar
Houck, Oliver A. (1991). The Regulation of Toxic Pollutants under the Clean Water Act. Environmental Law Reporter, 21, 1052810560.Google Scholar
Innes, Robert & Sam, Abdoul G. (2008). Voluntary Pollution Reductions and the Enforcement of Environmental Law: An Empirical Study of the 33/50 Program. Journal of Law and Economics, 51, 271296.Google Scholar
Khanna, Madhu & Damon, Lisa A. (1999). EPA’s Voluntary 33/50 Program: Impact on Toxic Releases and Economic Performance of Firms. Journal of Environmental Economics and Management, 37, 125.Google Scholar
Khanna, Madhu, Quimio, Wilma & Bojilova, Dora (1998). Toxic Release Information: A Policy Tool for Environmental Protection. Journal of Environmental Economics and Management, 36, 243266.Google Scholar
Khanna, Neha & Vidovic, Martina (2001). In Facility Participation in Voluntary Pollution Prevention Programs and the Role of Community Characteristics: Evidence from the 33/50 Program. Working paper. Binghamton, NY: Department of Economics, Binghamton University.Google Scholar
Konar, Shameek & Cohen, Mark (1997). Information as Regulation: The Effect of Community Right to Know Laws on Toxic Emissions. Journal of Environmental Economics and Management, 32(1), 109124.Google Scholar
Kopits, Elizabeth, McGartland, Al, Morgan, Cynthia, Pasurka, Carl, Shadbegian, Ronald, Simon, Nathalie B., Simpson, David & Wolverton, Ann (2014). Retrospective Cost Analyses of EPA Regulations: A Case Study Approach. Journal of Benefit-Cost Analysis, 5(2), 173193.Google Scholar
Kraft, Michael E., Stephan, Mark & Abel, Troy D. (2011). Coming Clean: Information Disclosure and Environmental Performance. Cambridge, MA: MIT Press.Google Scholar
Morgan, Cynthia, Pasurka, Carl A. & Shadbegian, Ronald J. (2014). Ex Ante and Ex Post Cost Estimates of the Cluster Rule and MACT II Rule. Journal of Benefit-Cost Analysis, 5(2), 19224.Google Scholar
Morgenstern, Richard (2015). “The RFF Regulatory Performance Initiative: What Have We Learned?Resources for the Future Discussion Paper, DP15-47.Google Scholar
Office of Information and Regulatory Affairs (OIRA) (2010a). Facilitating Scientific Research by Streamlining the Paperwork Reduction Act Process. In Memorandum for the Heads of Executive Departments and Agencies. Washington, DC: Office of Management and Budget; http//www.whitehouse.gov/omb/inforeg_infocoll.Google Scholar
Office of Information and Regulatory Affairs (OIRA) (2010b). Paperwork Reduction Act—Generic Clearances. In Memorandum for the Heads of Executive Departments and Agencies. Washington, DC: Office of Management and Budget; http://www.whitehouse.gov/omb/inforeg_infocoll.Google Scholar
Office of Information and Regulatory Affairs (OIRA) (2011a). Executive Order 13563: Improving Regulation and Regulatory Review. Washington, DC: Office of Management and Budget; www.whitehouse.gov/omb/inforeg_regmatters#eo12866.Google Scholar
Office of Information and Regulatory Affairs (OIRA) (2011b). Retrospective Analysis of Existing Significant Regulations. Washington, DC: Office of Management and Budget; www.whitehouse.gov/omb/inforeg_regmatters#eo12866.Google Scholar
Poje, Gerald V. & Horwitz, Daniel M. (1990). Phantom Reductions: Tracking Toxic Trends. Washington, DC: National Wildlife Federation.Google Scholar
Sam, Abdoul G., Khanna, Mardhu & Innes, Robert (2009). How Do Voluntary Pollution Reduction Programs (VPRs) Work? An Empirical Study of Links between VPRs, Environmental Management and Environmental Performance. Land Economics, 85(4), 692711.Google Scholar
Shadbegian, Ronald J. & Gray, Wayne B. (2006). Assessing Multi-dimensional Performance: Environmental and Economic Outcomes. Journal of Productivity Analysis, 26, 213234.Google Scholar
Tietenberg, Tom & Wheeler, David (1998). “Empowering the Community: Information Strategies for Pollution Control”. Paper presented at Frontiers of Environmental Economics Conference, Warrenton, VA, October.Google Scholar
Vidovic, Martina & Khanna, Neha (2012). Is Voluntary Pollution Abatement in the Absence of a Carrot or Stick Effective? Evidence from Facility Participation in the EPA’s 33/50 Program. Environmental Resource Economics, 52, 369393.Google Scholar
Figure 0

Table 1 EPA estimates of HAP emissions reductions and costs for five industry categories.

Figure 1

Table 2 Ex ante, ex post, and compliance periods for covered industries.

Figure 2

Table 3 Difference-in-difference results for MACT-regulated plants.

Figure 3

Table 4 First-difference results for MACT-regulated plants.

Figure 4

Table 5 Plants reported by EPA versus sample size in Model 1.

Figure 5

Table A1 EPA projected emissions reductions and costs for eight manufacturing MACT rules excluded from study because of data limitations.

Figure 6

Table B1 Pulp and paper: Plywood.

Figure 7

Table B2 Pulp and paper: Potpourri control.

Figure 8

Table B3 Pharmaceutical manufacturing: Potpourri control.

Figure 9

Table B4 Petroleum refining: Potpourri control.

Figure 10

Table B5 Printing and publishing: Paper and other web surface-coating control.

Figure 11

Table B6 Printing and publishing: Potpourri control.

Figure 12

Table B7 Wood furniture: Potpourri control.

Figure 13

Table C1 Difference-in-difference with paired industries.

Figure 14

Table C2 Difference-in-difference with potpourri industries.

Figure 15

Table C3 First-difference with paired industries.

Figure 16

Table C4 First-difference with potpourri industries.