With an Extended Example of a Weighted Regression Alternative to Matching

In the last chapter, we argued that traditional regression estimators of casual effects have substantial weaknesses, especially when individual-level causal effects are heterogeneous in ways that are not explicitly parameterized. In this chapter, we will introduce weighted regression estimators that solve these problems by appropriately averaging individual-level heterogeneity across the treatment and control groups using estimated propensity scores. In part because of this capacity, weighted regression estimators are now at the frontier of causal effect estimation, alongside the latest matching estimators that are also designed to properly handle such heterogeneity.

In the long run, we expect that weighted regression estimators will prove to be a common choice among alternative conditioning procedures that are used to estimate causal effects. In fact, we expect that weighted regression estimators will be used more frequently than the matching estimators presented in Chapter 5 when there is good overlap in the distributions of adjustment variables across the treatment and control groups. We have four primary reasons for this prediction, each of which we will explain in this chapter. First, weighted regression estimators allow the analyst to adopt the spirit of matching, and the clear thinking that it promotes, within a mode of data analysis that utilizes widely available software and that is familiar to most social scientists.

What role should the counterfactual approach to observational data analysis play in causal analysis in the social sciences? Some scholars see its elaboration as a justification for experimental methodology as an alternative to observational data analysis. We agree that by laying bare the challenges that confront causal analysis with observational data, the counterfactual approach does indirectly support experimentation as an alternative to observation. But, because experiments are often (perhaps usually) infeasible for most of the causal questions that practicing social scientists appear to want to answer, this implication, when considered apart from others, is understandably distressing.

We see the observational data analysis methods associated with the potential outcome model, motivated using directed graphs, as useful tools that can help to improve the investigation of causal relationships within the social sciences, especially when experiments are infeasible. Accordingly, we believe that the methods associated with the counterfactual approach complement and extend older approaches to causal analysis with observational data by shaping the goals of an analysis, requiring explicit consideration of individual-level heterogeneity of causal effects, encouraging a wider consideration of available identification strategies, and clarifying standards for credible interpretations.

In this chapter, we first shore up our presentation of the counterfactual approach by considering several critical perspectives on its utility. We weigh in with the arguments that we find most compelling, and it will not be surprising to the reader that we find these objections less serious than do those who have formulated them.

Social scientists have recognized for decades that the best explanations for how causes bring about their effects must specify in empirically verifiable ways the causal pathways between causes and their outcomes. This valuation of depth of causal explanation applies to the counterfactual tradition as well. Accordingly, it is widely recognized that a consistent estimate of a counterfactually defined causal effect of D on Y may not qualify as a sufficiently deep causal account of how D effects Y, based on the standards that prevail in a particular field of study.

In this chapter, we first discuss the dangers of insufficiently deep explanations of causal effects, reconsidering the weak explanatory power of some of the natural experiments discussed already in Chapter 9. We then consider the older literature on intervening variables in the social sciences as a way to introduce the mechanism-based estimation strategy proposed by Pearl (2009). In some respects, Pearl's approach is completely new, and it shows in a novel and sophisticated way how intervening variables can be used to identify causal effects, even when unblocked back-door paths between a causal variable and an outcome variable are present. In other respects, however, Pearl's approach is refreshingly familiar, as it helps to clarify the appropriate usage of intervening variables when attempting to deepen the explanation of a causal claim.

Independent of Pearl's important work, a diverse group of social scientists has appealed recently for the importance of mechanisms to all explanation in social science research.

In this chapter, we consider how analysts can proceed when no observed variables are available to point-identify and then estimate causal effects using the procedures explained in prior chapters. We discuss three complementary approaches. First, we will review the early literature on selection-bias adjustment, which shows clearly how distributional assumptions about unobservables, when harnessed from within a structural model, can identify causal parameters of interest. Point identification and estimation utilizing this strategy was frequent before it became clear to researchers in the 1990s how rarely the required distributional assumptions were warranted for their applications. The more recent selection-bias adjustment literature offers less restrictive semiparametric methods, but it also reemphasizes the relative value of instrumental variables in contrast to distributional assumptions for unobservables.

We will then consider the exact opposite strategy. Rather than place strong and usually untestable assumptions on the distributional characteristics of unobservables, the set-identification approach asks what can be learned about particular causal parameters by asserting only weak but defendable assumptions about unobserved variables. Instead of attempting to point-identify average causal effects, the set-identification approach suggests that it is more credible to try to limit the interval within which an average treatment effect must fall.

Finally, we will consider the related approach known as sensitivity analysis. Here, the analyst offers an estimate based on the provisional maintenance of an identifying assumption – most often, ignorability or selection on the observables – and then assesses the extent to which the estimate would vary as violations of the identifying assumption increase in severity.

In this chapter, we will lay the groundwork for our presentation of three strategies to estimate causal effects when simple conditioning on observed variables that lie along back-door paths will not suffice. These strategies will be taken up in Chapters 9, 10, and 11, where we will explain instrumental variable estimators, front-door identification with causal mechanisms, and conditioning estimators that use data on pretreatment values of the outcome variable. Under very specific assumptions, these three strategies will identify average causal effects of interest, even though selection is on the unobservables and treatment assignment is nonignorable.

In this chapter, we will first review the related concepts of nonignorable treatment assignment and selection on the unobservables, using the directed graphs presented in prior chapters. To deepen the understanding of these concepts, we will then demonstrate why the usage of additional posttreatment data on the outcome of interest is unlikely to aid in the point identification of the treatment effects of most central concern. One indirect goal of this demonstration is to convince the reader that oft-heard claims such as “I would be able to establish that this association is causal if I had longitudinal data” are nearly always untrue if the longed-for longitudinal data are additional measurements taken only after treatment exposure. Instead, longitudinal data are most useful, as we will later explain in detail in Chapter 11, when pretreatment measures are available for those who are subsequently exposed to the treatment.

As discussed in previous chapters, the fundamental challenge of causal inference is that an individual cannot be simultaneously observed in both the treatment and control states. In some situations, however, it is possible to observe the same individual or unit of observation in the treatment and control states at different points in time. If the potential outcomes do not evolve in time for reasons other than the treatment, then the causal effect of a treatment can be estimated as the difference between an individual's observed outcome in the control state at time 1 and the same individual's observed outcome in the treatment state at time 2. The assumption that potential outcomes are stable in time (and thus age for individuals) is often heroic. If, however, potential outcomes evolve in a predictable way, then it may be possible to use the longitudinal structure of the data to predict the counterfactual outcomes of each individual.

We begin our discussion with the interrupted time series (ITS) design, which we introduced already with the example of the year of the fire horse in Section 2.8.1. The ITS design is the simplest case where the goal is to determine the degree to which a treatment shifts the underlying trajectory of an outcome.

The rise of the counterfactual model to prominence has increased the popularity of data analysis routines that are most clearly useful for estimating the effects of causes. The matching estimators that we will review and explain in this chapter are perhaps the best example of a classic technique that has reemerged in the past three decades as a promising procedure for estimating causal effects. Matching represents an intuitive method for addressing causal questions, primarily because it pushes the analyst to confront the process of causal exposure as well as the limitations of available data. Accordingly, among social scientists who adopt a counterfactual perspective, matching methods are fast becoming an indispensable technique for prosecuting causal questions, even though they usually prove to be the beginning rather than the end of causal analysis on any particular topic.

We begin with a brief discussion of the past use of matching methods. Then, we present the fundamental concepts underlying matching, including stratification of the data, weighting to achieve balance, and propensity scores. Thereafter, we discuss how matching is usually undertaken in practice, including an overview of various matching algorithms.

In the course of presentation, we will offer four hypothetical examples that demonstrate some of the essential claims of the matching literature, progressing from idealized examples of stratification and weighting to the implementation of alternative matching algorithms on simulated data for which the treatment effects of interest are known by construction.