Before moving to Harvard University in the summer of 1983, I spent two wonderful years as a Professor at the University of Chicago, 75% in the Department of Statistics and 25% in the Department of Education. During that brief period of time, I supervised several great PhD students, one of whom was Neal Thomas, who, despite a strong mathematics background, had a deep appreciation for applied statistics. The issue of the subasymptotic properties of propensity score matching was of real interest to me at the time, and remains an important one today. His PhD thesis addressed this topic, but he and I made more progress after he was out and working at RAND, and then ETS. The result was a sequence of four articles, which started with quite general theoretical results and then moved to apply that theory.

The first of these, Chapter 15, Rubin and Thomas (1992a), was published in the Annals of Statistics and was built on a substantial extension of EPBR matching methods studied in Chapter 6, Rubin (1976b). Affinely invariant matching methods are ones that are closed under any wide-sense linear transformation of the data, such as discriminant matching (using either the estimated or the population discriminant), propensity score matching (again, using either the population or estimated, e.g., by logistic regression), as well as Mahalanobis-metric matching, and many other methods.

Matched samples are usually created to aid in the design of a study to assess the causal effect of some active treatment or intervention relative to some control treatment, based on nonrandomized observational data. Consequently, this summary of advice on matching will also offer some general suggestions for the design of such studies based on my four decades of work on them. The theoretical perspective for this advice, the “Rubin Causal Model” (RCM – Holland, 1986b; Rubin, 2006), has two essential parts: the definition of the scientific situation using “potential outcomes” to define causal effect estimands, and the formulation of a real or hypothetical “assignment mechanism”; and a third optional part, the modeling of the science to produce imputations of missing potential outcomes. Matched sampling is focused on the second step. As stated in the initial introduction, a full-length textbook from this perspective is Imbens and Rubin (2006b); and recent summaries of the RCM appear in Imbens and Rubin (2006a) and Rubin (2006a).

The first part of the RCM implies that we should always start by carefully defining all causal estimands in terms of potential outcomes, which are all values that could be observed in some real or hypothetical experiment comparing the results under an active treatment to the results under a control treatment. That is, causal effects are defined by a comparison of (a) the values that would be observed if the active treatment were applied and (b) the values that would be observed if instead the control treatment were applied.

Few issues in behavior genetics have received more public and scientific attention than that given to the possible role of an extra Y chromosome in human aggression. Soon after the literature began to suggest an elevated frequency of the XYY genotype among inmates of institutions for criminals and delinquents, interest in this issue had a meteoric rise; and it has been sustained ever since. This happened for several reasons. Stories about a few men who had or were presumed to have an extra Y chromosome and who had committed serious crimes were given prominent attention in the press, suggesting the intriguing idea that the single Y chromosome normally found in males contributes to “aggressive tendencies” in that sex and that an extra Y carries these tendencies beyond their usual bounds. Reports of antisocial behavior in XYY men, often based on a single case, soon began to appear in the scientific literature and were taken as evidence of an XYY-aggression linkage. The serious moral and legal implications of such a linkage attracted the interest of social scientists and legal groups to the XYY phenomenon (Shah, 1970), and students of genetics and psychology saw in it, as Lederberg (1973) has said, “one of the most tangible leads for connecting genetic constitution with behavior in man.”

A number of studies have supported the earlier finding of an elevated frequency of cases with an XYY complement among men in institutions, particularly in penal-mental institutions.

Abstract: The problem of drawing causal inferences from retrospective case-control studies is considered. A model for causal inference in prospective studies is reviewed and then applied to retrospective studies. The limitations of case-control studies are formulated in terms of the level of causally relevant parameters that can be estimated in such studies. An example using data from a large retrospective study of coffee-drinking and myocardial infarctions is used to illustrate the ideas of the article.

Philosophical discussions of causality often emphasize the meaning of causation. Scientists are usually concerned with understanding causal mechanisms. Purely statistical discussions of causality are substantially more limited in scope, because the unique contribution of statistics is to the measuring of causal effects and not to the understanding of causal mechanisms or to the meaning of causation. This distinction is sometimes expressed as “statistics can establish correlation, but not causation.” We feel our emphasis on measurement is more appropriate, because it focuses on what statistical theory can contribute to discussions of causality. Measuring causal effects accurately without any understanding whatsoever of the causal mechanisms involved or of the meaning of causation is not only possible but is, of course, a commonplace experience of everyday life; that is, people are quite capable of using automobiles, ovens, calculators, and typewriters safely and effectively without any knowledge of how these devices work. Of course, careful measurements of causal effects often lead to a better understanding of the causal mechanisms involved.

Abstract: Several matching methods that match all of one sample from another larger sample on a continuous matching variable are compared with respect to their ability to remove the bias of the matching variable. One method is a simple mean-matching method and three are nearest available pair-matching methods. The methods' abilities to remove bias are also compared with the theoretical maximum given fixed distributions and fixed sample sizes. A summary of advice to an investigator is included.

INTRODUCTION

Matched sampling is a method of data collection and organization designed to reduce bias and increase precision in observational studies, i.e., in those studies in which the random assignment of treatments to units (subjects) is absent. Although there are examples of observational studies which could have been conducted as properly randomized experiments, in many other cases the investigator could not randomly assign treatments to subjects. For example, consider the Kihlberg and Robinson [1968] study comparing severity of injury in automobile accidents for motorists using and not using seatbelts. One would not want to randomly assign subjects to “seatbelt” and “no seatbelt” treatments and then have them collide at varying speeds, angles of impact, etc. Neither, however, would one want to simply compare the severity of injury in “random” samples of motorists in accidents using and not using seatbelts; important variables such as “speed of automobile at time of accident” may be differently distributed in the two groups (i.e., seatbelted motorists are generally more cautious and therefore tend to drive more slowly).

Abstract: Propensity score matching refers to a class of multivariate methods used in comparative studies to construct treated and matched control samples that have similar distributions on many covariates. This matching is the observational study analog of randomization in ideal experiments, but is far less complete as it can only balance the distribution of observed covariates, whereas randomization balances the distribution of all covariates, both observed and unobserved. An important feature of propensity score matching is that it can be easily combined with model-based regression adjustments or with matching on a subset of special prognostic covariates or combinations of prognostic covariates that have been identified as being especially predictive of the outcome variables. We extend earlier results by developing approximations for the distributions of covariates in matched samples created with linear propensity score methods for the practically important situation where matching uses both the estimated linear propensity scores and a set of special prognostic covariates. Such matching on a subset of special prognostic covariates is an observational study analog of blocking in a randomized experiment. An example combining propensity score matching with Mahalanobis metric matching and regression adjustment is presented that demonstrates the flexibility of these methods for designing an observational study that effectively reduces both bias due to many observed covariates and bias and variability due to a more limited subset of covariates.

Abstract: Investigators in observational studies have no control over treatment assignment. As a result, large differences can exist between the treatment and control groups on observed covariates, which can lead to badly biased estimates of treatment effects. Propensity score methods are an increasingly popular method for balancing the distribution of the covariates in the two groups to reduce this bias; for example, using matching or subclassification, sometimes in combination with model-based adjustment. To estimate propensity scores, which are the conditional probabilities of being treated given a vector of observed covariates, we must model the distribution of the treatment indicator given these observed covariates. Much work has been done in the case where covariates are fully observed. We address the problem of calculating propensity scores when covariates can have missing values. In such cases, which commonly arise in practice, the pattern of missing covariates can be prognostically important, and then propensity scores should condition both on observed values of covariates and on the observed missing-data indicators. Using the resulting generalized propensity scores to adjust for the observed background differences between treatment and control groups leads, in expectation, to balanced distributions of observed covariates in the treatment and control groups, as well as balanced distributions of patterns of missing data. The methods are illustrated using the generalized propensity scores to create matched samples in a study of the effects of postterm pregnancy.

Patients write DNR orders to avoid a vegetative state or an otherwise low-quality life, which they fear will follow resuscitation, but policy makers are concerned that DNR orders hasten death, especially for vulnerable subgroups. Both patients and policy makers would be well informed by knowing how often and for whom obeying a DNR order will cause the early termination of life and how often those terminated lives would have been high quality.

There are two ways to learn about the causal effects of DNR orders: an observational study, such as performed by Shepardson et al. (1999) and a randomized clinical trial (RCT). At the end of our discussion, we consider an alternative to both an observational study and a traditional RCT that involves random assignment, yet may be both ethical and more policy relevant than a traditional RCT.

When treatment evaluation does not benefit from any random assignment, observational study methods must be used to assess treatment effects. Here, the authors use propensity score methods and conclude that a patient's choice to write a DNR order on average hastens death. At first glance, it may seem strange to use formal quantitative methods to evaluate this question. After all, can forbidding “resuscitation to preserve life” do anything but hasten death? However, the authors' give their attention to assessing the magnitude of the effect, which has far more policy relevance than the more mundane question of assessing statistical significance.

Abstract: Matched sampling is a standard technique in the evaluation of treatments in observational studies. Matching on estimated propensity scores comprises an important class of procedures when there are numerous matching variables. Recent theoretical work (Rubin, D. B., and Thomas, N., 1992a, reprinted in this volume as Chapter 15) on affinely invariant matching methods with ellipsoidal distributions provides a general framework for evaluating the operating characteristics of such methods. Moreover, Rubin and Thomas (1992b, reprinted in this volume as Chapter 16) uses this framework to derive several analytic approximations under normality for the distribution of the first two moments of the matching variables in samples obtained by matching on estimated linear propensity scores. Here we provide a bridge between these theoretical approximations and actual practice. First, we complete and refine the nomal-based analytic approximations, thereby making it possible to apply these results to practice. Second, we perform Monte Carlo evaluations of the analytic results under normal and nonnormal ellipsoidal distributions, which confirm the accuracy of the analytic approximations, and demonstrate the predictable ways in which the approximations deviate from simulation results when normal assumptions are violated within the ellipsoidal family. Third, we apply the analytic approximations to real data with clearly nonellipsoidal distributions, and show that the thoretical expressions, although derived under artificial distributional conditions, produce useful guidance for practice. Our results delineate the wide range of settings in which matching on estimated linear propensity scores performs well, thereby providing useful information for the design of matching studies.

Abstract: Observational studies comparing groups of treated and control units are often used to estimate the effects caused by treatments. Matching is a method for sampling a large reservoir of potential controls to produce a control group of modest size that is ostensibly similar to the treated group. In practice, there is a trade-off between the desires to find matches for all treated units and to obtain matched treated–control pairs that are extremely similar to each other. We derive expressions for the bias in the average matched pair difference due to (i) the failure to match all treated units – incomplete matching, and (ii) the failure to obtain exact matches – inexact matching. A practical example shows that the bias due to incomplete matching can be severe, and moreover, can be avoided entirely by using an appropriate multivariate nearest available matching algorithm, which, in the example, leaves only a small residual bias due to inexact matching.

INTRODUCTION

The Effects Caused by Treatments

A treatment is an intervention that can, in principle, be given to or withheld from any experimental unit under study. With an experimental treatment and a control treatment, each unit has two potential responses: a response r1 that would be observed if the unit received the experimental treatment, and a response r0 that would be observed if the unit received the control treatment.

Abstract: This paper proposes a simple technique for assessing the range of plausible causal conclusions from observational studies with a binary outcome and an observed categorical covariate. The technique assesses the sensitivity of conclusions to assumptions about an unobserved binary covariate relevant to both treatment assignment and response. A medical study of coronary artery disease is used to illustrate the technique.

INTRODUCTION AND NOTATION

Inevitably, the results of clinical studies are subject to dispute. In observational studies, one basis for dispute is obvious: since patients were not assigned to treatments at random, patients at greater risk may be over-represented in some treatment groups. This paper proposes a method for assessing the sensitivity of causal conclusions to an unmeasured patient characteristic relevant to both treatment assignment and response. Despite their limitations, observational studies will continue to be a valuable source of information, and therefore it is prudent to develop appropriate methods of analysis for them.

Our sensitivity analysis consists of the estimation of the average effect of a treatment on a binary outcome variable after adjustment for observed categorical covariates and an unobserved binary covariate u, under several sets of assumptions about u. Both Cornfield et al. (1959) and Bross (1966) have proposed guidelines for determining whether an unmeasured binary covariate having specified properties could explain all of the apparent effect of a treatment, that is, whether the treatment effect, after adjustment for u could be zero.

Abstract: The ability of matched sampling and linear regression adjustment to reduce the bias of an estimate of the treatment effect in two sample observational studies is investigated for a simple matching method and five simple estimates. Monte Carlo results are given for moderately linear exponential response surfaces and analytic results are presented for quadratic response surfaces. The conclusions are (1) in general both matched sampling and regression adjustment can be expected to reduce bias, (2) in some cases when the variance of the matching variable differs in the two populations both matching and regression adjustment can increase bias, (3) when the variance of the matching variable is the same in the two populations and the distributions of the matching variable are symmetric the usual covariance adjusted estimate based on random samples is almost unbiased, and (4) the combination of regression adjustment in matched samples generally produces the least biased estimate.

INTRODUCTION

This paper is an extension of Rubin [1973a] to include regression adjusted estimates and parallel nonlinear response surfaces. The reader is referred to Sections 1 and 2 of that paper for the statement of the general problem and an introduction to the notation.

After presenting the estimates of the treatment effect to be considered in the remainder of Section 1, we go on in Section 2 to present Monte Carlo results for the expected bias of the estimates assuming four exponential response surfaces, normally distributed X, and the random order, nearest available matching method.

Abstract: The aim of many analyses of large databases is to draw causal inferences about the effects of actions, treatments, or interventions. Examples include the effects of various options available to a physician for treating a particular patient, the relative efficacies of various health care providers, and the consequences of implementing a new national health care policy. A complication of using large databases to achieve such aims is that their data are almost always observational rather than experimental. That is, the data in most large data sets are not based on the results of carefully conducted randomized clinical trials, but rather represent data collected through the observation of systems as they operate in normal practice without any interventions implemented by randomized assignment rules. Such data are relatively inexpensive to obtain, however, and often do represent the spectrum of medical practice better than the settings of randomized experiments. Consequently, it is sensible to try to estimate the effects of treatments from such large data sets, even if only to help design a new randomized experiment or shed light on the generalizability of results from existing randomized experiments. However, standard methods of analysis using available statistical software (such as linear or logistic regression) can be deceptive for these objectives because they provide no warnings about their propriety. Propensity score methods are more reliable tools for addressing such objectives because the assumptions needed to make their answers appropriate are more assessable and transparent to the investigator.

Abstract: Propensity score methodology can be used to help design observational studies in a way analogous to the way randomized experiments are designed: without seeing any answers involving outcome variables. The typical models used to analyze observational data (e.g., least squares regressions, difference of difference methods) involve outcomes, and so cannot be used for design in this sense. Because the propensity score is a function only of covariates, not outcomes, repeated analyses attempting to balance covariate distributions across treatment groups do not bias estimates of the treatment effect on outcome variables. This theme will be the primary focus of this article: how to use the techniques of matching, subclassification and/or weighting to help design observational studies. The article also proposes a new diagnostic table to aid in this endeavor, which is especially useful when there are many covariates under consideration. The conclusion of the initial design phase may be that the treatment and control groups are too far apart to produce reliable effect estimates without heroic modeling assumptions. In such cases, it may be wisest to abandon the intended observational study, and search for a more acceptable data set where such heroic modeling assumptions are not necessary. The ideas and techniques will be illustrated using the initial design of an observational study for use in the tobacco litigation based on the NMES data set.

Abstract: Matched sampling is a method of data collection designed to reduce bias and variability due to specific matching variables. Although often used to control for bias in studies in which randomization is practically impossible, there is virtually no statistical literature devoted to investigating the ability of matched sampling to control bias in the common case of many matching variables. An obvious problem in studying the multivariate matching situation is the variety of sampling plans, underlying distributions, and intuitively reasonable matching methods. This article considers one class of multivariate matching methods which yield the same percent reduction in expected bias for each of the matching variables. The primary result is the derivation of the expression for the maximum attainable percent reduction in bias given fixed distributions and fixed sample sizes. An examination of trends in this maximum leads to a procedure for estimating minimum ratios of sample sizes needed to obtain well-matched samples.

INTRODUCTION

This introduction is brief; the reader is referred to Rubin [1976b] for a more detailed explanation of the notation and terminology.

Let G1 and G2 be two random samples of sizes N1 and N2 from two populations, P1 and P2. Matched sampling is an attempt to find subsamples of G1 and G2, G1* and G2* of sizes N1* and N2*, such that the distributions of the p matching variables X are more similar in G1* and G2* than in G1 and G2.

Abstract: Matched sampling is a method for selecting units from a large reservoir of potential controls to produce a control group of modest size that is similar to a treated group with respect to the distribution of observed covariates. We illustrate the use of multivariate matching methods in an observational study of the effects of prenatal exposure to barbiturates on subsequent psychological development. A key idea is the use of the propensity score as a distinct matching variable.

INTRODUCTION: BACKGROUND; WHY MATCH?

Matched Sampling in Observational Studies. In many observational studies, there is a relatively small group of subjects exposed to a treatment and a much larger group of control subjects not exposed. When the costs associated with obtaining outcome or response data from subjects are high, some sampling of the control reservoir is often necessary. Matched sampling attempts to choose the controls for further study so that they are similar to the treated subjects with respect to background variables measured on all subjects.

The Danish Cohort. We examine multivariate matched sampling using initial data from a proposed study of the effects on psychological development of prenatal exposure to barbiturates. The analyses presented are preliminary and intended only to explore methodological options; none of the matched samples are the actual ones to be used for study of in utero exposure to barbiturates. The children under study were born between 1959 and 1961 and have been the object of other studies (e.g., Mednick et al. 1971; Zachau-Christiansen and Ross 1975).