2 A Basic Simulation Exercise

The data structure explored in this letter is a simple multiple regression model of the formFootnote ²

(1) $$\begin{eqnarray}Y=\unicode[STIX]{x1D6FD}_{1}X_{1}+\unicode[STIX]{x1D6FD}_{2}X_{2}+\unicode[STIX]{x1D716}.\end{eqnarray}$$

In the initial exercise, data is missing only for $X_{2}$ , so both $X_{1}$ and $Y_{1}$ are fully observed. In all simulations, I set $\unicode[STIX]{x1D6FD}_{1}=\unicode[STIX]{x1D6FD}_{2}=5$ . The theoretical variable of interest is $X_{2}$ , so the estimation problem is to estimate $\unicode[STIX]{x1D6FD}_{2}$ . Restricting missingness to $X_{2}$ only is the best case scenario for listwise deletion with MNAR data because under the assumption that (1) is the true data generating process, then listwise deletion is unbiased (see Little Reference Little1992, 1229; Arel-Bundock and Pelc Reference Arel-Bundock and Pelc2018). It serves as a useful benchmark from which to compare listwise deletion to multiple imputation, and then to build more complicated simulations.

To capture the possibility that there might be other useful variables in the dataset, $X_{2}$ is modeled as a linear combination of two other variables and (possibly) an error term:

(2) $$\begin{eqnarray}X_{2}=.5U_{1}+.5U_{2}+\unicode[STIX]{x1D702}.\end{eqnarray}$$

These are drawn following a rule that ensures that in expectation, $X_{2}\sim N(0,1)$ : for values of $q$ between 0 and approaching 1,

(3) $$\begin{eqnarray}\displaystyle & \displaystyle \unicode[STIX]{x1D702}\sim N(0,q) & \displaystyle\end{eqnarray}$$

(4) $$\begin{eqnarray}\displaystyle & \displaystyle U_{1},U_{2}\sim N(0,(1-q)/.5). & \displaystyle\end{eqnarray}$$

When $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D702}}^{2}$ —the variance of $\unicode[STIX]{x1D702}$ in (2)—is $0$ , then $X_{2}$ is an exact linear combination of $U_{1}$ and $U_{2}$ . Both $X_{1}$ and $\unicode[STIX]{x1D716}\sim N(0,1)$ , so by construction $X_{1}$ and $X_{2}$ are independent (this will be relaxed later). It is useful to observe that given this setup, with $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D702}}^{2}=.2$ , the fully observed data contain a great deal of information about $X_{2}$ , the variable that will be plagued by missingness. These are propitious conditions for using observed data to predict missing data.

To compare multiple imputation with listwise deletion, I first discard the lowest 20% of the values of $X_{2}$ . I then run four standard analyses: one on the full data, one where I drop all observations that are missing, and then two multiple imputation analyses using the amelia package in R (Honaker, King, and Blackwell Reference Honaker, King and Blackwell2011), which I process through the zelig package (R Core Team Reference Choirat, Honaker, Imai, King and Lau2007). One includes the proxies $U_{1},U_{2}$ and one omits them.

From there, I extract four quantities: $\hat{b}_{2,\text{Full}}$ is the estimate when $X_{2}$ is fully observed, $\hat{b}_{2,\text{LD}}$ is the listwise deletion estimate (LD stands for listwise deletion), $\hat{b}_{2,\text{NoProxies}}$ employs multiple imputation without including the proxy variable $U_{1}$ or $U_{2}$ , and $\hat{b}_{2,\text{Proxies}}$ employs multiple imputation but includes the proxies.Footnote ³

To compare results, I repeat the above procedure 1000 times. Figure 1 displays the results of these 1000 simulations (left-hand side) alongside a density plot of the true versus simulated data in one run of the multiple imputation algorithm. The top left plot shows density plots of the four estimates of $\hat{b}_{2}$ . The bottom left plot calculates the absolute error of each estimator, calculated as $|\hat{b}_{2}-\unicode[STIX]{x1D6FD}_{2}|$ . Listwise deletion is unbiased, but less efficient than estimates with no missing data. Importantly, the two figures clearly show that multiple imputation is more biased than listwise deletion. And surprisingly, the inclusion of $U_{1}$ and $U_{2}$ does not help multiple imputation much, although the proxies do return estimates that are less biased than multiple imputation without proxies.

Figure 1.

Simulation results.

Why would multiple imputation underperform listwise deletion? Because multiple imputation imputes values for the missing data by assuming that the complete data is representative of the missing data, it predicts the relationship between $X_{2}$ and both $Y$ and $X_{1}$ without access to the lowest 20% of the observations for $X_{2}$ . This produces a slightly biased prediction of the relationships among the three, and as a result a slightly biased prediction of the missing $X_{2}$ (see the right side of Figure 1). Here, there is upward bias in the predictions of the missing $X_{2}$ . Because listwise deletion is unbiased, in this case it will outperform multiple imputation.

To compare coverage rates for multiple imputation and listwise deletion, for each simulation I check whether the true value of $\unicode[STIX]{x1D6FD}_{2}=5$ falls within two standard errors of the estimates $\hat{b}_{2,\text{LD}}$ , $\hat{b}_{2,\text{NoProxies}}$ , and $\hat{b}_{2,\text{Proxies}}$ . The resulting coverage rates across the 1000 simulations are 95.3% for listwise deletion, 22.9% for multiple imputation without proxies, and 31.6% for multiple imputation with proxies.

Before proceeding, I pause here to reiterate that this result should be unsurprising: if missingness is purely a function of $X_{2}$ then even when data are MNAR, listwise deletion is unbiased (see the proof in Allison (Reference Allison2002), footnote 1, p. 87). What is possibly surprising is that multiple imputation fares worse in this application than listwise deletion. Commenting on this issue, Allison (Reference Allison2014) observes that for NI missing data where missingness is unrelated to $Y$ , missingness “certainly could lead to bias if you use standard implementations of multiple imputation.” The findings here suggest that the dangers of bias when using multiple imputation are not just “possible” (see Lall Reference Lall2016, footnote 17); they are substantial when missingness is confined to $X$ . The results above also reveal that having proxy variables in the multiple imputation dataset that are known to be strongly predictive of the missing values does not help very much.

2.1 Variation: Missingness and the Informativeness of Proxies

To probe the performance of multiple imputation versus listwise deletion further, I repeat the above simulations, but vary the missingness on $X_{2}$ by deleting from 10% to 90% of the data from $X_{2}$ . For these simulations, $P(\text{Missing})$ is a deterministic function of $X_{2}$ : every observation lower than the cutoff threshold $p$ is missing, and every observation above that threshold is fully observed. As a result, $P(Y|X_{2}=\text{Observed})\neq P(Y|X_{2}=\text{Missing})$ , so data are MNAR, but simulated data are “closer to” MAR the lower the value of $p$ is. The number of simulations is set at 1000 for each value of $p$ .

Figure 2 displays the results. The top figure calculates the proportion of simulations of where the error of $\hat{b}_{2,\text{NoProxies}}$ (or $\hat{b}_{2,\text{Proxies}}$ ) is greater than the error of $\hat{b}_{2,\text{LD}}$ . The bottom left figure summarizes the empirical mean squared error (MSE), $(\hat{b}_{2}-\unicode[STIX]{x1D6FD}_{2})^{2}$ , for the 1000 simulations of $\hat{b}_{2,LD}$ , $\hat{b}_{2,\text{NoProxies}}$ , and $\hat{b}_{2,\text{Proxies}}$ across the values of $p$ . And finally, the bottom right figure summarizes the coverage rates, which again correspond to the percentage of simulations in which $\hat{b}_{2,\text{LD}}$ , $\hat{b}_{2,\text{NoProxies}}$ , and $\hat{b}_{2,\text{Proxies}}$ lie within two standard deviations of the true value of $\unicode[STIX]{x1D6FD}_{2}$ . The top figure shows that although there are some simulations for which the multiple imputation estimate has lower error than the listwise deletion estimate, in most cases the error for multiple imputation exceeds that of listwise deletion. For both multiple imputation and listwise deletion, MSE is larger as missingness is larger. However, the average MSE for multiple imputation is higher, even with highly informative proxies included in the multiple imputation data, than it is for listwise deletion. When missingness is low, the three estimators fare comparably (although listwise deletion usually has lower MSE anyway), but as missingness increases the relative performance of multiple imputation declines faster than listwise deletion.

Figure 2.

Simulation results by missingness.

I also vary the “informativeness” of the proxies. To do so, I fix $p=.2$ , deleting the lowest 20% of the observations for $X_{2}$ , but vary $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D702}}^{2}$ from 0 to .9. Recall that when $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D702}}^{2}=0$ then $X_{2}$ is a perfect linear combination of $U_{1}$ and $U_{2}$ . In this case, the data are actually MAR when these proxies are included in multiple imputation. As $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D702}}^{2}$ increases, the data depart ever more substantially from the MAR assumption.

Figure 3 below is identical to those above, but with $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D702}}^{2}$ as the X-axis. It shows that when $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D702}}^{2}=0$ , multiple imputation is superior to listwise deletion, as expected, because the data are MAR. However, with just the smallest bit of randomness, in this case as low as $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D702}}^{2}=.01$ , listwise deletion performs comparably to multiple imputation. As the data depart ever more systematically from MAR, the performance of multiple imputation relative to listwise deletion deteriorates as well.

Figure 3.

Simulation results by informativeness of the proxies.

3 Extended Simulations

In the Supplemental Appendix I introduce gradually more complicated data structures, and examine the relative performance of multiple imputation and listwise deletion using similar metrics. These extensions introduce probabilistic missingness in $X_{2}$ , NI missingness in $Y$ as well as $X_{2}$ , missingness in $X_{2}$ determined by the proxies $U_{1},U_{2}$ rather than $X_{2}$ itself, and allow $X_{2}$ to be correlated with $X_{1}$ . I also present an “exotic” multiple regression case with more independent variables, more kinds of missingness, discrete predictors, and non-normal distributions. In these simulations, multiple imputation performs relatively better than it does in the motivating case with missingness confined to $X_{2}$ only, although in no simulations does multiple imputation outperform listwise deletion in terms of bias, efficiency, or coverage. Here, I entertain one particularly important example: a cluster randomized experiment with heterogeneous treatment effects.

3.1 A Cluster Randomized Example

In this example, the setup is a cluster randomized experiment where groups $k$ are assigned to treatment $T_{k}$ , and the researcher believes that there is an individual level characteristic $D_{i}$ that differentiates the effect of $T$ . Consider a model of individuals nested within villages with treatment assignment at the village level, and differences by ethnic group within the village; or students nested within schools with treatment assignment at the school level, and differences by gender within the school. The model to be estimated is

(5) $$\begin{eqnarray}Y_{i}=\unicode[STIX]{x1D6FC}_{i}+\unicode[STIX]{x1D6FD}T_{k}+\unicode[STIX]{x1D6FE}D_{i}+\unicode[STIX]{x1D6FF}D_{i}\ast T_{k}+\unicode[STIX]{x1D719}_{k}+\unicode[STIX]{x1D716}_{i}.\end{eqnarray}$$

Fixed effects $\unicode[STIX]{x1D719}_{k}$ are included but they are uncorrelated with $T$ so their inclusion only increases efficiency. I set $\unicode[STIX]{x1D6FC}=1$ and $\unicode[STIX]{x1D6FD}=\unicode[STIX]{x1D6FE}=\unicode[STIX]{x1D6FF}=5$ for the simulations.

Missingness in $Y$ depends on the value of $D$ as well as the value of a fully observed pre-treatment covariate $X$ . In the case of individuals within villages, consider $Y$ to be a measure of political efficacy, and $X$ a measure of income. Perhaps data on political efficacy is less likely to be recorded from ethnic group A versus ethnic group B, but in both cases is more likely to be recorded from higher-income individuals. Missingness is nonrandom, but confined to $Y$ as a function of $D$ and $X$ :

(6) $$\begin{eqnarray}\displaystyle P(\text{Missing}\,|D=1) & = & \displaystyle \unicode[STIX]{x1D6F7}(X+p+.25)\end{eqnarray}$$

(7) $$\begin{eqnarray}\displaystyle P(\text{Missing}\,|D=0) & = & \displaystyle \unicode[STIX]{x1D6F7}(X+p-.05).\end{eqnarray}$$

As above, $p$ varies across simulations in order to increase the overall level of missingness.

Figure 4.

Simulation results from a cluster randomized experiment.

The results appear in Figure 4; in this example I compare performance for the parameters $\unicode[STIX]{x1D6FD}$ , $\unicode[STIX]{x1D6FE}$ , and $\unicode[STIX]{x1D6FF}$ . In this example multiple imputation has a lower MSE than listwise deletion when estimating $\unicode[STIX]{x1D6FD}$ but a higher MSE when estimating $\unicode[STIX]{x1D6FE}$ and $\unicode[STIX]{x1D6FF}$ . Coverage rates for $\unicode[STIX]{x1D6FD}$ are the opposite: low, but better for listwise deletion when estimating $\unicode[STIX]{x1D6FD}$ , and worse for listwise deletion when estimating $\unicode[STIX]{x1D6FE}$ and $\unicode[STIX]{x1D6FF}$ . Although neither listwise deletion nor multiple imputation performs particularly well by any metric, whether one prefers the former or the latter in this example depends both on the parameter of interest and the metric by which one compares them.

4 Application to Real Data

A final way to explore the effects of different strategies for handling missing data is to compare multiple imputation and listwise deletion using real-world data. To do so, I replicate the analysis of incumbency advantage in Indonesia’s legislative elections presented in Dettman, Pepinsky, and Pierskalla (Reference Dettman, Pepinsky and Pierskalla2017). I focus on Model 1 in Table 3 (p. 117), their baseline results that show the effects of incumbency and other variables on candidate vote share. Here, I confine my analysis to the relationship between incumbency, gender, and candidate age (as independent variables) and candidate vote share (as the dependent variable). A useful feature of Dettman, Pepinsky, and Pierskalla (Reference Dettman, Pepinsky and Pierskalla2017)’s data are that the dependent variable is highly skewed (median vote share is 0.48% and mean vote share is 1.19%) and the independent variable of interest is rare (433 out of 6043 or 7.2% of legislative candidates were incumbents). This mimics many real-world data problems, and thus allows me to explore a more realistic joint density using both listwise deletion and multiple imputation.

To simulate NI missingness, I generate a missing data situation in which data on incumbency status (the authors’ key theoretical variable of interest) is missing only among candidates who are not incumbents, and vote share is missing only among candidates who received the highest vote share. Specifically, I adopted the following procedure.

• ∙ I randomly select 25% of the observations in the dataset, and set Incumbency to missing when $\mathit{Incumbency}=0$ .

• ∙ I randomly select another 25% of the observations (with replacement, so some observations will be selected twice), and assign $\text{Vote}~\text{Share}$ to missing if the candidate’s true vote share was in the highest quartile in the dataset.

I then estimate $\hat{b}_{\text{Incumbency}}$ , $\hat{b}_{\text{Gender}}$ , and $\hat{b}_{\text{Age}}$ using both multiple imputation and listwise deletion. In this exercise I increase the number of imputed datasets substantially in the hopes of increasing the precision of the multiple imputation estimates: if 30% of the observations in a particular simulated dataset are missing, for example, I create and then analyze 30 imputed datasets. I repeat the above process 100 times, and display the results in Figure 5.

Figure 5.

Replication of Dettman, Pepinsky, and Pierskalla (Reference Dettman, Pepinsky and Pierskalla2017).

The “true” coefficients and T-statistics estimated in Dettman, Pepinsky, and Pierskalla (Reference Dettman, Pepinsky and Pierskalla2017), Table 3 are plotted as black dotted lines. We discover that in general, multiple imputation underestimates $\hat{b}_{\text{Incumbency}}$ by a factor of more than three—the median multiple imputation estimate is 0.87, whereas the true estimate is 2.66). Listwise deletion returns estimates of this key theoretical variable of interest that are roughly equivalent to the true estimate (the median listwise deletion estimate is 2.68), with T-statistics that are smaller but much closer to the true T-statistics.Footnote ⁴ In this case, given how large and statistically significant incumbency effects are in Indonesia’s legislative elections, an analysis using multiple imputation would still conclude that incumbency effects exist. But estimates of the magnitude of this effect will be much too small.

The same is not true for the estimates of $\hat{b}_{\text{Age}}$ . Here, the effect of candidate age is statistically significant at the 95% level for 94% of the simulations using listwise deletion but only 51% of the simulations when using multiple imputation. In this particular case, then, multiple imputation is much more likely to result in Type-2 error than listwise deletion. Looking to $\hat{b}_{\text{Gender}}$ , still another pattern emerges: here, both multiple imputation and listwise deletion perform about the same when estimating $\hat{b}_{\text{Gender}}$ , but now listwise deletion has T-statistics that are relatively more conservative.

The results from this simulated missing data exercise amplify the discussion above. In this particular example, multiple imputation produces results that are significantly worse in terms of bias and precision than listwise deletion for the theoretical variable of interest. For two (fully observed) covariates, results are in one case worse, and in one case better (at least in terms of precision). Of course, only knowing the true values from the fully observed data allows us to compare the performance of the two. But this exercise clearly reveals that multiple imputation can change substantive conclusions in real-world applications, and in this case, those conclusions will frequently be more misleading than those that come from listwise deletion.