Skip to main content
    • Aa
    • Aa

Combining Double Sampling and Bounds to Address Nonignorable Missing Outcomes in Randomized Experiments

  • Alexander Coppock (a1), Alan S. Gerber (a1), Donald P. Green (a2) and Holger L. Kern (a3)

Missing outcome data plague many randomized experiments. Common solutions rely on ignorability assumptions that may not be credible in all applications. We propose a method for confronting missing outcome data that makes fairly weak assumptions but can still yield informative bounds on the average treatment effect. Our approach is based on a combination of the double sampling design and nonparametric worst-case bounds. We derive a worst-case bounds estimator under double sampling and provide analytic expressions for variance estimators and confidence intervals. We also propose a method for covariate adjustment using poststratification and a sensitivity analysis for nonignorable missingness. Finally, we illustrate the utility of our approach using Monte Carlo simulations and a placebo-controlled randomized field experiment on the effects of persuasion on social attitudes with survey-based outcome measures.

Corresponding author
* Email:
Hide All

Authors’ note: The authors thank Sebastian Bauhoff, Bill Berry, Chris Blattman, Jake Bowers, Matias Cattaneo, Kosuke Imai, Molly Offer-Westort, Rocio Titiunik, and participants of the 2013 Joint Statistical Meetings for very helpful comments on previous versions of this manuscript. The authors especially thank Peter Aronow for his contributions to previous versions of this paper. This research was approved by the Columbia University IRB (Protocol AAAP1312) and the empirical analyses were preregistered at (ID: 20150702AA). Easy-to-use software for the statistical programming language R that implements the methods described in this paper is available at The replication materials for all analyses reported here are available at

Contributing Editor: Kosuke Imai

Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

Ming-Wen An , Constantine E. Frangakis , Beverly S. Musick , and Constantin T. Yiannoutsos . 2009. The need for double-sampling designs in survival studies: An application to monitor PEPFAR. Biometrics 65(1):301306.

Peter M. Aronow , and Donald K. K. Lee . 2013. Interval estimation of population means under unknown but bounded probabilities of sample selection. Biometrika 100(1):235240.

Z. W. Birnbaum , and Monroe G. Sirken . 1950b. Bias due to non-availability in sampling surveys. Journal of the American Statistical Association 45(249):98111.

Mitali Das , Whitney K. Newey , and Francis Vella . 2003. Nonparametric estimation of sample selection models. Review of Economic Studies 70(1):3358.

W. A. Ericson 1967. Optimal sample design with nonresponse. Journal of the American Statistical Association 62(317):6378.

Gary Fraser , and Ru Yan . 2007. Guided multiple imputation of missing data. Epidemiology 18(2):246252.

David A. Freedman , and Jasjeet S. Sekhon . 2010. Endogeneity in probit response models. Political Analysis 18(2):138150.

Laura Lazarus Frankel , and D. Sunshine Hillygus . 2014. Looking beyond demographics: Panel attrition in the ANES and GSS. Political Analysis 22(3):336353.

Alan S. Gerber , Donald P. Green , Edward H. Kaplan , and Holger L. Kern . 2010. Baseline, placebo, and treatment: Efficient estimation for three-group experiments. Political Analysis 18(3):297315.

Robert J. Glynn , Nan M. Laird , and Donald B. Rubin . 1993. Multiple imputation in mixture models for non-ignorable nonresponse with follow-ups. Journal of the American Statistical Association 88(423):984993.

Jeffrey Grogger . 2012. Bounding the effects of social experiments: Accounting for attrition in administrative data. Evaluation Review 36(6):449474.

M. H. Hansen , and William N. Hurwitz . 1946. The problem of non-response in sample surveys. Journal of the American Statistical Association 41(236):517529.

Erin Hartman , Richard Grieve , Roland Ramsahai , and Jasjeet S. Sekhon . 2015. From sample average treatment effect to population average treatment effect on the treated: Combining experimental with observational studies to estimate population treatment effects. Journal of the Royal Statistical Society, Series A 178(3):757778.

James J. Heckman 1979. Sample selection bias as a specification error. Econometrica 47(1):153161.

Joel L. Horowitz , and Charles F. Manski . 1998. Censoring of outcomes and regressors due to survey nonresponse: Identification and estimation using weights and imputations. Journal of Econometrics 84(1):3758.

Joel L. Horowitz , and Charles F. Manski . 2000. Nonparametric analysis of randomized experiments with missing covariate and outcome data. Journal of the American Statistical Association 95(449):7784.

Kosuke Imai . 2008. Sharp bounds on the causal effects in randomized experiments with ‘truncation-by-death’. Statistics and Probability Letters 78(1):144149.

Kosuke Imai , Gary King , and Elizabeth A. Stuart . 2008. Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society: Series A (Statistics in Society) 171(2):481502.

Kosuke Imai . 2009. Statistical analysis of randomized experiments with non-ignorable missing binary outcomes: an application to a voting experiment. Journal of the Royal Statistical Society: Series C (Applied Statistics) 58(1):83104.

Guido W. Imbens , and Charles F. Manski . 2004. Confidence intervals for partially identified parameters. Econometrica 72(6):18451857.

Guido W. Imbens , and Donald B. Rubin . 2015. Causal inference for statistics, social, and biomedical sciences: An introduction . New York, NY: Cambridge University Press.

Paul Jenkins , Charles Scheim , Jen-Ting Wang , Roberta Reed , and Allan Green . 2004. Assessment of coverage rates and bias using double sampling methodology. Journal of Clinical Epidemiology 57(2):123130.

Joseph D. Y. Kang , and Joseph L. Schafer . 2007. Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science 22(4):523539.

G. M. Kaufman , and Benjamin King . 1973. A Bayesian analysis of nonresponse in dichotomous processes. Journal of the American Statistical Association 68(343):670678.

Luke Keele , and William Minozzi . 2013. How much is Minnesota like Wisconsin? Assumptions and counterfactuals in causal inference with observational data. Political Analysis 21(2):193216.

Holger L. Kern , Elizabeth A. Stuart , Jennifer Hill , and Donald P. Green . 2016. Assessing methods for generalizing experimental impact estimates to target populations. Journal of Research on Educational Effectiveness 9(1):103127.

David S. Lee 2010. Training, wages, and sample selection: Estimating sharp bounds on treatment effects. Review of Economic Studies 76(3):10711102.

Matthew Levendusky , and Neil Malhotra . 2016. Does media coverage of partisan polarization affect political attitudes? Political Communication 33(2):283301.

Roderick J. A. Little 2008. Selection and pattern-mixture models. In Longitudinal data analysis , ed. Garrett Fitzmaurice , Marie Davidian , Geert Verbeke , and Geert Molenberghs . ch. 18, Chapman & Hall/CRC.

Charles F. Manski , and Daniel S. Nagin . 1998. Bounding disagreements about treatment effects: A case study of sentencing and recidivism. Sociological Methodology 28(1):99137.

Sheena McConnell , Elizabeth A. Stuart , and Barbara Devaney . 2008. The truncation-by-death problem: What to do in an experimental evaluation when the outcome is not always defined. Evaluation Review 32(2):157186.

Walter R. Mebane Jr, and Paul Poast . 2013. Causal inference without ignorability: Identification with nonrandom assignment and missing treatment data. Political Analysis 21(2):233251.

Luke W. Miratrix , Jasjeet S. Sekhon , and Bin Yu . 2013. Adjusting treatment effect estimates by post-stratification in randomized experiments. Journal of the Royal Statistical Society, Series B 75(2):369396.

J. Neyman 1938. Contribution to the theory of sampling human populations. Journal of the American Statistical Association 33(201):101116.

Michael Peress . 2010. Correcting for survey nonresponse using variable response propensity. Journal of the American Statistical Association 105(492):14181430.

J. N. K. Rao 1973. On double sampling for stratification and analytical surveys. Biometrika 60(1):125133.

Paul R. Rosenbaum 2002. Observational studies . 2nd edn. New York, NY: Springer.

Yajuan Si , Jerome P. Reiter , and D. Sunshine Hillygus . 2014. Semi-parametric selection models for potentially non-ignorable attrition in panel studies with refreshment samples. Political Analysis 23(1):92112.

Elie Tamer . 2010. Partial identification in econometrics. Annual Review of Economics 2:167195.

Junni L. Zhang , and Donald B. Rubin . 2004. Estimation of causal effects via principal stratification when some outcomes are truncated by ‘death’. Journal of Educational and Behavioral Statistics 28(4):353368.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
Type Description Title
Supplementary Materials

Coppock supplementary material
Online Appendix

 Unknown (238 KB)
238 KB


Altmetric attention score

Full text views

Total number of HTML views: 9
Total number of PDF views: 136 *
Loading metrics...

Abstract views

Total abstract views: 355 *
Loading metrics...

* Views captured on Cambridge Core between 23rd February 2017 - 26th September 2017. This data will be updated every 24 hours.