Hostname: page-component-5d84bcc8dc-4dnnq Total loading time: 0 Render date: 2026-08-11T06:45:05.369Z Has data issue: false hasContentIssue false
  • English
  • Français

Combining Double Sampling and Bounds to Address Nonignorable Missing Outcomes in Randomized Experiments

Published online by Cambridge University Press:  23 February 2017

Alexander Coppock ,
Alan S. Gerber ,
Donald P. Green  and
Holger L. Kern
Alexander Coppock
Affiliation:
Department of Political Science, Yale University, New Haven, CT 06520, USA
Alan S. Gerber
Affiliation:
Department of Political Science, Yale University, New Haven, CT 06520, USA
Donald P. Green
Affiliation:
Department of Political Science, Columbia University, New York, NY 10027, USA
Holger L. Kern*
Affiliation:
Department of Political Science, Florida State University, Tallahassee, FL 32306, USA. Email: hkern@fsu.edu
*
*Email: hkern@fsu.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Missing outcome data plague many randomized experiments. Common solutions rely on ignorability assumptions that may not be credible in all applications. We propose a method for confronting missing outcome data that makes fairly weak assumptions but can still yield informative bounds on the average treatment effect. Our approach is based on a combination of the double sampling design and nonparametric worst-case bounds. We derive a worst-case bounds estimator under double sampling and provide analytic expressions for variance estimators and confidence intervals. We also propose a method for covariate adjustment using poststratification and a sensitivity analysis for nonignorable missingness. Finally, we illustrate the utility of our approach using Monte Carlo simulations and a placebo-controlled randomized field experiment on the effects of persuasion on social attitudes with survey-based outcome measures.

Information

Type
Articles
Information
Political Analysis , Volume 25 , Issue 2 , April 2017 , pp. 188 - 206
Copyright
Copyright © The Author(s) 2017. Published by Cambridge University Press on behalf of the Society for Political Methodology. 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Authors’ note: The authors thank Sebastian Bauhoff, Bill Berry, Chris Blattman, Jake Bowers, Matias Cattaneo, Kosuke Imai, Molly Offer-Westort, Rocio Titiunik, and participants of the 2013 Joint Statistical Meetings for very helpful comments on previous versions of this manuscript. The authors especially thank Peter Aronow for his contributions to previous versions of this paper. This research was approved by the Columbia University IRB (Protocol AAAP1312) and the empirical analyses were preregistered at egap.org (ID: 20150702AA). Easy-to-use software for the statistical programming language R that implements the methods described in this paper is available at github.com/acoppock/attrition. The replication materials for all analyses reported here are available at dx.doi.org/10.7910/DVN/AQB4MP.

Contributing Editor: Kosuke Imai

References

An, Ming-Wen, Frangakis, Constantine E., Musick, Beverly S., and Yiannoutsos, Constantin T.. 2009. The need for double-sampling designs in survival studies: An application to monitor PEPFAR. Biometrics 65(1):301–306.Google Scholar
Aronow, Peter M., and Lee, Donald K. K.. 2013. Interval estimation of population means under unknown but bounded probabilities of sample selection. Biometrika 100(1):235–240.Google Scholar
Baird, Sarah, Hamory, Joan, and Miguel, Edward. 2008. Tracking, attrition and data quality in the Kenyan life panel survey round 1 (KLPS-1). Working Paper.Google Scholar
Birnbaum, Z. W., and Sirken, Monroe G.. 1950a. On the total error due to non-interview and to random sampling. International Journal of Opinion and Attitude Research 4:179–191.Google Scholar
Birnbaum, Z. W., and Sirken, Monroe G.. 1950b. Bias due to non-availability in sampling surveys. Journal of the American Statistical Association 45(249):98–111.Google Scholar
Cassel, C. M., Särndal, C. E., and Wretman, J. H.. 1983. Some uses of statistical models in connection with the non-response problem. In Incomplete data in sample surveys III. Symposium on incomplete data, proceedings , ed. Madow, W. G. and Olkin, I.. New York, NY: Academic Press, pp. 143–160.Google Scholar
Cochran, William G., Mosteller, Frederick, and Tukey, John W.. 1954. Statistical problems of the Kinsey report on sexual behavior in the human male . Washington, DC: American Statistical Association.Google Scholar
Cochran, William G. 1977. Sampling techniques . 3rd edn. New York, NY: John Wiley & Sons.Google Scholar
Coppock, Alexander, Gerber, Alan S., Green, Donald P., and Kern, Holger L.. 2016. Replication Data for: Combining double sampling and bounds to address non-ignorable missing outcomes in randomized experiments. Harvard Dataverse. dx.doi.org/10.7910/DVN/AQB4MP.Google Scholar
Das, Mitali, Newey, Whitney K., and Vella, Francis. 2003. Nonparametric estimation of sample selection models. Review of Economic Studies 70(1):33–58.Google Scholar
DiNardo, John, McCrary, Justin, and Sanbonmatsu, Lisa. 2006. Constructive proposals for dealing with attrition: An empirical example. Working Paper.Google Scholar
Ericson, W. A. 1967. Optimal sample design with nonresponse. Journal of the American Statistical Association 62(317):63–78.Google Scholar
Fraser, Gary, and Yan, Ru. 2007. Guided multiple imputation of missing data. Epidemiology 18(2):246–252.Google Scholar
Freedman, David, Pisani, Robert, and Purves, Roger. 2007. Statistics . 4th edn. New York, NY: W. W. Norton & Company.Google Scholar
Freedman, David A., and Sekhon, Jasjeet S.. 2010. Endogeneity in probit response models. Political Analysis 18(2):138–150.Google Scholar
Frankel, Laura Lazarus, and Hillygus, D. Sunshine. 2014. Looking beyond demographics: Panel attrition in the ANES and GSS. Political Analysis 22(3):336–353.Google Scholar
Gerber, Alan S., Green, Donald P., Kaplan, Edward H., and Kern, Holger L.. 2010. Baseline, placebo, and treatment: Efficient estimation for three-group experiments. Political Analysis 18(3):297–315.Google Scholar
Gerber, Alan S., and Green, Donald P.. 2012. Field experiments: Design, analysis, and interpretation . New York, NY: W. W. Norton & Company.Google Scholar
Glynn, Robert J., Laird, Nan M., and Rubin, Donald B.. 1993. Multiple imputation in mixture models for non-ignorable nonresponse with follow-ups. Journal of the American Statistical Association 88(423):984–993.Google Scholar
Grogger, Jeffrey. 2012. Bounding the effects of social experiments: Accounting for attrition in administrative data. Evaluation Review 36(6):449–474.Google Scholar
Hansen, M. H., and Hurwitz, William N.. 1946. The problem of non-response in sample surveys. Journal of the American Statistical Association 41(236):517–529.Google Scholar
Hartman, Erin, Grieve, Richard, Ramsahai, Roland, and Sekhon, Jasjeet S.. 2015. From sample average treatment effect to population average treatment effect on the treated: Combining experimental with observational studies to estimate population treatment effects. Journal of the Royal Statistical Society, Series A 178(3):757–778.Google Scholar
Heckman, James J. 1979. Sample selection bias as a specification error. Econometrica 47(1):153–161.Google Scholar
Horowitz, Joel L., and Manski, Charles F.. 1998. Censoring of outcomes and regressors due to survey nonresponse: Identification and estimation using weights and imputations. Journal of Econometrics 84(1):37–58.Google Scholar
Horowitz, Joel L., and Manski, Charles F.. 2000. Nonparametric analysis of randomized experiments with missing covariate and outcome data. Journal of the American Statistical Association 95(449):77–84.Google Scholar
Imai, Kosuke. 2008. Sharp bounds on the causal effects in randomized experiments with ‘truncation-by-death’. Statistics and Probability Letters 78(1):144–149.Google Scholar
Imai, Kosuke, King, Gary, and Stuart, Elizabeth A.. 2008. Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society: Series A (Statistics in Society) 171(2):481–502.Google Scholar
Imai, Kosuke. 2009. Statistical analysis of randomized experiments with non-ignorable missing binary outcomes: an application to a voting experiment. Journal of the Royal Statistical Society: Series C (Applied Statistics) 58(1):83–104.Google Scholar
Imbens, Guido W., and Manski, Charles F.. 2004. Confidence intervals for partially identified parameters. Econometrica 72(6):1845–1857.Google Scholar
Imbens, Guido W., and Rubin, Donald B.. 2015. Causal inference for statistics, social, and biomedical sciences: An introduction . New York, NY: Cambridge University Press.Google Scholar
Jenkins, Paul, Scheim, Charles, Wang, Jen-Ting, Reed, Roberta, and Green, Allan. 2004. Assessment of coverage rates and bias using double sampling methodology. Journal of Clinical Epidemiology 57(2):123–130.Google Scholar
Kang, Joseph D. Y., and Schafer, Joseph L.. 2007. Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science 22(4):523–539.Google Scholar
Kaufman, G. M., and King, Benjamin. 1973. A Bayesian analysis of nonresponse in dichotomous processes. Journal of the American Statistical Association 68(343):670–678.Google Scholar
Keele, Luke, and Minozzi, William. 2013. How much is Minnesota like Wisconsin? Assumptions and counterfactuals in causal inference with observational data. Political Analysis 21(2):193–216.Google Scholar
Kern, Holger L., Stuart, Elizabeth A., Hill, Jennifer, and Green, Donald P.. 2016. Assessing methods for generalizing experimental impact estimates to target populations. Journal of Research on Educational Effectiveness 9(1):103–127.Google Scholar
Lee, David S. 2010. Training, wages, and sample selection: Estimating sharp bounds on treatment effects. Review of Economic Studies 76(3):1071–1102.Google Scholar
Levendusky, Matthew, and Malhotra, Neil. 2016. Does media coverage of partisan polarization affect political attitudes? Political Communication 33(2):283–301.Google Scholar
Little, Roderick J. A., and Rubin, Donald B.. 2002. Statistical analysis with missing data . 2nd edn. New York, NY: Wiley-Interscience.Google Scholar
Little, Roderick J. A. 2008. Selection and pattern-mixture models. In Longitudinal data analysis , ed. Fitzmaurice, Garrett, Davidian, Marie, Verbeke, Geert, and Molenberghs, Geert. ch. 18, Chapman & Hall/CRC.Google Scholar
Lohr, Sharon L. 2010. Sampling: Design and analysis . 2nd edn. Pacific Grove, CA: Brooks/Cole.Google Scholar
Manski, Charles F. 1990. Nonparametric bounds on treatment effects. American Economic Review Papers and Proceedings 80(2):319–323.Google Scholar
Manski, Charles F. 1995. Identification problems in the social sciences . Cambridge, MA: Harvard University Press.Google Scholar
Manski, Charles F. 2007. Identification for prediction and decision . Cambridge, MA: Harvard University Press.Google Scholar
Manski, Charles F., and Nagin, Daniel S.. 1998. Bounding disagreements about treatment effects: A case study of sentencing and recidivism. Sociological Methodology 28(1):99–137.Google Scholar
Manski, Charles F., and Pepper, John V.. 2011. Deterrence and the death penalty: Partial identification analysis using repeated cross sections. NBER Working Paper 17455.Google Scholar
McConnell, Sheena, Stuart, Elizabeth A., and Devaney, Barbara. 2008. The truncation-by-death problem: What to do in an experimental evaluation when the outcome is not always defined. Evaluation Review 32(2):157–186.Google Scholar
Mebane, Walter R. Jr, and Poast, Paul. 2013. Causal inference without ignorability: Identification with nonrandom assignment and missing treatment data. Political Analysis 21(2):233–251.Google Scholar
Miratrix, Luke W., Sekhon, Jasjeet S., and Yu, Bin. 2013. Adjusting treatment effect estimates by post-stratification in randomized experiments. Journal of the Royal Statistical Society, Series B 75(2):369–396.Google Scholar
Neyman, J. 1938. Contribution to the theory of sampling human populations. Journal of the American Statistical Association 33(201):101–116.Google Scholar
Peress, Michael. 2010. Correcting for survey nonresponse using variable response propensity. Journal of the American Statistical Association 105(492):1418–1430.Google Scholar
Rao, J. N. K. 1973. On double sampling for stratification and analytical surveys. Biometrika 60(1):125–133.Google Scholar
Robins, James M., Rotnitzky, Andrea, and Scharfstein, Daniel O.. 1999. Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models. In Statistical models in epidemiology , ed. Halloran, M. E. and Berry, D.. New York, NY: Springer, pp. 1–92.Google Scholar
Rosenbaum, Paul R. 2002. Observational studies . 2nd edn. New York, NY: Springer.Google Scholar
Rosenbaum, P. R., and Rubin, Donald B.. 1983. Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome. Journal of the Royal Statistical Society, Series B 45(2):212–218.Google Scholar
Sharma, R., Gupta, M., and Kapoor, G.. 2010. Some better bounds on the variance with applications. Journal of Mathematical Inequalities 4(3):355–363.Google Scholar
Si, Yajuan, Reiter, Jerome P., and Hillygus, D. Sunshine. 2014. Semi-parametric selection models for potentially non-ignorable attrition in panel studies with refreshment samples. Political Analysis 23(1):92–112.Google Scholar
Stoye, Jörg. 2009. More on confidence intervals for partially identified parameters. Econometrica 77(4):1299–1315.Google Scholar
Tamer, Elie. 2010. Partial identification in econometrics. Annual Review of Economics 2:167–195.Google Scholar
Zhang, Junni L., and Rubin, Donald B.. 2004. Estimation of causal effects via principal stratification when some outcomes are truncated by ‘death’. Journal of Educational and Behavioral Statistics 28(4):353–368.Google Scholar
Supplementary material: File

Coppock supplementary material

Online Appendix

Download Coppock supplementary material(File)
File 238.5 KB