Skip to main content
×
×
Home

Worth Weighting? How to Think About and Use Weights in Survey Experiments

  • Luke W. Miratrix (a1), Jasjeet S. Sekhon (a2), Alexander G. Theodoridis (a3) and Luis F. Campos (a4)
Abstract

The popularity of online surveys has increased the prominence of using sampling weights to enhance claims of representativeness. Yet, much uncertainty remains regarding how these weights should be employed in survey experiment analysis: should they be used? If so, which estimators are preferred? We offer practical advice, rooted in the Neyman–Rubin model, for researchers working with survey experimental data. We examine simple, efficient estimators, and give formulas for their biases and variances. We provide simulations that examine these estimators as well as real examples from experiments administered online through YouGov. We find that for examining the existence of population treatment effects using high-quality, broadly representative samples recruited by top online survey firms, sample quantities, which do not rely on weights, are often sufficient. We found that sample average treatment effect (SATE) estimates did not appear to differ substantially from their weighted counterparts, and they avoided the substantial loss of statistical power that accompanies weighting. When precise estimates of population average treatment effects (PATE) are essential, we analytically show poststratifying on survey weights and/or covariates highly correlated with outcomes to be a conservative choice. While we show substantial gains in simulations, we find limited evidence of them in practice.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Worth Weighting? How to Think About and Use Weights in Survey Experiments
      Available formats
      ×
      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Worth Weighting? How to Think About and Use Weights in Survey Experiments
      Available formats
      ×
      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Worth Weighting? How to Think About and Use Weights in Survey Experiments
      Available formats
      ×
Copyright
Corresponding author
Footnotes
Hide All

Authors’ note: For research support and funding of the experimental studies analyzed here, Theodoridis thanks the University of California, Merced, the National Science Foundation Awards #1430505, #1225750 and #0924191, the Empirical Implications of Theoretical Models program and the Integrative Graduate Education and Research Traineeship program, and the Berkeley Institute of Governmental Studies and its Mike Synar Graduate Research Fellowship; Miratrix thanks the Institute of Education Sciences, U.S. Department of Education, through grant R305D150040; and Sekhon thanks Office of Naval Research (ONR) grants N00014-15-1-2367 and N00014-17-1-2176. The opinions expressed are those of the authors and do not represent views of the Institute, the U.S. Department of Education, ONR, or any of the organizations mentioned above. All studies were approved or deemed exempt by the appropriate institutional research ethics committees (University of California, Berkeley or University of California, Merced). Please send comments to lmiratrix@stat.harvard.edu, sekhon@berkeley.edu, atheodoridis@ucmerced.edu, or luiscampos@g.harvard.edu. Replication materials are publicly available, see Miratrix et al. (2017).

Contributing Editor: R. Michael Alvarez

Footnotes
References
Hide All
Aronow, P. M., and Middleton, J. A.. 2013. A class of unbiased estimators of the average treatment effect in randomized experiments. Journal of Causal Inference 1(1):135154.
Berinsky, A. J., Huber, G. A., and Lenz, G. S.. 2012. Evaluating online labor markets for experimental research: Amazon.com’s mechanical turk. Political Analysis 20(3):351368.
Cochran, W. G. 1977. Sampling techniques . 3rd edn. New York: John Wiley and Sons.
Cole, S. R., and Stuart, E. A.. 2010. Generalizing evidence from randomized clinical trials to target populations: the ACTG 320 trial. American Journal of Epidemiology 172(1):107115.
Ding, P., Feller, A., and Miratrix, L.. 2018. Decomposing treatment effect variation. Journal of the American Statistical Association , doi:10.1080/01621459.2017.1407322.
DuGoff, E. H., Schuler, M., and Stuart, E. A.. 2013. Generalizing observational study results: applying propensity score methods to complex surveys. Health Services Research 49(1):284303.
Franco, A., Simonovits, G., Malhotra, N., and Zigerell, L. J.. 2017. Developing standards for post-hoc weighting in population-based survey experiments. Journal of Experimental Political Science 4(2):161172.
Gaines, B. J., Kuklinski, J. H., and Quirk, P. J.. 2007. The logic of the survey experiment reexamined. Political Analysis 15(1):120.
Groves, R. M., Fowler, F. J. Jr., Couper, M. P., Lepkowski, J. M., Singer, E., and Tourangeau, R.. 2009. Survey methodology . Hoboken, NJ: John Wiley & Sons.
Hàjek, J. 1958. On the theory of ratio estimates. Applied Mathematics 3:384398.
Hartman, E., Grieve, R., Ramsahai, R., and Sekhon, J.. 2015. From sample average treatment effect to population average treatment effect on the treated: combining experimental with observational studies to estimate population treatment effects. Journal of the Royal Statistical Society: Series A (Statistics in Society) 3:757778.
Hirano, K., Imbens, G., and Ridder, G.. 2000. Efficient estimation of average treatment effects using the estimated propensity score. Econometrica 71(4):11611189.
Horvitz, D. G., and Thompson, D. J.. 1952. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association 47(260):663685.
Imai, K., King, G., and Stuart, E. A.. 2008. Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society. Series A (Statistics in Society) 171(2):481502.
Lin, W. 2013. Agnostic notes on regression adjustments to experimental data: reexamining freedman’s critique. The Annals of Applied Statistics 7(1):295318.
Miratrix, L. W., Sekhon, J. S., Theodoridis, A. G., and Campos, L. F.. 2017. Replication data for: Worth weighting? How to think about and use weights in survey experiments, https://doi.org/10.7910/DVN/52UGJT, Harvard Dataverse, V1.
Miratrix, L. W., Sekhon, J. S., and Yu, B.. 2013. Adjusting treatment effect estimates by post-stratification in randomized experiments. Journal of the Royal Statistical Society Series B 75(2):369396.
Mutz, D. C. 2011. Population-Based Survey Experiments . Princeton, NJ: Princeton University Press.
O’Muircheartaigh, C., and Hedges, L. V.. 2014. Generalizing from unrepresentative experiments: a stratified propensity score approach. Journal of the Royal Statistical Society: Series C (Applied Statistics) 63(2):195210.
Rivers, D. 2006. Sample matching: representative sampling from internet panels. Polimetrix White Paper Series .
Santoso, L. P., Stein, R., and Stevenson, R.. 2016. Survey experiments with google consumer surveys: promise and pitfalls for academic research in social science. Political Analysis 24(3):356373.
Särndal, C.-E., Swensson, B., and Wretman, J.. 2003. Model assisted survey sampling . New York: Springer-Verlag.
Si, Y., Pillai, N. S., and Gelman, A.. 2015. Bayesian nonparametric weighted sampling inference. Bayesian Analysis 10(3):605625.
Sniderman, P. M. 2011. The logic and design of the survey experiment. In Cambridge handbook of experimental political science , ed. Druckman, J. N., Green, D. P., Kuklinski, J. H., and Lupia, A.. Cambridge: Cambridge University Press, p. 102.
Splawa-Neyman, J., Dabrowska, D. M., and Speed, T. P.. 1990. On the application of probability theory to agricultural experiments. Essay on principles. Section 9. Statistical Science 5(4):465472.
Stuart, E. A., Cole, S. R., Bradshaw, C. P., and Leaf, P. J.. 2010. The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society (Series A) 174(3):118.
Tipton, E. 2013. Improving generalizations from experiments using propensity score subclassification: assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics 38(3):239266.
Weiss, M. J., Bloom, H. S., Verbitsky-Savitz, N., Gupta, H., Vigil, A. E., and Cullinan, D. N.. 2017. How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness 10(4):843876.
Wood, J. 2008. On the covariance between related Horvitz–Thompson estimators. Journal of Official Statistics 24(1):5378.
Zheng, H., and Little, R. J. A.. 2003. Penalized spline model-based estimation of the finite populations total from probability-proportional-to-size samples. Journal of Official Statistics 19(2):99117.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax

Keywords

Type Description Title
PDF
Supplementary materials

Miratrix et al. supplementary material
Miratrix et al. supplementary material 1

 PDF (396 KB)
396 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 26
Total number of PDF views: 445 *
Loading metrics...

Abstract views

Total abstract views: 1789 *
Loading metrics...

* Views captured on Cambridge Core between 25th May 2018 - 19th August 2018. This data will be updated every 24 hours.