Skip to main content Accessibility help
×
Home

A Note on Dropping Experimental Subjects who Fail a Manipulation Check

  • Peter M. Aronow (a1), Jonathon Baron (a1) and Lauren Pinson (a1)

Abstract

Dropping subjects based on the results of a manipulation check following treatment assignment is common practice across the social sciences, presumably to restrict estimates to a subpopulation of subjects who understand the experimental prompt. We show that this practice can lead to serious bias and argue for a focus on what is revealed without discarding subjects. Generalizing results developed in Zhang and Rubin (2003) and Lee (2009) to the case of multiple treatments, we provide sharp bounds for potential outcomes among those who would pass a manipulation check regardless of treatment assignment. These bounds may have large or infinite width, implying that this inferential target is often out of reach. As an application, we replicate Press, Sagan, and Valentino (2013) with a design that does not drop subjects that failed the manipulation check and show that the findings are likely stronger than originally reported. We conclude with suggestions for practice, namely alterations to the experimental design.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      A Note on Dropping Experimental Subjects who Fail a Manipulation Check
      Available formats
      ×

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      A Note on Dropping Experimental Subjects who Fail a Manipulation Check
      Available formats
      ×

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      A Note on Dropping Experimental Subjects who Fail a Manipulation Check
      Available formats
      ×

Copyright

Corresponding author

Footnotes

Hide All

Authors’ note: Peter M. Aronow is Assistant Professor, Departments of Political Science and Biostatistics, Yale University, 77 Prospect St., New Haven, CT 06520, USA (Email: peter.aronow@yale.edu). Jonathon Baron is Doctoral Student, Department of Political Science, Yale University, 115 Prospect St., New Haven, CT 06511, USA. Lauren Pinson is Doctoral Student, Department of Political Science, Yale University, 115 Prospect St., New Haven, CT 06511, USA. Author names are in alphabetical order and do not reflect relative contributions, which the authors consider to be equal. We thank Allan Dafoe, Don Green, Daniel Masterson, Ben Miller, Molly Offer-Westort, and Betsy Levy Paluck for helpful comments and conversations. Special thanks to Daryl Press, Scott Sagan, and Ben Valentino for generous assistance and materials in replication. We also thank the Yale Institution for Social and Policy Studies Summer Research Lunch group for valuable feedback. Replication data are available in Aronow, Baron, and Pinson (2018).

Contributing Editor: J. Grimmer

Footnotes

References

Hide All
Angrist, J. D., Imbens, G. W., and Rubin, D. B.. 1996. “Identification of Causal Effects using Instrumental Variables.” Journal of the American Statistical Association 91(434):444455.
Aronow, P. M., Baron, J., and Pinson, L.. 2018 “Replication Data for: A Note on Dropping Experimental Subjects who Fail a Manipulation Check.” https://doi.org/10.7910/DVN/GXXYMH, Harvard Dataverse, V1.
Berinsky, A. J., Margolis, M. F., and Sances, M. W.. 2014. “Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys.” American Journal of Political Science 58(3):739753.
Crawford, J. T., Brady, J. L., Pilanski, J. M., and Erny, H.. 2013. “Differential Effects of Right-Wing Authoritarianism and Social Dominance Orientation on Political Candidate Support: The Moderating Role of Message Framing.” Journal of Social and Political Psychology 1(1):528.
De Oliveira, P., Guimond, S., and Dambrun, M.. 2012. “Power and Legitimizing Ideologies in Hierarchy-Enhancing versus Hierarchy-Attenuating Environments.” Political Psychology 33(6):867885.
Gerber, A. S., and Green, D. P.. 2012. Field Experiments: Design, Analysis, and Interpretation . WW Norton.
Hauser, D. J., and Schwarz, N.. 2016. “Attentive Turkers: MTurk Participants Perform Better on Online Attention Checks than do Subject Pool Participants.” Behavior Research Methods 48(1):400407.
Hoffman, A. M., Agnew, C. R., VanderDrift, L. E., and Kulzick, R.. 2013. “Norms, Diplomatic Alternatives, and the Social Psychology of War Support.” Journal of Conflict Resolution 59(1):328.
Lee, D. S. 2009. “Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects.” The Review of Economic Studies 76(3):10711102.
Maoz, I. 2006, 10. “The Effect of News Coverage Concerning the Opponents’ Reaction to a Concession on its Evaluation in the Israeli–Palestinian Conflict.” The Harvard International Journal of Press/Politics 11(4):7088.
Oppenheimer, D. M., Meyvis, T., and Davidenko, N.. 2009. “Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power.” Journal of Experimental Social Psychology 45(4):867872.
Peer, E., Vosgerau, J., and Acquisti, A.. 2014. “Reputation as a Sufficient Condition for Data Quality on Amazon Mechanical Turk.” Behavior Research Methods 46(4):10231031.
Press, D. G., Sagan, S. D., and Valentino, B. A.. 2013. “Atomic Aversion: Experimental Evidence on Taboos, Traditions, and the Non-Use of Nuclear Weapons.” American Political Science Review 107(01):188206.
Rubin, D. B. 1980. “Comment.” Journal of the American Statistical Association 75(371):591593.
Small, D. A., Lerner, J. S., and Fischhoff, B.. 2006. “Emotion Priming and Attributions for Terrorism: Americans’ Reactions in a National Field Experiment.” Political Psychology 27(2):289298.
Turner, J. 2007. “The Messenger Overwhelming the Message: Ideological Cues and Perceptions of Bias in Television News.” Political Behavior 29(4):441464.
Wilson, T. D., Aronson, E., and Carlsmith, K.. 2010. “The Art of Laboratory Experimentation.” In Handbook of Social Psychology , 5th edn, edited by Fiske, S. T., Gilbert, D. T., Lindzey, G., and Jongsma, A. E., 5181. Hoboken, NJ: Wiley.
Zhang, J. L., and Rubin, D. B.. 2003. “Estimation of Causal Effects via Principal Stratification When Some Outcomes are Truncated by ‘Death.’” Journal of Educational and Behavioral Statistics 28(4):353368.
MathJax
MathJax is a JavaScript display engine for mathematics. For more information see http://www.mathjax.org.

Keywords

Type Description Title
UNKNOWN
Supplementary materials

Aronow et al. supplementary material
Aronow et al. supplementary material 1

 Unknown (11 KB)
11 KB

A Note on Dropping Experimental Subjects who Fail a Manipulation Check

  • Peter M. Aronow (a1), Jonathon Baron (a1) and Lauren Pinson (a1)

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed