Skip to main content
×
Home
    • Aa
    • Aa

Underreporting in Political Science Survey Experiments: Comparing Questionnaires to Published Results

  • Annie Franco (a1), Neil Malhotra (a2) and Gabor Simonovits (a3)
Abstract

The accuracy of published findings is compromised when researchers fail to report and adjust for multiple testing. Preregistration of studies and the requirement of preanalysis plans for publication are two proposed solutions to combat this problem. Some have raised concerns that such changes in research practice may hinder inductive learning. However, without knowing the extent of underreporting, it is difficult to assess the costs and benefits of institutional reforms. This paper examines published survey experiments conducted as part of the Time-sharing Experiments in the Social Sciences program, where the questionnaires are made publicly available, allowing us to compare planned design features against what is reported in published research. We find that: (1) 30% of papers report fewer experimental conditions in the published paper than in the questionnaire; (2) roughly 60% of papers report fewer outcome variables than what are listed in the questionnaire; and (3) about 80% of papers fail to report all experimental conditions and outcomes. These findings suggest that published statistical tests understate the probability of type I errors.

Copyright
Corresponding author
e-mail: neilm@stanford.edu (corresponding author)
Footnotes
Hide All

Author's note: Supplementary materials for this article are available on the Political Analysis Web site. Replication data are available on the Dataverse site for this article, http://dx.doi.org/10.7910/DVN/28766.

Footnotes
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

Richard G. Anderson 2013. Registration and replication: A comment. Political Analysis 21:3839.

Richard G. Anderson 2013. Registration and replication: A comment. Political Analysis 21:3839.

J. B. Asendorpf , M. Conner , F. De Fruyt , J. De Houwer , J. J. A. Denissen , K. Fiedler , S. Fiedler , D. C. Funder , R. Kliegl , B. A. Nosek , M. Perugini , B. W. Roberts , M. Schmitt , M. A. G. van Aken , H. Weber , and J. M. Wicherts 2013. Recommendations for increasing replicability in psychology. European Journal of Personality 27:108–19.

J. B. Asendorpf , M. Conner , F. De Fruyt , J. De Houwer , J. J. A. Denissen , K. Fiedler , S. Fiedler , D. C. Funder , R. Kliegl , B. A. Nosek , M. Perugini , B. W. Roberts , M. Schmitt , M. A. G. van Aken , H. Weber , and J. M. Wicherts 2013. Recommendations for increasing replicability in psychology. European Journal of Personality 27:108–19.

Katherine Casey , Rachel Glennerster , and Edward Miguel . 2012. Reshaping institutions: Evidence on aid impacts using a preanalysis plan. Quarterly Journal of Economics 127:17551812.

Katherine Casey , Rachel Glennerster , and Edward Miguel . 2012. Reshaping institutions: Evidence on aid impacts using a preanalysis plan. Quarterly Journal of Economics 127:17551812.

Christopher D. Chambers 2013. Registered reports: A new publishing initiative at Cortex. Cortex 49:609–10.

Christopher D. Chambers 2013. Registered reports: A new publishing initiative at Cortex. Cortex 49:609–10.

An-Wen Chan , and Douglas G. Altman 2005. Identifying outcome reporting bias in randomised trials on PubMed: Review of publications and survey of authors. British Medical Journal 330:753.

An-Wen Chan , and Douglas G. Altman 2005. Identifying outcome reporting bias in randomised trials on PubMed: Review of publications and survey of authors. British Medical Journal 330:753.

Annie Franco , Neil Malhotra , and Gabor Simonovits . 2014. Publication bias in the social sciences: Unlocking the file drawer. Science 345:15021505.

Annie Franco , Neil Malhotra , and Gabor Simonovits . 2014. Publication bias in the social sciences: Unlocking the file drawer. Science 345:15021505.

Andrew Gelman , Jennifer Hill , and Masanao Yajima . 2012. Why we (usually) don't have to worry about multiple comparisons. Journal of Research on Educational Effectiveness 5:189211.

Andrew Gelman , Jennifer Hill , and Masanao Yajima . 2012. Why we (usually) don't have to worry about multiple comparisons. Journal of Research on Educational Effectiveness 5:189211.

Andrew Gelman , and Francis Tuerlinckx . 2000. Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics 15:373–90.

Andrew Gelman , and Francis Tuerlinckx . 2000. Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics 15:373–90.

Alan Gerber , and Neil Malhotra . 2008. Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science 3:313–26.

Alan Gerber , and Neil Malhotra . 2008. Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science 3:313–26.

Macartan Humphreys , Raul Sanchez de la Sierra , and Peter van der Windt . 2013. Fishing, commitment, and communication: A proposal for comprehensive nonbinding research registration. Political Analysis 21:120.

Macartan Humphreys , Raul Sanchez de la Sierra , and Peter van der Windt . 2013. Fishing, commitment, and communication: A proposal for comprehensive nonbinding research registration. Political Analysis 21:120.

Gary King , Emmanuela Gakidou , Ninnala Ravishankar , Ryan T. Moore , Jason Lakin , Manett Vargas , Martha María Téllez-Rojo , Juan Eugenio Hernández Ávila , Mauricio Hernández Ávila , and Héctor Hernández Llamas . 2007. A “politically robust” experimental design for public policy evaluation, with application to the Mexican Universal Health Insurance Program. Journal of Policy Analysis and Management 26:479506.

Gary King , Emmanuela Gakidou , Ninnala Ravishankar , Ryan T. Moore , Jason Lakin , Manett Vargas , Martha María Téllez-Rojo , Juan Eugenio Hernández Ávila , Mauricio Hernández Ávila , and Héctor Hernández Llamas . 2007. A “politically robust” experimental design for public policy evaluation, with application to the Mexican Universal Health Insurance Program. Journal of Policy Analysis and Management 26:479506.

David D. Laitin 2013. Fisheries management. Political Analysis 21:4247.

David D. Laitin 2013. Fisheries management. Political Analysis 21:4247.

E. Miguel , C. Camerer , K. Casey , J. Cohen , K. M. Esterling , A. Gerber , R. Glennerster , D. P. Green , M. Humphreys , G. Imbens , D. Laitin , T. Madon , L. Nelson , B. A. Nosek , M. Petersen , R. Sedlmayr , J. P. Simmons , U. Simonsohn , and M. Van der Laan 2014. Promoting transparency in social science research. Science 343:3031.

E. Miguel , C. Camerer , K. Casey , J. Cohen , K. M. Esterling , A. Gerber , R. Glennerster , D. P. Green , M. Humphreys , G. Imbens , D. Laitin , T. Madon , L. Nelson , B. A. Nosek , M. Petersen , R. Sedlmayr , J. P. Simmons , U. Simonsohn , and M. Van der Laan 2014. Promoting transparency in social science research. Science 343:3031.

III Monogan , E. James 2013. A case for registering studies of political outcomes: An application in the 2010 House elections. Political Analysis 21:2137.

III Monogan , E. James 2013. A case for registering studies of political outcomes: An application in the 2010 House elections. Political Analysis 21:2137.

Santiago G. Moreno , Alex J. Sutton , Erick H. Turner , Keith R. Abrams , Nicola J. Cooper , Tom M. Palmer , and A. E. Ades 2009. Novel methods to deal with publication biases: Secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. British Medical Journal 339:b2981.

Santiago G. Moreno , Alex J. Sutton , Erick H. Turner , Keith R. Abrams , Nicola J. Cooper , Tom M. Palmer , and A. E. Ades 2009. Novel methods to deal with publication biases: Secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. British Medical Journal 339:b2981.

Kristin Rising , Peter Bacchetti , and Lisa Bero . 2008. Reporting bias in drug trials submitted to the Food and Drug Administration: Review of publication and presentation. PLoS Medicine 5:217.

Kristin Rising , Peter Bacchetti , and Lisa Bero . 2008. Reporting bias in drug trials submitted to the Food and Drug Administration: Review of publication and presentation. PLoS Medicine 5:217.

Joseph P. Simmons , Leif D. Nelson , and Uri Simonsohn . 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22:13591366.

Joseph P. Simmons , Leif D. Nelson , and Uri Simonsohn . 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22:13591366.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax
Type Description Title
PDF
Supplementary Materials

Franco et al. supplementary material
Appendix

 PDF (193 KB)
193 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 32 *
Loading metrics...

Abstract views

Total abstract views: 156 *
Loading metrics...

* Views captured on Cambridge Core between 4th January 2017 - 22nd September 2017. This data will be updated every 24 hours.