Skip to main content
×
Home
    • Aa
    • Aa

Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration

  • Macartan Humphreys (a1), Raul Sanchez de la Sierra (a2) and Peter van der Windt (a3)
Abstract

Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this “fishing” problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete “mock report” of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but nonbinding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but nonbinding registration are discussed, the principal advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.

Copyright
Corresponding author
e-mail: mh2245@columbia.edu (corresponding author)
Footnotes
Hide All

Authors' note: Our thanks to Ali Cirone, Andy Gelman, Grant Gordon, Alan Jacobs, Ryan Moore, and Ferran Elias Moreno for helpful comments. Our thanks to the Population Center at Columbia for providing access to the High Performance Computing (HPC) Cluster. This research was undertaken in the context of a field experiment in DRC; we thank the International Rescue Committee and CARE International for their partnership in that research and the International Initiative for Impact Evaluation (3IE) for their support. M. H. thanks the Trudeau Foundation for support while this work was undertaken. Replication data and code for tables and figures can be found at http://hdl.handle.net/1902.1/18182. Supplementary materials for this article are available on the Political Analysis Web site.

Footnotes
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

E. Callaway 2011. Report finds massive fraud at Dutch universities. Nature 479: 15.

E. Callaway 2011. Report finds massive fraud at Dutch universities. Nature 479: 15.

C. D. De Angelis , J. M. Drazen , F. A. Frizelle , C. Haug , J. Hoey , R. Horton , S. Kotzin , C. Laine , A. Marusic , A. J. P. Overbeke , T. V. Schroeder , H. C. Sox , and M. B. Van Der Weyden 2004. Clinical trial registration: A statement from the International Committee of Medical Journal Editors. New England Journal of Medicine 351: 1250–1.

C. D. De Angelis , J. M. Drazen , F. A. Frizelle , C. Haug , J. Hoey , R. Horton , S. Kotzin , C. Laine , A. Marusic , A. J. P. Overbeke , T. V. Schroeder , H. C. Sox , and M. B. Van Der Weyden 2004. Clinical trial registration: A statement from the International Committee of Medical Journal Editors. New England Journal of Medicine 351: 1250–1.

C. D. De Angelis , J. M. Drazen , F. A. Frizelle , C. Haug , J. Hoey , R. Horton , S. Kotzin , C. Laine , A. Marušic , A. J. P. Overbeke , T. V. Schroeder , H. C. Sox , and M. B. V. D. Weyden 2005. Is this clinical trial fully registered? A statement from the International Committee of Medical Journal Editors. Canadian Medical Association Journal 172: 1700–2.

C. D. De Angelis , J. M. Drazen , F. A. Frizelle , C. Haug , J. Hoey , R. Horton , S. Kotzin , C. Laine , A. Marušic , A. J. P. Overbeke , T. V. Schroeder , H. C. Sox , and M. B. V. D. Weyden 2005. Is this clinical trial fully registered? A statement from the International Committee of Medical Journal Editors. Canadian Medical Association Journal 172: 1700–2.

J. D. Fearon , M. Humphreys , and J. M. Weinstein 2009. Can development aid contribute to social cohesion after Civil War? Evidence from a field experiment in post-conflict Liberia. American Economic Review: Papers & Proceedings 99: 287–91.

J. D. Fearon , M. Humphreys , and J. M. Weinstein 2009. Can development aid contribute to social cohesion after Civil War? Evidence from a field experiment in post-conflict Liberia. American Economic Review: Papers & Proceedings 99: 287–91.

A. Gelman , and F. Tuerlinckx 2000. Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics 15: 373–90.

A. Gelman , and F. Tuerlinckx 2000. Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics 15: 373–90.

A. S. Gerber , D. P. Green , and D. Nickerson 2001. Testing for publication bias in political science. Political Analysis 9: 385–92.

A. S. Gerber , D. P. Green , and D. Nickerson 2001. Testing for publication bias in political science. Political Analysis 9: 385–92.

A. Gerber , and N. Malhotra 2008. Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science 3: 313–26.

A. Gerber , and N. Malhotra 2008. Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science 3: 313–26.

E. L. Glaeser 2006. Researcher incentives and empirical methods. Working paper (DP2122).

E. L. Glaeser 2006. Researcher incentives and empirical methods. Working paper (DP2122).

J. P. A. Ioannidis 2005. Why most published research findings are false. PLoS Medicine 2: 696701.

J. P. A. Ioannidis 2005. Why most published research findings are false. PLoS Medicine 2: 696701.

G. King , E. Gakidou , K. Imai , J. Lakin , R. T. Moore , C. Nall , N. Ravishankar , M. Vargas , M. M. Téllez-Rojo , J. E. H. Avila , M. H. Avila , and H. H. Llamas 2009. Public policy for the poor? A randomized assessment of the Mexican Universal Health Insurance Program. Lancet 373: 1447–54.

G. King , E. Gakidou , K. Imai , J. Lakin , R. T. Moore , C. Nall , N. Ravishankar , M. Vargas , M. M. Téllez-Rojo , J. E. H. Avila , M. H. Avila , and H. H. Llamas 2009. Public policy for the poor? A randomized assessment of the Mexican Universal Health Insurance Program. Lancet 373: 1447–54.

J. R. Kling , J. B. Liebman , and L. F. Katz 2007. Experimental analysis of neighborhood effects. Econometrics 75: 83119.

J. R. Kling , J. B. Liebman , and L. F. Katz 2007. Experimental analysis of neighborhood effects. Econometrics 75: 83119.

C. Laine , R. Horton , C. D. Deangelis , J. M. Drazen , F. A. Frizelle , F. Godlee , C. Haug , P. C. Hébert , and S. Kotzin 2007. Clinical trial registration: Looking back and moving ahead. Canadian Medical Association Journal 177: 78.

C. Laine , R. Horton , C. D. Deangelis , J. M. Drazen , F. A. Frizelle , F. Godlee , C. Haug , P. C. Hébert , and S. Kotzin 2007. Clinical trial registration: Looking back and moving ahead. Canadian Medical Association Journal 177: 78.

Lancet. 2010. Should protocols for observational research be registered? Lancet 375: 348.

Lancet. 2010. Should protocols for observational research be registered? Lancet 375: 348.

S. Mathieu , I. Boutron , D. Moher , D. G. Altman , and P. Ravaud 2009. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 302: 977–84.

S. Mathieu , I. Boutron , D. Moher , D. G. Altman , and P. Ravaud 2009. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 302: 977–84.

L. Reveiz , A.-W. Chan , K. Krleza-Jerić , C. E. Granados , M. Pinart , I. Etxeandia , D. Rada , M. Martinez , X. Bonfill , and A. F. Cardona 2010. Reporting of methodologic information on trial registries for quality assessment: A study of trial records retrieved from the WHO search portal. PloS One 5: 16.

L. Reveiz , A.-W. Chan , K. Krleza-Jerić , C. E. Granados , M. Pinart , I. Etxeandia , D. Rada , M. Martinez , X. Bonfill , and A. F. Cardona 2010. Reporting of methodologic information on trial registries for quality assessment: A study of trial records retrieved from the WHO search portal. PloS One 5: 16.

J. P. Simmons , L. D. Nelson , and U. Simonsohn 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22: 1359–66.

J. P. Simmons , L. D. Nelson , and U. Simonsohn 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22: 1359–66.

T. D. Sterling 1959. Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association 54: 3034.

T. D. Sterling 1959. Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association 54: 3034.

H. White 2000. A reality check for data snooping. Econometrica 68: 1097–126.

H. White 2000. A reality check for data snooping. Econometrica 68: 1097–126.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax
Type Description Title
UNKNOWN
Supplementary Materials

Humphreys et al. supplementary material
Appendix A

 Unknown (10 KB)
10 KB
UNKNOWN
Supplementary Materials

Humphreys et al. supplementary material
Appendix C

 Unknown (11 KB)
11 KB
UNKNOWN
Supplementary Materials

Humphreys et al. supplementary material
Appendix B

 Unknown (15 KB)
15 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 27 *
Loading metrics...

Abstract views

Total abstract views: 84 *
Loading metrics...

* Views captured on Cambridge Core between 4th January 2017 - 17th August 2017. This data will be updated every 24 hours.