Skip to main content
×
×
Home

Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration

  • Macartan Humphreys (a1), Raul Sanchez de la Sierra (a2) and Peter van der Windt (a3)
Abstract

Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this “fishing” problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete “mock report” of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but nonbinding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but nonbinding registration are discussed, the principal advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.

Copyright
Corresponding author
e-mail: mh2245@columbia.edu (corresponding author)
Footnotes
Hide All

Authors' note: Our thanks to Ali Cirone, Andy Gelman, Grant Gordon, Alan Jacobs, Ryan Moore, and Ferran Elias Moreno for helpful comments. Our thanks to the Population Center at Columbia for providing access to the High Performance Computing (HPC) Cluster. This research was undertaken in the context of a field experiment in DRC; we thank the International Rescue Committee and CARE International for their partnership in that research and the International Initiative for Impact Evaluation (3IE) for their support. M. H. thanks the Trudeau Foundation for support while this work was undertaken. Replication data and code for tables and figures can be found at http://hdl.handle.net/1902.1/18182. Supplementary materials for this article are available on the Political Analysis Web site.

Footnotes
References
Hide All
Beath, A., Christia, F., and Enikolopov, R. 2011. Elite capture of local institutions: Evidence from a field experiment in Afghanistan. Working paper.
Benjamini, Y., and Hochberg, Y. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological) 57: 289300.
Callaway, E. 2011. Report finds massive fraud at Dutch universities. Nature 479: 15.
Casey, K., Glennerster, R., and Miguel, E. 2011. Reshaping institutions: Evidence on external aid and local collective action. Working paper.
Collier, D., Seawright, J., and Munck, G. L. 2010. The quest for standards: King, Keohane, and Verba's designing social inquiry. In Rethinking social inquiry: Diverse tools, shared standards, 2nd ed., eds. Collier, D. and Brady, H. E., 3365. London: Rowman & Littlefield Publishers.
De Angelis, C. D., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P., Schroeder, T. V., Sox, H. C., and Van Der Weyden, M. B. 2004. Clinical trial registration: A statement from the International Committee of Medical Journal Editors. New England Journal of Medicine 351: 1250–1.
De Angelis, C. D., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marušic, A., Overbeke, A. J. P., Schroeder, T. V., Sox, H. C., and Weyden, M. B. V. D. 2005. Is this clinical trial fully registered? A statement from the International Committee of Medical Journal Editors. Canadian Medical Association Journal 172: 1700–2.
Duflo, E. 2008. Using randomization in development economics research: A toolkit. In Handbook of Development Economics, eds. Strauss, J. A. and Schultz, T. P. 3895–962. North Holland: Elsevier.
Fearon, J. D., Humphreys, M., and Weinstein, J. M. 2009. Can development aid contribute to social cohesion after Civil War? Evidence from a field experiment in post-conflict Liberia. American Economic Review: Papers & Proceedings 99: 287–91.
Finkelstein, A., Taubman, S., Allen, H., Gruber, J., Newhouse, J. P., Wright, B., and Baicker, K. 2010. The short-run impact of extending public health insurance to low-income adults: Evidence from the first year of the Oregon Medicaid Experiment. Working paper.
Gelman, A., Hill, J., and Yajima, M. 2009. Why we (usually) dont have to worry about multiple comparisons. Working paper.
Gelman, A., and Tuerlinckx, F. 2000. Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics 15: 373–90.
Gerber, A. S., Green, D. P., and Nickerson, D. 2001. Testing for publication bias in political science. Political Analysis 9: 385–92.
Gerber, A., and Malhotra, N. 2008. Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science 3: 313–26.
Glaeser, E. L. 2006. Researcher incentives and empirical methods. Working paper (DP2122).
Hansen, P. R. 2005. A test for superior predictive ability. Journal of Business & Economic Statistics2 23: 365–80.
Ioannidis, J. P. A. 2005. Why most published research findings are false. PLoS Medicine 2: 696701.
King, G., Gakidou, E., Imai, K., Lakin, J., Moore, R. T., Nall, C., Ravishankar, N., Vargas, M., Téllez-Rojo, M. M., Avila, J. E. H., Avila, M. H., and Llamas, H. H. 2009. Public policy for the poor? A randomized assessment of the Mexican Universal Health Insurance Program. Lancet 373: 1447–54.
King, G., Keohane, R. O., and Verba, S. 1994. Designing social inquiry: Scientific inference in qualitative research. Princeton, NJ: Princeton University Press.
Kling, J. R., Liebman, J. B., and Katz, L. F. 2007. Experimental analysis of neighborhood effects. Econometrics 75: 83119.
Laine, C., Horton, R., Deangelis, C. D., Drazen, J. M., Frizelle, F. A., Godlee, F., Haug, C., Hébert, P. C., and Kotzin, S. 2007. Clinical trial registration: Looking back and moving ahead. Canadian Medical Association Journal 177: 78.
Lancet. 2010. Should protocols for observational research be registered? Lancet 375: 348.
Mathieu, S., Boutron, I., Moher, D., Altman, D. G., and Ravaud, P. 2009. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 302: 977–84.
Monogan, J. E. III. 2010. The immigration issue and the 2010 House elections: A research design. Working paper.
Mutz, D., and Pemantle, R. 2011. The perils of randomization checks in the analysis of experiments. Working paper.
Rasmussen, O. D., Malchow-Møller, N., and Barnebeck Andersen, T. 2011. Walking the talk: The need for a trial registry for development interventions. Journal of Development Effectiveness 3: 129.
Reveiz, L., Chan, A.-W., Krleza-Jerić, K., Granados, C. E., Pinart, M., Etxeandia, I., Rada, D., Martinez, M., Bonfill, X., and Cardona, A. F. 2010. Reporting of methodologic information on trial registries for quality assessment: A study of trial records retrieved from the WHO search portal. PloS One 5: 16.
Simmons, J. P., Nelson, L. D., and Simonsohn, U. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22: 1359–66.
Sterling, T. D. 1959. Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association 54: 3034.
White, H. 2000. A reality check for data snooping. Econometrica 68: 1097–126.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax
Type Description Title
UNKNOWN
Supplementary materials

Humphreys et al. supplementary material
Appendix A

 Unknown (10 KB)
10 KB
UNKNOWN
Supplementary materials

Humphreys et al. supplementary material
Appendix C

 Unknown (11 KB)
11 KB
UNKNOWN
Supplementary materials

Humphreys et al. supplementary material
Appendix B

 Unknown (15 KB)
15 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed