Skip to main content Accessibility help
×
Home

Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration

Published online by Cambridge University Press:  04 January 2017

Macartan Humphreys
Affiliation:
Columbia University, Department of Political Science, 7th floor, IAB Building, 420 West 118th St., New York, NY 10027
Raul Sanchez de la Sierra
Affiliation:
Columbia University, Department of Economics, 1022 IAB Building, 420 West 118th St., New York, NY 10027 e-mail: rs2861@columbia.edu
Peter van der Windt
Affiliation:
Columbia University, Department of Political Science, 7th floor, IAB Building, 420 West 118th St., New York, NY 10027 e-mail: pv2160@columbia.edu
Corresponding

Abstract

Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this “fishing” problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete “mock report” of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but nonbinding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but nonbinding registration are discussed, the principal advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.

Type
Symposium on Research Registration
Copyright
Copyright © The Author 2013. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below.

Footnotes

Authors' note: Our thanks to Ali Cirone, Andy Gelman, Grant Gordon, Alan Jacobs, Ryan Moore, and Ferran Elias Moreno for helpful comments. Our thanks to the Population Center at Columbia for providing access to the High Performance Computing (HPC) Cluster. This research was undertaken in the context of a field experiment in DRC; we thank the International Rescue Committee and CARE International for their partnership in that research and the International Initiative for Impact Evaluation (3IE) for their support. M. H. thanks the Trudeau Foundation for support while this work was undertaken. Replication data and code for tables and figures can be found at http://hdl.handle.net/1902.1/18182. Supplementary materials for this article are available on the Political Analysis Web site.

References

Beath, A., Christia, F., and Enikolopov, R. 2011. Elite capture of local institutions: Evidence from a field experiment in Afghanistan. Working paper.Google Scholar
Benjamini, Y., and Hochberg, Y. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological) 57: 289300.Google Scholar
Callaway, E. 2011. Report finds massive fraud at Dutch universities. Nature 479: 15.CrossRefGoogle ScholarPubMed
Casey, K., Glennerster, R., and Miguel, E. 2011. Reshaping institutions: Evidence on external aid and local collective action. Working paper.Google Scholar
Collier, D., Seawright, J., and Munck, G. L. 2010. The quest for standards: King, Keohane, and Verba's designing social inquiry. In Rethinking social inquiry: Diverse tools, shared standards, 2nd ed., eds. Collier, D. and Brady, H. E., 3365. London: Rowman & Littlefield Publishers.Google Scholar
De Angelis, C. D., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P., Schroeder, T. V., Sox, H. C., and Van Der Weyden, M. B. 2004. Clinical trial registration: A statement from the International Committee of Medical Journal Editors. New England Journal of Medicine 351: 1250–1.CrossRefGoogle ScholarPubMed
De Angelis, C. D., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marušic, A., Overbeke, A. J. P., Schroeder, T. V., Sox, H. C., and Weyden, M. B. V. D. 2005. Is this clinical trial fully registered? A statement from the International Committee of Medical Journal Editors. Canadian Medical Association Journal 172: 1700–2.CrossRefGoogle ScholarPubMed
Duflo, E. 2008. Using randomization in development economics research: A toolkit. In Handbook of Development Economics, eds. Strauss, J. A. and Schultz, T. P. 3895–962. North Holland: Elsevier.Google Scholar
Fearon, J. D., Humphreys, M., and Weinstein, J. M. 2009. Can development aid contribute to social cohesion after Civil War? Evidence from a field experiment in post-conflict Liberia. American Economic Review: Papers & Proceedings 99: 287–91.CrossRefGoogle Scholar
Finkelstein, A., Taubman, S., Allen, H., Gruber, J., Newhouse, J. P., Wright, B., and Baicker, K. 2010. The short-run impact of extending public health insurance to low-income adults: Evidence from the first year of the Oregon Medicaid Experiment. Working paper.Google Scholar
Gelman, A., Hill, J., and Yajima, M. 2009. Why we (usually) dont have to worry about multiple comparisons. Working paper.Google Scholar
Gelman, A., and Tuerlinckx, F. 2000. Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics 15: 373–90.CrossRefGoogle Scholar
Gerber, A. S., Green, D. P., and Nickerson, D. 2001. Testing for publication bias in political science. Political Analysis 9: 385–92.CrossRefGoogle Scholar
Gerber, A., and Malhotra, N. 2008. Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science 3: 313–26.CrossRefGoogle Scholar
Glaeser, E. L. 2006. Researcher incentives and empirical methods. Working paper (DP2122).Google Scholar
Hansen, P. R. 2005. A test for superior predictive ability. Journal of Business & Economic Statistics2 23: 365–80.Google Scholar
Ioannidis, J. P. A. 2005. Why most published research findings are false. PLoS Medicine 2: 696701.CrossRefGoogle ScholarPubMed
King, G., Gakidou, E., Imai, K., Lakin, J., Moore, R. T., Nall, C., Ravishankar, N., Vargas, M., Téllez-Rojo, M. M., Avila, J. E. H., Avila, M. H., and Llamas, H. H. 2009. Public policy for the poor? A randomized assessment of the Mexican Universal Health Insurance Program. Lancet 373: 1447–54.CrossRefGoogle Scholar
King, G., Keohane, R. O., and Verba, S. 1994. Designing social inquiry: Scientific inference in qualitative research. Princeton, NJ: Princeton University Press.Google Scholar
Kling, J. R., Liebman, J. B., and Katz, L. F. 2007. Experimental analysis of neighborhood effects. Econometrics 75: 83119.CrossRefGoogle Scholar
Laine, C., Horton, R., Deangelis, C. D., Drazen, J. M., Frizelle, F. A., Godlee, F., Haug, C., Hébert, P. C., and Kotzin, S. 2007. Clinical trial registration: Looking back and moving ahead. Canadian Medical Association Journal 177: 78.CrossRefGoogle ScholarPubMed
Lancet. 2010. Should protocols for observational research be registered? Lancet 375: 348.CrossRefGoogle Scholar
Mathieu, S., Boutron, I., Moher, D., Altman, D. G., and Ravaud, P. 2009. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 302: 977–84.CrossRefGoogle ScholarPubMed
Monogan, J. E. III. 2010. The immigration issue and the 2010 House elections: A research design. Working paper.Google Scholar
Mutz, D., and Pemantle, R. 2011. The perils of randomization checks in the analysis of experiments. Working paper.Google Scholar
Rasmussen, O. D., Malchow-Møller, N., and Barnebeck Andersen, T. 2011. Walking the talk: The need for a trial registry for development interventions. Journal of Development Effectiveness 3: 129.Google Scholar
Reveiz, L., Chan, A.-W., Krleza-Jerić, K., Granados, C. E., Pinart, M., Etxeandia, I., Rada, D., Martinez, M., Bonfill, X., and Cardona, A. F. 2010. Reporting of methodologic information on trial registries for quality assessment: A study of trial records retrieved from the WHO search portal. PloS One 5: 16.CrossRefGoogle ScholarPubMed
Simmons, J. P., Nelson, L. D., and Simonsohn, U. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22: 1359–66.CrossRefGoogle ScholarPubMed
Sterling, T. D. 1959. Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association 54: 3034.Google Scholar
White, H. 2000. A reality check for data snooping. Econometrica 68: 1097–126.CrossRefGoogle Scholar

Humphreys et al. supplementary material

Appendix A

File 10 KB

Humphreys et al. supplementary material

Appendix B

File 15 KB

Humphreys et al. supplementary material

Appendix C

File 11 KB

Altmetric attention score

Full text views

Full text views reflects PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views.

Total number of HTML views: 0
Total number of PDF views: 526 *
View data table for this chart

* Views captured on Cambridge Core between 04th January 2017 - 21st January 2021. This data will be updated every 24 hours.

Hostname: page-component-76cb886bbf-tmbpq Total loading time: 0.455 Render date: 2021-01-21T06:27:05.088Z Query parameters: { "hasAccess": "0", "openAccess": "0", "isLogged": "0", "lang": "en" } Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": false, "newCiteModal": false }

Send article to Kindle

To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration
Available formats
×

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration
Available formats
×

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration
Available formats
×
×

Reply to: Submit a response


Your details


Conflicting interests

Do you have any conflicting interests? *