Hostname: page-component-77f85d65b8-8wtlm Total loading time: 0 Render date: 2026-04-14T21:13:21.737Z Has data issue: false hasContentIssue false
  • English
  • Français

A practical guide to Registered Reports for economists

Published online by Cambridge University Press:  17 January 2025

Thibaut Arpinon  and
Romain Espinosa Open the ORCID record for Romain Espinosa [Opens in a new window]
Thibaut Arpinon
Affiliation:
CREM, University of Rennes 1, Rennes, France
Romain Espinosa*
Affiliation:
CIRED, CNRS, Nogent-sur-Marne, France
*
e-mail: romain.espinosa@cnrs.fr
Get access Rights & Permissions [Opens in a new window]

Abstract

The current publication system in economics has encouraged the inflation of positive results in empirical papers. Registered Reports, also called Pre-Results Reviews, are a new submission format for empirical work that takes pre-registration one step further. In Registered Reports, researchers write their papers before running the study and commit to a detailed data collection process and analysis plan. After a first-stage review, a journal can give an In-Principle-Acceptance guaranteeing that the paper will be published if the authors carry out their data collection and analysis as pre-specified. We here propose a practical guide to Registered Reports for empirical economists. We illustrate the major problems that Registered Reports address (p-hacking, HARKing, forking, and publication bias), and present practical guidelines on how to write and review Registered Reports (e.g., the data-analysis plan, power analysis, and correction for multiple-hypothesis testing), with R and STATA codes. We provide specific examples for experimental economics, and show how research design can be improved to maximize statistical power. Last, we discuss some tools that authors, editors, and referees can use to evaluate Registered Reports (checklist, study-design table, and quality assessment).

Keywords

Registered ReportsPractical guidePre-registrationp-hackingHARKingMultiple-hypothesis testingPower analysisThe smallest effect size of interest

JEL classification

A10: General C12: Hypothesis Testing: General C9: Design of Experiments

Information

Type
Methodology Paper
Information
Journal of the Economic Science Association , Volume 9 , Issue 1 , June 2023 , pp. 90 - 122
Copyright
Copyright © The Author(s), under exclusive licence to Economic Science Association 2023.

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

The authors thank Lionel Page, Emma Henderson, Daniel Lakens, Zoltan Dienes, Jens Rommel, Anna Dreber Almenberg, Andrew Clark, Marianne Lefebvre, and Etienne Dagorn for useful comments.

References

Abrams, E., Libgober, J., & List, J. A. (2020). Research registries: Facts, myths, and possible improvements. NBER Working Paper.Google Scholar
Agnoli, F., Wicherts, J. M., Veldkamp, C. L., Albiero, P., Cubelli, R. (2017). Questionable research practices among Italian research psychologists. PloS One, 12(3), 10.1371/journal.pone.0172792Google ScholarPubMed
Albers, C., Lakens, D. (2018). When power analyses based on pilot data are biased: Inaccurate effect size estimators and follow-up bias. Journal of Experimental Social Psychology, 74, 187–195. 10.1016/j.jesp.2017.09.004Google Scholar
Althouse, A. D. (2021). Post hoc power: Not empowering, just misleading. Journal of Surgical Research, 259, A3–A6. 10.1016/j.jss.2019.10.049Google ScholarPubMed
Altman, D. G., Bland, J. M. (1995). Statistics notes: Absence of evidence is not evidence of absence. BMJ, 311(7003),485. 10.1136/bmj.311.7003.485Google Scholar
Amrhein, V., Greenland, S., McShane, B. (2019). Scientists rise up against statistical significance. Nature, 567, 305–307. 10.1038/d41586-019-00857-9Google ScholarPubMed
Bakker, M., Veldkamp, C. L., van Assen, M. A., Crompvoets, E. A., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., Wicherts, J. M. (2020). Ensuring the quality and specificity of preregistrations. PLoS Biology, 18(12), 10.1371/journal.pbio.3000937Google ScholarPubMed
Beaumont, J.-F., Rivest, L.-P. (2009). Dealing with outliers in survey data Handbook of statistics, Elsevier 247–279.Google Scholar
Bellemare, C., Bissonnette, L., Kröger, S. (2016). Simulating power of economic experiments: The powerbbk package. Journal of the Economic Science Association, 2(2), 157–168. 10.1007/s40881-016-0028-4Google Scholar
Brodeur, A., Cook, N., Hartley, J., & Heyes, A. (2022). Do pre-registration and pre-analysis plans reduce p-hacking and publication bias? Available at SSRN.Google Scholar
Brodeur, A., Lé, M., Sangnier, M., Zylberberg, Y. (2016). Star wars: The empirics strike back. American Economic Journal: Applied Economics, 8(1), 1–32.Google Scholar
Bruns, S., Deressa, T. K., Stanley, T., Doucouliagos, C., & Ioannidis, J. (2022). Estimating the extent of inflated significance in economics. MetaArXiv—Preprints.Google Scholar
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T. et al., (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436. 10.1126/science.aaf0918Google ScholarPubMed
Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D., Etchells, P. (2014). Instead of playing the game it is time to change the rules: Registered reports at aims neuroscience and beyond. AIMS Neuroscience, 1(1), 4–17. 10.3934/Neuroscience.2014.1.4Google Scholar
Chambers, C. D., Tzavella, L. (2022). The past, present and future of registered reports. Nature Human Behaviour, 6, 29–42. 10.1038/s41562-021-01193-7Google ScholarPubMed
Chen, R., Chen, Y., Riyanto, Y. E. (2021). Best practices in replication: A case study of common information in coordination games. Experimental Economics, 24(1), 2–30. 10.1007/s10683-020-09658-8Google Scholar
Christensen, G., Miguel, E. (2018). Transparency, reproducibility, and the credibility of economics research. Journal of Economic Literature, 56(3), 920–80. 10.1257/jel.20171350Google Scholar
Clarke, D., Romano, J. P., Wolf, M. (2020). The Romano–Wolf multiple-hypothesis correction in Stata. The Stata Journal, 20(4), 812–843. 10.1177/1536867X20976314Google Scholar
Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science, 6(3), 274–290. 10.1177/1745691611406920Google Scholar
Dienes, Z. (2020). The inner workings of registered reports. PsyArXiv—Preprints.Google Scholar
Dienes, Z. (2021). Obtaining evidence for no effect. Collabra: Psychology, 7(1),28202. 10.1525/collabra.28202Google Scholar
Dienes, Z. (2022). Testing theories with Bayes factors. PsyArXiv—Preprints.Google Scholar
Dutilh, G., Sarafoglou, A., Wagenmakers, E.-J. (2021). Flexible yet fair: Blinding analyses in experimental psychology. Synthese, 198(23), 5745–5772. 10.1007/s11229-019-02456-7Google Scholar
Espinosa, R., Arpinon, T., Maginot, P., Demange, S., & Peureux, F. (2022). Removing barriers to plant-based diets: assisting doctors with vegan patients. Stage-2 Registered Report accepted at PCI RR.Google Scholar
Espinosa, R., Treich, N. (2021). Moderate versus radical NGOs. American Journal of Agricultural Economics, 103(4), 1478–1501. 10.1111/ajae.12156Google Scholar
Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PloS One, 4(5), 10.1371/journal.pone.0005738Google ScholarPubMed
Fanelli, D. (2010). Positive results increase down the hierarchy of the sciences. PloS One, 5(4), 10.1371/journal.pone.0010068Google ScholarPubMed
Ferraro, P. J., & Shukla, P. (2020). Feature—Is a replicability crisis on the horizon for environmental and resource economics? Review of Environmental Economics and Policy, 14(2).Google Scholar
Fiedler, K., Schwarz, N. (2016). Questionable research practices revisited. Social Psychological and Personality Science, 7(1), 45–52. 10.1177/1948550615612150Google Scholar
Franco, A., Malhotra, N., Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. 10.1126/science.1255484Google ScholarPubMed
Gelman, A., Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no fishing expedition or p-hacking and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University, 348, 1–17.Google Scholar
Glennerster, R., Takavarasha, K. (2013). Running randomized evaluations Running randomized evaluations, Princeton University Press 10.2307/j.ctt4cgd52Google Scholar
Heckelei, T., Hüttel, S., Odening, M., Rommel, J. (2022). The replicability crisis and the p-value debate—What are the consequences for the agricultural and food economics community? Preprints.Google Scholar
Henderson, E. (2022). A guide to preregistration and registered reports. MetaArXiv.Google Scholar
Henderson, E. L., Chambers, C. D. (2022). Ten simple rules for writing a registered report. PLoS Computational Biology, 18(10), 10.1371/journal.pcbi.1010571Google ScholarPubMed
Henderson, E., Vallée-Tourangeau, F., & Simons, D. (2019). The effect of concrete wording on truth judgements: A preregistered replication and extension of Hansen & Wänke (2010). Registered Report—Stage 1, retrieved from osf.io/f9jh6.Google Scholar
Jasielska, D., Rogoza, R., Zajenkowska, A., Russa, M. B. (2021). General trust scale: Validation in cross-cultural settings. Current Psychology, 40(10), 5019–5029. 10.1007/s12144-019-00435-2Google Scholar
John, L. K., Loewenstein, G., Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. 10.1177/0956797611430953Google ScholarPubMed
Lakens, D. (2014). Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44(7), 701–710. 10.1002/ejsp.2023Google Scholar
Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–362. 10.1177/1948550617697177Google Scholar
Lakens, D. (2022). Sample size justification. Collabra: Psychology, 8(1),33267. 10.1525/collabra.33267Google Scholar
Lakens, D., Scheel, A. M., Isager, P. M. (2018). Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science, 1(2), 259–269. 10.1177/2515245918770963Google Scholar
LeBel, E. P., Borsboom, D., Giner-Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., Smith, C. T. (2013). Psychdisclosure.org: Grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8(4), 424–432. 10.1177/1745691613491437Google ScholarPubMed
Leys, C., Delacre, M., Mora, Y. L., Lakens, D., & Ley, C. (2019). How to classify, detect, and manage univariate and multivariate outliers, with emphasis on pre-registration. International Review of Social Psychology, 32(1).Google Scholar
List, J. A., Shaikh, A. M., Xu, Y. (2019). Multiple hypothesis testing in experimental economics. Experimental Economics, 22(4), 773–793. 10.1007/s10683-018-09597-5Google Scholar
Loken, E., Gelman, A. (2017). Measurement error and the replication crisis. Science, 355(6325), 584–585. 10.1126/science.aal3618Google ScholarPubMed
MacCoun, R. J., & Perlmutter, S. (2017). Blind analysis as a correction for confirmatory bias in physics and in psychology. In Psychological science under scrutiny: Recent challenges and proposed solutions.Google Scholar
McCloskey, D. N., Ziliak, S. T. (1996). The standard error of regressions. Journal of Economic Literature, 34(1), 97–114.Google Scholar
Miguel, E. (2021). Evidence on research transparency in economics. Journal of Economic Perspectives, 35(3), 193–214. 10.1257/jep.35.3.193Google Scholar
Necker, S. (2014). Scientific misbehavior in economics. Research Policy, 43(10), 1747–1759. 10.1016/j.respol.2014.05.002Google Scholar
Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45(3), 137–141.Google Scholar
O’Boyle, E. H. Jr, Banks, G. C., Gonzalez-Mulé, E. (2017). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43(2), 376–399. 10.1177/0149206314527133Google Scholar
Ofosu, G. K., & Posner, D. N. (2021). Pre-analysis plans: An early stocktaking. In Perspectives on Politics, pp. 1–17. https://www.cambridge.org/core/journals/perspectives-on-politics/article/preanalysis-plans-an-earlystocktaking/94E7FAE76001C45A04E8F5E272C773CE.Google Scholar
Olken, B. A. (2015). Promises and perils of pre-analysis plans. Journal of Economic Perspectives, 29(3), 61–80. 10.1257/jep.29.3.61Google Scholar
Page, L., Noussair, C. N., Slonim, R. (2021). The replication crisis, the rise of new research practices and what it means for experimental economics. Journal of the Economic Science Association, 7(2), 210–225. 10.1007/s40881-021-00107-7Google Scholar
Romano, J. P., Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica, 73(4), 1237–1282. 10.1111/j.1468-0262.2005.00615.xGoogle Scholar
Romano, J. P., Wolf, M. (2016). Efficient computation of adjusted p-values for resampling-based stepdown multiple testing. Statistics & Probability Letters, 113, 38–40. 10.1016/j.spl.2016.02.012Google Scholar
Schafmeister, F. (2021). The effect of replications on citation patterns: Evidence from a large-scale reproducibility project. Psychological Science, 32(10), 1537–1548. 10.1177/09567976211005767Google ScholarPubMed
Scheel, A. M., Schijen, M. R., Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science, 4(2),25152459211007467. 10.1177/25152459211007467Google Scholar
Schooler, J. W. (2014). Metascience could rescue the replication crisis. Nature, 515(7525), 9–9. 10.1038/515009aGoogle ScholarPubMed
Serra-Garcia, M., Gneezy, U. (2021). Nonreplicable publications are cited more than replicable ones. Science Advances, 7(21),eabd1705. 10.1126/sciadv.abd1705Google ScholarPubMed
Swanson, N., Christensen, G., Littman, R., Birke, D., Miguel, E., Paluck, E. L., & Wang, Z. (2020). Research transparency is on the rise in economics. In In AEA Papers and Proceedings, vol. 110, pp. 61–65.Google Scholar
Thissen, D., Steinberg, L., Kuang, D. (2002). Quick and easy implementation of the Benjamini–Hochberg procedure for controlling the false positive rate in multiple comparisons. Journal of Educational and Behavioral Statistics, 27(1), 77–83. 10.3102/10769986027001077Google Scholar
Van’t Veer, A. E., Giner-Sorolla, R. (2016). Pre-registration in social psychology—A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12. 10.1016/j.jesp.2016.03.004Google Scholar
Wicherts, J. M., Veldkamp, C. L., Augusteijn, H. E., Bakker, M., Van Aert, R., & Van Assen, M. A. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. In Frontiers in psychology, p. 1832.Google Scholar
Yamagishi, T., Yamagishi, M. (1994). Trust and commitment in the United States and Japan. Motivation and Emotion, 18(2), 129–166. 10.1007/BF02249397Google Scholar
Young, C., Holsteen, K. (2017). Model uncertainty and robustness: A computational framework for multimodel analysis. Sociological Methods & Research, 46(1), 3–40. 10.1177/0049124115610347Google Scholar
Ziliak, S. T., McCloskey, D. N. (2004). Size matters: The standard error of regressions in the American Economic Review. Journal of Socio-Economics, 33(5), 527–546. 10.1016/j.socec.2004.09.024Google Scholar
Supplementary material: File

Arpinon and Espinosa supplementary material

Arpinon and Espinosa supplementary material
Download Arpinon and Espinosa supplementary material(File)
File 997 KB