Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-5b777bbd6c-sbgtn Total loading time: 0 Render date: 2025-06-23T11:32:58.406Z Has data issue: false hasContentIssue false

14 - The Inner Workings of Registered Reports

from Part II - Important Methodological Considerations

Published online by Cambridge University Press:  12 December 2024

By
Zoltan Dienes
Edited by
John E. Edlund  and
Austin Lee Nichols
John E. Edlund
Affiliation:
Rochester Institute of Technology, New York
Austin Lee Nichols
Affiliation:
Central European University, Vienna
Get access

Summary

Registered Reports provide one way to address shortcomings in the current way we manage research – from the design of studies to their publication. The format requires pre-specifying (1) why a design may crucially test a theory, (2) what auxiliary assumptions are required for the experiment to be such a test, (3) what outcome-neutral tests are required to test those assumptions, (4) what specific crucial tests will be used to test the theory (of the many tests that could be used), and (5) why those tests could provide evidence for no effect of interest given the proposed numbers of trials and participants. Reviewers and authors are then constructively involved in optimizing the study before it is performed. The agreement between reviewers and authors, as adjudicated by the editors, defines, in advance, the proposed method and analytic protocol, virtually guaranteeing acceptance of the paper, no matter what position, if any, the results support. In this chapter, I go through what problems the format solves and why it must be approached in a way that is little understood by people coming to it for the first time. Common pitfalls are also discussed. In all, the paper provides a roadmap for how readers, authors, and editors can approach Registered Reports.

Type
Chapter
Information
The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
Volume 2: Performing Research
 , pp. 305 - 326
Publisher: Cambridge University Press
Print publication year: 2024

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Book purchase

Temporarily unavailable

References

Alger, B. E. (2019). Defense of the Scientific Hypothesis: From Reproducibility Crisis to Big Data. Oxford University Press.CrossRefGoogle Scholar
Allen, C., & Mehler, D. M. A. (2019). Open science challenges, benefits and tips in early career and beyond. PLOS Biology, 17(5), e3000246.CrossRefGoogle ScholarPubMed
Alogna, V. K., Attaya, M. K., Aucoin, P., Bahnik, S., Birch, S., Birt, A. R., et al. (2014). Registered replication report: Schooler & Engstler-Schooler (1990). Perspectives on Psychological Science, 9, 556–578.CrossRefGoogle ScholarPubMed
Baribault, B., Donkin, C., Little, D. R., Trueblood, J., Oravecz, Z., van Ravenzwaaij, D., et al. (2018). Metastudies for robust tests of theory. Proceedings of the National Academy of Sciences, 115, 2607–2612.CrossRefGoogle ScholarPubMed
Baumeister, R. F. (2016). Charting the future of social psychology on stormy seas: Winners, losers, and recommendations. Journal of Experimental Social Psychology, 66, 153–158CrossRefGoogle Scholar
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, R. B., Bollen, K. A., et al. (2017). Redefine statistical signfiicance. Nature Human Behaviour, 2, 6–10. https://doi.org/10.1038/s41562-017-0189-zCrossRefGoogle Scholar
Braithwaite, V. (2010). Do Fish Feel Pain? Oxford University Press.Google Scholar
Carney, D. R., Cuddy, A. J. C., & Yap, A. J. (2010). Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological Science, 21, 1363–1368.CrossRefGoogle ScholarPubMed
Carney, D. R., Cuddy, A. J. C., & Yap, A. J. (2015). Review and summary of research on the embodied effects of expansive (vs. contractive) nonverbal displays. Psychological Science, 26, 657–663.CrossRefGoogle ScholarPubMed
Carp, J. (2012). On the plurality of (methodological) worlds: Estimating the analytic flexibility of fMRI experiments. Frontiers in Neuroscience, 6. http://doi.org/10.3389/fnins.2012.00149CrossRefGoogle ScholarPubMed
Carroll, C., Patterson, M., Wood, S. Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40. https://doi.org/10.1186/1748-5908-2-40CrossRefGoogle ScholarPubMed
Center for Open Science. (2023). Registered Reports: Peer review before results are known to align scientific values and practices [webpage]. www.cos.io/initiatives/registered-reports (viewed March 16, 2023.Google Scholar
Cesario, J., Jonas, K. J., & Carney, D. R. (2017). CRSP special issue on power poses: What was the point and what did we learn? Comprehensive Results in Social Psychology, 2(1), 1–5. https://doi.org/10.1080/23743603.2017.1309876CrossRefGoogle Scholar
Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49, 609–610.CrossRefGoogle ScholarPubMed
Chambers, C. D. (2019). What’s next for registered reports? Nature, 573(7773), 187–189.CrossRefGoogle ScholarPubMed
Chambers, C. D., & Tzavella, L. (2020).The past, present and future of Registered Reports. MetaArXiv Preprints. https://doi.org/10.31222/osf.io/43298CrossRefGoogle Scholar
Chavalarias, D., & Ioannidis, J. P. A. (2010). Science mapping analysis characterizes 235 biases in biomedical research. Journal of Clinical Epidemiology, 63, 1205–1215.CrossRefGoogle ScholarPubMed
Chuard, P. J. C, Vrtílek, M., Head, M. L., & Jennions, M. D. (2019). Evidence that nonsignificant results are sometimes preferred: Reverse P-hacking or selective reporting? PLOS Biology, 17(1): e3000127. https://doi.org/10.1371/journal.pbio.3000127CrossRefGoogle ScholarPubMed
Deci, E. L., & Ryan, R. M. (2000). The “what” and “why” of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry, 11, 227–268.CrossRefGoogle Scholar
Deutsch, D. (2011). The Beginning of Infinity: Explanations that Transform the World. Penguin.Google Scholar
Devezer, B., Navarro, D. J., Vandekerckhove, J., & Buzbas, E. O. (2021). The case for formal methodology in scientific reform. Royal Society Open Science, 8, 200805. https://doi.org/10.1098/rsos.200805CrossRefGoogle ScholarPubMed
Dienes, Z. (2016). How Bayes factors change scientific practice. Journal of Mathematical Psychology, 72, 78–89.CrossRefGoogle Scholar
Dienes, Z. (2019). How do I know what my theory predicts? Advances in Methods and Practices in Psychological Science, 2, 364–377. https://doi.org/10.1177/2515245919876960CrossRefGoogle Scholar
Dienes, Z. (2021a). Obtaining evidence for no effect. Collabra: Psychology, 7(1), 28202. https://doi.org/10.1525/collabra.28202CrossRefGoogle Scholar
Dienes, Z. (2021b). How to use and report Bayesian hypothesis tests. Psychology of Consciousness: Theory, Research, and Practice, 8, 9–26.Google Scholar
Dienes, Z. (2023). Testing theories with Bayes factors. In Nichols, A. L. & Edlund, J. E. (eds.), Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences (vol. 1, pp. 494–512). Cambridge University Press.CrossRefGoogle Scholar
Dienes, Z., Palfi, B., & Lush, P. (2022). Controlling phenomenology by being unaware of intentions. In Weisberg, J. (ed.), Qualitative Consciousness: Themes from the Philosophy of David Rosenthal (pp. 229–242). Cambridge University Press.CrossRefGoogle Scholar
Dutilh, G., Vandekerckhove, J., Ly, A., Matzke, D., Pedroni, A., Frey, R., et al. (2017). A test of the diffusion model explanation for the worst performance rule using preregistration and blinding. Attention, Perception, & Psychophysics, 79, 713–725. https://doi.org/10.3758/s13414-017-1304-yCrossRefGoogle ScholarPubMed
Fiedler, K. (2017). What constitutes strong psychological science? The (neglected) role of diagnosticity and a priori theorizing. Perspectives on Psychological Science, 12, 46–61.CrossRefGoogle Scholar
Flake, J. K., Davidson, I. J., Wong, O., & Pek, J. (2022). Construct validity and the validity of replication studies: A systematic review. American Psychologist, 77(4), 576–588.CrossRefGoogle ScholarPubMed
Flake, J. K., & Fried, E. I. (2019). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456–465. https://doi.org/10.1177/2515245920952393CrossRefGoogle Scholar
Francis, G. (2012). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19, 151–156.CrossRefGoogle ScholarPubMed
Friese, M., & Frankenbach, J. (2020). p-Hacking and publication bias interact to distort meta-analytic effect size estimates. Psychological Methods, 25(4), 456–471.CrossRefGoogle ScholarPubMed
Gelman, A., & Loken, E. (2014). The statistical crisis in science: Data-dependent analysis – a “garden of forking paths” – explains why many statistically significant comparisons don’t hold up. American Scientist, 102(6), 460. https://doi.org/10.1511/2014.111.460CrossRefGoogle Scholar
Goldacre, B, DeVito, N. J., Heneghan, C., Irving, F., Bacon, S., Fleminger, J., & Curtis, H. (2018). Compliance with requirement to report results on the EU Clinical Trials Register: Cohort study and web resource. BMJ, 362, k3218.CrossRefGoogle ScholarPubMed
Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82(1), 1–20. https://doi.org/10.1037/h0076157CrossRefGoogle Scholar
Gronau, Q. F., van Erp, S., Heck, D. A., Cesario, J., Jonas, K. J., & Wagenmakers, E.-J. (2017). Bayesian model-averaged meta-analysis of the power pose effect with informed and default priors: The case of felt power. Comprehensive Results in Social Psychology, 2, 123–138.CrossRefGoogle Scholar
Guest, O., & Martin, A. E. (2021). How computational modeling can force theory building in psychological science. Perspectives on Psychological Science, 16(4), 789–802. https://doi.org/10.1177/1745691620970585CrossRefGoogle ScholarPubMed
Haig, B. D. (2014). Investigating the Psychological World: Scientific Method in the Behavioural Sciences. MIT Press.CrossRefGoogle Scholar
Ikeda, A., Xu, H., Fuji, N., Zhu, S., & Yamada, Y. (2019). Questionable research practices following pre-registration. Japanese Psychological Review, 62(3), 281–295.Google Scholar
Jeffreys, H. (1939). The Theory of Probability. Oxford University Press.Google Scholar
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality & Social Psychology Review, 2, 196–217.CrossRefGoogle ScholarPubMed
Khan, K. A., & Cheung, P. (2020). Presence of mismatches between diagnostic PCR assays and coronavirus SARS-CoV-2 genome. Royal Society Open Science, 7, 200636. http://doi.org/10.1098/rsos.200636CrossRefGoogle ScholarPubMed
Klein, S. B. (2014). What can recent replication failures tell us about the theoretical commitments of psychology?Theory & Psychology, 24(3), 326–338. https://doi.org/10.1177/0959354314529616CrossRefGoogle Scholar
Kruschke, J. K. (2018). Rejecting or accepting parameter values in Bayesian estimation. Advances in Methods and Practices in Psychological Science, 1, 270–280.CrossRefGoogle Scholar
Lakatos, I. (1970). Falsification and the methodology of scientific research programmes. In Lakatos, I. & Musgrave, A. (eds.), Criticism and the Growth of Knowledge (pp. 91–196). Cambridge University Press.CrossRefGoogle Scholar
MacCoun, R., & Perlmutter, S. (2015). Hide results to seek the truth. Nature, 526, 187–189.CrossRefGoogle ScholarPubMed
Mayo, D. G. (2018). Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars. Cambridge University press.CrossRefGoogle Scholar
McIntosh, R. D. (2017). Exploratory reports: A new article type for Cortex. Cortex, 96, A1–A4.CrossRefGoogle ScholarPubMed
Nature Human Behaviour. (2020). Registered Reports [webpage]. www.nature.com/nathumbehav/submission-guidelines/registeredreports (retrieved April 20, 2020).Google Scholar
Newman, J., & Taylor, A. (1992). Effect of a means-end contingency on young children’s food preferences. Journal of Experimental Child Psychology, 64, 200–216.CrossRefGoogle Scholar
Obels, P., Lakens, D., Coles, N. A., Gottfried, J., & Green, S. A. (2020). Analysis of open data and computational reproducibility in Registered Reports in psychology. Advances in Methods and Practices in Psychological Science, 3(2), 229–237. https://doi.org/10.1177/2515245920918872CrossRefGoogle Scholar
Palfi, B., & Dienes, Z. (2019a). When and how to calculate the Bayes factor with an interval null hypothesis. PsyArXiv Preprints. https://doi.org/10.31234/osf.io/9chmwCrossRefGoogle Scholar
Palfi, B., & Dienes, Z. (2019b). The role of Bayes factors in testing interactions. PsyArXiv Preprints. https://doi.org/10.31234/osf.io/qjrg4CrossRefGoogle Scholar
Palfi, B., & Dienes, Z. (2020). Why Bayesian “evidence for H1” in one condition and “evidence for H0” in another does not mean Bayesian evidence for a difference between conditions. Advances in Methods and Practices in Psychological Science, 3, 300–308.CrossRefGoogle Scholar
PCI RR (Peer Community In Registered Reports). (2023). Guide for Authors [webpage]. https://rr.peercommunityin.org/help/guide_for_authorsGoogle Scholar
Popper, K. R. (1959). The Logic of Scientific Discovery. Hutchinson.Google Scholar
Popper, K. R. (1963). Conjectures and Refutations: The Growth of Scientific Knowledge. Routledge & Kegan Paul.Google Scholar
Popper, K. R. (1972). Objective Knowledge: An Evolutionary Approach. Oxford University Press.Google Scholar
Ritchie, S. (2020). Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth. Metropolitan Books.Google Scholar
Royal Society Open Science. (2020). Registered Reports [webpage]. https://royalsocietypublishing.org/rsos/registered-reports (retrieved April 20, 2020).Google Scholar
Scheel, A. M., Schijen, M., & Lakens, D. (2020). An excess of positive results: Comparing the standard psychology literature with Registered Reports. Advances in Methods and Practices in Psychological Science, 4(2). https://doi.org/10.1177/25152459211007467Google Scholar
Schönbrodt, F. D., & Wagenmakers, E.-J. (2018). Bayes factor design analysis: Planning for compelling evidence. Psychonomic Bulletin & Review, 25, 128–142.CrossRefGoogle ScholarPubMed
Schönbrodt, F. D., Wagenmakers, E.-J., Zehetleitner, M., & Perugini, M. (2017). Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences. Psychological Methods, 22, 322–339.CrossRefGoogle ScholarPubMed
Schooler, J. W., & Engstler-Schooler, T. Y. (1990). Verbal overshadowing of visual memories: Some things are better left unsaid. Cognitive Psychology, 22, 36–71.CrossRefGoogle ScholarPubMed
Simmons, J., Nelson, L., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allow presenting anything as significant. Psychological Science, 22, 1359–1366.CrossRefGoogle ScholarPubMed
Simmons, J., & Simonsohn, U. (2015, May). Power posing: Reassessing the evidence behind the most popular TED talk. Data Colada, May 8. http://datacolada.org/2015/05/08/37-power-posing-reassessing-the-evidence-behind-the-most-popular-ted-talkGoogle Scholar
Simons, D. J., Holcombe, A. O., & Spellman, B. A. (2014). An introduction to Registered Replication Reports at Perspectives on Psychological Science. Perspectives on Psychological Science, 9, 552–555.CrossRefGoogle ScholarPubMed
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2019). Specification curve: Descriptive and inferential statistics on all reasonable specifications. SSRN. http://doi.org/10.2139/ssrn.2694998CrossRefGoogle Scholar
Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., et al. (2021). Initial evidence of research quality of registered reports compared with the standard publishing model. Nature Human Behaviour, 5(8), 990–997.CrossRefGoogle ScholarPubMed
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11, 702–712.CrossRefGoogle ScholarPubMed
Wagenmakers, E.-J., Verhagen, A. J., Ly, A., Matzke, D., Steingroever, H., Rouder, J . N., & Morey, R. D. (2017). The need for Bayesian hypothesis testing in psychological science. In Lilienfeld, S. & Waldman, I. (eds.), Psychological Science under Scrutiny: Recent Challenges and Proposed Solutions (pp. 123–138). John Wiley and Sons.CrossRefGoogle Scholar
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 627–633.CrossRefGoogle ScholarPubMed
Westlake, W. J. (1972). Use of confidence intervals in analysis of comparative bioavailability trials. Journal of Pharmaceutical Sciences, 61, 1340–1341. https://doi.org/10.1002/jps.2600610845CrossRefGoogle ScholarPubMed
Yanai, I., & Lercher, M. (2020). A hypothesis is a liability. Genome Biology, 21, 231. https://doi.org/10.1186/s13059-020-02133-wCrossRefGoogle ScholarPubMed

Save book to Kindle

To save this book to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×