Hostname: page-component-84b7d79bbc-fnpn6 Total loading time: 0 Render date: 2024-07-26T18:11:34.318Z Has data issue: false hasContentIssue false
  • English
  • Français

The Division of Replication Labor

Published online by Cambridge University Press:  01 January 2022

Felipe Romero
Get access Rights & Permissions [Opens in a new window]

Abstract

Scientists are becoming increasingly aware of a “replicability crisis” in the behavioral, social, and biomedical sciences. Researchers have made progress identifying statistical and methodological causes of the crisis. However, the social structure of science is also to blame. In the fields affected by the crisis, nobody is explicitly responsible and rewarded for doing confirmation and replication work. This article makes the case for a social structural reform to address the problem. I argue that we need to establish a reward system that supports a dedicated group of confirmation researchers and formulate a proposal that would achieve this.

Type
Values, Policy, and Social Epistemology
Information
Philosophy of Science , Volume 87 , Issue 5: PSA 2018 - Proceedings of the 2018 biennial meeting of the Philosophy of Science Association. Part II Symposia Papers , December 2020 , pp. 1014 - 1025
Copyright
Copyright © The Philosophy of Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

†

I am grateful to Ryan Doody, Jan Sprenger, Teresa Ai, Carl Craver, and the editor for their useful comments on previous drafts. I also thank the audience at the Philosophy of Science Association Biennial Meeting, Seattle, November 2018, for helpful discussion.

References

Bacon, F. 1627/2000. New Atlantis. Project Gutenberg. http://www.gutenberg.org/ebooks/2434.Google Scholar
Baker, M. 2016. “1,500 Scientists Lift the Lid on Reproducibility.” Nature 533 (7604): 452–54.CrossRefGoogle ScholarPubMed
Begley, C. G., and Ellis, L. M.. 2012. “Drug Development: Raise Standards for Preclinical Cancer Research.” Nature 483:531–33.CrossRefGoogle ScholarPubMed
Bissell, M. 2013. “Reproducibility: The Risks of the Replication Drive.” Nature 503:333–34.CrossRefGoogle ScholarPubMed
Bohannon, J. 2014. “Replication Effort Provokes Praise—and ‘Bullying’ Charges.” Science 344:788–89.CrossRefGoogle Scholar
Campbell, K. E., and Jackson, T. T.. 1979. “The Role and Need for Replication Research in Social Psychology.” Replications in Social Psychology 1 (1): 3–14.Google Scholar
Center for Open Science. 2013. “Reproducibility Initiative Receives $1.3M Grant to Validate 50 Landmark Cancer Studies.” Press release, October 16. https://cos.io/about/news/reproducibility-initiative-receives-grant-validate-50-landmark-cancer-studies/.Google Scholar
Cesario, J. 2014. “Priming, Replication, and the Hardest Science.” Perspectives on Psychological Science 9:40–48.CrossRefGoogle ScholarPubMed
Chambers, C. D. 2013. “Registered Reports: A New Publishing Initiative at Cortex.” Cortex 49:609–10.CrossRefGoogle ScholarPubMed
Cumming, G. 2012. Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-analysis. New York: Routledge.Google Scholar
Ebersole, C. R., et al. 2016. “Many Labs 3: Evaluating Participant Pool Quality across the Academic Semester via Replication.” Journal of Experimental Social Psychology 67:68–82.CrossRefGoogle Scholar
Feest, U. 2019. “Why Replication Is Overrated.” Philosophy of Science 86 (5): 895–905.CrossRefGoogle Scholar
Ioannidis, J. P. A. 2008. “Why Most Discovered True Associations Are Inflated.” Epidemiology 19:640–48.CrossRefGoogle ScholarPubMed
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., and David, S. P.. 2014. “Publication and Other Reporting Biases in Cognitive Sciences: Detection, Prevalence, and Prevention.” Trends in Cognitive Sciences 18:235–41.CrossRefGoogle Scholar
John, L. K., Loewenstein, G., and Prelec, D.. 2012. “Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling.” Psychological Science 23:524–32.CrossRefGoogle ScholarPubMed
Kerr, N. L. 1998. “Harking: Hypothesizing after the Results Are Known.” Personality and Social Psychology Review 2:196–217.CrossRefGoogle ScholarPubMed
Kitcher, P. 1990. “The Division of Cognitive Labor.” Journal of Philosophy 87:5–22.CrossRefGoogle Scholar
Klein, R. A., et al. 2014. “Investigating Variation in Replicability: A ‘Many Labs’ Replication Project.” Social Psychology 45:142–52.CrossRefGoogle Scholar
Koole, S. L., and Lakens, D.. 2012. “Rewarding Replications.” Perspectives on Psychological Science 7:608–14.CrossRefGoogle ScholarPubMed
Kunert, R. 2016. “Internal Conceptual Replications Do Not Increase Independent Replication Success.” Psychonomic Bulletin and Review 23 (5): 1631–38.CrossRefGoogle Scholar
Lee, M. D., and Wagenmakers, E.-J.. 2013. Bayesian Cognitive Modeling: A Practical Course. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Leonelli, S. 2018. “Re-thinking Reproducibility as a Criterion for Research Quality.” In Including a Symposium on Mary Morgan: Curiosity, Imagination, and Surprise, 129–46. Bingley: Emerald.Google Scholar
Machery, E. 2019. “The Alpha War.” Review of Philosophy and Psychology. https://doi.org/10.1007/s13164-019-00440-1.CrossRefGoogle Scholar
Makel, M. C., Plucker, J. A., and Hegarty, B.. 2012. “Replications in Psychology Research: How Often Do They Really Occur?” Perspectives on Psychological Science 7:537–42.CrossRefGoogle ScholarPubMed
Mayo, D. 2018. Statistical Inference as Severe Testing: How to Get beyond the Science Wars. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Merton, R. K. 1957. “Priorities in Scientific Discovery: A Chapter in the Sociology of Science.” American Sociological Review 22:635–59.CrossRefGoogle Scholar
Nosek, B. A., Spies, J. R., and Motyl, M.. 2012. “Scientific Utopia.” Pt. 2, “Restructuring Incentives and Practices to Promote Truth over Publishability.” Perspectives on Psychological Science 7:615–31.CrossRefGoogle Scholar
Nuzzo, R. 2015. “How Scientists Fool Themselves—and How They Can Stop.” Nature 526:182–85.CrossRefGoogle Scholar
NWO (Nederlandse Organisatie voor Wetenschappelijk Onderzoek). 2016. “NWO Makes 3 Million Available for Replication Studies Pilot.” NWO, July 19. https://www.nwo.nl/en/news-and-events/news/2016/nwo-makes-3-million-available-for-replication-studies-pilot.html.Google Scholar
Open Science Collaboration. 2012. “An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science.” Perspectives on Psychological Science 7:657–60.Google Scholar
NWO (Nederlandse Organisatie voor Wetenschappelijk Onderzoek). 2015. “Estimating the Reproducibility of Psychological Science.” Science 349: aac4716.CrossRefGoogle Scholar
Peirce, C. S. 1901/1958. “The Logic of Drawing History from Ancient Documents.” In The Collected Papers of Charles Sanders Peirce, ed. Burks, A. W., vol. 4, 89–107. Cambridge, MA: Belknap.Google Scholar
Prinz, F., Schlange, T., and Asadullah, K.. 2011. “Believe It or Not: How Much Can We Rely on Published Data on Potential Drug Targets?” Nature Reviews Drug Discovery 10: 712.CrossRefGoogle ScholarPubMed
Reichenbach, H. 1938. Experience and Prediction. Chicago: Cambridge University Press.Google Scholar
Romero, F. 2017. “Novelty versus Replicability: Virtues and Vices in the Reward System of Science.” Philosophy of Science 84 (5): 1031–43.CrossRefGoogle Scholar
Romero, F.. 2018. “Who Should Do Replication Labor?” Advances in Methods and Practices in Psychological Science 1 (4): 516–37.CrossRefGoogle Scholar
Romero, F.. 2019. “Philosophy of Science and the Replicability Crisis.” Philosophy Compass 14 (11): e12633.CrossRefGoogle Scholar
Sakaluk, J. K. 2016. “Exploring Small, Confirming Big: An Alternative System to the New Statistics for Advancing Cumulative and Replicable Psychological Research.” Journal of Experimental Social Psychology 66:47–54.CrossRefGoogle Scholar
Schönbrodt, F., Heene, M., Maier, M., and Zehetleitner, M.. 2015. “The Replication-/Credibility-Crisis in Psychology: Consequences at LMU?” Center for Open Science. https://osf.io/nptd9/.Google Scholar
Simmons, J. P., Nelson, L. D., and Simonsohn, U.. 2011. “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.” Psychological Science 22:1359–66.CrossRefGoogle ScholarPubMed
Simons, D. J., Holcombe, A. O., and Spellman, B. A.. 2014. “An Introduction to Registered Replication Reports at Perspectives on Psychological Science.” Perspectives on Psychological Science 9:552–55.CrossRefGoogle ScholarPubMed
Steinle, F. 1997. “Entering New Fields: Exploratory Uses of Experimentation.” Philosophy of Science 64:S65–S74.CrossRefGoogle Scholar
Stephan, P. E. 2012. How Economics Shapes Science. Cambridge, MA: Harvard University Press.CrossRefGoogle Scholar
Vazire, S. 2015. Editorial. Social Psychological and Personality Science 7:3–7.CrossRefGoogle Scholar
Weisberg, M., and Muldoon, R.. 2009. “Epistemic Landscapes and the Division of Cognitive Labor.” Philosophy of Science 76:225–52.CrossRefGoogle Scholar
Wright, A. G. C. 2017. “The Current State and Future of Factor Analysis in Personality Disorder Research.” Personality Disorders: Theory, Research, and Treatment 8:14–25.CrossRefGoogle ScholarPubMed
Yong, E. 2012. “A Failed Replication Attempt Draws a Scathing Personal Attack from a Psychology Professor.” Blog post. Discover Magazine Blog, March 10. http://blogs.discovermagazine.com/notrocketscience/2012/03/10/failed-replication-bargh-psychology-study-doyen/.Google Scholar