Skip to main content
×
×
Home

Making replication mainstream

  • Rolf A. Zwaan (a1), Alexander Etz (a2), Richard E. Lucas (a3) and M. Brent Donnellan (a4)
Abstract

Many philosophers of science and methodologists have argued that the ability to repeat studies and obtain similar results is an essential component of science. A finding is elevated from single observation to scientific evidence when the procedures that were used to obtain it can be reproduced and the finding itself can be replicated. Recent replication attempts show that some high profile results – most notably in psychology, but in many other disciplines as well – cannot be replicated consistently. These replication attempts have generated a considerable amount of controversy, and the issue of whether direct replications have value has, in particular, proven to be contentious. However, much of this discussion has occurred in published commentaries and social media outlets, resulting in a fragmented discourse. To address the need for an integrative summary, we review various types of replication studies and then discuss the most commonly voiced concerns about direct replication. We provide detailed responses to these concerns and consider different statistical ways to evaluate replications. We conclude there are no theoretical or statistical obstacles to making direct replication a routine aspect of psychological science.

Copyright
Footnotes
Hide All
1.

Current address: Department of Psychology, Psychology Building, 316 Physics Road, Room 249A, Michigan State University, East Lansing, MI 48824.

Footnotes
References
Hide All
Alogna, V. K., Attaya, M. K., Aucoin, P., Bahnik, Š., Birch, S., Birt, A. R., Bornstein, B. H., Bouwmeester, S., Brandimonte, M. A., Brown, C., Buswell, K., Carlson, C., Carlson, M., Chu, S., Cislak, A., Colarusso, M., Colloff, M. F., Dellapaolera, K. S., Delvenne, J.-F., Di Domenico, A., Drummond, A., Echterhoff, G., Edlund, J. E., Eggleston, C. M., Fairfield, B., Franco, G., Gabbert, F., Gamblin, B. W., Garry, M., Gentry, R., Gilbert, E. A., Greenberg, D. L., Halberstadt, J., Hall, L., Hancock, P. J. B., Hirsch, D., Holt, G., Jackson, J. C., Jong, J., Kehn, A., Koch, C., Kopietz, R., Körner, U., Kunar, M. A., Lai, C. K., Langton, S. R. H., Leite, F. P., Mammarella, N., Marsh, J. E., McConnaughy, K. A., McCoy, S., McIntyre, A. H., Meissner, C. A., Michael, R. B., Mitchell, A. A., Mugayar-Baldocchi, M., Musselman, R., Ng, C., Nichols, A. L., Nunez, N. L., Palmer, M. A., Pappagianopoulos, J. E., Petro, M. S., Poirier, C. R., Portch, E., Rainsford, M., Rancourt, A., Romig, C., Rubínová, E., Sanson, M., Satchell, L., Sauer, J. D., Schweitzer, K., Shaheed, J., Skelton, F., Sullivan, G. A., Susa, K. J., Swanner, J. K., Thompson, W. B., Todaro, R., Ulatowska, J., Valentine, T., Verkoijen, P. P. J. L., Vranka, M., Wade, K. A., Was, C. A., Weatherford, D., Wiseman, K., Zaksaite, T., Zuj, D. V. & Zwaan, R. A. (2014) Registered replication report: Schooler & Engstler-Schooler (1990). Perspectives on Psychological Science 9:556–78.
Anderson, C. J., Bahník, Š., Barnett-Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., Cheung, F., Christopherson, C. D., Cordes, A., Cremata, E. J., Della Penna, N., Estel, V., Fedor, A., Fitneva, S. A., Frank, M. C., Grange, J. A., Hartshorne, J. K., Hasselman, F., Henninger, F., van der Hulst, M., Jonas, K. J., Lai, C. K., Levitan, C. A., Miller, J. K., Moore, K. S., Meixner, J. M., Munafò, M. R., Neijenhuijs, K. I., Nilsonne, G., Nosek, B. A., Plessow, F., Prenoveau, J. M., Ricker, A. A., Schmidt, K., Spies, J. R., Steiger, S., Strohminger, N., Sullivan, G. B., van Aert, R. C. M., van Assen, M. A. L. M., Vanpaemel, W., Vianello, M., Voracek, M. & Zuni, K. (2016) Response to comment on “estimating the reproducibility of psychological science.Science 351(6277):1037.
Baker, M. (2016) Is there a reproducibility crisis? Nature 533:452–54.
Barsalou, L. W. (2016) Situated conceptualization offers a theoretical account of social priming. Current Opinion in Psychology 12:611.
Baumeister, R. F. (2016) Charting the future of social psychology on stormy seas: Winners, losers, and recommendations. Journal of Experimental Social Psychology 66:153–58. Available at: http://doi.org/10.1016/j.jesp.2016.02.003.
Bem, D. J. (2003) Writing the empirical journal article. In: The compleat academic: A career guide, 2nd edition, ed . Darley, J. M., Zanna, M. P. & Roediger, H. L. III, pp . 185219. American Psychological Association.
Bem, D. J. (2011) Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology 100:407–25.
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. & van ‘t Veer, A. (2014) The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology 50:217–24.
Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. & Munafò, M. R. (2013) Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience 14(5):365–76.
Camerer, C. F., Dreber, A., Forsell, E., Ho, T. H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmedj, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M. & Wu, H. (2016) Evaluating replicability of laboratory experiments in economics. Science 351(6280):1433–36.
Cesario, J. (2014) Priming, replication, and the hardest science. Perspectives on Psychological Science 9:4048.
Chambers, C. (2017) The 7 deadly sins of psychology: A manifesto for reforming the culture of scientific practice. Princeton University Press.
Cheung, I., Campbell, L., LeBel, E. P., Ackerman, R. A., Aykutoğlu, B., Bahník, Š., Bowen, J. D., Bredbow, C. A., Bromberg, C., Caprariello, P. A., Carcedo, R. J., Carson, K. J., Cobb, R. J., Collins, N. L., Corretti, C. A., DiDonato, T. E., Ellithorpe, C., Fenrnández-Rouco, N., Fuglestad, P. T., Goldberg, R. M., Golom, F. D., Gündoğdu-Aktürk, E., Hoplock, L. B., Houdek, P., Kane, H. S., Kim, J. S., Kraus, S., Leone, C. T., Li, N. P., Logan, J. M., Millman, R. D., Morry, M. M., Pink, J. C., Ritchey, T., Root Luna, L. M., Sinclair, H. C., Stinson, D. A., Sucharyna, T. A., Tidwell, N. D., Uysal, A., Vranka, M., Winczewski, L. A. & Yong, J. C. (2016) Registered replication report: Study 1 from Finkel, Rusbult, Kumashiro & Hannon (2002). Perspectives on Psychological Science 11(5):750–64.
Cohen, J. (1990) Things I have learned (so far). American Psychologist 45(12):1304.
Cook, D. J., Guyatt, G. H., Ryan, G., Clifton, J., Buckingham, L., Willan, A., Mcllor, W. & Oxman, A. D. (1993) Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 269(21):2749–53.
Coyne, J. C. (2016) Replication initiatives will not salvage the trustworthiness of psychology. BMC Psychology 4:28. Available at: http://doi.org/10.1186/s40359-016-0134-3.
Crandall, C. S. & Sherman, J. W. (2016) On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology 66:9399. Available at: http://doi.org/10.1016/j.jesp.2015.10.002.
Dunlap, K. (1926) The experimental methods of psychology. In: Psychologies of 1925, ed . Murchison, C., pp . 331–53. Clark University Press.
Ebersole, C. R., Axt, J. R. & Nosek, B. A. (2016b) Scientists’ reputations are based on getting it right, not being right. PLoS Biology 14(5):e1002460. Available at: https://doi.org/10.1371/journal.pbio.1002460.
Eerland, A., Sherrill, A. M., Magliano, J. P., Zwaan, R. A., Arnal, J. D., Aucoin, P., Berger, S. A., Birt, A. R., Capezza, N., Carlucci, M., Crocker, C., Ferretti, T. R., Kibbe, M. R., Knepp, M. M., Kurby, C. A., Melcher, J. M., Michael, S. W., Poirier, C. & Prenoveau, J. M. (2016) Registered replication report: Hart & Albarracín (2011). Perspectives on Psychological Science 11:158–71.
Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J. & Nosek, B. A. (2014) An open investigation of the reproducibility of cancer biology research. Elife 3:e04333.
Etz, A. (2015, August 30) The Bayesian Reproducibility Project . Weblog post . Retrieved 23 August 2017 from: https://web.archive.org/web/20160407113631/http://alexanderetz.com:80/2015/08/30/the-bayesian-reproducibility-project/.
Etz, A. & Vandekerckhove, J. (2016) A Bayesian perspective on the reproducibility project: Psychology. PLoS ONE 11(2):e0149794. Available at: http://doi.org/10.1371/journal.pone.0149794.
Etz, A. & Wagenmakers, E. J. (2017) JBS Haldane's contribution to the Bayes factor hypothesis test. Statistical Science 32(2):313–29.
Fabrigar, L. R. & Wegener, D. T. (2016) Conceptualizing and evaluating the replication of research results. Journal of Experimental Social Psychology 66:6880.
Ferguson, C. J. & Brannick, M. T. (2012) Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods 17(1):120–28. Available at: http://doi.org/10.1037/a0024445.
Ferguson, C. J. & Heene, M. (2012) A vast graveyard of undead theories publication bias and psychological science's aversion to the null. Perspectives on Psychological Science 7(6):555–61.
Fetterman, A. K. & Sassenberg, K. (2015) The reputational consequences of failed replications and wrongness admission among scientists. PLoS One 10(12):e0143723. Available at: https://doi.org/10.1371/journal.pone.0143723.
Fiedler, K. & Schwarz, N. (2015) Questionable research practices revisited. Social Psychological and Personality Science 7:4552. Available at: http://doi.org/10.1177/1948550615612150.
Finkel, E. J., Eastwick, P. W. & Reis, H. T. (2015) Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science. Journal of Personality and Social Psychology 108:275–97. Available at: http://doi.org/10.1037/pspi0000007.
Franco, A., Malhotra, N. & Simonovits, G. (2014) Publication bias in the social sciences: Unlocking the file drawer. Science 345(6203):1502–505.
Gelman, A. & Loken, E. (2014) The statistical crisis in science data-dependent analysis – a “garden of forking paths” – explains why many statistically significant comparisons don't hold up. American Scientist 102(6):460.
Gilbert, D. T., King, G., Pettigrew, S, & Wilson, T. D. (2016) Comment on “estimating the reproducibility of psychological science.Science 351(6277):1037. Available at: http://doi.org/10.1126/science.aad7243.
Greenwald, A. G. (1975) Significance, nonsignificance, and interpretation of an ESP experiment. Journal of Experimental Social Psychology 11:180–91.
Hagger, M. S., Chatzisarantis, N. L. D., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., Brand, R., Brandt, M. J., Brewer, G., Bruyneel, S., Calvillo, D. P., Campbell, W. K., Cannon, P. R., Carlucci, M., Carruth, N. P., Cheung, T., Crowell, A., De Ridder, D. T. D., Dewitte, S., Elson, M., Evans, J. R., Fay, B. A., Fennis, B. M, Finley, A., Francis, Z., Heise, E., Hoemann, H., Inzlicht, M., Koole, S. L., Koppel, L., Kroese, F., Lange, F., Lau, K., Lynch, B. P., Martijn, C., Merckelbach, H., Mills, N. V., Michirev, A., Miyake, A., Mosser, A. E., Muise, M., Muller, D., Muzi, M., Nalis, D., Nurwanti, R., Otgaar, H, Philipp, M. C., Primoceri, P., Rentzsch, K., Ringos, L., Schlinkert, C., Schmeichel, B. J., Schoch, S. F., Schrama, M., Schütz, A., Stamos, A., Tinghög, G., Ullrich, J., vanDellen, M., Wimbarti, S., Wolff, W., Yusainy, C., Zerhouni, O. & Zwienenberg, M. (2016) A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science 11(4):546–73.
Hewitt, J. K. (2012) Editorial policy on candidate gene association and candidate gene-by-environment interaction studies of complex traits. Behavior Genetics 42(1):12.
Hofstee, W. K. B. (1984) Methodological decision rules as research policies: A betting reconstruction of empirical research. Acta Psychologica 56:93109.
Hüffmeier, J., Mazei, J. & Schultze, T. (2016) Reconceptualizing replication as a sequence of different studies: A replication typology. Journal of Experimental Social Psychology 66:8192.
Ioannidis, J. P. (2005) Why most published research findings are false. PLoS Medicine 2(8):e124.
Jeffreys, H. (1961) Theory of probability. Oxford University Press.
John, L. K., Loewenstein, G. & Prelec, D. (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science 23(5):524–32.
Kahneman, D. (2003) Experiences of collaborative research. American Psychologist 58:723.
Kerr, N. L. (1998) HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review 2:196217. Available at: http://doi.org/10.1207/s15327957pspr0203_4.
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B. Jr., Bahník, S., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., Hasselman, F., Hicks, J. A., Hovermale, J. F., Hunt, S. J., Hunstinger, J. R., IJerzman, H., John, M.-S., Joy-Gaba, J. A., Kappes, H. B., Krueger, L. E., Kurtz, J., Levitan, C. A., Mallett, R. K., Morris, W. L., Nelson, A. J., Nier, J. A., Packard, G., Pilati, R., Rutchick, A. M., Schmidt, K., Skorinko, J. L., Smith, R., Steiner, T. G., Storbeck, J., Van Swol, L. M., Thompson, D., van't Veer, A. E., Vaughn, L. A., Vranka, M., Wichman, A. L., Woodzicka, J. A. & Nosek, B. A. (2014a) Investigating variation in replicability: A “Many Labs” replication project. Social Psychology 45(3):142–52. Available at: http://doi.org/10.1027/1864-9335/a000178.
Kühberger, A., Fritz, A. & Scherndl, T. (2014) Publication bias in psychology: A diagnosis based on the correlation between effect size and sample size. PLoS One 9(9):e105825.
Kunert, R. (2016) Internal conceptual replications do not increase independent replication success. Psychonomic Bulletin and Review 23(5):1631–38.
Lakatos, I. (1970) Falsification and the methodology of scientific research programmes. In: Criticism and the growth of knowledge, ed. Lakatos, I. & Musgrave, A. , pp . 91196. Cambridge University Press.
LeBel, E. P., Berger, D., Campbell, L. & Loving, T. J. (2017) Falsifiability is not optional. Journal of Personality and Social Psychology 113:254–61.
Levelt Committee Noort Committee & Drent Committee (2012, November 28) Flawed science: The fraudulent research practices of social psychologist Diederik Stapel. Retrieved 21 August 2017 from www.tilburguniversity.edu/upload/3ff904d7-547b-40ae85febea38e05a34a_Final%20report%20Flawed%20Science.pdf.
Lupia, A. & Elman, C. (2014) Openness in political science: Data access and research transparency. PS – Political Science and Politics 47:1942.
Ly, A., Etz, A., Marsman, M. & Wagenmakers, E. J. (2017) Replication Bayes factors from evidence updating. PsyArXiv preprints. Available at: https://psyarxiv.com/u8m2s/.
Ly, A., Verhagen, J. & Wagenmakers, E. J. (2016) Harold Jeffreys's default Bayes factor hypothesis tests: Explanation, extension, and application in psychology. Journal of Mathematical Psychology 72:1932.
Lykken, D. T. (1968) Statistical significance in psychological research. Psychological Bulletin 70:151–59.
Makel, M. C., Plucker, J. A. & Hegarty, B. (2012) Replications in psychology research: How often do they occur? Perspectives on Psychological Science 7(6):537–42.
Matzke, D., Nieuwenhuis, S., van Rijn, H., Slagter, H. A., van der Molen, M. W. & Wagenmakers, E.-J. (2015) The effect of horizontal eye movements on free recall: A preregistered adversarial collaboration. Journal of Experimental Psychology: General 144(1):e1–15. Available at: http://doi.org/10.1037/xge0000038.
Maxwell, S. E., Lau, M. Y. & Howard, G. S. (2015) Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist 70:487–98. Available at: http://dx.doi.org/10.1037/a0039400.
Meehl, P. E. (1990b) Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry 1:108–41. Available at: http://doi.org/10.1207/s15327965pli0102_1.
Mellers, B., Hertwig, R. & Kahneman, D. (2001) Do frequency representations eliminate conjunction effects? An exercise in adversarial collaboration. Psychological Science 12:269–75.
Morey, R. D. & Lakens, D. (2016) Why most of psychology is statistically unfalsifiable . Available at: https://github.com/richarddmorey/psychology_resolution/blob/master/paper/response.pdf
National Cancer Institute–National Human Genome Research Institute (NCI-NHGRI) Working Group on Replication in Association Studies (2007) Replicating genotype-phenotype associations. Nature 447:655–60.
Nosek, B. A. & Errington, T. M. (2017) Making sense of replications. eLife 6:e23383.
Open Science Collaboration (2015) Estimating the reproducibility of psychological science. Science 349(6251):aac4716. Available at: http://doi.org/10.1126/science.aac4716.
Pashler, H. & Harris, C. (2012) Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science 7:531–36.
Pashler, H. & Wagenmakers, E.-J. (2012) Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science 7:528–30.
Popper, K. R. (1959/2002) The logic of scientific discovery, translation of Logik der Forschung . Routledge.
Rohrer, D., Pashler, H. & Harris, C. R. (2015) Do subtle reminders of money change people's political views? Journal of Experimental Psychology: General 144(4):e73.
Rosenthal, R. (1979) The “file drawer problem” and tolerance for null results. Psychological Bulletin 86(3):638–41.
Rotello, C. M., Heit, E. & Dubé, C. (2015) When more data steer us wrong: Replications with the wrong dependent measure perpetuate erroneous conclusions. Psychonomic Bulletin and Review 22:944–54.
Rothstein, H. R. & Bushman, B. J. (2012) Publication bias in psychological science: Comment on Ferguson and Brannick (2012). Psychological Methods 17:129–36.
Schmidt, F. L. & Oh, I.-S. (2016) The crisis of confidence in research findings in psychology: Is lack of replication the real problem? Or is it something else? Archives of Scientific Psychology 4(1):3237. Available at: http://dx.doi.org/10.1037/arc0000029.
Schmidt, S. (2009) Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology 13:90100.
Schweinsberg, M., Madan, N., Vianello, M., Sommer, S. A., Jordan, J., Tierney, W., Awtrey, E., Zhu, L. L., Diermeier, D., Heinze, J. E., Srinivasan, M., Tannenbaum, D., Bivolaru, E., Dana, J., Davis-Stober, C. P., du Plessis, C., Gronau, Q. F., Hafenbrack, A. C., Liao, E. Y., Ly, A., Marsman, M., Murase, T., Qureshi, I., Schaerer, M., Thornley, N., Tworek, C. M., Wagenmakers, E.-J., Wong, L., Anderson, T., Bauman, C. W., Bedwell, W. L., Brescoll, V., Canavan, A., Chandler, J. J., Cheries, E., Cheryan, S., Cheung, F., Cimpian, A., Clark, M. A., Cordon, D., Cushman, F., Ditto, P. H., Donahue, T., Frick, S. E., Gamez-Djokic, M., Hofstein Grady, R., Graham, J., Gu, J., Hahn, A., Hanson, B. E., Hartwich, N. J., Hein, K. Inbar, Y., Jiang, L., Kellogg, T., Kennedy, D. M., Legate, N., Luoma, T. P., Maibuecher, H., Meindl, P., Miles, J., Mislin, A., Molden, D. C., Motyl, M., Newman, G., Ngo, H. H., Packham, H., Ramsay, P. S., Ray, J. L., Sackett, A. M., Sellier, A.-L., Sokolova, T., Sowden, W., Storage, D., Sun, X., Van Bavel, J. J., Washburn, A. N., Wei, C., Wetter, E., Wilson, C. T., Darrous, S.-C. & Uhlmann, E. L. (2016) The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline. Journal of Experimental Social Psychology 66:5567.
Shanks, D. R., Vadillo, M. A., Riedel, B., Clymo, A., Govind, S., Hickin, N., Tamman, A. J. & Puhlmann, L. (2015) Romance, risk, and replication: Can consumer choices and risk-taking be primed by mating motives? Journal of Experimental Psychology: General 144(6):e142–58.
Silberzahn, R., Uhlmann, E. L., Martin, D., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., Fong, N., Gamez-Djokic, M., Glenz, A., Gordon-McKeon, S., Heaton, T. J., Hederos, K., Heene, M., Hofelich Mohr, A. J., Högden, F., Hui, K., Johannesson, M., Kalodimos, J., Kaszubowski, E., Kennedy, D. M., Lei, R., Lindsay, T. A., Liverani, S., Madan, C. R., Molden, D., Molleman, E., Morey, R. D., Mulder, L. B., Nijstad, B. R., Pope, N. G., Pope, B., Prenoveau, J. M., Rink, F., Robusto, E., Roderique, H., Sandberg, A., Schlüter, E., Schönbrodt, F. D., Sherman, M. F., Sommer, S. A., Sotak, K., Spain, S., Spörlein, C., Stafford, T., Stefanutti, L., Tauber, S., Ullrich, J., Vianello, M., Wagenmakers, E.-J., Witkowiak, M., Yoon, S. & Nosek, B. A. (2017) Many analysts, one dataset: Making transparent how variations in analytical choices affect results. PsyArXiv Preprint. Available at: https://psyarxiv.com/qkwst/.
Simmons, J. P., Nelson, L. D. & Simonsohn, U. (2011) False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22:1359–66. Available at: http://doi.org/10.1177/0956797611417632.
Simons, D. J., Shoda, Y. & Lindsay, D. S. (2017) Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science 12:1123–28. Available at: http://doi.org/10.1177/1745691617708630.
Simonsohn, U. (2015) Small telescopes: Detectability and the evaluation of replication results. Psychological Science 26:559–69.
Simonsohn, U. (2016, March 3) [47] Evaluating replications: 40% full≠60% empty . Available at: https://web.archive.org/web/20170709184952/http://datacolada.org/47.
Smaldino, P. E. & McElreath, R. (2016) The natural selection of bad science. Royal Society Open Science 3:160384.
Smart, R. G. (1964) The importance of negative results in psychological research. Canadian Psychologist 5:225–32.
Spellman, B. A. (2015) A short (personal) future history of Revolution 2.0. Perspectives on Psychological Science 10:886–99.
Sterling, T. D. (1959) Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association 54(285):3034. Available at: http://doi.org/10.2307/2282137.
Sterling, T. D., Rosenbaum, W. L. & Weinkam, J. J. (1995) Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician 49:108–12.
Stroebe, W. & Strack, F. (2014) The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science 9:5971. Available at: http://doi.org/10.1177/1745691613514450.
van Aert, R. C. & van Assen, M. A. (2017) Bayesian evaluation of effect size after replicating an original study. PLoS One 12(4):e0175302.
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J. & Reinero, D. A. (2016) Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences of the United States of America 113(23):6454–59.
Verhagen, A. J. & Wagenmakers, E.-J. (2014) Bayesian tests to quantify the result of a replication attempt. Journal of Experimental Psychology: General 143:1457–75.
Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B. Jr., Albohn, D. N., Allard, E. S., Benning, S. D., Blouin-Hudon, E.-M., Bulnes, L. C., Caldwell, T. L., Calin-Jageman, R. J., Capaldi, C. A., Carfagno, N. S., Chasten, K. T., Cleeremans, A., Connell, L., DeCicco, J. M., Dijkstra, K, Fischer, A. H., Foroni, F., Hess, U., Holmes, K. J., Jones, J. L. H., Klein, O., Koch, C., Korb, S., Lewinski, P., Liao, J. D., Lund, S., Lupianez, J., Lynott, D., Nance, C. N., Oosterwijk, S., Ozdoğru, A. A., Pacheco-Unguetti, A. P., Pearson, B., Powis, C., Riding, S., Roberts, T.-A., Rumiati, R. I., Senden, M., Shea-Shumsky, N. B., Sobocko, K., Soto, J. A., Steiner, T. G., Talarico, J. M., van Allen, Z. M., Vandekerckhove, M., Wainwright, B., Wayand, J. F., Zeelenberg, R., Zetzer, E. E. & Zwaan, R. A. (2016a) Registered replication report: Strack, Martin & Stepper (1988). Perspectives on Psychological Science 11:917–28.
Wagenmakers, E.-J., Verhagen, A. J. & Ly, A. (2016b) How to quantify the evidence for the absence of a correlation. Behavior Research Methods 48:413–26.
Wrinch, D. & Jeffreys, H. (1921) XLII. On certain fundamental principles of scientific inquiry. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 42(249):369–90.
Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K. & Zeelenberg, R. (2017) Participant nonnaiveté and the reproducibility of cognitive psychology. Psychonomic Bulletin and Review. Available at: http://doi.org/10.3758/s13423-017-1348-y.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Behavioral and Brain Sciences
  • ISSN: 0140-525X
  • EISSN: 1469-1825
  • URL: /core/journals/behavioral-and-brain-sciences
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Keywords

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed