Skip to main content Accessibility help

Putting replication in its place

  • Evan Heit (a1) and Caren M. Rotello (a2)


Direct replication is valuable but should not be elevated over other worthwhile research practices, including conceptual replication and checking of statistical assumptions. As noted by Rotello et al. (2015), replicating studies without checking the statistical assumptions can lead to increased confidence in incorrect conclusions. Finally, successful replications should not be elevated over failed replications, given that both are informative.



Hide All

Parts of this commentary are a work of the U.S. Government and are not subject to copyright protection in the United States.


This material includes work performed by Evan Heit while serving at the National Science Foundation. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.



Hide All
Dubé, C., Rotello, C. M. & Heit, E. (2010) Assessing the belief bias effect with ROCs: It's a response bias effect. Psychological Review 117:831–63.
Evans, J. St. B. T., Barston, J. L. & Pollard, P. (1983) On the conflict between logic and belief in syllogistic reasoning. Memory and Cognition 11:295306.
Heit, E., Hahn, U. & Feeney, A. (2005) Defending diversity. In: Categorization inside and outside the laboratory: Essays in honor of Douglas L. Medin, ed. Ahn, W.-K., Goldstone, R. L., Love, B. C., Markman, A. B. & Wolff, P., pp. 8799. American Psychological Association.
Heit, E. & Rotello, C. M. (2014) Traditional difference-score analyses of reasoning are flawed. Cognition 131:7591.
Nuijten, M. B., Hartgerink, C. H. J., Van Assen, M. A. L. M., Epskamp, S. & Wicherts, J. M. (2016) The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods 48(4):1205–26. Available at:
Open Science Collaboration (2015) Estimating the reproducibility of psychological science. Science 349(6251):aac4716. Available at:
Rotello, C. M., Heit, E. & Dubé, C. (2015) When more data steer us wrong: Replications with the wrong dependent measure perpetuate erroneous conclusions. Psychonomic Bulletin and Review 22:944–54.
Rotello, C. M., Masson, M. E. J. & Verde, M. F. (2008) Type I error rates and power analyses for single-point sensitivity measures. Perception and Psychophysics 70:389401.
Salmon, W. C. (1984) Scientific explanation and the causal structure of the world. Princeton University Press.


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed