Skip to main content Accessibility help
×
Home

Beyond psychology: prevalence of p value and confidence interval misinterpretation across different fields

  • Xiao-Kang Lyu (a1), Yuepei Xu (a2) (a3), Xiao-Fan Zhao (a1), Xi-Nian Zuo (a2) and Chuan-Peng Hu (a4) (a5)...

Abstract

P values and confidence intervals (CIs) are the most widely used statistical indices in scientific literature. Several surveys have revealed that these two indices are generally misunderstood. However, existing surveys on this subject fall under psychology and biomedical research, and data from other disciplines are rare. Moreover, the confidence of researchers when constructing judgments remains unclear. To fill this research gap, we surveyed 1,479 researchers and students from different fields in China. Results reveal that for significant (i.e., p < .05, CI does not include zero) and non-significant (i.e., p > .05, CI includes zero) conditions, most respondents, regardless of academic degrees, research fields and stages of career, could not interpret p values and CIs accurately. Moreover, the majority were confident about their (inaccurate) judgements (see osf.io/mcu9q/ for raw data, materials, and supplementary analyses). Therefore, as misinterpretations of p values and CIs prevail in the whole scientific community, there is a need for better statistical training in science.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Beyond psychology: prevalence of p value and confidence interval misinterpretation across different fields
      Available formats
      ×

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Beyond psychology: prevalence of p value and confidence interval misinterpretation across different fields
      Available formats
      ×

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Beyond psychology: prevalence of p value and confidence interval misinterpretation across different fields
      Available formats
      ×

Copyright

This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike licence (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the same Creative Commons licence is included and the original work is properly cited. The written permission of Cambridge University Press must be obtained for commercial re-use.

Corresponding author

Author for correspondence: Chuan-Peng Hu, Email: hcp4715@gmail.com

Footnotes

Hide All
#

Lyu X-K and Xu Y are equally contributed to this work.

Footnotes

References

Hide All
Aczel, B., Palfi, B., Szollosi, A., Kovacs, M., Szaszi, B., Szecsi, P.Wagenmakers, E.-J. (2018). Quantifying support for the null hypothesis in psychology: An empirical investigation. Advances in Methods and Practices in Psychological Science, 1, 357366.
Amrhein, V. and Greenland, S. (2017). Remove, rather than redefine, statistical significance. Nature Human Behaviour, 2, 4.
Amrhein, V., Greenland, S. and McShane, B. (2019). Retire statistical significance. Nature, 567, 305307.
Badenes-Ribera, L., Frias-Navarro, D., Iotti, B., Bonilla-Campos, A. and Longobardi, C. (2016). Misconceptions of the p-value among Chilean and Italian academic psychologists. Frontiers in Psychology, 7, 1247.
Badenes-Ribera, L., Frías-Navarro, D., Monterde-i-Bort, H. and Pascual-Soler, M. (2015). Interpretation of the p value: A national survey study in academic psychologists from Spain. Psicothema, 27, 290295.
Bakker, M., Hartgerink, C.H.J., Wicherts, J.M. and van der Maas, H.L.J. (2016). Researchers’ Intuitions About Power in Psychological Research. Psychological Science, 27, 10691077.
Benjamin, D.J., Berger, J.O., Johannesson, M., Nosek, B.A., Wagenmakers, E.J., Berk, R.Johnson, V.E. (2018). Redefine statistical significance. Nature Human Behaviour, 2, 610.
Button, K.S., Ioannidis, J.P.A., Mokrysz, C., Nosek, B.A., Flint, J., Robinson, E.S.J. and Munafo, M.R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365376.
Cassidy, S.A., Dimova, R., Giguère, B., Spence, J.R. and Stanley, D.J. (2019). Failing grade: 89% of Introduction-to-Psychology textbooks that define or explain statistical significance do so incorrectly. Advances in Methods and Practices in Psychological Science, 2, 233239.
Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. The Journal of Abnormal and Social Psychology, 65, 145153.
Cohen, J. (1994). The earth is round (p <. 05). American Psychologist, 49, 9971003.
Colquhoun, D. (2014). An investigation of the false discovery rate and the misinterpretation of p-values. Royal Society Open Science, 1, 140216.
Coulson, M., Healey, M., Fidler, F. and Cumming, G. (2010). Confidence intervals permit, but do not guarantee, better inference than statistical significance testing. Frontiers in Psychology, 1, 26.
Cristea, I.A. and Ioannidis, J.P.A. (2018). P values in display items are ubiquitous and almost invariably significant: A survey of top science journals. PLoS One, 13, e0197440.
Cumming, G. (2013). The New Statistics: Why and How. Psychological Science, 25, 729.
Cumming, G., Fidler, F., Leonard, M., Kalinowski, P., Christiansen, A., Kleinig, A.Wilson, S. (2007). Statistical reform in psychology: Is anything changing? Psychological Science, 18, 230232.
Franco, A., Malhotra, N. and Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345, 15021505.
Gigerenzer, G. (2004). Mindless statistics. The Journal of Socio-Economics, 33, 587606.
Gigerenzer, G. (2018). Statistical rituals: The replication delusion and how we got there. Advances in Methods and Practices in Psychological Science, 1, 198218.
Gigerenzer, G., Swijtink, Z., Porter, T. and Daston, L. (1990). The empire of chance: How probability changed science and everyday life (vol. 12). Cambridge University Press.
Goodman, S. (2008). A dirty dozen: Twelve p-value misconceptions (Vol. 45). Seminars in Hematology, 45, 135140.
Greenland, S., Senn, S.J., Rothman, K.J., Carlin, J.B., Poole, C., Goodman, S.N. and Altman, D.G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31, 337350.
Haller, H. and Krauss, S. (2002). Misinterpretations of significance: A problem students share with their teachers? Methods of Psychological Research, 7, 120.
Head, M.L., Holman, L., Lanfear, R., Kahn, A.T. and Jennions, M.D. (2015). The extent and consequences of p-hacking in science. PLOS Biology, 13, e1002106.
Hoekstra, R., Morey, R.D., Rouder, J.N. and Wagenmakers, E.-J. (2014). Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review, 21, 11571164.
Hu, C.-P., Kong, X.-Z., Wagenmakers, E.-J., Ly, A. and Peng, K. (2018). Bayes factor and Its implementation in JASP: A practical primer [in Chinese]. Advances in Psychology Science, 26, 951965.
Hu, C.-P., Wang, F., Guo, J., Song, M., Sui, J. and Peng, K. (2016). The replication crisis in psychological research [in Chinese]. Advances in Psychology Science, 24, 15041518.
Jeffreys, H. (1961). Theory of Probability (3rd ed.). Oxford, UK: Oxford University Press.
John, L.K., Loewenstein, G. and Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524532.
Lakens, D., Adolfi, F.G., Albers, C.J., Anvari, F., Apps, M.A.J., Argamon, S.E.Zwaan, R.A. (2018). Justify your alpha. Nature Human Behaviour, 2, 168171.
Love, J., Selker, R., Marsman, M., Jamil, T., Dropmann, D., Verhagen, J.Wagenmakers, E.-J. (2019). JASP: Graphical statistical software for common statistical designs. Journal of Statistical Software, 1, 117.
Lyu, Z., Peng, K. and Hu, C.-P. (2018). P-value, confidence intervals, and statistical inference: A new dataset of misinterpretation. Frontiers in Psychology, 9, 868.
Maxwell, S.E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147163.
McCloskey, D.N. and Ziliak, S. (2008). The Cult of Statistical Significance. Ann Arbor, MI: University of Michigan Press.
Meehl, P.E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806834.
Miller, J. and Ulrich, R. (2015). Interpreting confidence intervals: A comment on Hoekstra, Morey, Rouder, and Wagenmakers (2014). Psychonomic Bulletin & Review, 23, 124130.
Montag, C., Becker, B. and Gan, C. (2018). The multipurpose application WeChat: A review on recent research. Frontiers in Psychology, 9, 2247.
Morey, R.D., Hoekstra, R., Rouder, J.N. and Wagenmakers, E.-J. (2016). Continued misinterpretation of confidence intervals: Response to Miller and Ulrich. Psychonomic Bulletin & Review, 23, 131140.
Nuijten, M.B., Hartgerink, C.H.J., van Assen, M.A.L.M., Epskamp, S. and Wicherts, J.M. (2016). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 48, 12051226.
Oakes, M. (1986). Statistical Inference: A Commentary for the Social and Behavioural Sciences. Wiley.
Simmons, J.P., Nelson, L.D. and Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 13591366.
Vidgen, B. and Yasseri, T. (2016). P-values: Misunderstood and misused. Frontiers in Physics, 4, 6.

Keywords

Type Description Title
WORD
Supplementary materials

Lyu et al. supplementary material
Lyu et al. supplementary material

 Word (315 KB)
315 KB

Beyond psychology: prevalence of p value and confidence interval misinterpretation across different fields

  • Xiao-Kang Lyu (a1), Yuepei Xu (a2) (a3), Xiao-Fan Zhao (a1), Xi-Nian Zuo (a2) and Chuan-Peng Hu (a4) (a5)...

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed