Hostname: page-component-848d4c4894-wg55d Total loading time: 0 Render date: 2024-06-01T20:03:48.177Z Has data issue: false hasContentIssue false
  • English
  • Français

Historical trends in reporting effect sizes in clinical neuropsychology journals: A call to venture beyond the results section

Published online by Cambridge University Press:  10 February 2023

Steven Paul Woods Open the ORCID record for Steven Paul Woods [Opens in a new window] ,
Andrea Mustafa Open the ORCID record for Andrea Mustafa [Opens in a new window] ,
Ilex Beltran-Najera ,
Anastasia Matchanova ,
Jennifer L. Thompson  and
Natalie C. Ridgely
Steven Paul Woods*
Affiliation:
Department of Psychology, University of Houston, Houston, TX 77204, USA
Andrea Mustafa
Affiliation:
Department of Psychology, University of Houston, Houston, TX 77204, USA
Ilex Beltran-Najera
Affiliation:
Department of Psychology, University of Houston, Houston, TX 77204, USA
Anastasia Matchanova
Affiliation:
Department of Psychology, University of Houston, Houston, TX 77204, USA
Jennifer L. Thompson
Affiliation:
Department of Psychology, University of Houston, Houston, TX 77204, USA
Natalie C. Ridgely
Affiliation:
Department of Psychology, University of Houston, Houston, TX 77204, USA
*
Corresponding author: Steven Paul Woods, email: spwoods@uh.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Objective:

For decades, quantitative psychologists have recommended that authors report effect sizes to convey the magnitude and potential clinical relevance of statistical associations. However, fewer than one-third of neuropsychology articles published in the early 2000s reported effect sizes. This study re-examines the frequency and extent of effect size reporting in neuropsychology journal articles by manuscript section and over time.

Methods:

A sample of 326 empirical articles were drawn from 36 randomly selected issues of six neuropsychology journals at 5-year intervals between 1995 and 2020. Four raters used a novel, reliable coding system to quantify the extent to which effect sizes were included in the major sections of all 326 articles.

Results:

Findings showed medium-to-large increases in effect size reporting in the Methods and Results sections of neuropsychology journal articles that plateaued in recent years; however, there were only very small and nonsignificant changes in effect size reporting in the Abstract, Introduction, and Discussion sections.

Conclusions:

Authors in neuropsychology journals have markedly improved their effect size reporting in the core Methods and Results sections, but are still unlikely to consider these valuable metrics when motivating their study hypotheses and interpreting the conceptual and clinical implications of their findings. Recommendations are provided to encourage more widespread integration of effect sizes in neuropsychological research.

Keywords

Bibliometricsdata interpretationbehavioral sciencesinformation sciencestatistical techniquesclinical neuropsychology
Type
Research Article
Information
Journal of the International Neuropsychological Society , Volume 29 , Issue 9 , November 2023 , pp. 885 - 892
Copyright
Copyright © INS. Published by Cambridge University Press, 2023

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

American Psychological Association. (1996). Task force on statistical inference. https://www.apa.org/science/leadership/bsa/statistical/tfsi-initial-report.pdf Google Scholar
American Psychological Association. (2020). Publication manual, 7th ed. Washington, DC: American Psychological Association.Google Scholar
Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. The American Psychologist, 73, 3–25. https://doi.org/10.1037/amp0000191 CrossRefGoogle ScholarPubMed
Bezeau, S., & Graves, R. (2001). Statistical power and effect sizes of clinical neuropsychology research. Journal of Clinical and Experimental Neuropsychology, 23, 399–406. https://doi.org/10.1076/jcen.23.3.399.1181 CrossRefGoogle ScholarPubMed
Bieliauskas, L. A., Fastenau, P. S., Lacy, M. A., & Roper, B. L. (1997). Use of the odds ratio to translate neuropsychological test scores into real-world outcomes: From statistical significance to clinical significance. Journal of Clinical and Experimental Neuropsychology, 19, 889–896. https://doi.org/10.1080/01688639708403769 CrossRefGoogle ScholarPubMed
Centers for Disease Control and Prevention (2022). Precision health: Improving health for each of us and all of us. https://www.cdc.gov/genomics/about/precision_med.htm Google Scholar
Chelune, G.J. (2010). Evidence-based research and practice in clinical neuropsychology. The Clinical Neuropsychologist, 24, 454–467. https://doi.org/10.1080/13854040802360574 CrossRefGoogle ScholarPubMed
Chelune, G. J., Naugle, R. I., Lüders, H., Sedlak, J., & Awad, I. A. (1993). Individual change after epilepsy surgery: Practice effects and base-rate information. Neuropsychology, 7, 41–52. https://doi.org/10.1037/0894-4105.7.1.41 CrossRefGoogle Scholar
Cicchetti, D. V. (1998). Role of null hypothesis significance testing (NHST) in the design of neuropsychologic research. Journal of Clinical and Experimental Neuropsychology, 20, 293–295. https://doi.org/10.1076/jcen.20.2.293.1165 CrossRefGoogle ScholarPubMed
Clark, C. M. (1999). Further considerations of null hypothesis testing. Journal of Clinical and Experimental Neuropsychology, 21, 283–284. DOI: 10.1076/jcen.20.2.293.1165 CrossRefGoogle ScholarPubMed
Cohen, J. (1994). The Earth is round (p <. 05). American Psychologist, 49, 97–1003. https://doi.org/10.1037/0003-066X.50.12.1103 CrossRefGoogle Scholar
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed). Hillside, N.J.: Lawrence Erlbaum Associates.Google Scholar
Demakis, G. J. (2006) Meta-analysis in neuropsychology: Basic approaches, findings, and applications. The Clinical Neuropsychologist, 20, 10–26. DOI: 10.1080/13854040500203282 CrossRefGoogle ScholarPubMed
Donders, J. (2000) From null hypothesis to clinical significance, Journal of Clinical and Experimental Neuropsychology, 22, 265–266. DOI: 10.1076/1380-3395(200004)22:2;1-1;FT265 CrossRefGoogle ScholarPubMed
Fisher, R. A. (1925). Statistical methods for research workers. London: Oliver & Boyd Google Scholar
Fritz, A., Scherndl, T., & Kuhberger. (2012). A comprehensive review of reporting practices in psychological journals: Are effect sizes really enough? Theory & Psychology, 23, 98–122. https://doi.org/10.1177/0959354312436870 CrossRefGoogle Scholar
Funder, D. C., & Ozer, D. J. (2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2, 156–168. https://doi.org/10.1177/2515245919847202.CrossRefGoogle Scholar
Heaton, R. K., Franklin, D. R., Ellis, R. J., McCutchan, J. A., Letendre, S. L., Leblanc, S., Corkran, S. H., Duarte, N. A., Clifford, D. B., Woods, S.P., Collier, A. C., Marra, C. M., Morgello, S., Mindt, M. R., Taylor, M. J., Marcotte, T. D., Atkinson, J. H., Wolfson, T., Gelman, B. B., McArthur, J. C., & The HNRC Group (2011). HIV-associated neurocognitive disorders before and during the era of combination antiretroviral therapy: differences in rates, nature, and predictors. Journal of Neurovirology, 17, 3–16. https://doi.org/10.1007/s13365-010-0006-1 CrossRefGoogle ScholarPubMed
Holdnack, J. A, Drozdick, L. W., Weiss, L. G., & Iverson, G. L. (2013). WAIS-IV, WMS-IV, and ACS: Advanced Clinical Interpretation. Elsevier Inc.Google Scholar
Huettner, M. I., Rosenthal, B. L., & Hynd, G. W. (1989). Regional cerebral blood flow (rCBF) in normal readers: Bilateral activation with narrative text. Archives of Clinical Neuropsychology, 4, 71–78. https://doi.org/10.1093/arclin/4.1.71 CrossRefGoogle ScholarPubMed
Huygelier, H., Gillebert, C. R., & Moors, P. (2021). The value of Bayesian methods for accurate and efficient neuropsychological assessment. Journal of the International Neuropsychological Society. https://doi.org/10.1017/S1355617721001120 Google ScholarPubMed
Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746–759. https://doi.org/10.1177/0013164496056005002 CrossRefGoogle Scholar
Millis, S. (2003). Statistical practices: The seven deadly sins. Child Neuropsychology, 9, 221–233. https://doi.org/10.1076/chin.9.3.221.16455.CrossRefGoogle ScholarPubMed
Peng, C. Y. J., Chen, L. T., Chiang, H. M., & Chiang, Y. C. (2013). The impact of APA and AERA guidelines on effect size reporting. Educational Psychology Review, 25, 157–209. https://doi.org/10.1007/s10648-013-9218-2 CrossRefGoogle Scholar
Reitan, R. M. (1955). The relation of the trail making test to organic brain damage. Journal of Consulting Psychology, 19, 393–394. https://doi.org/10.1037/h0044509 CrossRefGoogle ScholarPubMed
Schatz, P., Jay, K. A., McComb, J., & McLaughlin, J. R. (2005). Misuse of statistical tests in Archives of Clinical Neuropsychology publications. Archives of Clinical Neuropsychology, 20, 1053–1059. DOI: https://doi.org/10.1016/j.acn.2005.06.006 CrossRefGoogle ScholarPubMed
Smith, G. E. (2002). What is the outcome we seek? A commentary on Keith et al. (2002). Neuropsychology, 16, 432–433. https://doi.org/10.1037//0894-4105.16.3.432 CrossRefGoogle Scholar
Sun, S., Pan, W., & Wang, L. L. (2010). A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology. Journal of Educational Psychology, 102, 989–1004. https://doi.org/10.1037/a0019507 CrossRefGoogle Scholar
Tierney, S., Woods, S. P., Verduzco, M., Beltran, J., Massman, P. J., & Hasbun, R. (2018). Semantic memory in HIV-associated Neurocognitive Disorders: An evaluation of the “cortical” versus “subcortical” hypothesis. Archives of Clinical Neuropsychology, 33, 406–416. https://doi.org/10.1093/arclin/acx083 CrossRefGoogle ScholarPubMed
Wilkinson, L., & The Task Force on Statistical Inference (1999). Statistical methods in psychology journals: Guidelines and explanations. The American Psychologist, 54, 594–604. https://doi.org/10.1037/0003-066X.54.8.594 CrossRefGoogle Scholar
Woods, S. P., Rippeth, J. D., Conover, E., Carey, C. L., Parsons, T. D., & Tröster, A. I. (2006). Statistical power of studies examining the cognitive effects of subthalamic nucleus deep brain stimulation in Parkinson’s disease. The Clinical Neuropsychologist, 20, 27–38. https://doi.org/10.1080/13854040500203290 CrossRefGoogle ScholarPubMed
Woods, S. P., Weinborn, M., & Lovejoy, D. W. (2003). Are classification accuracy statistics underused in neuropsychological research? Journal of Clinical and Experimental Neuropsychology, 25, 431–439. https://doi.org/10.1076/jcen.25.3.431.13800 CrossRefGoogle ScholarPubMed
Sweet, J. J., Meyer, D. G., Nelson, N. W., & Moberg, P. J. (2011). The TCN/AACN 2010 “salary survey”: Professional practices, beliefs, and incomes of US neuropsychologists. The Clinical Neuropsychologist, 25, 12–61. https://doi.org/10.1080/13854046.2010.544165 CrossRefGoogle ScholarPubMed
Yates, F. (1951). The influence of “statistical methods for research workers” on the development of the science of statistics. Journal of the American Statistical Association, 46, 19–34. https://doi.org/10.1080/01621459.1951.10500764 Google Scholar
Zakzanis, K. K. (2001). Statistics to tell the truth, the whole truth, and nothing but the truth: Formulae, illustrative numerical examples, and heuristic interpretation of effect size analyses for neuropsychological researchers. Archives of Clinical Neuropsychology, 16, 653–667. https://doi.org/10.1093/arclin/16.7.653 CrossRefGoogle ScholarPubMed
Supplementary material: File

Woods et al. supplementary material

Woods et al. supplementary material

Download Woods et al. supplementary material(File)
File 20.5 KB