Skip to main content
×
×
Home

Pre-post effect sizes should be avoided in meta-analyses

  • P. Cuijpers (a1) (a2), E. Weitz (a1) (a2), I. A. Cristea (a3) and J. Twisk (a2)
Abstract
Aims

The standardised mean difference (SMD) is one of the most used effect sizes to indicate the effects of treatments. It indicates the difference between a treatment and comparison group after treatment has ended, in terms of standard deviations. Some meta-analyses, including several highly cited and influential ones, use the pre-post SMD, indicating the difference between baseline and post-test within one (treatment group).

Methods

In this paper, we argue that these pre-post SMDs should be avoided in meta-analyses and we describe the arguments why pre-post SMDs can result in biased outcomes.

Results

One important reason why pre-post SMDs should be avoided is that the scores on baseline and post-test are not independent of each other. The value for the correlation should be used in the calculation of the SMD, while this value is typically not known. We used data from an ‘individual patient data’ meta-analysis of trials comparing cognitive behaviour therapy and anti-depressive medication, to show that this problem can lead to considerable errors in the estimation of the SMDs. Another even more important reason why pre-post SMDs should be avoided in meta-analyses is that they are influenced by natural processes and characteristics of the patients and settings, and these cannot be discerned from the effects of the intervention. Between-group SMDs are much better because they control for such variables and these variables only affect the between group SMD when they are related to the effects of the intervention.

Conclusions

We conclude that pre-post SMDs should be avoided in meta-analyses as using them probably results in biased outcomes.

Copyright
Corresponding author
*Address for correspondence: Professor P. Cuijpers, Ph.D., Department of Clinical, Neuro and Developmental Psychology, Vrije Universiteit Amsterdam, Van der Boechorststraat 1, 1081 BT Amsterdam, The Netherlands. (Email: p.cuijpers@vu.nl)
References
Hide All
Balk, EM, Earley, A, Patel, K, Trikalinos, TA, Dahabreh, IJ (2012). Empirical Assessment of Within-arm Correlation Imputation in Trials of Continuous Outcomes (Methods Research Report No. AHRQ Publication No. 12(13)-EHC141-EF). Agency for Healthcare Research and Quality: Rockville, MD. Retrieved from www.effectivehealthcare.ahrq.gov/reports/final.cfm
Beck, AT, Ward, CH, Mendelson, M, Mock, J, Erbaugh, J (1961). An inventory for measuring depression. Archives of General Psychiatry 4, 561571.
Beck, AT, Steer, RA, Brown, GK (1996). BDI-II. Beck Depression Inventory, 2nd edn. Manual. Psychological Corporation: San Antonio.
Biostat Inc (2015). Comprehensive Meta-Analysis (Version 2.2.064). Retrieved from www.meta-analysis.com
Cuijpers, P, Karyotaki, E, Weitz, E, Andersson, G, Hollon, SD, van Straten, A (2014 a). The effects of psychotherapies for major depression in adults on remission, recovery and improvement: a meta-analysis. Journal of Affective Disorders 159, 118126.
Cuijpers, P, Weitz, E, Twisk, J, Kuehner, C, Cristea, I, David, D, Hollon, SD (2014 b). Gender as predictor and moderator of outcome in cognitive behavior therapy and pharmacotherapy for adult depression: an ‘individual patient data’ meta-analysis. Depression and Anxiety 31, 941951.
Hamilton, M (1960). A rating scale for depression. Journal of Neurology, Neurosurgery, and Psychiatry 23, 5662.
Hesser, H, Weise, C, Westin, VZ, Andersson, G (2011). A systematic review and meta-analysis of randomized controlled trials of cognitive–behavioral therapy for tinnitus distress. Clinical Psychology Review 31, 545553.
Higgins, JPT, Green, S (2011). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration: Chicester, England.
Hofmann, SG, Wu, JQ, Boettcher, H (2014). Effect of cognitive-behavioral therapy for anxiety disorders on quality of life: a meta-analysis. Journal of Consulting and Clinical Psychology 82, 375391.
Johnsen, TJ, Friborg, O (2015). The effects of cognitive behavioral therapy as an anti-depressive treatment is falling: a meta-analysis. Psychological Bulletin, Online First Publication 141, 747768.
Kirsch, I, Deacon, BJ, Huedo-Medina, TB, Scoboria, A, Moore, TJ, Johnson, BT (2008). Initial severity and antidepressant benefits: a meta-analysis of data submitted to the Food and Drug Administration. PLoS Medicine 5, e45.
Leichsenring, F, Rabung, S (2008). Effectiveness of long-term psychodynamic psychotherapy: a meta-analysis. JAMA 300, 15511565.
van der Lem, R, de Wever, WW, van der Wee, NJ, van Veen, T, Cuijpers, P, Zitman, FG (2012 a). The generalizability of psychotherapy efficacy trials in major depressive disorder: an analysis of the influence of patient selection in efficacy trials on symptom outcome in daily practice. BMC Psychiatry 12, 192.
van der Lem, R, van der Wee, NJA, van Veen, T, Zitman, FG (2012 b). Efficacy versus effectiveness: a direct comparison of the outcome of treatment for mild to moderate depression in randomized controlled trials and daily practice. Psychotherapy and Psychosomatics 81, 226234.
Vittengl, JR, Jarrett, RB, Weitz, E, Hollon, SD, Twisk, J, Cristea, I, Cuijpers, P (2016). Divergent outcomes in cognitive-behavioral therapy and pharmacotherapy for adult depression. American Journal of Psychiatry 173, 481490.
Weitz, ES, Hollon, SD, Twisk, J, van Straten, A, Huibers, MJ, David, D, Cuijpers, P (2015). Baseline depression severity as moderator of depression outcomes between cognitive behavioral therapy vs pharmacotherapy: an individual patient data meta-analysis. JAMA Psychiatry 72, 11021109.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Epidemiology and Psychiatric Sciences
  • ISSN: 2045-7960
  • EISSN: 2045-7979
  • URL: /core/journals/epidemiology-and-psychiatric-sciences
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Keywords:

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 154
Total number of PDF views: 225 *
Loading metrics...

Abstract views

Total abstract views: 978 *
Loading metrics...

* Views captured on Cambridge Core between 28th October 2016 - 26th April 2018. This data will be updated every 24 hours.