Hostname: page-component-55f67697df-gmt7q Total loading time: 0 Render date: 2025-05-22T11:13:30.715Z Has data issue: false hasContentIssue false
  • English
  • Français

Development of a model to predict psychotherapy response for depression among Veterans

Published online by Cambridge University Press:  11 February 2022

Hannah N. Ziobrowski ,
Ruifeng Cui ,
Eric L. Ross ,
Howard Liu ,
Victor Puac-Polanco ,
Brett Turner ,
Lucinda B. Leung ,
Robert M. Bossarte ,
Corey Bryant  and
Wilfred R. Pigeon
...Show all authors
Hannah N. Ziobrowski
Affiliation:
Department of Health Care Policy, Harvard Medical School, Boston, MA, USA
Ruifeng Cui
Affiliation:
Department of Veterans Affairs, VISN 4 Mental Illness Research, Education and Clinical Center, VA Pittsburgh Health Care System, Pittsburgh, PA, USA Department of Psychiatry, University of Pittsburgh School of Medicine, Pittsburgh, PA, USA
Eric L. Ross
Affiliation:
Department of Psychiatry, McLean Hospital, Belmont, MA, USA Department of Psychiatry, Massachusetts General Hospital, Boston, MA, USA Department of Psychiatry, Harvard Medical School, Boston, MA, USA
Howard Liu
Affiliation:
Department of Health Care Policy, Harvard Medical School, Boston, MA, USA Center of Excellence for Suicide Prevention, Canandaigua VA Medical Center, Canandaigua, NY, USA
Victor Puac-Polanco
Affiliation:
Department of Health Care Policy, Harvard Medical School, Boston, MA, USA
Brett Turner
Affiliation:
Department of Health Care Policy, Harvard Medical School, Boston, MA, USA Harvard T.H. Chan School of Public Health, Boston, MA, USA
Lucinda B. Leung
Affiliation:
Center for the Study of Healthcare Innovation, Implementation, and Policy, VA Greater Los Angeles Healthcare System, Los Angeles, CA, USA Division of General Internal Medicine and Health Services Research, UCLA David Geffen School of Medicine, Los Angeles, CA, USA
Robert M. Bossarte
Affiliation:
Center of Excellence for Suicide Prevention, Canandaigua VA Medical Center, Canandaigua, NY, USA Department of Behavioral Medicine and Psychiatry, West Virginia University, Morgantown, WV, USA
Corey Bryant
Affiliation:
Center for Clinical Management Research, VA, Ann Arbor, MI, USA
Wilfred R. Pigeon
Affiliation:
Center of Excellence for Suicide Prevention, Canandaigua VA Medical Center, Canandaigua, NY, USA Department of Psychiatry, University of Rochester Medical Center, Rochester, NY, USA
David W. Oslin
Affiliation:
VISN 4 Mental Illness Research, Education, and Clinical Center, Corporal Michael J. Crescenz VA Medical Center, Philadelphia, PA, USA Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
Edward P. Post
Affiliation:
Center for Clinical Management Research, VA, Ann Arbor, MI, USA Department of Medicine, University of Michigan Medical School, Ann Arbor, MI, USA
Alan M. Zaslavsky
Affiliation:
Department of Health Care Policy, Harvard Medical School, Boston, MA, USA
Jose R. Zubizarreta
Affiliation:
Department of Health Care Policy, Harvard Medical School, Boston, MA, USA Department of Statistics, Harvard University, Cambridge, MA, USA Department of Biostatistics, Harvard University, Cambridge, MA, USA
Andrew A. Nierenberg
Affiliation:
Department of Psychiatry, Harvard Medical School, Boston, MA, USA Department of Psychiatry, Dauten Family Center for Bipolar Treatment Innovation, Massachusetts General Hospital, Boston, MA, USA
Alex Luedtke
Affiliation:
Department of Statistics, University of Washington, Seattle, WA, USA Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Research Center, Seattle, WA, USA
Chris J. Kennedy
Affiliation:
Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
Ronald C. Kessler*
Affiliation:
Department of Health Care Policy, Harvard Medical School, Boston, MA, USA
*
Author for correspondence: Ronald C. Kessler, E-mail: kessler@hcp.med.harvard.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Background

Fewer than half of patients with major depressive disorder (MDD) respond to psychotherapy. Pre-emptively informing patients of their likelihood of responding could be useful as part of a patient-centered treatment decision-support plan.

Methods

This prospective observational study examined a national sample of 807 patients beginning psychotherapy for MDD at the Veterans Health Administration. Patients completed a self-report survey at baseline and 3-months follow-up (data collected 2018–2020). We developed a machine learning (ML) model to predict psychotherapy response at 3 months using baseline survey, administrative, and geospatial variables in a 70% training sample. Model performance was then evaluated in the 30% test sample.

Results

32.0% of patients responded to treatment after 3 months. The best ML model had an AUC (SE) of 0.652 (0.038) in the test sample. Among the one-third of patients ranked by the model as most likely to respond, 50.0% in the test sample responded to psychotherapy. In comparison, among the remaining two-thirds of patients, <25% responded to psychotherapy. The model selected 43 predictors, of which nearly all were self-report variables.

Conclusions

Patients with MDD could pre-emptively be informed of their likelihood of responding to psychotherapy using a prediction tool based on self-report data. This tool could meaningfully help patients and providers in shared decision-making, although parallel information about the likelihood of responding to alternative treatments would be needed to inform decision-making across multiple treatments.

Keywords

Depressionmachine learningpsychotherapytreatment responseVeterans Health Administration
Type
Original Article
Information
Psychological Medicine , Volume 53 , Issue 8 , June 2023 , pp. 3591 - 3600
Check for updates
Copyright
Copyright © The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Austin, P. C., & Steyerberg, E. W. (2014) Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Statistics in Medicine, 33(3), 517–535. doi:10.1002/sim.5941CrossRefGoogle ScholarPubMed
Austin, P. C., & Steyerberg, E. W. (2019) The integrated calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Statistics in Medicine, 38(21), 4051–4065. doi:10.1002/sim.8281CrossRefGoogle ScholarPubMed
Blais, M. A., Malone, J. C., Stein, M. B., Slavin-Mulford, J., O'Keefe, S. M., Renna, M., & Sinclair, S. J. (2013) Treatment as usual (TAU) for depression: A comparison of psychotherapy, pharmacotherapy, and combined treatment at a large academic medical center. Psychotherapy, 50(1), 110–118. doi:10.1037/a0031385CrossRefGoogle Scholar
Bone, C., Simmonds-Buckley, M., Thwaites, R., Sandford, D., Merzhvynska, M., Rubel, J., … Delgadillo, J. (2021) Dynamic prediction of psychological treatment outcomes: Development and validation of a prediction model using routinely collected symptom data. The Lancet Digital Health, 3(4), e231–e240. doi:10.1016/s2589-7500(21)00018-2CrossRefGoogle ScholarPubMed
Chipman, H. A., George, E. I., & McCulloch, R. E. (2010) BART: Bayesian additive regression trees. The Annals of Applied Statistics, 4(1), 266–298. doi:10.1214/09-AOAS285CrossRefGoogle Scholar
Coley, R. Y., Boggs, J. M., Beck, A., & Simon, G. E. (2021) Predicting outcomes of psychotherapy for depression with electronic health record data. Journal of Affective Disorders Reports, 6(100198). doi:10.1016/j.jadr.2021.100198CrossRefGoogle ScholarPubMed
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. (2015) Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement. British Journal of Surgery, 102(3), 148–158. doi:10.1002/bjs.9736CrossRefGoogle ScholarPubMed
Cuijpers, P., Berking, M., Andersson, G., Quigley, L., Kleiboer, A., & Dobson, K. S. (2013) A meta-analysis of cognitive-behavioural therapy for adult depression, alone and in comparison with other treatments. Canadian Journal of Psychiatry, 58(7), 376–385. doi:10.1177/070674371305800702CrossRefGoogle ScholarPubMed
Cuijpers, P., Karyotaki, E., Ciharova, M., Miguel, C., Noma, H., & Furukawa, T. A. (2021) The effects of psychotherapies for depression on response, remission, reliable change, and deterioration: A meta-analysis. Acta Psychiatrica Scandinavica, 144(3), 288–299. doi:10.1111/acps.13335CrossRefGoogle ScholarPubMed
Delgadillo, J., & Gonzalez Salas Duhne, P. (2020) Targeted prescription of cognitive-behavioral therapy versus person-centered counseling for depression using a machine learning approach. Journal of Consulting and Clinical Psychology, 88(1), 14–24. doi:10.1037/ccp0000476CrossRefGoogle ScholarPubMed
DeRubeis, R. J., Cohen, Z. D., Forand, N. R., Fournier, J. C., Gelfand, L. A., & Lorenzo-Luaces, L. (2014) The personalized advantage Index: Translating research on prediction into individualized treatment recommendations. A demonstration. PLoS ONE, 9(1), e83875. doi:10.1371/journal.pone.0083875CrossRefGoogle ScholarPubMed
GBD 2019 Diseases and Injuries Collaborators (2020) Global burden of 369 diseases and injuries in 204 countries and territories, 1990-2019: A systematic analysis for the global burden of disease study 2019. Lancet, 396(10258), 1204-1222. doi:10.1016/s0140-6736(20)30925-9CrossRefGoogle Scholar
Gelhorn, H. L., Sexton, C. C., & Classi, P. M. (2011) Patient preferences for treatment of major depressive disorder and the impact on health outcomes: A systematic review. Primary Care Companion for CNS Disorders, 13(5). doi:10.4088/PCC.11r01161Google ScholarPubMed
Greden, J. F., Parikh, S. V., Rothschild, A. J., Thase, M. E., Dunlop, B. W., DeBattista, C., … Dechairo, B. (2019) Impact of pharmacogenomics on clinical outcomes in major depressive disorder in the GUIDED trial: A large, patient- and rater-blinded, randomized, controlled study. Journal of Psychiatric Research, 111, 59–67. doi:10.1016/j.jpsychires.2019.01.003CrossRefGoogle Scholar
Herrman, H., Patel, V., Kieling, C., Berk, M., Buchweitz, C., Cuijpers, P., … Wolpert, M. (in press) Time for united action on depression: A lancet-world psychiatric association commission. The Lancet.Google Scholar
Huibers, M. J., Cohen, Z. D., Lemmens, L. H., Arntz, A., Peeters, F. P., Cuijpers, P., & DeRubeis, R. J. (2015) Predicting optimal outcomes in cognitive therapy or interpersonal psychotherapy for depressed individuals using the personalized advantage Index approach. PLoS ONE, 10(11), e0140771. doi:10.1371/journal.pone.0140771CrossRefGoogle ScholarPubMed
Kabir, M. F., & Ludwig, S. A. (2019) Enhancing the performance of classification using super learning. Data-Enabled Discovery and Applications, 3(1), 5. doi:10.1007/s41688-019-0030-0CrossRefGoogle Scholar
Kappelmann, N., Rein, M., Fietz, J., Mayberg, H. S., Craighead, W. E., Dunlop, B. W., … Kopf-Beck, J. (2020) Psychotherapy or medication for depression? Using individual symptom meta-analyses to derive a symptom-oriented therapy (SOrT) metric for a personalised psychiatry. BMC Medicine, 18(1), 170. doi:10.1186/s12916-020-01623-9CrossRefGoogle ScholarPubMed
Karrer, T. M., Bassett, D. S., Derntl, B., Gruber, O., Aleman, A., Jardri, R., … Bzdok, D. (2019) Brain-based ranking of cognitive domains to predict schizophrenia. Human Brain Mapping, 40(15), 4487–4507. doi:10.1002/hbm.24716CrossRefGoogle ScholarPubMed
Katz, I. R., Liebmann, E. P., Resnick, S. G., & Hoff, R. A. (2021) Performance of the PHQ-9 across conditions and comorbidities: Findings from the veterans outcome assessment survey. Journal of Affective Disorders, 294, 864–867. doi:10.1016/j.jad.2021.07.108CrossRefGoogle ScholarPubMed
Kessler, R. C., & Luedtke, A. (2021) Pragmatic precision psychiatry-A New direction for optimizing treatment selection. JAMA Psychiatry, 78(12), 1384–1390. doi: 10.1001/jamapsychiatry.2021.2500.CrossRefGoogle ScholarPubMed
King, P. R., Beehler, G. P., Buchholz, L. J., Johnson, E. M., & Wray, L. O. (2019) Functional concerns and treatment priorities among veterans receiving VHA primary care behavioral health services. Families, Systems & Health, 37(1), 68–73. doi:10.1037/fsh0000393CrossRefGoogle ScholarPubMed
Koeser, L., Donisi, V., Goldberg, D. P., & McCrone, P. (2015) Modelling the cost-effectiveness of pharmacotherapy compared with cognitive-behavioural therapy and combination therapy for the treatment of moderate to severe depression in the UK. Psychological Medicine, 45(14), 3019–3031. doi:10.1017/s0033291715000951CrossRefGoogle ScholarPubMed
LeDell, E., van der Laan, M. J., & Petersen, M. (2016) AUC-Maximizing Ensembles through metalearning. International Journal of Biostatistics, 12(1), 203–218. doi:10.1515/ijb-2015-0035CrossRefGoogle ScholarPubMed
Leeuwenberg, A. M., van Smeden, M., Langendijk, J. A., van der Schaaf, A., Mauer, M. E., Moons, K. G. M., … Schuit, E. (2022) Performance of binary prediction models in high-correlation low-dimensional settings: A comparison of methods. Diagnostic and Prognostic Research, 6(1), 1. doi:10.1186/s41512-021-00115-5CrossRefGoogle ScholarPubMed
Leon, A. C., Olfson, M., Portera, L., Farber, L., & Sheehan, D. V. (1997) Assessing psychiatric impairment in primary care with the Sheehan disability scale. International Journal of Psychiatry in Medicine, 27(2), 93–105. doi:10.2190/t8em-c8yh-373n-1uwdCrossRefGoogle ScholarPubMed
Leung, L. B., Rubenstein, L. V., Yoon, J., Post, E. P., Jaske, E., Wells, K. B., & Trivedi, R. B. (2019) Veterans health administration investments In primary care And mental health integration improved care access. Health Affairs, 38(8), 1281–1288. doi:10.1377/hlthaff.2019.00270CrossRefGoogle ScholarPubMed
Leung, L. B., Ziobrowski, H. N., Puac-Polanco, V., Bossarte, R. M., Bryant, C., Keusch, J., … Kessler, R. C. (2021) Are veterans getting their preferred depression treatment? A national observational study in the veterans health administration. Journal of General Internal Medicine, Advance online publication. doi:10.1007/s11606-021-07136-2Google ScholarPubMed
Maj, M., Stein, D. J., Parker, G., Zimmerman, M., Fava, G. A., De Hert, M., … Wittchen, H. U. (2020) The clinical characterization of the adult patient with depression aimed at personalization of management. World Psychiatry, 19(3), 269–293. doi:10.1002/wps.20771CrossRefGoogle ScholarPubMed
Naeini, M. P., Cooper, G. F., & Hauskrecht, M. (2015). Obtaining Well Calibrated Probabilities Using Bayesian Binning. Retrieved from https://people.cs.pitt.edu/~milos/research/AAAI_Calibration.pdf.Google Scholar
Naimi, A. I., & Balzer, L. B. (2018) Stacked generalization: An introduction to super learning. European Journal of Epidemiology, 33(5), 459–464. doi:10.1007/s10654-018-0390-zCrossRefGoogle ScholarPubMed
Park, T., & Casella, G. (2008) The Bayesian lasso. Journal of the American Statistical Association, 103(482), 681–686. doi:10.1198/016214508000000337CrossRefGoogle Scholar
Pearson, R., Pisner, D., Meyer, B., Shumake, J., & Beevers, C. G. (2019) A machine learning ensemble to predict treatment outcomes following an internet intervention for depression. Psychological Medicine, 49(14), 2330–2341. doi:10.1017/s003329171800315xCrossRefGoogle ScholarPubMed
Polley, E., LeDell, E., Kennedy, C., Lendle, S, & van der Laan, M. J. (2021). Superlearner: Super learner prediction, version 2.0-28. Retrieved from https://CRAN.R-project.org/package=SuperLearner.Google Scholar
Polley, E. C., Rose, S., & van der Laan, M. J. (2011) Super learning. In Targeted learning: Casual inference for observational and experimental data (ed. van der Laan, M. J. and Rose, S.), pp. 43–66. Springer: New York.CrossRefGoogle Scholar
Puac-Polanco, V., Leung, L. B., Bossarte, R. M., Bryant, C., Keusch, J. N., Liu, H., … Kessler, R. C. (2021) Treatment differences in primary and specialty settings in veterans with major depression. Journal of the American Board of Family Medicine, 34(2), 268–290. doi:10.3122/jabfm.2021.02.200475CrossRefGoogle ScholarPubMed
Qaseem, A., Barry, M. J., & Kansagara, D. (2016) Nonpharmacologic versus pharmacologic treatment of adult patients with major depressive disorder: A clinical practice guideline from the American college of physicians. Annals of Internal Medicine, 164(5), 350–359. doi:10.7326/m15-2570CrossRefGoogle ScholarPubMed
R Core Team. (2021). R: A language and environment for statistical computing. Retrieved from https://www.R-project.org/.Google Scholar
Roelofs, R., Shankar, V., Recht, B., Fridovich-Keil, S., Hardt, M., Miller, J., & Schmidt, L. (2019). A Meta-Analysis of Overfitting in Machine Learning, Part of Advances in Neural Information Processing Systems 32 (NeurIPS 2019). Retrieved from https://papers.nips.cc/paper/2019/hash/ee39e503b6bedf0c98c388b7e8589aca-Abstract.html.Google Scholar
Ross, E. L., Vijan, S., Miller, E. M., Valenstein, M., & Zivin, K. (2019) The cost-effectiveness of cognitive behavioral therapy versus second-generation antidepressants for initial treatment of major depressive disorder in the United States: A decision-analytic model. Annals of Internal Medicine, 171(11), 785–795. doi:10.7326/m18-1480CrossRefGoogle ScholarPubMed
Rush, A. J., Trivedi, M. H., Ibrahim, H. M., Carmody, T. J., Arnow, B., Klein, D. N., … Keller, M. B. (2003) The 16-item quick inventory of depressive symptomatology (QIDS), clinician rating (QIDS-C), and self-report (QIDS-SR): A psychometric evaluation in patients with chronic major depression. Biological Psychiatry, 54(5), 573–583. doi:10.1016/s0006-3223(02)01866-8CrossRefGoogle ScholarPubMed
Rush, A. J., Trivedi, M. H., Wisniewski, S. R., Nierenberg, A. A., Stewart, J. W., Warden, D., … Fava, M. (2006) Acute and longer-term outcomes in depressed outpatients requiring one or several treatment steps: A STAR*D report. The American Journal of Psychiatry, 163(11), 1905–1917. doi:10.1176/ajp.2006.163.11.1905CrossRefGoogle ScholarPubMed
SAS Institute Inc (2013) SAS ®Software 9.4 edn. Cary, NC.Google Scholar
Saunders, R., Cohen, Z. D., Ambler, G., DeRubeis, R. J., Wiles, N., Kessler, D., … Buckman, J. E. J. (2021) A patient stratification approach to identifying the likelihood of continued chronic depression and relapse following treatment for depression. Journal of Personalized Medicine, 11(12). doi:10.3390/jpm11121295CrossRefGoogle ScholarPubMed
Serbanescu, I., Backenstrass, M., Drost, S., Weber, B., Walter, H., Klein, J. P., … Schoepf, D. (2020) Impact of baseline characteristics on the effectiveness of disorder-specific cognitive behavioral analysis system of psychotherapy (CBASP) and supportive psychotherapy in outpatient treatment for persistent depressive disorder. Frontiers in Psychiatry, 11, 607300. doi:10.3389/fpsyt.2020.607300CrossRefGoogle ScholarPubMed
Stolzmann, K., Meterko, M., Miller, C. J., Belanger, L., Seibert, M. N., & Bauer, M. S. (2019) Survey response rate and quality in a mental health clinic population: Results from a randomized survey comparison. The Journal of Behavioral Health Services & Research, 46(3), 521–532. doi:10.1007/s11414-018-9617-8CrossRefGoogle Scholar
Tymofiyeva, O., Yuan, J. P., Huang, C. Y., Connolly, C. G., Henje Blom, E., Xu, D., & Yang, T. T. (2019) Application of machine learning to structural connectome to predict symptom reduction in depressed adolescents with cognitive behavioral therapy (CBT). NeuroImage: Clinical, 23(101914). doi:10.1016/j.nicl.2019.101914Google ScholarPubMed
van Klaveren, D., Balan, T. A., Steyerberg, E. W., & Kent, D. M. (2019) Models with interactions overestimated heterogeneity of treatment effects and were prone to treatment mistargeting. Journal of Clinical Epidemiology, 114, 72–83. doi:10.1016/j.jclinepi.2019.05.029CrossRefGoogle ScholarPubMed
van Schaik, D. J., Klijn, A. F., van Hout, H. P., van Marwijk, H. W., Beekman, A. T., de Haan, M., & van Dyck, R. (2004) Patients’ preferences in the treatment of depressive disorder in primary care. General Hospital Psychiatry, 26(3), 184–189. doi:10.1016/j.genhosppsych.2003.12.001CrossRefGoogle ScholarPubMed
Yuan, M., Kumar, V., Ahmad, M. A., & Teredesai, A. (2021). Assessing Fairness in Classification Parity of Machine Learning Models in Healthcare. Retrieved from https://arxiv.org/abs/2102.03717.Google Scholar
Ziobrowski, H. N., Kennedy, C. J., Ustun, B., House, S. L., Beaudoin, F. L., An, X., … van Rooij, S. J. H. (2021a) Development and validation of a model to predict posttraumatic stress disorder and major depression after a motor vehicle collision. JAMA Psychiatry, 78(11), 1228–1237. doi:10.1001/jamapsychiatry.2021.2427CrossRefGoogle Scholar
Ziobrowski, H. N., Leung, L. B., Bossarte, R. M., Bryant, C., Keusch, J. N., Liu, H., … Kessler, R. C. (2021b) Comorbid mental disorders, depression symptom severity, and role impairment among veterans initiating depression treatment through the veterans health administration. Journal of Affective Disorders, 290, 227–236. doi:10.1016/j.jad.2021.04.033CrossRefGoogle ScholarPubMed
Zou, G. (2004) A modified Poisson regression approach to prospective studies with binary data. American Journal of Epidemiology, 159(7), 702–706. doi:10.1093/aje/kwh090CrossRefGoogle ScholarPubMed
Zubizarreta, J. R. (2015) Stable weights that balance covariates for estimation with incomplete outcome data. Journal of the American Statistical Association, 110(511), 910–922. doi:10.1080/01621459.2015.1023805CrossRefGoogle Scholar
Zubizarreta, J. R., Li, Y., Allouah, A., & Greifer, N. (2021). sbw: Stable balancing weights for causal inference and estimation with incomplete outcome data (Version 1.1.1). Retrieved from https://cran.rstudio.com/web/packages/sbw/.Google Scholar
Supplementary material: File

Ziobrowski et al. supplementary material

Ziobrowski et al. supplementary material 1

Download Ziobrowski et al. supplementary material(File)
File 28.7 KB
Supplementary material: File

Ziobrowski et al. supplementary material

Ziobrowski et al. supplementary material 2

Download Ziobrowski et al. supplementary material(File)
File 73 KB