Hostname: page-component-77c78cf97d-v4t4b Total loading time: 0 Render date: 2026-04-23T12:57:11.263Z Has data issue: false hasContentIssue false
  • English
  • Français

Characterizing Veteran suicide decedents that were not classified as high-suicide-risk

Published online by Cambridge University Press:  16 September 2024

Maxwell Levis Open the ORCID record for Maxwell Levis [Opens in a new window] ,
Monica Dimambro ,
Joshua Levy ,
Vincent Dufort ,
Abby Fraade ,
Max Winer  and
Brian Shiner
Maxwell Levis*
Affiliation:
White River Junction VA Medical Center, White River Junction, VT, USA Department of Psychiatry, Geisel School of Medicine at Dartmouth, Hanover, NH, USA
Monica Dimambro
Affiliation:
White River Junction VA Medical Center, White River Junction, VT, USA
Joshua Levy
Affiliation:
Pathology and Computational Biomedicine, Cedars Sinai Medical Center, Los Angeles, CA, USA
Vincent Dufort
Affiliation:
White River Junction VA Medical Center, White River Junction, VT, USA
Abby Fraade
Affiliation:
Long Island University, Brooklyn, NY, USA
Max Winer
Affiliation:
White River Junction VA Medical Center, White River Junction, VT, USA
Brian Shiner
Affiliation:
White River Junction VA Medical Center, White River Junction, VT, USA Department of Psychiatry, Geisel School of Medicine at Dartmouth, Hanover, NH, USA National Center for PTSD Executive Division, White River Junction, VTS, USA
*
Corresponding author: Maxwell Levis; Email: maxwelle.levis@va.gov
Get access Rights & Permissions [Opens in a new window]

Abstract

Background

Although the Department of Veterans Affairs (VA) has made important suicide prevention advances, efforts primarily target high-risk patients with documented suicide risk, such as suicidal ideation, prior suicide attempts, and recent psychiatric hospitalization. Approximately 90% of VA patients that go on to die by suicide do not meet these high-risk criteria and therefore do not receive targeted suicide prevention services. In this study, we used national VA data to focus on patients that were not classified as high-risk, but died by suicide.

Methods

Our sample included all VA patients who died by suicide in 2017 or 2018. We determined whether patients were classified as high-risk using the VA's machine learning risk prediction algorithm. After excluding these patients, we used principal component analysis to identify moderate-risk and low-risk patients and investigated demographics, service-usage, diagnoses, and social determinants of health differences across high-, moderate-, and low-risk subgroups.

Results

High-risk (n = 452) patients tended to be younger, White, unmarried, homeless, and have more mental health diagnoses compared to moderate- (n = 2149) and low-risk (n = 2209) patients. Moderate- and low-risk patients tended to be older, married, Black, and Native American or Pacific Islander, and have more physical health diagnoses compared to high-risk patients. Low-risk patients had more missing data than higher-risk patients.

Conclusions

Study expands epidemiological understanding about non-high-risk suicide decedents, historically understudied and underserved populations. Findings raise concerns about reliance on machine learning risk prediction models that may be biased by relative underrepresentation of racial/ethnic minorities within health system.

Keywords

machine learning biasmachine learningsuicide preventionveterans and military

Information

Type
Original Article
Information
Psychological Medicine , Volume 54 , Issue 11 , August 2024 , pp. 3135 - 3144
Check for updates
Creative Commons
This is a work of the US Government and is not subject to copyright protection within the United States. Published by Cambridge University Press
Copyright
Copyright © US Department of Veterans Affairs, 2024

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Afrose, S., Song, W., Nemeroff, C. B., Lu, C., & Yao, D. (2022). Subpopulation-specific machine learning prognosis for underrepresented patients with double prioritized bias correction. Communications Medicine, 2(1), 111. https://doi.org/10.1038/s43856-022-00165-wCrossRefGoogle ScholarPubMed
Ahmedani, B. K., & Vannoy, S. (2014). National pathways for suicide prevention and health services research. American Journal of Preventive Medicine, 47(3), S222–S228. https://doi.org/10.1016/j.amepre.2014.05.038CrossRefGoogle ScholarPubMed
Bailey, R., Mokonogho, J., & Kumar, A. (2019). Racial and ethnic differences in depression: Current perspectives. Neuropsychiatric Disease and Treatment, 15, 603–609. https://doi.org/10.2147/NDT.S128584CrossRefGoogle ScholarPubMed
Botero, G., Rivera, N. I., Calloway, S. C., Ortiz, P. L., Edwards, E., Chae, J., & Geraci, J. C. (2020). A lifeline in the dark: Breaking through the stigma of Veteran mental health and treating America's combat Veterans. Journal of Clinical Psychology, 76(5), 831–840. https://doi.org/10.1002/jclp.22918CrossRefGoogle ScholarPubMed
Cannizzaro, K. (2017). REACH VET and the possible impact on integrated healthcare. VA Health Care. Rocky Mountain Regional VAMC, Aurora, CO. https://avapl.org/conference/pubs/2018%20Conference%20Presentations/Cannizzaro%20-%20slides%20-%20Open%20Plenary.pdfGoogle Scholar
Ding, C., & He, X. (2004). K-means clustering via principal component analysis. Twenty-first international conference on machine learning – ICML ’04, 29. https://doi.org/10.1145/1015330.1015408CrossRefGoogle Scholar
Elnitsky, C. A., Andresen, E. M., Clark, M. E., McGarity, S., Hall, C. G., & Kerns, R. D. (2013). Access to the US Department of Veterans Affairs health system: Self-reported barriers to care among returnees of operations enduring freedom and Iraqi freedom. BMC Health Services Research, 13(1), 498. https://doi.org/10.1186/1472-6963-13-498CrossRefGoogle Scholar
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018). Potential biases in machine learning algorithms using electronic health record data. JAMA Internal Medicine, 178(11), 1544. https://doi.org/10.1001/jamainternmed.2018.3763CrossRefGoogle ScholarPubMed
Hein, T. C., Peltzman, T., Hallows, J., Theriot, N., & McCarthy, J. F. (2021). Suicide mortality among Veterans health administration care recipients with suicide risk record flags. Psychiatric Services, 73(3), 259–264. https://doi.org/10.1176/appi.ps.202000771CrossRefGoogle ScholarPubMed
Huang, J., Galal, G., Etemadi, M., & Vaidyanathan, M. (2022). Evaluation and mitigation of racial bias in clinical machine learning models: Scoping review. JMIR Medical Informatics, 10(5), e36388. https://doi.org/10.2196/36388CrossRefGoogle ScholarPubMed
Jobes, D. A., Haddock, L. A., & Olivares, M. R. (2019). Military and Veteran suicide prevention. In Ritchie, E. C., & Llorente, M. D. (Eds.), Veteran psychiatry in the US (pp. 51–71). New York, NY: Springer International Publishing. https://doi.org/10.1007/978-3-030-05384-0_5CrossRefGoogle Scholar
Jolliffe, I. T. (2002). Principal component analysis. New York, NY: Springer-Verlag. https://doi.org/10.1007/b98835Google Scholar
Kessler, R. C., Bossarte, R. M., Luedtke, A., Zaslavsky, A. M., & Zubizarreta, J. R. (2020). Suicide prediction models: A critical review of recent research with recommendations for the way forward. Molecular Psychiatry, 25(1), 168–179. https://doi.org/10.1038/s41380-019-0531-0CrossRefGoogle ScholarPubMed
Kessler, R. C., Hwang, I., Hoffmire, C. A., McCarthy, J. F., Petukhova, M. V., Rosellini, A. J., … Bossarte, R. M. (2017). Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans health administration. International Journal of Methods in Psychiatric Research, 26(3), e1575. https://doi.org/10.1002/mpr.1575CrossRefGoogle ScholarPubMed
Klimes-Dougan, B., Klingbeil, D. A., & Meller, S. J. (2013). The impact of universal suicide-prevention programs on the help-seeking attitudes and behaviors of youths. Crisis, 34(2), 82–97. https://doi.org/10.1027/0227-5910/a000178CrossRefGoogle ScholarPubMed
Lavingia, R., Jones, K., & Asghar-Ali, A. A. (2020). A systematic review of barriers faced by older adults in seeking and accessing mental health care. Journal of Psychiatric Practice, 26(5), 367–382. https://doi.org/10.1097/PRA.0000000000000491CrossRefGoogle ScholarPubMed
Levis, M., Levy, J., Dent, K. R., Dufort, V., Gobbel, G. T., Watts, B. V., & Shiner, B. (2023a). Leveraging natural language processing to improve electronic health record suicide risk prediction for Veterans health administration users. The Journal of Clinical Psychiatry, 84(4), 22m14568. https://doi.org/10.4088/JCP.22m14568CrossRefGoogle ScholarPubMed
Levis, M., Levy, J., Dufort, V., Russ, C. J., & Shiner, B. (2023b). Dynamic suicide topic modelling: Deriving population-specific, psychosocial and time-sensitive suicide risk variables from electronic health record psychotherapy notes. Clinical Psychology & Psychotherapy, 30(4), 795810. https://doi.org/10.1002/cpp.2842CrossRefGoogle ScholarPubMed
Mann, J. J., Michel, C. A., & Auerbach, R. P. (2021). Improving suicide prevention through evidence-based strategies: A systematic review. American Journal of Psychiatry, 178(7), 611–624. https://doi.org/10.1176/appi.ajp.2020.20060864CrossRefGoogle ScholarPubMed
Matarazzo, B. B., Brenner, L. A., & Reger, M. A. (2019). Positive predictive values and potential success of suicide prediction models. JAMA Psychiatry, 76(8), 869. https://doi.org/10.1001/jamapsychiatry.2019.1519CrossRefGoogle ScholarPubMed
Matarazzo, B. B., Eagan, A., Landes, S. J., Mina, L. K., Clark, K., Gerard, G. R., … Reger, M. A. (2023). The Veterans health administration REACH VET program: Suicide predictive modeling in practice. Psychiatric Services, 74(2), 206–209. https://doi.org/10.1176/appi.ps.202100629CrossRefGoogle ScholarPubMed
Mattocks, K. M., Cunningham, K., Elwy, A. R., Finley, E. P., Greenstone, C., Mengeling, M. A., Pizer, S. D., … Bastian, L. A. (2019). Recommendations for the evaluation of cross-system care coordination from the VA state-of-the-art working group on VA/non-VA care. Journal of General Internal Medicine, 34(S1), 18–23. https://doi.org/10.1007/s11606-019-04972-1CrossRefGoogle Scholar
McCarthy, J. F., Bossarte, R. M., Katz, I. R., Thompson, C., Kemp, J., Hannemann, C. M., … Schoenbaum, M. (2015). Predictive modeling and concentration of the risk of suicide: Implications for preventive interventions in the US Department of Veterans Affairs. American Journal of Public Health, 105(9), 1935–1942. https://doi.org/10.2105/AJPH.2015.302737CrossRefGoogle ScholarPubMed
McCarthy, J. F., Cooper, S. A., Dent, K. R., Eagan, A. E., Matarazzo, B. B., Hannemann, C. M., … Katz, I. R. (2021). Evaluation of the recovery engagement and coordination for health – Veterans enhanced treatment suicide risk modeling clinical program in the Veterans health administration. JAMA Network Open, 4(10), e2129900. https://doi.org/10.1001/jamanetworkopen.2021.29900CrossRefGoogle ScholarPubMed
Meffert, B. N., Morabito, D. M., Sawicki, D. A., Hausman, C., Southwick, S. M., Pietrzak, R. H., & Heinz, A. J. (2019). US Veterans who do and do not utilize Veterans Affairs health care services: Demographic, military, medical, and psychosocial characteristics. The Primary Care Companion for CNS Disorders, 21(1), 18m02350. https://doi.org/10.4088/PCC.18m02350CrossRefGoogle Scholar
Miller-Matero, L. R., Yeh, H.-H., Maffett, A., Mooney, J. T., Sala-Hamrick, K., Frank, C. B., … Ahmedani, B. K. (2023). Racial-ethnic differences in receipt of past-year health care services among suicide decedents: A case-control study. Psychiatric Services, 75(2), 124–130. https://doi.org/10.1176/appi.ps.20220578CrossRefGoogle ScholarPubMed
Nock, M. K., Ramirez, F., & Rankin, O. (2019). Advancing our understanding of the who, when, and why of suicide risk. JAMA Psychiatry, 76(1), 11. https://doi.org/10.1001/jamapsychiatry.2018.3164CrossRefGoogle ScholarPubMed
Nong, P., Williamson, A., Anthony, D., Platt, J., & Kardia, S. (2022). Discrimination, trust, and withholding information from providers: Implications for missing data and inequity. SSM – Population Health, 18, 101092. https://doi.org/10.1016/j.ssmph.2022.101092CrossRefGoogle ScholarPubMed
OMHSP, . (2021). National Veteran Suicide Prevention annual report. VA Office of Mental Health and Suicide Prevention, Washington DC. https://www.mentalhealth.va.gov/docs/data-sheets/2021/2021-National-Veteran-Suicide-Prevention-Annual-Report-FINAL-9-8-21.pdfGoogle Scholar
Peltzman, T., Rice, K., Jones, K. T., Washington, D. L., & Shiner, B. (2022). Optimizing data on race and ethnicity for Veterans Affairs patients. Military Medicine, 187(7–8), e955–e962. https://doi.org/10.1093/milmed/usac066CrossRefGoogle ScholarPubMed
Serneels, S., & Verdonck, T. (2008). Principal component analysis for data containing outliers and missing elements. Computational Statistics & Data Analysis, 52(3), 1712–1727. https://doi.org/10.1016/j.csda.2007.05.024CrossRefGoogle Scholar
Shiner, B., Levis, M., Dufort, V. M., Patterson, O. V., Watts, B. V., DuVall, S. L., … Maguen, S. (2021). Improvements to PTSD quality metrics with natural language processing. Journal of Evaluation in Clinical Practice, 28(4), 520–530. https://doi.org/10.1111/jep.13587CrossRefGoogle ScholarPubMed
Stanley, B., Brown, G., Brent, D. A., Wells, K., Poling, K., Curry, J., … Hughes, J. (2009). Cognitive-behavioral therapy for suicide prevention (CBT-SP): Treatment model, feasibility, and acceptability. Journal of the American Academy of Child & Adolescent Psychiatry, 48(10), 1005–1013. https://doi.org/10.1097/CHI.0b013e3181b5dbfeCrossRefGoogle ScholarPubMed
Steinley, D. (2004). Properties of the Hubert-Arable adjusted rand index. Psychological Methods, 9(3), 386–396. https://doi.org/10.1037/1082-989X.9.3.386CrossRefGoogle ScholarPubMed
Sullivan, J. L., Simons, K. V., Mills, W. L., Hilgeman, M. M., Freytes, I. M., Morin, R. T., … Byers, A. L. (2023). A paucity of data on Veterans 65 and older and risk of suicide: A systematic review. The American Journal of Geriatric Psychiatry, 31(7), 525–539. https://doi.org/10.1016/j.jagp.2022.11.005CrossRefGoogle ScholarPubMed
VA/DoD. (2017). Joint Department of Veterans Affairs (VA) and Department of Defense (DoD) Mortality Data Repository – National Death Index (NDI). Center of Excellence for Suicide Prevention, Washington DC. https://www.mirecc.va.gov/suicideprevention/documents/VA_DoD-MDR_Flyer.pdfGoogle Scholar
VA/DoD. (2019). VA/DoD clinical practice guideline for the assessment and management of patients at risk for suicide (2.0). The Assessment and Management of Suicide Risk Work Group, Washington DC. https://www.healthquality.va.gov/guidelines/mh/srb/Google Scholar
Wasserman, D., Tadić, I., & Bec, C. (2023). Vision zero in suicide prevention and suicide preventive methods. In Björnberg, K. E., Hansson, S. O., Belin, M.-Å., & Tingvall, C. (Eds.), The vision zero handbook (pp. 1117–1142). New York, NY: Springer International Publishing. https://doi.org/10.1007/978-3-030-76505-7_43CrossRefGoogle Scholar