Suicide prediction tools, traditionally pen-and-paper tests that derive a score by the accumulation of suicide risk factors, have been used for decades in an attempt to identify patients at particularly significant risk of suicide. Reference Pokorny1–Reference Steeg, Kapur, Webb, Applegate, Stewart and Hawton4 Many researchers and some healthcare systems have concluded that suicide risk prediction tools do not provide sufficient information to be clinically useful, Reference Steeg, Quinlivan, Nowland, Carroll, Casey and Clements5–Reference Ribeiro, Franklin, Fox, Bentley, Kleiman and Chang9 chiefly because they produce far too many false positives and miss too many future suicides. Reference Large, Ryan, Carter and Kapur10,Reference Graney, Hunt, Quinlivan, Rodway, Turnbull and Gianatsi11 In recent years a new wave of tools, some assisted by machine learning, have been published Reference Kessler, Hwang, Hoffmire, McCarthy, Petukhova and Rosellini12–Reference Kennedy, Kearns, Geraci, Gildea, Hwang and King14 and advocated for. Reference Seyedsalehi and Fazel15
The appeal of such tools – the promise to identify which patients are likely to go on to die by suicide so that they can be targeted for treatment – appears clear. However, these tools also have a number of limitations and hidden dangers. Here, we explore what clinicians might make of the next generation of tools by considering one of the recently published and accessible suicide risk assessment scales, OxSATS. Reference Fazel, Vazquez-Montes, Molero, Runeson, D’Onofrio and Larsson13 We do so using eight plausible but fictional clinical vignettes.
OxSATS
OxSATS is a tool developed to assess risk of suicide in people who have presented to an accident and emergency department with self-harm. It uses 11 items (Table 1), determined by a model from 35 possible items, to produce a probabilistic (percentage) estimate of suicide for each person within 6 and 12 months. It was developed and tested on the record data of 53 172 patients in Sweden between 2008 and 2012.
Items used by OxSATS
OxSATS can be completed quickly, does not rely on a detailed assessment and is objective. What is new about OxSATS is that, rather than producing risk categories, it provides a probability estimate expressed as percentage suicide mortality within 6 and 12 months, for each patient. This score can be derived using a freely available online calculator. Reference Fazel, Vazquez-Montes, Molero, Runeson, D’Onofrio and Larsson13 OxSATS selects variables through stepwise selection rather than machine learning. Unlike some machine learning tools, it is transparent in letting us see how risk is scored. It is designed to be used alongside clinicians as a clinical adjunct, not as a replacement. However, the authors suggest that it could be used to allocate treatment to those most at risk.
While OxSATS’ statistical performance is not the focus of this paper, and no single statistic captures the value of a tool, its predictive accuracy is comparable to older methods when thresholds are applied; such thresholds would probably be required for use in clinical practice. Using a 2% risk threshold, OxSATS’ sensitivity (43%) and specificity (81%) closely match those in Pokorny’s 1983 study (45 and 84%, respectively), although OxSATS has a lower positive predictive value (PPV) (2.5 v. 3.7%). Reference Pokorny1 OxSATS’ sensitivity here (43%) means that more than half of those who died by suicide fell below the threshold; at a 1% threshold, OxSATS’ odds ratio (5.36) aligns with that of a previous meta-analysis (4.84). Reference Large, Kaneson, Myles, Myles, Gunaratne and Ryan16 OxSATS’ area under the curve (AUC) score (0.75) is similar to that of traditional tools (0.71). Reference Steeg, Quinlivan, Nowland, Carroll, Casey and Clements5
Vignettes
In Box 1 we present eight fictionalised clinical vignettes of typical patients who present to accident and emergency departments with self-harm. None require overnight admission to a medical bed. The information is brief, as might be recorded in accident and emergency triage, but has sufficient data to calculate OxSATS scores. For each of the presentations we have recorded the 12-month estimated probability of suicide according to OxSATS. A discussion follows about the probabilistic estimates produced by OxSATS and how they might affect patients, clinicians and the systems in which they are used.
Case vignettes of eight fictionalised patients and their OxSATS scores
EUPD, emotionally unstable personality disorder (ICD-11: 6D11.5); GP, general practitioner.
Results
Once seen by clinicians, risk scores will seem salient, and most clinicians would accept that probability estimates generated by OxSATS provide some information about future suicide risk. The tool is both intended to, and in all likelihood would, influence clinician decision-making: sometimes nudging clinicians’ decision-making in the direction of the tool’s score, sometimes overtly influencing or even obligating decisions and actions by clinicians. It is intended to help decision-making, but the actual potential effect on both clinicians and patients requires careful consideration.
Vignette 1: false reassurance, risk of storing up problems for the future
This 13-year-old girl clearly needs a careful assessment to understand, and help her understand, the causes of her distress. Would knowing that her 12-month risk score is 0.0% help her, help her clinician or help her access the support she needs? OxSATS authors suggest that, in future, tools could ‘act as a screen for more detailed assessment’ and ‘allow for more efficient allocation of clinical resources’. Reference Fazel, Vazquez-Montes, Molero, Runeson, D’Onofrio and Larsson13 Should clinicians take this to mean that the patient in vignette 1 might not need a full psychosocial assessment or follow-up support? Even if her immediate risk of suicide is low, her distress is real and the need for intervention urgent. The focus of mental healthcare should be on her many and immediate needs.
This case demonstrates the risk that such a tool would de-prioritise ‘lower-risk’ patients. Unseen and untreated, will her risk stay at 0%? Today’s lower-risk patients will become tomorrow’s ‘higher-risk’ patients, Reference Gianatsi, Burns, Hunt, Ibrahim, Windfuhr and While17 let alone tomorrow’s unhappy person failing to thrive. As such, in cases like these the use of OxSATS may provide unwarranted reassurance and increase the likelihood of leaving illness untreated and distress unexplored, only to present more severely in the future.
Vignette 2: underestimation of groups and vulnerability to exclusion culture
This vignette represents a patient diagnosed with emotionally unstable personality disorder (EUPD) experiencing a crisis. At present, around 10% of patients with EUPD die by suicide in their lifetime. Reference Paris18 Clinicians may feel that 0.6% represents a significant underestimation of risk for this patient, who needs careful assessment, validation, psychoeducation and referral to appropriate treatment pathways. Furthermore, although 0.6% is below the median score in the OxSATS sample, it translates as 65 times the 2021 global suicide rate (9.1/100 000). 19
This vignette illustrates a danger regarding the way in which OxSATS scores could be used in overstretched services. When attempting onward referral, would her relatively low suicide risk score be used as an excuse for exclusion? Reference Beale20 We fear that it may be.
Furthermore, would the illusions of certainty, precision and objectivity from the score particularly affect less experienced clinicians Who may rely on the number and overlook or ignore worrying aspects of the presentation?
Vignette 3: missed opportunities, prioritising perceived suicide risk over other debilitating mental illness experiences
The young man here probably has emergent schizophrenia. He too needs careful assessment to identify, explain and treat the underlying psychiatric cause. Were OxSATS to be used as a screen for more detailed assessment, he may not get that assessment at all. The role of psychiatric assessment in accident and emergency departments is about much more than predicting risk. In this case, assessment serves as a hugely important screen for a treatable mental illness. As in vignette 1, untreated, this patient’s risk of suicide is only likely to rise.
However, the most pressing problems faced by the patient in vignette 3 are not suicide – he has significant functional impairments and is falling off his developmental trajectory. Identifying the underlying cause, and understanding and addressing his social withdrawal and academic difficulties, can emerge only in a more complete, relational, assessment.
Vignettes 4 and 8: priming overtreatment and iatrogenic harm of higher-risk patients
The patient in vignette 4 may well be experiencing an adjustment reaction to a significant social stressor (marital breakdown). He requires an assessment to screen for mental illness, identify strengths and form a mutually agreed plan. The patient in vignette 8 is probably experiencing a psychotic relapse after stopping risperidone, and needs assessment of why he stopped and how and where to restart.
High scores according to OxSATS may alert staff to high-risk individuals who might otherwise be missed. However, this benefit comes with a cost. On seeing either of these OxSATS scores, clinicians, who are already prone to be being risk averse, Reference Beale20 may feel obliged to act – or be seen to act – on the high-risk scores. At the most restrictive end this may involve admitting someone to hospital, perhaps coercively or directly against their will. Alternatively, it may involve inappropriate prescribing or interventions and referrals that the patient does not want. Treatment such as involuntary admission may serve to help a patient but may also represent defensive psychiatry, Reference Reuveni, Pelov, Reuveni, Bonne and Canetti21 serving not to benefit that patient but limit a clinician’s liability. Reference Riolo and Babici22,Reference Brown and Rayne23
Hospitalisation does not prevent suicide, and it may even increase the risk. Reference Large, Ryan, Walsh, Stein-Parbury and Patfield24 Suicide rates are notably high following discharge from psychiatric hospital and subsequently remain elevated for decades. The suicide rate among discharged patients is around 25 times that in the general population, even over 10 years following discharge. Reference Chung, Ryan, Hadzi-Pavlovic, Singh, Stanton and Large25 Suicide risk following discharge for patients admitted with suicidal thoughts/behaviours is particularly high – around 200 times the baseline rate (2078/100 000). Reference Chung, Ryan, Hadzi-Pavlovic, Singh, Stanton and Large25 The extent to which this is the effect of selecting a very high-risk group, versus hospitalisation at times being harmful, is unclear. Reference Large and Kapur26 However, it is clear that the benefits of hospitalisation need to be carefully weighed against its risks, and it is possible that hospitalisation may delay – or even magnify – the high-risk period following self-harm. In this case, admitting the man in vignette 4, perhaps influenced by his OxSATS score, would probably be inappropriate, unhelpful and may even serve to increase, not decrease, his risk of suicide.
As has always been the case, the great majority of so-called higher-risk patients do not go on to die by suicide. We do not have effective, low-side-effect interventions for suicide Reference Fox, Huang, Guzmán, Funsch, Cha and Ribeiro27 and, if we did, they should not be denied to lower-risk patients, among whom many suicides occur. With our present interventions, it is these higher-risk patients, as in vignette 4, who pay the cost of interventions, including coercive treatment and occasionally harmful hospitalisation, even when their individual likelihood of suicide is low.
Vignettes 5 and 6: systematic de-prioritisation of demographic groups (especially young and adolescent women)
Aside from demographic factors, these are identical presentations. It would be clear to clinicians that vignette 5 carries a higher risk of death by suicide than vignette 6. To what extent should that determine assessment or treatment? If used clinically, the influence of OxSATS on clinicians would probably introduce a systematic gender and age bias, prioritising services towards attending to the risk of older men over the suffering of younger women. Because adolescent women are the fastest-growing demographic with mental illness, Reference McManus, Bebbington, Jenkins and Brugha28 de-prioritising them would be contentious to say the least. If clinicians and services wanted to introduce such tools, their introduction of such biases would necessitate candid, thoughtful discussions among professionals, patients and healthcare funders.
Vignette 7: impact of OxSATS score on patients
With this vignette, what are we to make of her score of 1.7%? What degree of confidence can we have in this estimate? Should we give her this information? And, if so, how should she understand it and what would she make of it? She may feel that the score is high. Perhaps it serves as an important message that she needs change in her life; but perhaps the score implies a mechanical, destined process that disempowers her and takes away her agency. Alternatively, she may feel that the score is low. Perhaps it devalues her feelings and experiences, making her feel unheard. Perhaps it could make her think that the only way she can get the help she needs is to increase her risk score. There are many ways in which communicating this score to patients could be harmful. In truth, we cannot truly know how an algorithmically assigned score would impact a patient in significant distress in the aftermath of a suicide attempt in a hospital emergency department.
Discussion
There are a variety of risk prediction tools published and under development in this area. Reference Pigoni, Delvecchio, Turtulici, Madonna, Pietrini and Cecchetti29 They vary in terms of their setting, the population being screened and the outcomes they predict. In taking people who have presented following self-harm, OxSATS is trying to parse a high-risk, very distressed group in which we argue that all patients, even those scoring 0% risk, require careful assessment and consideration. Importantly, the vignettes demonstrate where tools like OxSATS have the potential to do harm. These also risk overshadowing the role of the clinical assessment of the needs of people who self-harm. That assessment is about much more than attempting to predict the risk of suicide.
Psychiatry’s perceived lack of risk prediction tools is sometimes compared unfavourably with the many risk prediction tools in general medicine. Reference Seyedsalehi and Fazel15 A particular comparison made is with cardiovascular disease when using tools such as Framingham or QRISK, Reference Seyedsalehi and Fazel15 which give patients a percentage risk of having a cardiovascular event within 10 years. Reference Hippisley-Cox, Coupland and Brindle30 The closest analogy would be that a suicide attempt (rather than death by suicide) is akin to a cardiovascular event. OxSATS is designed for use following a suicide attempt. This is like using a cardiac risk score after a patient experiences a heart attack.
More importantly, we think the comparison with cardiovascular disease fundamentally misunderstands what suicide is. Cardiovascular disease, in addition to well-described risk factors, has a known, well-understood final pathway to an event, which follows biological and largely predictable processes (insufficient perfusion of tissue, usually via a clot). No such comparison exists for suicidal thoughts and behaviours. These have innumerable pathways leading to them and are part of a complex syndrome with biological, psychological, social and existential dimensions, many origins and huge stochasticity. 31,Reference Soper, Ocejo and Large32 As such, likening cardiovascular disease to suicide is inappropriate and a category error. Furthermore, it carries implications of disempowering people as being at the mercy of their neurobiology or destiny, rather than promoting their agency.
Comparison of these suicide risk tools to QRISK is also currently of limited use. QRISK allows patients to explore how their risk of cardiovascular events would alter if they were to make changes – lose weight, lower blood pressure, stop smoking, etc. OxSATS is purely predictive and cannot do this. If vignette 5 or 6 tried to use the tool in this way, they would perversely find that stopping their psychotropic (lithium) had reduced their risk score, despite randomised trials showing that lithium reduces suicide risk. Reference Cipriani, Hawton, Stockton and Geddes33 OxSATS does not help clinicians in the search with our patients for modifiable factors in their presentation.
Importantly, there is no equivalent of a statin for suicide, no low-side-effect intervention that can be safely and effectively given to all higher-risk patients. In contrast, specific interventions to treat suicidal thoughts and behaviours are limited, with small effect sizes across outcomes, a situation largely unchanged for over 50 years. Reference Fox, Huang, Guzmán, Funsch, Cha and Ribeiro27 Classifying people by perceived risk has little value if we lack interventions for those identified as being at greater risk.
Suicide is as much a cultural phenomenon as a medical one. To what extent can a Swedish sample from 2008 to 2012 be applied to a different country today? World Health Organization data from Sweden and the UK in 2012 show significant differences in suicide rates. 34 Overall age-adjusted rates (per 100 000) were nearly twice as high in Sweden (11.1 v. 6.2), with a particular increase in suicide in Swedish versus UK women (×2.35) and different rates in each country by age distribution. 34 OxSATS was validated on a sample of the same Swedish population on which it was developed. Reference Fazel, Vazquez-Montes, Molero, Runeson, D’Onofrio and Larsson13 Given the difference in populations and interventions, its performance on patients in other countries is likely to be limited. In practice, if applied to the UK, the scores are likely to overestimate risk in most people, particularly women.
OxSATS is based on observational data; the patients on which it was trained would have received assessments and interventions routinely. As such, lower-risk patients may be lower risk because they responded well to the intervention. It is not possible to justify removing the intervention based on a model developed from such observational data.
Tools such as OxSATS are usually deliberately designed to achieve high reliability by including only easily collected, unambiguous, quantitative data. However, a person’s true risk is likely formed of a combination of (a) nomothetic data – generalisable, group-level data, (b) idiographic data – based on the unique, variable circumstances of each individual and (c) data that cannot be known or accessed. The acknowledgement that tools can only be adjuncts to clinicians is partially because tools cannot easily access idiographic data. However, even the best assessment will leave unknowns that contribute to a patient’s true risk. Five minutes after leaving accident and emergency, a person may go through a relationship breakdown, or have an old friend unexpectedly reach out, or find their home burgled, or receive good news on a much-longed-for job. None of us can predict these unknowns, but clinicians can get a sense of the idiographic data and the patient in front of them. Experienced clinicians, who also implicitly use idiographic data, outperform some predictive tools in physical healthcare. Reference Penaloza, Verschuren, Meyer, Quentin-Georget, Soulie and Thys35 Whether clinicians outperform suicide and self-harm tools is very hard to test, because clinicians will inevitably do all they can to reduce risk in patients they think are high-risk, in effect trying to prove themselves wrong. Given this, we should keep an open mind that good clinicians can and will outperform these tools.
As discussed in vignette 7, tools like this risk disempowering patients and tend to be disliked by them. Reference Hawton, Lascelles, Pitman, Gilbert, Silverman and Keith7,Reference Graney, Hunt, Quinlivan, Rodway, Turnbull and Gianatsi11 We must be conscious too that the focus on suicide risk over and above other aspects of mental illness can disempower clinicians. We become risk predictors and managers, rather than professionals using our experience and skills to develop a therapeutic relationship, diagnose and treat mental illness and help people in distress. We must guard against a culture that renders patients bearers of risk to be feared, rather than people whom we have the opportunity to help. Reference Gibbons36,Reference Appel37
Alternative
At present, the key intervention for people who have self-harmed, attempted suicide or are acutely suicidal is an individualised therapeutic assessment. There is convergence on this point from colleagues from different perspectives of suicide research. Reference Hawton, Lascelles, Pitman, Gilbert, Silverman and Keith7,Reference Large, Ryan, Carter and Kapur10,Reference Kapur, Tham, Turnbull, Richards, Rodway and Clements38 Gibbons describes suicide as an ‘action take(ing) the place of feeling’. She posits that one role of professionals is to help patients ‘put the feelings into words to encourage expression of pain’, ultimately aiming to ‘reduce the risk of action’. Reference Gibbons36 Each presentation of a suicidal person is unique, with different complexities, circumstances, risk factors and best approaches to management. Key things for clinicians to bear in mind during assessments are shown in Box 2.
Suggestions for areas to focus on during a psychosocial assessment, drawing on Large et al and Hawton et alReference Hawton, Lascelles, Pitman, Gilbert, Silverman and Keith7,Reference Large, Ryan, Carter and Kapur10
There are other important areas for clinicians and researchers to focus on, such as: how to build better therapeutic relationships; training clinical skills in helping patients manage strong emotions and crises; carefully assessing the effectiveness of newer interventions such as safety plans; incorporating patient experiences into practice; reducing burnout in clinicians; working to reduce bureaucracy so that more clinician time can be spent with patients and less completing documentation (just 17% of nursing time on in-patient wards was spent on direct care in one study Reference Glantz and Sunnqvist39 ); and, as a society, how to help the 74% of people who are not known to psychiatric services in the year prior to suicide. 40
It may be that, in the future, new tools – perhaps aided by machine learning or artificial intelligence – will allow us to identify certain patients, in certain settings, at particularly high or low risk of suicide. However, the impact and value of such tools would need to be very carefully considered, with a number of pitfalls to avoid. Reference Kirtley, van Mens, Hoogendoorn, Kapur and de Beurs41 As we have argued here, implementation of tools may even be harmful and the role of clinicians is much broader than risk prediction.
Presently it remains the case that tools are not able to give clinicians sufficient meaningful information regarding who will go on to die by suicide to guide assessment or management. Our conclusions here align with and support the current National Institute for Health and Care Excellence guidance to that effect. 6 Suicidal actions are expressions of distress that demand our respect and attention. Every patient who presents with self-harm and suicidal behaviour should be met with a comprehensive, sensitive, patient-centred, psychosocial assessment, with onward treatment decisions made collaboratively on the basis of person-specific needs. The efforts of many healthcare services around the world to deliver this standard of care should be celebrated.
Data availability
The data produced by the OxSATS tool can be recreated at its website (at the time of publishing), https://oxrisk.com/oxsats/, or are available on request from the corresponding author (A.C).
Acknowledgements
A.C. thanks the following colleagues for comments on a draft: Dr Karen Lascelles, psychiatry nurse consultant; Julie Cooper, expert by experience; Fiona Brand, psychiatric liaison and research nurse; and Dr Kezia Lange, MRCPsych.
Author contributions
Both authors contributed to the drafting and final version of the manuscript, and approved it for submission.
Funding
This research received no specific grant from any funding agency, commercial or not-for-profit sectors. A.C. is supported by an Academic Clinical Fellowship funded by the University of Oxford Department of Psychiatry.
Declaration of interest
A.C., none. M.L. has provided evidence in legal matters following suicides.
Open access
eLetters
No eLetters have been published for this article.