Main findings

Our results indicate that individual items from risk scales performed relatively poorly in terms of predicting repeat self-harm. The individual scale items, including history of self-harm, previous psychiatric treatment and suicidal intent failed to outperform clinician and patient global estimations of risk. Scale items relating to previous suicide attempt and previous psychiatric treatment performed slightly better than the SAD PERSONS and Modified SAD PERSONS scales they were part of. Other scale items performed about the same or worse than the overall scales. The scale that we constructed using a combination of the items with the highest predictive accuracy did not enhance the predictive performance for repeat self-harm. Global accuracy was generally poor for the newly constructed scale and did not outperform patient and clinician estimations of risk. Across the individual scale items and constructed scale, more highly sensitive items often had poor specificity, and vice versa. Our findings suggest that despite the potential importance of these items as part of assessments, there is little clinical utility in the use of the items in the prediction of self-harm, either in isolation or combination.

Strengths and limitations

The patients in our study are similar to patient samples in prior representative multicentre studies in terms of gender, method of self-harm and age.^{Reference Cooper, Kapur, Dunning, Guthrie, Appleby and Mackway-Jones19,Reference Bergen, Hawton, Waters, Cooper and Kapur36} However, patients who did not complete the research assessments or who did not meet eligibility criteria (for example, those who did not speak English, those who were too medically unstable, or those who were actively psychotic) were not included, which could affect generalisability to other populations. The cohort we analysed had a high proportion of participants with a prior history of self-harm (74%) and a high repetition rate (30%), which could suggest that our participants had higher levels of clinical need.^{Reference Quinlivan, Cooper, Meehan, Longson, Potokar and Hulme11}

We used multiple methods to test predictive accuracy of scale items used in real-world clinical settings, including the use of algorithmic methods. However, it is possible that alternative machine learning approaches using larger samples would result in improved predictive accuracy. The sample size in the present study was too small to enable prediction of suicide to be assessed. A recent study using routine clinical data, collected as part of a self-harm monitoring system, was sufficiently powered to examine the accuracy of risk scales for predicting suicide.^{Reference Steeg, Quinlivan, Nowland, Carroll, Casey and Clements14} The four tools measured (Manchester Self-Harm Rule, ReACT, SAD PERSONS and Modified SAD PERSONS) were found to predict suicide deaths less accurately than repeat self-harm episodes. Suicide is a comparatively rare outcome and even greater accuracy is needed in order for predictive tools to be clinically useful.^{Reference Velupillai, Hadlaczky, Baca-Garcia, Gorrell, Werbeloff and Nguyen37}

Whereas two of the scales (Manchester Self-Harm Rule and ReACT) were developed to be used in self-harm risk assessment, the Barratt Impulsiveness Scale is a scale that is not validated specifically as a risk scale for patients who have presented with self-harm (it is a measure of impulsiveness), and the SAD PERSONS Scale was initially developed as a training tool. However, a systematic review conducted prior to the cohort study found that the SAD PERSONS Scale and the Barratt Impulsiveness Scale were being used to predict future suicidal behaviour or repeat self-harm.^{Reference Quinlivan, Cooper, Steeg, Davies, Hawton and Gunnell10,Reference Quinlivan, Cooper, Davies, Hawton, Gunnell and Kapur25,Reference Saunders, Brand, Lascelles and Hawton38}

Our methodological approach was based on our primary objectives that were to compare the predictive accuracy of psychometrically tested scale items and combine the top-performing items from each scale. However, the use of a simple additive scale, giving each factor the same weight, may underestimate the predictive accuracy if a key risk factor, such as previous self-harm, is considered equally to an item with lower predictive ability.

We used CART modelling as a sensitivity analysis to test the robustness of our approach and ensure the performance of the new scale was accurately estimated. Although we could have used CART analysis as the primary analysis, this would not have taken into account the risk scale that each item was extracted from, which is how they are used clinically. Logistic regression could have allowed us to select items based on the strength of association between each scale item and the outcome of repeat self-harm. However, we focused on selecting items based on predictive accuracy as the scales are already psychometrically developed, tested and used in clinical practice. Additionally, we categorised the ordinal data in the Barratt Impulsiveness Scale as categorical data, which may result in a loss of information and potential overlap between the groups because of the statistically imposed cut-off. However, recoding was necessary in order to combine the Barratt Impulsiveness Scale items with the binary scale items. Finally, validating the findings on a larger, population-level sample would provide greater accuracy of our findings.

Comparison with previous research

The previous study by Quinlivan et al^{Reference Quinlivan, Cooper, Meehan, Longson, Potokar and Hulme11} found that the diagnostic accuracy of the full risk scales was not sufficient for them to be of clinical use when used by mental health professionals, and that clinician or global assessment of risk had greater predictive ability for repeat self-harm. A more recent study by Steeg et al^{Reference Steeg, Quinlivan, Nowland, Carroll, Casey and Clements14} found that utilising risk scales to predict repeat self-harm and future suicide following presentation to hospital is also unsuitable for episodes where mental health professionals are not involved in patient management. A meta-analysis of risk scales used for predicting suicidal behaviour also found that risk scales were not sufficient to be used to predict repeat self-harm and thus treatment allocation.^{Reference Carter, Milder, McGill, Pirkis, Kapur and Spittal12} Although that study examined the individual components of risk scales rather than the risk scales as a whole, our results are similar and suggest that even though some individual items performed slightly better than the scale they originated from, when combined they did not perform better than patient or clinician assessment of risk.

Machine learning methods may present new opportunities for risk prediction.^{Reference Velupillai, Hadlaczky, Baca-Garcia, Gorrell, Werbeloff and Nguyen37,Reference Walsh, Ribeiro and Franklin39–Reference Siddaway, Quinlivan, Kapur, O'Connor and De Beurs41} Machine learning has been used in several prospective studies on the diagnosis of physical health conditions^{Reference Alaa, Bolton, Di Angelantonio, Rudd and Van der Schaar42} including prediction of cancer survival outcomes.^{Reference Ashley, Gupta, Tran, Wei, Lorgelly and Thomas43} Internationally there is an increasing drive to develop risk scales using machine learning techniques such as random forest, decision trees and support vector machines on ‘big data’.^{Reference Fazel and O'Reilly44} Machine learning techniques offer some advantages for analysing non-linear observations and large numbers of variables. However, there are a number of potential limitations to using machine learning techniques in mental health settings. When predictive accuracy is prioritised over clinical interpretation, risk prediction models can become highly complex, limiting their clinical utility.^{Reference Siddaway, Quinlivan, Kapur, O'Connor and De Beurs41,Reference Luo45} Complex algorithms may also shift the focus to prediction and away from clinicians formulating a management plan based on patients’ individual needs and circumstances.^{Reference Siddaway, Quinlivan, Kapur, O'Connor and De Beurs41}

A recent systematic scoping review of the use of machine learning in mental health found that most research is focused on detection, such as natural language processing techniques to detect suicidal ideation from therapy transcripts, and short-term diagnosis rather than on predicting long-term outcomes.^{Reference Shatte, Hutchinson and Teague46} Furthermore, studies did not have a prospective design. One study found that machine learning models could correctly predict a sufficient proportion of suicides among soldiers for the prediction model to have implications for targeting interventions; however, the study focused on a specific population and would not be generalisable to the wider population.^{Reference Kessler, Stein, Petukhova, Bliese, Bossarte and Bromet47} Furthermore, the study did not compare the prediction models to clinician judgement. Other studies using machine learning techniques have been conducted in highly select samples with limited generalisability.^{Reference Hawes, Yaseen, Briggs and Galynker48} A recent systematic review^{Reference Belsher, Smolenski, Pruitt, Bush, Beech and Workman49} found that although suicide prediction models produce overall accurate classification models, their accuracy of predicting a future event is near zero. If machine learning approaches can overcome these limitations, they may result in greater clinical utility. However, our results indicate that no combination or individual items from widely used tested scales outperformed patient and clinician estimations of risk when predicting real-world patient outcomes which suggests there may be limitations to the prediction of suicidal behaviour.

Clinical implications

This study adds to the growing evidence that risk scales, including components of risk scales, are not suitable for predicting repeat episodes of self-harm or future suicide. The overall performance of all items as measured by the AUC did not exceed a ‘fair’ level of prediction (defined as between 0.7 and 0.8) and none of the items exceeded clinicians’ global assessments of risk for repeat self-harm.^{Reference Quinlivan, Cooper, Meehan, Longson, Potokar and Hulme11} Measures of global accuracy were used to enable head-to-head comparison of scale items. However, measures of dual accuracy highlighted the limitations of scale items for accurately identifying those who repeated and accuracy for those who did not. This has implications for their utility. For example, although some items, such as the highly sensitive ‘lifetime history of self-harm’ item from the Manchester Self-Harm Rule, could be used as a guide for either emergency department or mental health clinicians in their own assessments, the high sensitivity items had poor specificity (and vice versa) so should not be used alone to identify patients as high or low risk and thus affect their treatment allocation. Identifying patients as high risk simply on the basis of meeting one criterion with high sensitivity but poor specificity could result in significant challenges to service provision if risk scales were used to determine patient management because of a potentially large number of referrals to more intensive services.

Factors such as prior psychiatric treatment and lifetime history of self-harm have been previously identified as risk factors for repeat self-harm, however, some Barratt Impulsiveness Scale items such as ‘rarely self-controlled’ and ‘usually changes jobs frequently’ were more specific or sensitive than the other items (although none were both sensitive and specific). Impulsivity and aggression have been correlated with self-harm in a recent systematic review,^{Reference O'Donnell, House and Waterman50} so awareness that impulsive/aggressive behaviours may increase the risk of future self-harm may help clinicians assess needs.

Risk classification scales remain in widespread use despite the evidence of their poor predictive abilities.^{Reference Quinlivan, Cooper, Steeg, Davies, Hawton and Gunnell10,Reference Carter, Milder, McGill, Pirkis, Kapur and Spittal12} This may be because they act as prompts for factors to consider in formulating a management plan, or because the risk scale results in a score that can be easily interpreted and can instruct the clinician on the management plan (for example, high risk requiring more intensive follow-up). This may be reassuring for less experienced clinicians but is unlikely to be beneficial to their practice or to patients. Furthermore, for relatively rare outcomes such as suicide, classification into high and low risk is unlikely to be clinically useful because of the relatively low positive predictive values that can be obtained. Carter et al^{Reference Carter, Milder, McGill, Pirkis, Kapur and Spittal12} suggest that modifiable risk factors, such as isolation or improving physical health, should be a focus for improvement as part of a holistic assessment of the patient. In addition, patients value a therapeutic alliance,^{Reference Hunter, Chantler, Kapur and Cooper51} so focusing on forging a positive and encouraging relationship, as part of a good-quality assessment, may itself reduce the risk of a repeat self-harm episode.^{Reference Kapur, Steeg, Webb, Haigh, Bergen and Hawton52,Reference Steeg, Emsley, Carr, Cooper and Kapur53}

The items we measured in the present study were not able to capture the quality and nature of the clinical encounter, which may also contribute to likelihood of repeat self-harm.^{Reference Hunter, Chantler, Kapur and Cooper51} Identifying those who are at particularly high risk is challenging; therefore safety planning should be a priority for all patients.^{Reference Stanley and Brown54} Instead of relying on risk scales to instruct clinicians on the most appropriate management, clinicians should receive comprehensive training and ongoing supervision to improve their knowledge and confidence in assessing patients who self-harm. Research into ‘de-implementation’ of the use of risk scales is an important next step in improving clinical practice for people who have self-harmed. Alternative strategies such as comprehensive patient-centred assessment and safety planning should be explored as part of this.

Given their range of predictive utilities, individual items of scales may be useful for clinicians to consider in their assessments, but they should not be used alone to predict repeat self-harm or determine patient management. Constructing new scales from higher-performing items did not significantly improved performance. The large number of items included in this new scale construction highlights potential limitations to predicting suicidal behaviour accurately.