Skip to main content
×
×
Home

Predictive accuracy of risk scales following self-harm: Multicentre, prospective cohort study

  • Leah Quinlivan (a1), Jayne Cooper (a1), Declan Meehan (a2), Damien Longson (a2), John Potokar (a3), Tom Hulme (a4), Jennifer Marsden (a5), Fiona Brand (a6), Kezia Lange (a6), Elena Riseborough (a7), Lisa Page (a7), Chris Metcalfe (a8), Linda Davies (a9), Rory O'Connor (a10), Keith Hawton (a11), David Gunnell (a8) and Nav Kapur (a12)...
Abstract
Background

Scales are widely used in psychiatric assessments following self-harm. Robust evidence for their diagnostic use is lacking.

Aims

To evaluate the performance of risk scales (Manchester Self-Harm Rule, ReACT Self-Harm Rule, SAD PERSONS scale, Modified SAD PERSONS scale, Barratt Impulsiveness Scale); and patient and clinician estimates of risk in identifying patients who repeat self-harm within 6 months.

Method

A multisite prospective cohort study was conducted of adults aged 18 years and over referred to liaison psychiatry services following self-harm. Scale a priori cut-offs were evaluated using diagnostic accuracy statistics. The area under the curve (AUC) was used to determine optimal cut-offs and compare global accuracy.

Results

In total, 483 episodes of self-harm were included in the study. The episode-based 6-month repetition rate was 30% (n = 145). Sensitivity ranged from 1% (95% CI 0–5) for the SAD PERSONS scale, to 97% (95% CI 93–99) for the Manchester Self-Harm Rule. Positive predictive values ranged from 13% (95% CI 2–47) for the Modified SAD PERSONS Scale to 47% (95% CI 41–53) for the clinician assessment of risk. The AUC ranged from 0.55 (95% CI 0.50–0.61) for the SAD PERSONS scale to 0.74 (95% CI 0.69–0.79) for the clinician global scale. The remaining scales performed significantly worse than clinician and patient estimates of risk (P < 0.001).

Conclusions

Risk scales following self-harm have limited clinical utility and may waste valuable resources. Most scales performed no better than clinician or patient ratings of risk. Some performed considerably worse. Positive predictive values were modest. In line with national guidelines, risk scales should not be used to determine patient management or predict self-harm.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Predictive accuracy of risk scales following self-harm: Multicentre, prospective cohort study
      Available formats
      ×
      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Predictive accuracy of risk scales following self-harm: Multicentre, prospective cohort study
      Available formats
      ×
      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Predictive accuracy of risk scales following self-harm: Multicentre, prospective cohort study
      Available formats
      ×
Copyright
This is an open access article distributed under the terms of the Creative Commons Attribution (CC BY) licence.
Corresponding author
Leah Quinlivan, Jean McFarlane Building, University of Manchester, Oxford Road, Manchester M13 9PL, UK. Email: leah.quinlivan@manchester.ac.uk
Footnotes
Hide All

See editorial, pp. 384–386, this issue.

Declaration of interest

D.G., K.H. and N.K. are members of the Department of Health's (England) National Suicide Prevention Advisory Group. N.K. chaired the NICE guideline development group for the longer-term management of self-harm and the NICE Topic Expert Group (which developed the quality standards for self-harm services). He is currently chair of the updated NICE guideline for depression. R.O.C. was a member of the NICE guideline development group for the longer-term management of self-harm and is a member of the Scottish Government's suicide prevention implementation and monitoring group.

Footnotes
References
Hide All
1 Bergen, H, Hawton, K, Waters, K, Cooper, J, Kapur, N. Psychosocial assessment and repetition of self-harm: the significance of single and multiple repeat episode analyses. J Affect Disord 2010; 127: 257–65.
2 Bergen, H, Hawton, K, Waters, K, Ness, J, Cooper, J, Steeg, S, et al. Premature death after self-harm: a multicentre cohort study. Lancet 2012; 380: 1568–74.
3 Hawton, K, Bergen, H, Cooper, J, Turnbull, P, Waters, K, Ness, J, et al. Suicide following self-harm: findings from the multicentre study of self-harm in England, 2000–2012. J Affect Disord 2015; 175: 147–51.
4 Haw, C, Bergen, H, Casey, D, Hawton, K. Repetition of deliberate self-harm: a study of the characteristics and subsequent deaths in patients presenting to a general hospital according to extent of repetition. Suicide Life Threat Behav 2007; 37: 379–96.
5 Zahl, DL, Hawton, K. Repetition of deliberate self-harm and subsequent suicide risk: long-term follow-up study of 11583 patients. Br J Psychiatry 2004; 185: 70–5.
6 National Institute for Health and Care Excellence. Self-harm. The NICE Guideline on Longer-Term Management. National Clinical Guideline Number 133. The British Psychological Society and The Royal College of Psychiatrists, 2011.
7 Royal Australian and New Zealand College of Psychiatrists Clinical Practice Guidelines Team for Deliberate Self-harm. Australian and New Zealand clinical practice guidelines for the management of adult deliberate self-harm. Aust NZ J Psychiatry 2004; 38: 868–84.
8 Bolton, JM, Gunnell, D, Turecki, G. Suicide risk assessment and intervention in people with mental illness. BMJ 2015; 35: h4978.
9 Kapur, N, Murphy, E, Cooper, J, Bergen, H, Hawton, K, Simkin, S, et al. Psychosocial assessment following self-harm: results from the multi-centre monitoring of self-harm project. J Affect Disord 2008; 106: 285–93.
10 Hawton, K. Psychiatric assessment and management of deliberate self-poisoning patients. Medicine 2016; 44: 103–5.
11 Royal College of Psychiatrists. Self-Harm, Suicide, and Risk: Helping People who Self-Harm. Final Report of a Working Group. Royal College of Psychiatrists, 2010.
12 Quinlivan, L, Cooper, J, Steeg, S, Davies, L, Hawton, K, Gunnell, D, et al. Scales for predicting risk following self-harm: an observational study in 32 hospitals in England. BMJ Open 2014; 4: e004732.
13 Quinlivan, L, Cooper, J, Davies, L, Hawton, K, Gunnell, D, Kapur, N, et al. Which are the most useful scales for predicting repeat self-harm? A systematic review evaluating risk scales using measures of diagnostic accuracy. BMJ Open 2016; 6: e009297.
14 Bilén, K, Ponzer, S, Ottosson, C, Castrén, M, Owe-Larsson, B, Ekdahl, K, et al. Can repetition of deliberate self-harm be predicted? A prospective multicenter study validating clinical decision rules. J Affect Disord 2013; 149: 253–8.
15 Bolton, JM, Spiwak, R, Sareen, J. Predicting suicide attempts with the SAD PERSONS scale: a longitudinal analysis. J Clin Psychiatry 2012; 76: e73541.
16 Carter, GL, Clover, KA, Bryant, JL, Whyte, IM. Can the Edinburgh Risk of Repetition Scale predict repetition of deliberate self-poisoning in an Australian clinical setting? Suicide Life Threat Behav 2002; 32: 230–9.
17 Cooper, J, Kapur, N, Dunning, J, Guthrie, E, Appleby, L, Mackway-Jones, K. A clinical tool for assessing risk after self-harm. Ann Emerg Med 2006; 48: 459–66.
18 Spittal, MJ, Pirkis, J, Miller, M, Carter, G, Studdert, DM. The Repeated Episodes of Self-Harm (RESH) score: a tool for predicting risk of future episodes of self-harm by hospital patients. J Affect Disord 2014; 161: 3642.
19 Steeg, S, Kapur, N, Webb, R, Applegate, E, Stewart, SL, Hawton, K, et al. The development of a population-level clinical screening tool for self-harm repetition and suicide: the ReACT Self-Harm Rule. Psychol Med 2012; 42: 2383–94.
20 Randall, JR, Rowe, BH, Colman, I. Emergency department assessment of self-harm risk using psychometric questionnaires. Can J Psychiatry 2012; 57: 21.
21 Waern, M, Sjöström, N, Marlow, T, Hetta, J. Does the Suicide Assessment Scale predict risk of repetition? A prospective study of suicide attempters at a hospital emergency department. Eur Psychiatry 2010; 25: 421–6.
22 Hockberger, RS, Rothstein, RJ. Assessment of suicide potential by nonpsychiatrists using the SAD PERSONS score. J Emerg Med 1988; 6: 99107.
23 Patton, JH, Stanford, MS. Factor structure of the Barratt Impulsiveness Scale. J Clin Psychol 1995; 51: 768–77.
24 Randall, JR, Colman, I, Rowe, BH. A systematic review of psychometric assessment of self-harm risk in the emergency department. J Affect Disord 2011; 134: 348–55.
25 Larkin, C, Di Blasi, Z, Arensman, E. Risk factors for repetition of self-harm: a systematic review of prospective hospital-based studies. PLoS ONE 2014; 9: e84282.
26 Chan, MK, Bhatti, H, Meader, N, Stockton, S, Evans, J, O'Connor, RC, et al. Predicting suicide following self-harm: systematic review of risk factors and risk scales. Br J Psychiatry 2016; 209: 277–83.
27 Bewick, V, Cheek, L, Ball, J. Statistics review 13: receiver operating characteristic curves. Critical Care 2004; 8: 508.
28 Kapur, N, Steeg, S, Webb, R, Haigh, M, Bergen, H, Hawton, K, et al. Does clinical management improve outcomes following self-harm? Results from the multicentre study of self-harm in England. PLoS ONE 2013; 8: e70434.
29 Majid, M, Tadros, M, Tadros, G, Singh, S, Broome, MR, Upthegrove, R, et al. Young people who self-harm: a prospective 1-year follow-up study. Soc Psychiatry Psychiatr Epidemiol 2016: 51: 171–81.
30 Kapur, N, Steeg, S, Turnbull, P, Webb, R, Bergen, R, Hawton, K, et al. Hospital management of suicidal behaviour and subsequent mortality: a prospective cohort study. Lancet Psychiatry 2015; 2: 809–16.
31 Steeg, S, Haigh, M, Webb, RT, Kapur, N, Awenat, Y, Gooding, P, et al. The exacerbating influence of hopelessness on other known risk factors for repeat self-harm and suicide. J Affect Disord 2016; 190: 522–8.
32 Kapur, N, Murphy, E, Cooper, J, Bergen, H, Hawton, K, Simkin, S, et al. Psychosocial assessment following self-harm: results from the multi-centre monitoring of self-harm project. J Affect Disord 2008; 106: 285–93.
33 Silverman, MM, Berman, AL, Sanddal, ND, O'Carroll, PW, Joiner, TE. Rebuilding the Tower of Babel: a revised nomenclature for the study of suicide and suicidal behaviors Part 1: background, rationale, and methodology. Suicide Life Threat Behav 2007; 37: 248–63.
34 O'Carroll, PW, Berman, AL, Maris, RW, Moscicki, EK, Tanney, BL, Silverman, MM. Beyond the Tower of Babel: a nomenclature for suicidology. Suicide Life Threat Behav 1996; 26: 237–52.
35 Andover, MS, Morris, BW, Wren, A, Bruzzese, ME. The co-occurrence of non-suicidal self-injury and attempted suicide among adolescents: distinguishing risk factors and psychosocial correlates. Child Adolesc Psychiatry Ment Health 2012; 6: 11.
36 Owens, D, Kelley, R, Munyombwe, T, Bergen, H, Hawton, K, Cooper, J, et al. Switching methods of self-harm at repeat episodes: findings from a multicentre cohort study. J Affect Disord 2015; 180: 4451.
37 Clements, C, Jones, S, Morriss, R, Peters, S, Cooper, J, While, D, et al. Self-harm in bipolar disorder: findings from a prospective clinical database. J Affect Disord 2015; 173: 113–9.
38 Hawton, K, Zahl, D, Weatherall, R. Suicide following deliberate self-harm: long-term follow-up of patients who presented to a general hospital. Br J Psychiatry 2003; 182: 537–42.
39 Owens, D, Horrocks, J, House, A. Fatal and non-fatal repetition of self-harm. Br J Psychiatry 2002; 181: 193–9.
40 Carroll, R, Metcalfe, C, Gunnell, D. Hospital presenting self-harm and risk of fatal and non-fatal repetition: systematic review and meta-analysis. PLoS ONE 2014; 9: e89944.
41 Cooper, J, Kapur, N, Mackway-Jones, K. A comparison between clinicians' assessment and the Manchester Self-Harm Rule: a cohort study. Emerg Med J 2007; 24: 720–1.
42 Patterson, WM, Dohn, HH, Bird, J, Patterson, GA. Evaluation of suicidal patients: the SAD PERSONS scale. Psychosomatics 1983; 24: 343–9.
43 Kapur, N, Cooper, J, Hiroeh, U, May, C, Appleby, L, House, A. Emergency department management and outcome for self-poisoning: a cohort study. Gen Hosp Psychiatry 2004; 26: 3641.
44 Cooper, J, Steeg, S, Bennewith, O, Lowe, M, Gunnell, D, House, A, et al. Are hospital services for self-harm getting better? An observational study examining management, service provision and temporal trends in England. BMJ Open 2013; 3: e003444.
45 DerSimonian, R, Laird, N. Meta-analysis in clinical trials. Control Clin Trials 1986; 7: 177–88.
46 Higgins, JP, Green, S. Cochrane Handbook for Systematic Reviews Of Interventions (vol 5). Wiley Online Library, 2008.
47 Hosmer, DW Jr, Lemeshow, S, Sturdivant, RX. Applied Logistic Regression (vol 398). John Wiley & Sons, 2013.
48 DeLong, ER, DeLong, DM, Clarke-Pearson, DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988; 44: 837–45.
49 Allison, PD. Missing Data (vol 136). Sage Publications, 2001.
50 Ness, J, Hawton, K, Bergen, H, Cooper, J, Steeg, S, Kapur, N, et al. Alcohol use and misuse, self-harm and subsequent mortality: an epidemiological and longitudinal study from the multicentre study of self-harm in England. Emerg Med J 2015; 32: 793–9.
51 Geulayov, G, Kapur, N, Turnbull, P, Clements, C, Waters, K, Ness, J, et al. Epidemiology and trends in non-fatal self-harm in three centres in England, 2000-2012: findings from the Multicentre Study of Self-Harm in England. BMJ Open 2016; 6: e010538.
52 Hawton, K, Witt, KG, Taylor Salisbury, TL, Arensman, E, Gunnell, D, Hazell, P, et al. Psychosocial Interventions for Self-Harm in Adults. The Cochrane Library, 2016.
53 O'Connor, E, Gaynes, BN, Burda, BU, Soh, C, Whitlock, EP. Screening for and treatment of suicide risk relevant to primary care: a systematic review for the US Preventive Services Task Force. Ann Antern Med 2013; 158: 741–54.
54 O'Connor, RC, Nock, MK. The psychology of suicidal behaviour. Lancet Psychiatry 2014; 1: 7385.
55 Saunders, K, Brand, F, Lascelles, K, Hawton, K. The sad truth about the SADPERSONS Scale: an evaluation of its clinical utility in self-harm patients. Emerg Med J 2014; 31: 796–8.
56 Carroll, R, Metcalfe, C, Steeg, S, Davies, NM, Cooper, J, Kapur, N, et al. Psychosocial assessment of self-harm patients and risk of repeat presentation: an instrumental variable analysis using time of hospital presentation. PLoS ONE 2016; 11: e0149713.
57 Smith, MJ, Bouch, J, Bradstreet, S, Lakey, T, Nightingale, A, O'Connor, RC. Health services, suicide, and self-harm: patient distress and system anxiety. Lancet Psychiatry 2015; 2: 275–80.
58 Hunter, C, Chantler, K, Kapur, N, Cooper, J. Service user perspectives on psychosocial assessment following self-harm and its impact on further help-seeking: a qualitative study. J Affect Disord 2013; 145: 315–23.
59 Abderhalden, C, Needham, I, Dassen, T, Halfens, R, Haug, HJ, Fischer, JE. Structured risk assessment and violence in acute psychiatric wards: randomised controlled trial. Br J Psychiatry 2008; 193: 4450.
60 Troquete, NA, van den Brink, RH, Beintema, H, Mulder, T, van Os, TW, Schoevers, RA, et al. Risk assessment and shared care planning in out-patient forensic psychiatry: cluster randomised controlled trial. Br J Psychiatry 2013; 202: 365–71.
61 Coope, C, Donovan, J, Wilson, C, Barnes, M, Metcalfe, C, Hollingworth, W, et al. Characteristics of people dying by suicide after job loss, financial difficulties and other economic stressors during a period of recession (2010–2011): a review of coroners records. J Affect Disord 2015; 183: 98105.
62 Haw, C, Hawton, K, Gunnell, D, Platt, S. Economic recession and suicidal behaviour: Possible mechanisms and ameliorating factors. Int J Soc Psychiatry 2015; 61: 7381.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

The British Journal of Psychiatry
  • ISSN: 0007-1250
  • EISSN: 1472-1465
  • URL: /core/journals/the-british-journal-of-psychiatry
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 3
Total number of PDF views: 149 *
Loading metrics...

Abstract views

Total abstract views: 367 *
Loading metrics...

* Views captured on Cambridge Core between 2nd January 2018 - 22nd June 2018. This data will be updated every 24 hours.

Predictive accuracy of risk scales following self-harm: Multicentre, prospective cohort study

  • Leah Quinlivan (a1), Jayne Cooper (a1), Declan Meehan (a2), Damien Longson (a2), John Potokar (a3), Tom Hulme (a4), Jennifer Marsden (a5), Fiona Brand (a6), Kezia Lange (a6), Elena Riseborough (a7), Lisa Page (a7), Chris Metcalfe (a8), Linda Davies (a9), Rory O'Connor (a10), Keith Hawton (a11), David Gunnell (a8) and Nav Kapur (a12)...
Submit a response

eLetters

Suicide risk assessment tools do not perform worse than clinical judgement

Seena Fazel, Wellcome Trust Senior Research Fellow, Department of Psychiatry, University of Oxford
Achim Wolf, DPhil Student & Research Assistant, Department of Psychiatry, University of Oxford
11 May 2017

The study by Quinlivan and colleagues (1) could be interpreted to suggest that clinician and patient ratings are better than actuarial tools in predicting self-harm after an emergency hospital presentation with self-harm. However, we would argue that this is an incorrect interpretation.



First, the clinical evaluation appears to have occurred after these tools were completed by the same clinician, and, although they were masked to the overall score, the clinical impression will therefore have been strongly informed by the items in these suicide risk assessment tools. In fact, the study does not appear to be a comparison between actuarial tools and a distinct, unstructured clinical judgement, but a comparison between actuarial tools and what is called a structured clinical judgement approach (where structured questions about relevant risk factors are asked, and then a clinical judgement is made about an individual’s overall risk level). Clarification of the exact procedure used is important for interpreting the findings.



Second, the authors correctly point out that, on the basis of ROC curves, risk assessment tools performed no better than clinician ratings. The other way of looking at this, however, is that clinician ratings performed no better than risk scales. In particular, the Manchester Self-Harm Rule (2), a 4-item tool, performed just as well. Importantly, the authors found no evidence of between-hospital heterogeneity for this tool’s performance. Clinician ratings on the other hand showed substantial heterogeneity between hospitals, with specificity ranging from 58% to 82%. The lack of variability in the actuarial tools could be argued to be an advantage when performance between clinician rating and assessment tool is no different. Furthermore, tools like this will be considerably quicker, leaving more clinician time for risk management (as opposed to assessment).



Third, the clinicians were based in teaching hospitals (Brighton, Bristol, Derby, Manchester, and Oxford) with longstanding research interests in self-harm. Whether the reported predictive accuracy of clinician ratings is generalizable to non-specialist centres is an empirical question.



Fourth, the patient rating may also have been influenced by the questions asked by the tools (which tend to be categorical and therefore easy to work out what constitutes a risk factor). In a sense, then, the patient rating is a form of structured judgement.



Comparing risk tools with clinicians may not be informative, or even feasible, as clinical interviews already include many of the items used in risk tools. Instead, future research should compare actuarial scores with or without additional clinician input. In other words, if clinicians disagree with the risk level provided by actuarial tools, does this reclassification lead to an improvement in predictive performance? As the AUCs for the tools in this study ranged from 0.55 to 0.72, there may be considerable room for improvement by incorporating novel and modifiable risk factors as has been shown in violence risk assessment in patients with severe mental illness (3). Ultimately, randomized studies will be required to establish the effects of different approaches to risk assessment on patient and service outcomes.



1 Quinlivan L, Cooper J, Meehan D, Longson D, Potokar J, Hulme T, Marsden J, Brand F, Lange K, Riseborough E, Page L. Predictive accuracy of risk scales following self-harm: multicentre, prospective cohort study. British Journal of Psychiatry. 2017. DOI: 10.1192/bjp.bp.116.189993

2 Cooper J, Kapur N, Dunning J, Guthrie E, Appleby L, Mackway-Jones K. A clinical tool for assessing risk after self-harm. Annals of Emergency Medicine. 2006; 48(4), 459-466.

3Fazel S, Wolf A, Larsson H, Lichtenstein P, Mallett S, Fanshawe TR. Identification of low risk of violent crime in severe mental illness with a clinical prediction tool (Oxford Mental Illness and Violence tool [OxMIV]): a derivation and validation study. Lancet Psychiatry. 2017. DOI: 10.1016/S2215-0366(17)30109-8
... More

Conflict of interest: S.F. has published on risk assessment, including a platform for risk assessment tools (OxRisk). A.W. is currently researching violence risk assessment.

Write a reply

Dear Leah Quinlivan et al.,

Emily Kruger, Assistant Psychologist, Child and Adolescent Mental Health Service (CAMHS)
26 March 2017

Thank you for publishing such an interesting, in depth study. As one who is very interested in ways to recognise and prevent self harm and crisis situations, I am pleased to see an article dedicating research to such an important area. Thus, in order to prevent crisis situations, it is paramount that reliable and valid measures are used to assess in clinical settings.

I only have one criticism of the paper. In terms of your introduction, I would have liked to see more information R.E self harm and risk factors of self-harming behaviours, in order to set the scene. In addition, it may have been useful to state what self harm consists of and the controversies surrounding this in order to inform a ready of whom may be unaware of the details.

Again, I thank you for a compelling read.

Many thanks,

Emily Kruger
... More

Conflict of interest: None Declared

Write a reply

×

Reply to: Submit a response


Your details


Conflicting interests

Do you have any conflicting interests? *