Health professionals are expected to have the skills to assess and manage patients with suicidal thoughts.Reference Weber, Michail, Thompson and Fiedorowicz1 The presence of thoughts of suicide and the expression of suicidal ideation are signs of significant distress, and all patients who express suicidal ideation require a careful clinical assessmentReference Rives2, Reference Hawley, James, Birkett, Baldwin, de Ruiter and Priest3 leading to a comprehensive treatment plan.Reference Hawley, James, Birkett, Baldwin, de Ruiter and Priest3 In addition, suicidal ideation is sometimes considered to be the most important sign of short-term suicide risk,Reference Morgan and Stanton4 and it has been argued that questions about suicidal ideation play a crucial role in suicide screening.Reference Boudreaux, Camargo, Arias, Sullivan, Allen and Goldstein5, Reference Horowitz, Snyder, Ludi, Rosenstein, Kohn-Godbout and Lee6
Despite the undoubted clinical importance of suicidal ideation, meta-analysis has only been used recently to estimate the statistical strength of suicidal ideation as a test for later suicide. Several meta-analyses have highlighted the modest strength of the association between suicidal ideation and later suicideReference Chapman, Mullin, Ryan, Kuffel, Nielssen and Large7–Reference Franklin, Ribeiro, Fox, Bentley, Kleiman and Huang10 and one has quantified the positive predictive value (PPV).Reference Hubers, Moaddine, Peersmann, Stijnen, van Duijn and van der Mast8 To date, no meta-analysis has examined the sensitivity and specificity of suicidal ideation for suicide. Knowledge of the sensitivity of suicidal ideation for later suicide is particularly important because screening tests should be sufficiently sensitive, such that a high proportion of affected people screen positive.Reference Wilson and Jungner11 Moreover, existing meta-analyses of the association between suicide and suicidal ideation have been limited to studies with a cohort design,Reference Ribeiro, Franklin, Fox, Bentley, Kleiman and Chang9 only included patients with particular diagnosesReference Chapman, Mullin, Ryan, Kuffel, Nielssen and Large7 or included studies where suicidal ideation was established post-mortem using psychological autopsy methods.Reference Hubers, Moaddine, Peersmann, Stijnen, van Duijn and van der Mast8
We report a meta-analysis of cohort and case–control studies that examined the presence or absence of expressed suicidal ideation among those who later did or did not die by suicide.
Aims and hypotheses
The first aim of this study was to calculate a comprehensive set of measures of prediction of suicidal ideation for later suicide, including of effect size as measured by the pooled odds ratio (OR), sensitivity, specificity, meta-analytically derived receiver operator curve (ROC) and area under the curve (AUC) and the PPV. The second aim was to explore the extent of, and moderators of, between-study heterogeneity in these predictive measures. We hypothesised that suicidal ideation would be a more sensitive and specific test for suicide in non-psychiatric settings, where patients with suicidal ideation can be expected to have fewer additional and potentially confounding risk factors.
We conducted a registered meta-analysis (Prospero 42017059236) according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (see supplementary Table 1 available at https://doi.org/10.1192/bjo.2018.88).Reference Moher, Liberati, Tetzlaff and Altman12
Multiple search strategies were initially explored using the databases Medline, Embase and PsycINFO from inception to January 2017. These searches yielded an excessive number of titles when the term ‘suicide’ was used as a keyword (>200 000) or in the title (>70 000). Moreover, placing limits in the searches using relevant search terms (such as ‘ideation’ or ‘mortality’) missed numerous relevant papers that were known to the authors through earlier hand searching. However, PubMed indexed all but one English-language publication located by searches of multiple databases in earlier studies.Reference Chapman, Mullin, Ryan, Kuffel, Nielssen and Large7, Reference Hubers, Moaddine, Peersmann, Stijnen, van Duijn and van der Mast8 Therefore, and in order to obtain a large and representative sample of relevant studies, one author (M.M.L.) examined the titles of English-language publications with an accompanying abstract that contained variants of the single term ‘suicide’ (suicid*) in their title and were published in PubMed from inception to 14 September 2017 and hand searched the references list of relevant review articles.Reference Chapman, Mullin, Ryan, Kuffel, Nielssen and Large7–Reference Franklin, Ribeiro, Fox, Bentley, Kleiman and Huang10, Reference Large, Smith, Sharma, Nielssen and Singh13, Reference Large, Sharma, Cannon, Ryan and Nielssen14 Two authors (M.M.L. and C.M.M.) winnowed the resulting abstracts and full-text publications (Fig. 1).
Studies were included if they reported the expression or non-expression of suicidal ideation among people who later died by suicide or did not, such that all individuals could be classified into the following groups: true positive (those with suicidal ideation and suicide), false positive (those with suicidal ideation but without suicide), false negative (those without suicidal ideation but with suicide) or true negative (those without suicidal ideation or suicide). Studies were included if they provided effect size or other data that was sufficient to enable the calculation of the numbers of people in the true positive, false positive, false negative and true negative categories.
We included studies of patients who had received psychiatric care and people recruited from non-psychiatric settings including primary healthcare, general population samples, military populations and prisons. We excluded studies in which the dependent variable was suicide attempts, or when suicidal ideation was assessed by psychological autopsy methods, when all the controls were deceased or all of the individuals had made suicide attempts. We excluded studies of patients with severe medical illnesses such as malignancies or human immunodeficiency virus.
Definitions of suicide and suicidal ideation
Two authors (M.M.L. and C.M.M.) independently extracted the numbers of individuals with suicidal ideation among those who had died by suicide and controls and the moderator variables. A third author (A.C.) re-examined the data points. The discrepant points were re-examined by two authors. We accepted the definition of suicidal ideation used in the primary research, acknowledging that the primary research might not have fully reflected differences in the way suicidal ideation was expressed. Studies were coded into four categories according to their definition of suicidal ideation: studies that reported a composite measure of suicidality (inclusive of both suicidal ideation and behaviour); studies that reported suicide plans (including suicide ‘threats’ as reported in the primary studies); studies that recorded a wish to die; and the majority of studies in which the nature of the suicidal ideation or suicidal thoughts was not specified.
We collected data about potential moderators of the association between suicidal ideation and suicide; (a) whether the individuals were from psychiatric or non-psychiatric samples; (b) whether the samples consisted of hospital-treated patients; (c) the mean sample age; (d) the duration of follow-up after the assessment for suicidal ideation; (e) whether the study had a cohort or control design; (f) whether patients with suicidal behaviour were regarded as having suicidal ideation; (g) the year of study publication; (h) the proportion of individuals included with suicidal ideation; and (i) the incidence of suicide in the study.
Assessment of strength of reporting
Two researchers (M.M.L. and C.M.M.) independently assessed the reporting strength (and hence the risk of bias) of each primary study's research using a scale with four further moderator variables (scored 0–4) derived from the Newcastle Ottawa Scale.Reference Wells, Shea, O'Connnell, Peterson, Welch and Losos15 One point was allocated according to each of the following criteria: (a) use of a structured method to assess suicidal ideation; (b) collection of data about suicidal ideation in a method that was masked to the patient's suicide (either by a masking method or by electronic recording at the point of assessment); (c) if all the suicides were defined using a mortality database; and (d) inclusion of open verdicts as suicide. Studies counting open verdicts were rated as having stronger reporting because open verdicts are thought to be similar to suicides in some jurisdictionsReference Linsley, Schapira and Kelly16 and because the presence of a history of suicidal ideation might make a coronial verdict of suicide more likely.
Random effects meta-analysis was chosen a priori for all estimates (OR, sensitivity, specificity, PPV) because of the diversity of study populations and the differences in methods used in the primary research. Between-study heterogeneity in effect sizes was examined using I2, Q-value statistics and prediction intervals. The possibility of publication bias was assessed using a funnel plot and Egger's regression,Reference Egger, Davey Smith, Schneider and Minder17 and was quantified using Duval & Tweedie's trim and fill method.Reference Duval and Tweedie18 Subgroup analysis (random effects within subgroups) and random-effects meta-regression (method of moments) were used to explore the extent to which between-study heterogeneity was explained by categorical and continuous variables (including the year of publication and the proportion of all participants with suicidal ideation).
Moderator variables (including strength of reporting score items) that were significantly associated with between-study heterogeneity in sensitivity and specificity at P≤0.05 were included in multiple meta-regression models. Multiple meta-regression was not used to examine heterogeneity in ORs because of the absence of significant moderator variables. Comprehensive Meta-Analysis (Version 3, Biostat, Englewood NJ) was used for the main analysis and a meta-analytic estimate of the ROC and AUC was calculated using Meta-DiSc.Reference Zamora, Abraira, Muriel, Khan and Coomarasamy19
Searches and data extraction
The initial searches yielded 320 potentially relevant papers that were examined in full text. There were 70 papers,Reference Allebeck, Varla, Kristjansson and Wistedt20–Reference Yim, Yip, Li, Dunn, Yeung and Miao89 reporting 71 studies in which true positive, false positive, false negative and true negative could be ascertained. Eight differences in the effect size in the independent data extraction were resolved by re-examination of the data (Table 1 and see supplementary Table 2 for the data used in meta-analysis).
The earliest study was published in 1961 and the latest in 2017. The median publication date was 2006. In total, 42 studies had a case–control design and 29 were cohort studies. Of the studies, 58 studies were of current or former psychiatric patients including 42 studies of current or former psychiatric in-patients.
In 52 studies suicidal ideation was defined clinically and 19 studies used a structured method. Twelve studies reported a composite measure of suicidality (inclusive of suicide attempts). Four studies reported on suicide plans and two studies recorded a wish to die rather than active suicidal ideation.
The 71 studies included in total of 4 669 303 individuals who had been assessed for suicidal ideation (mean 65 765, median 235) of whom 5 811 died by suicide (mean 81.8, median 49). In total there were 2082 people in the true positive, 125 034 in the false positive, 3729 in the false negative and 4 538 458 in the true negative groups.
Meta-analysis of odds ratios
The pooled OR of suicide associated with suicidal ideation was 3.41 (95% CI 2.59–4.49; 95% prediction interval (PI) 0.42–28.1). There was very marked between-study heterogeneity, I 2 89.4, Q-value = 661, d.f.(Q) = 70, P<0.001. The range of ORs was 0.28 to 81, the first quartile, 2.03, median, 2.96 and the third quartile 6.45 (Fig. 2).
This was associated with a meta-analytic AUC of 0.676 (s.e. = 0.021) (Fig. 3). The funnel plot appeared symmetrical (Fig. 4) but an Egger's regression was borderline significant (intercept 1.55, t-value = 2.07, d.f. = 69, P = 0.04). Duval & Tweedie's trim and fill did not identify any hypothetically missing studies.
Structured methods to assess suicidal ideation were associated with a non-significantly lower OR (OR = 2.38, 95% CI 1.14–4.99) than clinically defined suicidal ideation (OR = 3.72, 95% CI 2.96–4.67, Q-value = 1.28, d.f.(Q) = 1, P = 0.26). There was no evidence that studies that reported on a composite measure of suicidality spanning suicidal behaviour and suicidal ideation (OR = 3.09, 95% CI 2.02–4.72) differed from studies that simply reported on suicidal ideation (OR = 3.54, 95% CI 2.57–4.88, Q = 0.26, d.f.(Q) = 1, P = 0.61). Suicide plans were more strongly associated with suicide (four studies, OR = 8.51, 95% CI 5.51–13.06) than studies that reported a wish to die (two studies, OR = 3.01, 95% CI 1.49–6.06) and studies that did not specify the degree of intent or planning (65 studies, OR = 3.2, 95% CI 2.41–4.29, Q-value = 14.6, d.f.(Q) = 2, P = 0.001)
Studies that were considered to have stronger reporting on the basis of a total strength of reporting score of ≥2 had a non-significantly weaker association (Q-value = 3.58, d.f.(Q) = 1, P = 0.06) between suicidal ideation and later suicide (n = 37, OR = 2.70, 95% CI 1.81–4.01) than studies with a strength of reporting score of <2 (n = 34, OR = 4.41, 95% CI 3.20–6.08).
Between-study heterogeneity was not significantly explained by any of our predetermined moderator variables (Table 2). Studies of non-psychiatric samples (n = 13, OR = 3.86, 95% CI 2.67–3.97) had similar ORs to studies of psychiatric patients (n = 58, OR = 3.23, 95% CI 2.64–3.96; Q-value = 0.119, d.f.(Q) = 1, P = 0.7). The main result and the analyses of moderators were not sensitive to the exclusion of two non-psychiatric studies with very large sample sizes that when combined included over 90% of individuals and 3.5% of the study weight in the random-effects meta-analysis.Reference Crandall, Fullerton-Gleason, Aguero and LaValley36, Reference Hyman, Ireland, Frost and Cottrell57
a. 58 studies in the analysis.
b. 68 studies included in the analysis.
Meta-analysis of sensitivity
The pooled sensitivity of suicidal ideation for suicide was 41% (95% CI 35–48). There was very high between-study heterogeneity, I 2 = 93.2, Q-value = 1033, d.f.(Q) = 70, P≤0.001. The sensitivity ranged from 0 to 97%, first quartile 25%, median 44% and the third quartile 67%. An Egger's regression was non-significant (intercept 0.60, t-value = 0.61, d.f. = 69, P = 0.54).
Studies of non-psychiatric populations had a significantly lower sensitivity (n = 13, sensitivity 22%, 95% CI 13–53) than studies among psychiatric patients (n = 58, sensitivity 46%, 95% CI 40–52; Q-value = 10.6, d.f.(Q) = 1, P = 0.001). This result was not sensitive to the exclusion of the two non-psychiatric studies with very large sample sizes.
Use of a mortality database and more recent publication were associated with a lower sensitivity. Sensitivity was increased among studies of patients in hospital and in studies with a higher proportion of individuals with suicidal ideation (Table 3). The proportion of people with suicidal ideation was the only factor that was independently associated with between-study heterogeneity in a multiple meta-regression (supplementary Table 3). The multiple meta-regression model suggested that the proportion of people with suicidal ideation was the only independent moderator and explained 58% of the between-study heterogeneity in sensitivity.
a. 58 studies included in the analysis.
b. 68 studies in the analysis.
Meta-analysis of specificity
The pooled specificity of suicidal ideation for later suicide was 86% (95% CI 76–92). There was very high between-study heterogeneity, I 2 = 99.9, Q-value >10 000, d.f. (Q) = 70, P≤0.001. The specificity ranged from 28 to 100%, first quartile 69%, median 83% and the third quartile 93%. An Egger's regression was non-significant (intercept −7.27 t-value = 0.74, d.f. = 69, P = 0.46).
Studies of non-psychiatric populations had a higher specificity (n = 13, specificity 96%, 95% CI 84–99) than studies of psychiatric populations (n = 58, specificity 81%, 95% CI 76–85, Q-value = 4.78, d.f.(Q) = 1, P = 0.03). This result was not sensitive to the exclusion of the non-psychiatric studies with very large sample sizes.
Suicidal ideation was a less specific indicator of suicide among studies with a high proportion of people with suicidal ideation and among samples of hospital-treated patients. A multivariate model found that hospital treatment and the proportion of people with suicidal ideation in the study population were significant variables, explaining 79% of the between-study heterogeneity in specificity. (Table 4 and supplementary Table 4).
a. 58 studies in the analysis.
b. 68 studies included in the analysis.
Meta-analysis of PPV
The pooled PPV for suicidal ideation as a test for later suicide among the 29 cohort studies was 1.7% (95% CI 0.9–3.2) over an average duration of follow-up of 9.1 years. There was very high between-study heterogeneity, I 2 = 97.9, Q-value = 1186, d.f.(Q) = 28, P≤0.001. An Egger's regression was non-significant (intercept 2.81, t-value = 1.13, d.f. = 27, P = 0.27). The PPV among nine non-psychiatric cohorts (0.3% 95% CI 0.1–0.5) was significantly lower (Q-value = 35.6, d.f.(Q) = 1, P<0.001) than among psychiatric samples (3.9% 95% CI 2.2–6.6).
Clinicians sometimes rely on suicidal ideation as a crucial test for short-term suicide risk,Reference Morgan and Stanton4 and it has been argued that asking about suicidal ideation could form part of a screening test for later suicide.Reference Boudreaux, Camargo, Arias, Sullivan, Allen and Goldstein5, Reference Horowitz, Snyder, Ludi, Rosenstein, Kohn-Godbout and Lee6 Our finding of a pooled OR of suicide associated with suicidal ideation of 3.41 and the meta-analytic AUC of 0.676 indicate a statistical association between suicidal ideation and later suicide with moderate strength.Reference Rosenthal90 However, this should be interpreted very cautiously because of the very high and largely unexplained heterogeneity between studies. Moreover, the low PPV of suicidal ideation for suicide, which flows from the low incidence of suicide outcomes, even among psychiatric cohorts, highlights the limitations of suicidal ideation as a practical test of future suicide.
We had hypothesised that suicidal ideation would be a more sensitive and specific test for suicide in non-psychiatric settings, where patients with suicidal ideation might have fewer other risk factors. We found no evidence that suicidal ideation is more strongly associated with suicide in non-psychiatric settings than in psychiatric settings. Contrary to our hypothesis we found that suicide ideation was a less sensitive test for later suicide in non-psychiatric settings, and that the prevalence of suicidal ideation is associated with the sensitivity for later suicide.
Our results can be compared with those of recent meta-analyses of the psychometric properties of suicide risk scalesReference Carter, Milner, McGill, Pirkis, Kapur and Spittal91, Reference Chan, Bhatti, Meader, Stockton, Evans and O'Connor92 and multivariate models of suicide risk,Reference Large, Kaneson, Myles, Myles, Gunaratne and Ryan93 which found similar odds of suicide, sensitivity, specificity and PPV among studies conducted in psychiatric settings as well as similar between-study heterogeneity despite examining quite different study sets. Collectively, these studies highlight a high degree of uncertainty about the statistical strength of commonly used approaches to suicide risk assessment.
The current paper advances knowledge of the association between suicidal ideation and suicide by reporting the pooled sensitivity and specificity of suicidal ideation for suicide. The main finding is the limited sensitivity of suicidal ideation for suicide, such that approximately 60% of people who go on to die by suicide have not expressed suicidal ideation at a specified earlier time. In contrast, the non-expression of suicidal ideation is fairly specific to not dying by suicide in the study period, with only 14% of those not dying by suicide expressing suicidal ideation.
Knowledge of the sensitivity and specificity of suicidal ideation for suicide is important because any given OR can be achieved with differing trade-offs between sensitivity and specificity. In this study we found firm evidence that such trade-offs differ between studies, in part as a result of the proportion of individuals expressing suicidal ideation. The proportion of people with suicidal ideation was strongly associated with sensitivity and was inversely associated with specificity, such that similar odds of suicide were found in populations with higher rates of suicidal ideation (i.e. psychiatric patients) and lower rates of suicidal ideation (i.e. non-psychiatric populations).
In general, screening tests should be sensitive enough to capture most cases.Reference Wilson and Jungner11 Our study suggests that suicidal ideation is not sensitive enough to be very helpful as a stand-alone screening test for suicide in psychiatric or non-psychiatric settings, a limitation that is compounded by the modest specificity. Clinicians should not assume that patients experiencing mental distress without suicidal ideation are not at elevated risk of suicide. It is notable that our subgroup analysis of non-psychiatric settings found that nearly 80% of people who later die by suicide had not expressed suicidal ideation. Conversely suicidal ideation is somewhat specific to suicide, particularly in settings where the prevalence of suicidal ideation is low. In this sense, the presence of suicidal ideation conveys more salient information about later suicide than the absence of suicidal ideation.
The finding that the proportion of individuals with suicidal ideation was strongly associated with sensitivity and inversely associated with specificity requires some consideration. To some extent this result is a logical consequence of the definition of sensitivity as the proportion of all cases detected – hypothetically, an extremely sensitive test that included the slightest degree of suicidal ideation would likely select almost the entire population and would approach a 100% sensitivity. Although variation in the threshold test for suicidal ideation might have contributed to variation in the prevalence of suicidal ideation between studies in non-extreme examples with typical rates of suicidal ideation, a very wide variation in the proportion of people with suicidal ideation can be associated with a wide variation of sensitivities.
In addition to highlighting uncertainty in the strength of the association between suicidal ideation and suicide, this study highlights an important clinical dilemma about the trade-off between sensitivity and specificity. Although detailed questioning about suicidal ideation is often indicated, for example when there is suspicion of a suicide plan, and such questioning might bring about a greater sensitivity, it might be associated with a higher false positive rate than less detailed questioning.
In addition to the main limitation posed by the extent of between-study heterogeneity, this meta-analysis has a number of other limitations. In most studies there was a lack of clarity about how suicidal ideation was determined and defined. However, this limitation might not have had very much impact on our results because we found little evidence that structured methods of assessing suicidal ideation predicted suicide more accurately than clinical methods. Although it is logical that there may be important threshold issues on the spectrum of severity of suicidal ideation – a possibility supported by our finding of an association between the rate of suicidal ideation and the sensitivity of suicidal ideation for later suicide – there were too few studies that reported different points on the spectrum of ideation to properly explore threshold issues. Another limitation is that we lacked any data about how factors such as gender, comorbid conditions and wider cultural factors might affect the association between suicidal ideation and suicide.
Our results should also be interpreted in light of the knowledge that suicidal ideation can fluctuate over short periods of time.Reference Williams, Davidson and Montgomery94 The primary studies we included measured suicidal ideation at the beginning of a follow-up period, often a follow-up of years. In cases of individuals who died by suicide without earlier suicidal ideation it is likely that suicidal ideation emerged closer to the time of the suicide. It remains to be seen if real time or more temporally proximal measures of suicidal ideation will be more strongly associated with later suicide.Reference Kleiman, Turner, Fedor, Beale, Huffman and Nock95
In conclusion, enquiring about suicidal ideation will always be a central skill for mental health professionals, not because suicidal ideation is a meaningful forecast of suicide but because patients who express suicidal ideation are making important communications about their inner world and level of psychological distress. However, clinicians should be aware of the statistical limitations of ideation as a screening tool, and not be lured into either a false confidence generated by an absence of ideation, or determinism about the likelihood of suicide if it is present.
Supplementary material is available online at https://doi.org/10.1192/bjo.2018.88.