Association between suicidal ideation and suicide: meta-analyses of odds ratios, sensitivity, specificity and positive predictive value

Background The expression of suicidal ideation is considered to be an important warning sign for suicide. However, the predictive properties of suicidal ideation as a test of later suicide are unclear. Aims To assess the strength of the association between suicidal ideation and later suicide measured by odds ratio (OR), sensitivity, specificity and positive predictive value (PPV). Method We located English-language studies indexed in PubMed that reported the expression or non-expression of suicidal ideation among people who later died by suicide or did not. A random effects meta-analysis was used to assess the pooled OR, sensitivity, specificity and PPV of suicidal ideation for later suicide among groups of people from psychiatric and non-psychiatric settings. Results There was a moderately strong but highly heterogeneous association between suicidal ideation and later suicide (n = 71, OR = 3.41, 95% CI 2.59–4.49, 95% prediction interval 0.42–28.1, I2 = 89.4, Q-value = 661, d.f.(Q) = 70, P ≤0.001). Studies conducted in primary care and other non-psychiatric settings had similar pooled odds to studies of current and former psychiatric patients (OR = 3.86 v. OR = 3.23, P = 0.7). The pooled sensitivity of suicidal ideation for later suicide was 41% (95% CI 35–48) and the pooled specificity was 86% (95% CI 76–92), with high between-study heterogeneity. Studies of suicidal ideation expressed by current and former psychiatric patients had a significantly higher pooled sensitivity (46% v. 22%) and lower pooled specificity (81% v. 96%) than studies conducted in non-psychiatric settings. The PPV among non-psychiatric cohorts (0.3%, 95% CI 0.1%–0.5%) was significantly lower (Q-value = 35.6, P < 0.001) than among psychiatric samples (3.9%, 95% CI 2.2–6.6). Conclusions Estimates of the extent of the association between suicidal ideation and later suicide are limited by unexplained between-study heterogeneity. The utility of suicidal ideation as a test for later suicide is limited by a modest sensitivity and low PPV. Declaration interest M.M.L. and C.J.R. have provided expert evidence in civil, criminal and coronial matters. I.B.H. has been a Commissioner in Australia's National Mental Health Commission since 2012. He is the Co-Director, Health and Policy at the Brain and Mind Centre (BMC) University of Sydney. The BMC operates an early-intervention youth services at Camperdown under contract to Headspace. I.B.H. has previously led community-based and pharmaceutical industry-supported (Wyeth, Eli Lily, Servier, Pfizer, AstraZeneca) projects focused on the identification and better management of anxiety and depression. He is a Board Member of Psychosis Australia Trust and a member of Veterans Mental Health Clinical Reference group. He was a member of the Medical Advisory Panel for Medibank Private until October 2017. He is the Chief Scientific Advisor to, and an equity shareholder in, InnoWell. InnoWell has been formed by the University of Sydney and PricewaterhouseCoopers to administer the $30 M Australian Government Funded Project Synergy. Project Synergy is a 3-year programme for the transformation of mental health services through the use of innovative technologies.

Health professionals are expected to have the skills to assess and manage patients with suicidal thoughts. 1 The presence of thoughts of suicide and the expression of suicidal ideation are signs of significant distress, and all patients who express suicidal ideation require a careful clinical assessment 2,3 leading to a comprehensive treatment plan. 3 In addition, suicidal ideation is sometimes considered to be the most important sign of short-term suicide risk, 4 and it has been argued that questions about suicidal ideation play a crucial role in suicide screening. 5,6 Despite the undoubted clinical importance of suicidal ideation, meta-analysis has only been used recently to estimate the statistical strength of suicidal ideation as a test for later suicide. Several metaanalyses have highlighted the modest strength of the association between suicidal ideation and later suicide 7-10 and one has quantified the positive predictive value (PPV). 8 To date, no metaanalysis has examined the sensitivity and specificity of suicidal ideation for suicide. Knowledge of the sensitivity of suicidal ideation for later suicide is particularly important because screening tests should be sufficiently sensitive, such that a high proportion of affected people screen positive. 11 Moreover, existing meta-analyses of the association between suicide and suicidal ideation have been limited to studies with a cohort design, 9 only included patients with particular diagnoses 7 or included studies where suicidal ideation was established post-mortem using psychological autopsy methods. 8 We report a meta-analysis of cohort and case-control studies that examined the presence or absence of expressed suicidal ideation among those who later did or did not die by suicide.

Aims and hypotheses
The first aim of this study was to calculate a comprehensive set of measures of prediction of suicidal ideation for later suicide, including of effect size as measured by the pooled odds ratio (OR), sensitivity, specificity, meta-analytically derived receiver operator curve (ROC) and area under the curve (AUC) and the PPV. The second aim was to explore the extent of, and moderators of, between-study heterogeneity in these predictive measures. We hypothesised that suicidal ideation would be a more sensitive and specific test for suicide in non-psychiatric settings, where patients with suicidal ideation can be expected to have fewer additional and potentially confounding risk factors.

Searches
Multiple search strategies were initially explored using the databases Medline, Embase and PsycINFO from inception to January 2017. These searches yielded an excessive number of titles when the term 'suicide' was used as a keyword (>200 000) or in the title (>70 000). Moreover, placing limits in the searches using relevant search terms (such as 'ideation' or 'mortality') missed numerous relevant papers that were known to the authors through earlier hand searching. However, PubMed indexed all but one Englishlanguage publication located by searches of multiple databases in earlier studies. 7,8 Therefore, and in order to obtain a large and representative sample of relevant studies, one author (M.M.L.) examined the titles of English-language publications with an accompanying abstract that contained variants of the single term 'suicide' (suicid*) in their title and were published in PubMed from inception to 14 September 2017 and hand searched the references list of relevant review articles. [7][8][9][10]13,14 Two authors (M.M.L. and C.M.M.) winnowed the resulting abstracts and full-text publications ( Fig. 1).

Included studies
Studies were included if they reported the expression or nonexpression of suicidal ideation among people who later died by suicide or did not, such that all individuals could be classified into the following groups: true positive (those with suicidal ideation and suicide), false positive (those with suicidal ideation but without suicide), false negative (those without suicidal ideation but with suicide) or true negative (those without suicidal ideation or suicide). Studies were included if they provided effect size or other data that was sufficient to enable the calculation of the numbers of people in the true positive, false positive, false negative and true negative categories.
We included studies of patients who had received psychiatric care and people recruited from non-psychiatric settings including primary healthcare, general population samples, military populations and prisons. We excluded studies in which the dependent variable was suicide attempts, or when suicidal ideation was assessed by psychological autopsy methods, when all the controls were deceased or all of the individuals had made suicide attempts. We excluded studies of patients with severe medical illnesses such as malignancies or human immunodeficiency virus.

Definitions of suicide and suicidal ideation
Two authors (M.M.L. and C.M.M.) independently extracted the numbers of individuals with suicidal ideation among those who had died by suicide and controls and the moderator variables. A third author (A.C.) re-examined the data points. The discrepant points were re-examined by two authors. We accepted the definition of suicidal ideation used in the primary research, acknowledging that the primary research might not have fully reflected differences in the way suicidal ideation was expressed. Studies were coded into four categories according to their definition of suicidal ideation: studies that reported a composite measure of suicidality (inclusive of both suicidal ideation and behaviour); studies that reported suicide plans (including suicide 'threats' as reported in the primary studies); studies that recorded a wish to die; and the majority of studies in which the nature of the suicidal ideation or suicidal thoughts was not specified.

Moderator variables
We collected data about potential moderators of the association between suicidal ideation and suicide; (a) whether the individuals were from psychiatric or non-psychiatric samples; (b) whether the samples consisted of hospital-treated patients; (c) the mean sample age; (d) the duration of follow-up after the assessment for suicidal ideation; (e) whether the study had a cohort or control design; (f) whether patients with suicidal behaviour were regarded as having suicidal ideation; (g) the year of study publication; (h) the proportion of individuals included with suicidal ideation; and (i) the incidence of suicide in the study.

Assessment of strength of reporting
Two researchers (M.M.L. and C.M.M.) independently assessed the reporting strength (and hence the risk of bias) of each primary study's research using a scale with four further moderator variables (scored 0-4) derived from the Newcastle Ottawa Scale. 15 One point was allocated according to each of the following criteria: (a) use of a structured method to assess suicidal ideation; (b) collection of data about suicidal ideation in a method that was masked to the patient's suicide (either by a masking method or by electronic recording at the point of assessment); (c) if all the suicides were defined using a mortality database; and (d) inclusion of open verdicts as suicide. Studies counting open verdicts were rated as having stronger reporting because open verdicts are thought to be similar to suicides in some jurisdictions 16 and because the presence of a history of suicidal ideation might make a coronial verdict of suicide more likely.

Data synthesis
Random effects meta-analysis was chosen a priori for all estimates (OR, sensitivity, specificity, PPV) because of the diversity of study populations and the differences in methods used in the primary research. Between-study heterogeneity in effect sizes was examined using I 2 , Q-value statistics and prediction intervals. The possibility of publication bias was assessed using a funnel plot and Egger's regression, 17 and was quantified using Duval & Tweedie's trim and fill method. 18 Subgroup analysis (random effects within subgroups) and random-effects meta-regression (method of moments) were used to explore the extent to which between-study heterogeneity was explained by categorical and continuous variables (including the year of publication and the proportion of all participants with suicidal ideation).
Moderator variables (including strength of reporting score items) that were significantly associated with between-study heterogeneity in sensitivity and specificity at P≤0.05 were included in multiple meta-regression models. Multiple meta-regression was not used to examine heterogeneity in ORs because of the absence of significant moderator variables. Comprehensive Meta-Analysis (Version 3, Biostat, Englewood NJ) was used for the main analysis and a meta-analytic estimate of the ROC and AUC was calculated using Meta-DiSc. 19

Searches and data extraction
The initial searches yielded 320 potentially relevant papers that were examined in full text. There were 70 papers, 20-89 reporting 71 studies in which true positive, false positive, false negative and true negative could be ascertained. Eight differences in the effect size in the independent data extraction were resolved by re-examination of the data (Table 1 and see supplementary Table 2 for the data used in meta-analysis).
The earliest study was published in 1961 and the latest in 2017. The median publication date was 2006. In total, 42 studies had a case-control design and 29 were cohort studies. Of the studies, 58 studies were of current or former psychiatric patients including 42 studies of current or former psychiatric in-patients.
In 52 studies suicidal ideation was defined clinically and 19 studies used a structured method. Twelve studies reported a composite measure of suicidality (inclusive of suicide attempts). Four studies reported on suicide plans and two studies recorded a wish to die rather than active suicidal ideation.
Use of a mortality database and more recent publication were associated with a lower sensitivity. Sensitivity was increased among studies of patients in hospital and in studies with a higher proportion of individuals with suicidal ideation ( Table 3). The proportion of people with suicidal ideation was the only factor that was independently associated with between-study heterogeneity in a multiple meta-regression (supplementary Table 3). The multiple meta-regression model suggested that the proportion of people with suicidal ideation was the only independent moderator and explained 58% of the between-study heterogeneity in sensitivity.
Studies of non-psychiatric populations had a higher specificity (n = 13, specificity 96%, 95% CI 84-99) than studies of psychiatric  Suicidal ideation was a less specific indicator of suicide among studies with a high proportion of people with suicidal ideation and among samples of hospital-treated patients. A multivariate model found that hospital treatment and the proportion of people with suicidal ideation in the study population were significant variables, explaining 79% of the between-study heterogeneity in specificity. (Table 4 and supplementary Table 4).

Meta-analysis of PPV
The pooled PPV for suicidal ideation as a test for later suicide among the 29 cohort studies was 1.7% (95% CI 0.9-3.2) over an average duration of follow-up of 9.1 years. There was very

Main findings
Clinicians sometimes rely on suicidal ideation as a crucial test for short-term suicide risk, 4 and it has been argued that asking about suicidal ideation could form part of a screening test for later suicide. 5,6 Our finding of a pooled OR of suicide associated with suicidal ideation of 3.41 and the meta-analytic AUC of 0.676 indicate a statistical association between suicidal ideation and later suicide with moderate strength. 90 However, this should be interpreted very cautiously because of the very high and largely unexplained heterogeneity between studies. Moreover, the low PPV of suicidal ideation for suicide, which flows from the low incidence of suicide outcomes, even among psychiatric cohorts, highlights the limitations of suicidal ideation as a practical test of future suicide. We had hypothesised that suicidal ideation would be a more sensitive and specific test for suicide in non-psychiatric settings, where patients with suicidal ideation might have fewer other risk factors. We found no evidence that suicidal ideation is more strongly associated with suicide in non-psychiatric settings than in psychiatric settings. Contrary to our hypothesis we found that suicide ideation was a less sensitive test for later suicide in non-psychiatric settings, and that the prevalence of suicidal ideation is associated with the sensitivity for later suicide.
Our results can be compared with those of recent meta-analyses of the psychometric properties of suicide risk scales 91,92 and multivariate models of suicide risk, 93 which found similar odds of suicide, sensitivity, specificity and PPV among studies conducted in psychiatric settings as well as similar betweenstudy heterogeneity despite examining quite different study sets. Collectively, these studies highlight a high degree of uncertainty about the statistical strength of commonly used approaches to suicide risk assessment.
The current paper advances knowledge of the association between suicidal ideation and suicide by reporting the pooled sensitivity and specificity of suicidal ideation for suicide. The main finding is the limited sensitivity of suicidal ideation for suicide, such that approximately 60% of people who go on to die by suicide have not expressed suicidal ideation at a specified earlier time. In contrast, the non-expression of suicidal ideation is fairly specific to not dying by suicide in the study period, with only 14% of those not dying by suicide expressing suicidal ideation.

Implications
Knowledge of the sensitivity and specificity of suicidal ideation for suicide is important because any given OR can be achieved with differing trade-offs between sensitivity and specificity. In this study we found firm evidence that such trade-offs differ between studies, in part as a result of the proportion of individuals expressing suicidal ideation. The proportion of people with suicidal ideation was strongly associated with sensitivity and was inversely associated with specificity, such that similar odds of suicide were found in populations with higher rates of suicidal ideation (i.e. psychiatric patients) and lower rates of suicidal ideation (i.e. non-psychiatric populations).
In general, screening tests should be sensitive enough to capture most cases. 11 Our study suggests that suicidal ideation is not sensitive enough to be very helpful as a stand-alone screening test for suicide in psychiatric or non-psychiatric settings, a limitation that is compounded by the modest specificity. Clinicians should not assume that patients experiencing mental distress without suicidal ideation are not at elevated risk of suicide. It is notable that our subgroup analysis of non-psychiatric settings found that nearly 80% of people who later die by suicide had not expressed suicidal ideation. Conversely suicidal ideation is somewhat specific to suicide, particularly in settings where the prevalence of suicidal ideation is low. In this sense, the presence of suicidal ideation conveys more salient information about later suicide than the absence of suicidal ideation.
The finding that the proportion of individuals with suicidal ideation was strongly associated with sensitivity and inversely associated with specificity requires some consideration. To some extent this result is a logical consequence of the definition of sensitivity as the proportion of all cases detectedhypothetically, an extremely sensitive test that included the slightest degree of suicidal ideation would likely select almost the entire population and would approach a 100% sensitivity. Although variation in the threshold test for suicidal ideation might have contributed to variation in the prevalence of suicidal ideation between studies in non-extreme examples with typical rates of suicidal ideation, a very wide variation in the proportion of people with suicidal ideation can be associated with a wide variation of sensitivities.
In addition to highlighting uncertainty in the strength of the association between suicidal ideation and suicide, this study highlights an important clinical dilemma about the trade-off between Association between suicidal ideation and suicide sensitivity and specificity. Although detailed questioning about suicidal ideation is often indicated, for example when there is suspicion of a suicide plan, and such questioning might bring about a greater sensitivity, it might be associated with a higher false positive rate than less detailed questioning.

Limitations
In addition to the main limitation posed by the extent of betweenstudy heterogeneity, this meta-analysis has a number of other limitations. In most studies there was a lack of clarity about how suicidal ideation was determined and defined. However, this limitation might not have had very much impact on our results because we found little evidence that structured methods of assessing suicidal ideation predicted suicide more accurately than clinical methods. Although it is logical that there may be important threshold issues on the spectrum of severity of suicidal ideationa possibility supported by our finding of an association between the rate of suicidal ideation and the sensitivity of suicidal ideation for later suicidethere were too few studies that reported different points on the spectrum of ideation to properly explore threshold issues.
Another limitation is that we lacked any data about how factors such as gender, comorbid conditions and wider cultural factors might affect the association between suicidal ideation and suicide.
Our results should also be interpreted in light of the knowledge that suicidal ideation can fluctuate over short periods of time. 94 The primary studies we included measured suicidal ideation at the beginning of a follow-up period, often a follow-up of years. In cases of individuals who died by suicide without earlier suicidal ideation it is likely that suicidal ideation emerged closer to the time of the suicide. It remains to be seen if real time or more temporally proximal measures of suicidal ideation will be more strongly associated with later suicide. 95 In conclusion, enquiring about suicidal ideation will always be a central skill for mental health professionals, not because suicidal ideation is a meaningful forecast of suicide but because patients who express suicidal ideation are making important communications about their inner world and level of psychological distress. However, clinicians should be aware of the statistical limitations of ideation as a screening tool, and not be lured into either a false confidence generated by an absence of ideation, or determinism about the likelihood of suicide if it is present.