Confidence in the face of risk: the Risk Assessment and Management Self-Efficacy Study (RAMSES)

Aims and method To evaluate a comprehensive risk management programme. A Risk Assessment and Management Self-Efficacy Scale (RAMSES) was used to evaluate the impact of a clinical guideline and training course. Fifty-three psychological therapists were randomly allocated to training v. waiting list in a controlled, delayed-intervention design. Differences in mean self-efficacy scores between groups were examined using analysis of covariance (ANCOVA). Results The RAMSES measure had adequate factor structure, internal consistency and construct validity. When adjusting for baseline scores and cluster design, the group exposed to training had a higher mean self-efficacy score than controls. Mean differences between groups were not significant after the control group received training, nor at 6 months’ follow-up. Clinical implications Exposure to training and clinical guidelines can improve self-efficacy in risk assessment and management. An important advance put forward by this study is the specification of areas of competence in risk assessment and management, which can be measured using a psychometrically sound tool.


Risk in mental health services: existing practices
Since the 1990s risk assessment and risk management have become ingrained within the standard practice of mental health services. In the past decade, English government policy has emphasised the notion of safety as essential to good practice. 3,[13][14][15][16] With this emphasis on safety and risk in mental health services, a number of standardised assessment tools, management models and training programmes have been developed. 14,17,18 Risk assessment training has been suggested to be important in maintaining staff competence, although previous initiatives have mostly used non-validated satisfaction surveys and qualitative data as a means of evaluating training outcomes in uncontrolled study designs. [19][20][21] Based on such data, it is unclear whether these training programmes indeed improve competence in this area. Furthermore, to our knowledge, there is no consensus about what would constitute areas of competence in risk assessment and management.
Training programmes in this area typically instruct clinicians on how to use standardised risk assessment tools. Alongside standardised tools, however, practitioners use their experience, knowledge and judgement to formulate risk assessment and management plans. It is this combination of multiple factors that leads practitioners towards a more structured clinical judgement approach to risk. 14 However, healthcare professionals must feel competent and confident in employing what is essentially a complex, multifactorial approach to risk assessment and management. Qualitative research involving over 100 mental health staff, for example, suggests that confidence in professional judgements and practices is fundamental to safe and effective practice. 22

Self-efficacy in healthcare
Task-or work-specific confidence has often been conceptualised as related to the notion of 'self-efficacy'. Following the theory developed by Bandura, self-efficacy refers to an individual's perceived confidence in their capability to perform required actions to deal with prospective or future-oriented situations. 23,24 Self-efficacy has been found to influence professionals' behaviour, and enhanced self-efficacy leads to engaging more fully and effectively with tasks. 25 A number of studies have demonstrated this effect, and specifically in relation to improving assessment practices in healthcare settings. For example, self-efficacy related to physical assessments was positively correlated to the frequency of these assessments in a study involving acute care nurse practitioners. 26 In another study, a training programme designed to improve obstetrics and gynaecology residents' attitudes and confidence in caring for patients with depression led to 3-month improvements in perceived self-efficacy and in the use of formal diagnostic tools. 27 A lack of self-efficacy has also been suggested to be a potential barrier to healthcare professionals engaging in expected work behaviours, 28 and has been shown to lead to avoidance or reduced effort in tasks, whereas higher self-efficacy leads to intensification of efforts. 29 Furthermore, Bandura has argued that selfefficacy becomes more important when pursuing behaviour change in the face of adversity or obstacles, and that a high sense of self-efficacy in one domain does not necessarily transfer to other practice domains. 24 Since self-efficacy is likely to be associated with health professionals' behaviour, a deficit in the challenging domain of risk management may possibly result in poor effort or avoidance of these tasks, which in turn may result in neglect or poor response to risks and adverse events. In this sense, specifying the key domains and tasks pertinent to effective risk assessment and management is both relevant and important to mental healthcare, as is the consideration of self-efficacy within clinical training programmes.
This report describes the development and evaluation of a comprehensive risk assessment and management programme for mental health practitioners in primary care. The programme includes a clinical guideline, decisionmaking flowcharts, a training course and a structured questionnaire for use as a measure of self-efficacy in key areas of competence. The report presents two linked studies; the first study aimed to develop and validate the Risk Assessment and Management Self-Efficacy Scale (RAMSES) and the second study used this measure to evaluate the training programme and clinical guideline.

Participants and study design
Participating services included National Health Service (NHS) and voluntary-sector teams delivering evidencebased psychological interventions and addictions treatment and rehabilitation following national guidelines. [30][31][32][33] A cross-sectional sample of mental health and substance misuse practitioners (n = 76, across 8 teams and n = 34, across 4 teams respectively) completed the study questionnaires using a secure electronic survey. Table 1 provides participants' demographic characteristics.

Measures
RAMSES is a measure of task-specific self-efficacy, following the theory by Bandura. 24 It has been modelled on other valid and reliable self-efficacy measures used to evaluate risk management training programmes in other healthcare settings. 34 To our knowledge, there are no published comparable measures specific to risk management in mental healthcare. RAMSES contains a total of 18 items subdivided into three broad domains: assessment, management and referral (the actual questionnaire items are listed in Table 2). Each item is formulated as a specific task relating to one of the three broad domains or subscales. Respondents are prompted to rate their perceived selfefficacy on a Likert scale ranging from 0 (no confidence in ability to perform the task) to 10 (complete confidence in ability to perform the task). A composite self-efficacy score can be obtained by adding all item ratings and dividing the sum by 18.
This study included two comparative measures. The Addiction Counseling Self-Efficacy Scale (ACSES) measures self-efficacy in seven different areas: (1) clinical evaluation,   35 This valid and reliable measure provided a criterion standard to evaluate the convergent validity of the RAMSES tool, that is, the degree to which it measures a theoretically similar construct (self-efficacy). The Job Discrepancy and Satisfaction Scale (JDSS) addresses the extent to which practitioners are satisfied with their current working conditions including remuneration, supervision and autonomy. This instrument can be a useful tool to identify organisational factors that may need to be addressed to support and maintain a healthy workplace. 36 This tool was included to examine discriminant validity, that is, the degree to which RAMSES measures a theoretically different construct (self-efficacy) than the construct measured by the JDSS.

Data analysis
Factor analysis was used to examine the underlying structure of the data-set of 18 items in the RAMSES questionnaire. We hypothesised that the questionnaire measures an underlying construct with three components: (1) the assessment of risks, (2) the management of risks in clinical practice, and (3) the referral process that may ensue when risks are detected. Following the general rule outlined by Bryant & Yarnold, 37 we estimated a minimal sample size of 90 participants, based on a ratio of 5:1 between participants and scale items.
Principal components analysis (PCA) with varimax rotation and scree tests were used to perform factor analysis, which proceeds by extracting possible underlying factors and retains those which explain a large proportion of variance in the data. Conventional statistical tests were used to empirically evaluate the adequacy of the factor solution. The Kaiser-Meyer-Olkin (KMO) measure was calculated to determine whether any questionnaire items should be excluded from the final factor solution. In addition, Bartlett's test of sphericity was used, where P50.05 would be indicative of adequate factorability for the data-set as a whole.
After determining the factor structure of the questionnaire and deciding whether any items needed to be removed, we examined its validity and reliability using a series of conventional methods. Construct validity was tested by correlating RAMSES to another measure of self-efficacy (ACSES); and discriminant validity was assessed by correlating it to the JDSS scale which theoretically measures a different underlying construct ( job satisfaction). These correlations were calculated using averaged scores (sum of items divided by total number of items), and were only conducted in the subgroup of substance misuse practitioners, since we could only find a comparable self-efficacy measure (ACSES) that was relevant and specific to the practice of that professional group. Parametric (Pearson's) correlations between RAMSES scores and years of experience using the total sample were also performed, expecting that self-efficacy would positively correlate to experience. Conventional assumptions of normality and homoscedasticity were verified graphically and statistically (using Shapiro-Wilk test) before performing the above analyses. Finally, Cronbach's a was used as a measure of internal consistency of questionnaire items, using a = 0.70 as a cut-off, with a higher number indicating acceptable reliability. 38

Factor analysis
Using Kaiser's eigenvalue criteria (values 41.0), PCA indicated that three underlying factors accounted for 77.16% of variance in the data-set (factor 1 eigenvalue 11.50, accounting for 63.90% of variance; factor 2 eigenvalue 1.32, accounting for 7.35% of variance; factor 3 eigenvalue 1.07, accounting for 5.92% of variance). This was consistent with the scree test, which also indicated a three-factor solution based on eigenvalues above the cut-off of 1. After varimax rotation, factor 1 included items A4, B1, B2, B3, B4, B5 (eigenvalue 5.03, accounting for 27.92% of variance), and related to specific interventions to minimise risk. Factor 2 included items B6, B7, C1, C2, C3, C4 (eigenvalue 4.77, accounting for 26.50% of variance), and related to case management and referral in line with organisational policies. Factor 3 included items A1, A2, A3, A5, A6, B8 (eigenvalue 4.09, accounting for 22.73% of variance), and related to the assessment of risk. Table 2 presents the final rotated component matrix, displaying the correlations between the observed variables and the underlying factors. All factor loadings were above the minimal acceptable level of 0.40 on at least one factor, and we therefore decided to retain all 18 items in the final model. In addition, the adequacy of this set of variables for factor analysis was confirmed by Bartlett's test of sphericity, which was nonsignificant (approximate w 2 = 2099.05, d.f. = 153, P50.001).
In line with these findings, the overall KMO measure of sampling adequacy was 0.92, indicating excellent factorability for the set of variables, and suggesting that a large amount of variance within the data can be explained by the underlying factor structure.

Validity
Pearson's correlations were performed with the subgroup of substance misuse practitioners (n = 34) who had completed both the ACSES and JDSS measures. RAMSES was highly correlated to ACSES which theoretically measures selfefficacy (r = 0.71; P50.001), and was not correlated to JDSS which theoretically measures a different construct of job satisfaction (r = 70.33; P = 0.06). RAMSES scores were also positively correlated with years of experience in the whole sample (n = 109; r = 0.22; P = 0.02), indicative of a modest association between years of experience and self-reported self-efficacy levels.

Reliability
Cronbach's a for the RAMSES questionnaire using the whole sample (n = 110) was 0.96, indicative of high internal consistency based on a conventional cut-off of 0.70. Comparative a values between mental health and substance misuse practitioners (n = 76 and n = 34 respectively) were 0.97 and 0.95. These analyses all indicate a high level of internal consistency and reliability for the questionnaire in different professional groups.

Risk management programme
The intervention in this study was a specialist training programme which follows the structure of a clinical guideline on the assessment and management of risks of harm to self or others 39 and was based on an adaptation of the threshold model for risk assessment. 40 The threshold model is based on research evidence indicating that the risk of suicide is influenced by a combination of predisposing factors (genetic factors such as family history, biological factors such as serotonin dysfunction, psychosocial factors such as history of childhood abuse, environmental factors such as social isolation, etc.), current trigger events (e.g. social, financial or family crisis, substance misuse), and protective factors (lack of access to methods of lethal selfharm, current support networks, personal values and attitudes towards suicide or self-harm, etc.). 41 Following assessment, preventive measures for high-risk patients may include psychotherapeutic interventions, the negotiation of a risk management plan, and liaison with other professionals. 39 The threshold model for suicidal risk outlines key risk assessment and management practices including: detection of major depression, thorough assessment of high-risk patients and suicide risk, prescription of adequate antidepressant medication, regular education of health professionals regarding risk, reducing availability of selfharm methods, managing substance misuse, and identification of family members at risk. 41 The risk guideline and training programme contained detailed information on: (a) how to identify people at increased risk of self-harm, suicide or violence; (b) decision rules for risk management and appropriate interagency liaison; and (c) guidelines on defensible practice and clinical record-keeping. Participants were exposed to 2-hour workshops covering the above topics, and were provided with copies of the clinical guideline and decision-making flowcharts.

Participants and study design
A total of 53 mental health practitioners participated in risk management training, and completed baseline and follow-up measures. Demographic characteristics for this subsample are in Table 3. Participants were randomly allocated to cohort A (immediate training) and cohort B (waiting list control group with delayed training after 6 weeks). Cluster randomisation was used, grouping participants according to their teams to minimise potential contamination of the control group through routine peer discussions. All participants were asked to complete outcome measures at four time points: (1) pre-training; (2) after training cohort A; (3) after training cohort B; and (4) at 6 months' follow-up.

Measures
The RAMSES questionnaire was used as the primary outcome measure.

Data analysis
Given the cluster randomisation design, the data were summarised for each cluster (7 clusters) and average RAMSES scores for each cluster were calculated at different time points. This is a methodologically robust alternative when analysing a relatively small number of clusters. 42,43 To determine the effects of training, analysis of covariance (ANCOVA) was used with the post-intervention RAMSES score (time 2) as the dependent variable. A grouping variable (cohort) was entered as a fixed factor, a clustering variable (team) was entered as a random factor, and the pre-training RAMSES score was entered as a covariate to adjust for baseline self-efficacy measures. The analyses were weighted for the total number of participants in each cluster who responded at time point 2. Two further weighted ANCOVA models were used to compare group differences in mean self-efficacy scores at different time points, with time 3 and time 4 measures as the dependent variables. Model checking was performed by assessing residual plots to ensure all models fit the data. Finally, paired t-tests were used to compare pre-training v. 6 months' follow-up scores for each cohort separately. Conventional assumptions of normality and equality of variances were tested prior to performing t-tests.   training plus clinical guidelines, and these gains were sustained at 6 months' follow-up.

Main findings
The RAMSES questionnaire was found to have a robust factor structure consisting of three components: clinical interventions, case management and assessment. Evidence of adequate internal consistency, construct and discriminant validity was also described for this measure. Limitations in the validation of the questionnaire related to the relatively small sample; particularly the low response rate for the substance misuse treatment subgroup which provided comparative data to assess construct and discriminant validity. Although the overall sample size was sufficient for psychometric testing according to conventional guidelines, 37 the small sample size may have been influenced by response bias. Further validation of the measure in larger samples and in different clinical settings (e.g. social workers, forensic and hospital settings) would be advisable. Future validation studies could also consider the rotation of questionnaire items or the addition of reverse-keyed questions to control for acquiescence response bias. This study presents evidence that training and development of clinical guidelines can improve mental health practitioners' confidence in assessing and managing clinical risks. These data indicate that such gains in selfefficacy can be sustained for at least 6 months after training. Although statistically significant, observed improvements in self-efficacy measures were fairly modest. This was likely to be influenced by the high baseline measures in this professional group which led to ceiling effects, possibly owing to the large number of therapists with several years of experience. Our correlation analysis indicated modest positive associations between self-efficacy and years of experience; therefore, differences in mean years of experience between cohorts possibly account for the higher baseline self-efficacy mean in cohort A.

Limitations
Some limitations in this trial concerned the small number of clusters (7), which required data analysis using summary measures for cluster means as proposed by Kerry & Bland. 42 An important limitation is that this study did not investigate associations between changes in self-efficacy and any actuarial data such as measurable changes in practice or risk incidents. There is also the possibility that self-efficacy may not be a strong predictor of mental health practitioners' behaviour change or improved outcomes. It is possible, for example, that overconfidence could actually result in inaccurate or complacent risk assessment, so the sensitivity and specificity of confident practitioners' identification of high-risk cases is yet to be determined. In addition, estimating the predictive accuracy of clinicians' risk detection and management is complicated by the relative low incidence rate of high-risk events at the level of individual mental health practitioners; for example, three-quarters of all those dying by suicide in the UK have not had contact with mental health services in the year before death. 16 A further limitation of this study concerns the broad definition of clinical risk used in the RAMSES measure. This may not be specific enough to discriminate between acute or immediate risk, and lifetime risk or risk in relation to particular diagnoses or illnesses, and the presentation and subsequent management of these risks would potentially differ in clinical practice.

Implications for practice and research
Mental health practitioners have a crucial role to play in the identification and prevention of serious risk incidents. This is clear from the detection point of view since increasing numbers of people are accessing mental health interventions in primary care services in England, 44,45 and also from an intervention point of view since some psychological interventions (e.g. dialectical behaviour therapy, behaviour therapy for those who attempt suicide) are likely to help to increase treatment retention and to reduce the frequency of suicidal ideation and attempts. 46,47 An important advance put forward by this study is the specification of areas of competence in risk assessment and management, which can be measured using a psychometrically sound tool. The RAMSES tool may be of relevance to mandatory training schemes for mental health specialists and other professionals whose role involves managing risk.
The present study adds to the increasing body of evidence demonstrating that training programmes can improve self-efficacy of healthcare staff. 34,[48][49][50][51] We argue that self-efficacy is important insofar as it reflects perceived confidence and competence on key aspects of clinical practice. Self-efficacy is closely related to the constructs of perceived control and perceived behavioural control; together, these aspects of self-perception are likely to predict intentions and actual behaviours. 52 Bandura's theory 24,53 posits that task-specific behaviours can be increased by enhancing self-efficacy through vicarious learning, persuasion and enactive performance. However,  the state of the evidence is still inconclusive with regard to impact on patient outcomes. 54 It is still too early to assert that increasing self-efficacy will necessarily result in sustained changes in risk management practice or tangible outcomes such as net reductions in risk incidents in a given treatment population. Next steps in the advancement of knowledge in this field would require the collection of guideline-matched performance data from risk management plans as a means of evaluating behaviour change following exposure to clinical guidelines and training. The role of clinical supervision may also be a relevant focus of attention. For example, actuarial data on risk management practices and risk incidents may be collected as part of clinical supervision, which could be correlated with selfefficacy measures to explore the predictive value of this construct.
In conclusion, this study demonstrates that evidencebased risk training can enhance mental health professionals' self-efficacy, and these gains are sustained for at least 6 months. We propose that comprehensive programmes of risk assessment and management training such as the one described here can be potentially helpful to other groups of professionals who come into contact with vulnerable individuals, for example health visitors, social workers, GPs, forensic and criminal justice professionals.