Obtaining an accurate clinical diagnosis is a critical process in mental healthcare pathways. Valid and reliable diagnostic data are required to match services and funding to areas of greatest need, with inaccurate or incomplete diagnoses limiting patient access to specialist care. 1 Within services, diagnostic formulations can assist practitioners in evidence-based treatment planning, thereby contributing to improved therapeutic engagement and outcomes for service users. Reference Jensen-Doss and Weisz2
Given the importance of diagnosis in the targeting and delivery of mental healthcare, surprisingly little attention has been paid to workforce development in this area. Previous research has focused on training non-mental health staff to diagnose psychiatric disorders, demonstrating the effectiveness of educational interventions that address knowledge, skills and attitudes in combination. Reference Bentham, Heg, Van Leeuwen and Metsemakers3 However, such approaches have yet to be applied systematically in the mental health workforce, where key training and support needs around diagnosis are also apparent. In particular, the move towards ‘new ways of working’ in the National Health Service (NHS), 4 involving modified and extended roles for clinical staff, has meant that assessments in multidisciplinary mental health teams are routinely undertaken by practitioners without formal diagnostic training.
The present study aimed to develop and test a new diagnostic training intervention for multidisciplinary practitioners in out-patient child and adolescent mental health services (CAMHS). This pilot work was conducted in a locality where previous audits had revealed large inconsistencies in rates of recorded diagnoses. The imminent introduction of a new system of care pathways based on disorder classifications also meant that improving quality of diagnostic assessments was a key priority within the wider CAMHS directorate of the host NHS trust.
It was hypothesised that the training intervention would be (a) acceptable to practitioners, (b) associated with increased practitioner self-efficacy in diagnosing cases and (c) associated with higher rates of routinely recorded case diagnoses. Although it was beyond the scope of this project to introduce specific supervisory arrangements or other support following training, participants were encouraged to make personal plans for translating knowledge and skills acquisition into changes in practice. The activities of individuals and teams in supporting implementation were examined after 8 months in order to better understand the sustained effects of training.
Method
Participants
Sixty-three clinicians working across an inner London borough took part in training, representing 91.3% of all CAMHS practitioners in the locality. Participants were drawn from six clinical teams: tier II–generic (33.3%), tier III–adolescents (20.6%), tier III–children (15.5%), tier III–paediatric liaison (6.3%), tier III–looked after children (14.3%) and tier III–neurodevelopmental (7.9%). The sample included social workers (28.6%), nurses (17.5%), clinical psychologists (14.3%), psychotherapists (9.5%), occupational therapists (4.7%), psychiatrists (7.9%), art therapists (6.4%), counselling psychologists (3.2%), family therapists (3.2%) and one play therapist (1.6%). Experience of working in CAMHS ranged from less than 1 month to 30 years (mean 6.52 years, s.d. = 6.72).
Training intervention
The training intervention involved a one-off, full-day workshop that was delivered over consecutive weeks to three separate practitioner cohorts, each comprising two multidisciplinary clinical teams. The aims were to increase competencies in the following areas: structuring a diagnostic assessment; recognising symptom patterns and comorbidity; selecting and interpreting standardised screening measures; coming to an accurate and meaningful diagnostic formulation; presenting a formulation to colleagues; and feeding back to the young person, family and referrer. The workshop was facilitated by two specialist registrars in child and adolescent psychiatry, with the contents and format informed by existing literature on training interventions for health professionals as well as consultations with senior clinicians in local services. A variety of interactive teaching methods were used, including small group work with case vignettes, larger group discussion about challenging issues in diagnosis, and role-play using simulated patients. Didactic teaching focused on basics of the ICD-10 5 multi-axial system and practicalities of recording a diagnosis in electronic patient records. Participants were also provided with a specially developed handbook that gave an overview of the main emotional, behavioural and developmental disorders, with descriptions of diagnostic criteria, differential diagnoses, common comorbidities, interview prompts and relevant screening measures.
Measures and data collection procedures
Learning outcomes
Participants’ satisfaction was assessed at the end of training using a modified version of the Training Acceptability Rating Scale (TARS). Reference Davis, Rawana and Capponi6 The original format (a four-point Likert scale combined with open-ended questions enabling qualitative feedback) was preserved, with individual items modified to ensure relevance (all measures are available from the authors on request). Acquisition of diagnostic competencies was assessed using a newly developed Self-Efficacy Questionnaire (SEQ). Respondents rated their confidence in performing nine specified diagnostic tasks on a six-point Likert scale, with higher scores indicating greater self-efficacy. A repeated-measures design was used to compare pre-training ratings with scores immediately after training, and again at 8-month follow-up. Initial questionnaires were administered in person at training sites; follow-up data were obtained by email.
Practice outcomes
The impacts of training on diagnostic rates were examined in a comparative audit of electronic patient notes across two case cohorts. The pre-training cohort included cases referred to participating teams during a 2-month window starting 6 months prior to the training intervention. The post-training cohort included cases referred during a 3-month window starting 1 month after the training intervention. The extended duration of the post-training audit window was chosen so that a larger number of cases would be available for comparison. For both cohorts, we were interested in multi-axial ICD-10 diagnoses recorded in electronic case notes within 4 months of the referral date. This ensured that the two sets of audit data would reflect distinct, non-overlapping time periods with identical follow-up periods. Figure 1 illustrates how the cases were selected for inclusion in the pre- and post-training audit cohorts. Audit approval was obtained from the local CAMHS Clinical Governance Committee prior to commencement.
Selection process for audited cases.
Implementation processes
A newly developed Plan For Going Forward (PFGF) form was used to record practitioners’ personal action plans for implementing changes in diagnostic practice, as well as eliciting suggestions about how clinical teams could help to sustain changes in practice. This form was completed at the end of the training workshop. Eight months later, PFGF forms were emailed back to individuals in a modified format that included a six-point likert scale for rating the extent to which original plans had been implemented. Separate ‘practitioner progress’ and ‘team progress’ scores were obtained from mean ratings on this measure.
An additional ten-item questionnaire was constructed based on a thematic analysis of team-related responses on PFGF forms. This was circulated by email to the six team managers at follow-up. Respondents were asked to provide ‘yes’ or ‘no’ answers (and supporting examples) to indicate whether or not their team had provided specified sources of support to practitioners.
Results
Learning outcomes
Thirty-four participants (53.9%) completed the TARS at the end of training. The majority of respondents (67.9%) reported that their knowledge and understanding had improved ‘quite a lot’ or ‘a great deal’. Most participants (78.1%) expected to use their learning in routine practice ‘quite a lot’ or ‘a great deal’, and 83.6% were similarly satisfied with the training overall. The most frequently identified strengths of the training were the use of actors as simulated patients, role-play and opportunities for group discussion. Participants recommended that the training could be improved by tailoring clinical vignettes to the diagnostic and age groups seen by different teams, and by individualising the training to suit the previous experience and expertise of participants.
Matched pre- and post-training SEQs were available for 56 participants (88.8%); 29 respondents (46.0%) also returned follow-up measures after 8 months. A repeated measures ANOVA with a Greenhouse–Geisser correction determined that mean SEQ scores differed significantly between time points (F(1.46, 38.0) = 15.7, P>0.001). Post hoc tests using the Bonferroni correction revealed that post-training SEQ scores (mean 39.9, s.d. = 5.48) were significantly higher than pre-training SEQ scores (mean 34.7, s.d. = 7.14; P>0.001). No significant difference (P = 1.00) was observed in SEQ ratings from post-training (mean 39.9, s.d. = 5.48) to follow-up (mean 40.4, s.d. = 8.14).
Practice outcomes
Table 1 summarises the sociodemographic and diagnostic characteristics of cases included in the audit. Compared with the pre-training cohort, individuals referred after training were significantly more likely to have a confirmed Axis I diagnosis (31.5% v. 17.5%), and significantly less likely to have an Axis I diagnosis explicitly ruled out (5.6% v. 18.4%). However, there was no difference between the two groups in the proportion of cases with an unspecified Axis I classification, which was more than 60% both before and after training. In addition, there were no discernible differences in relative frequencies of Axis I disorder subtypes, and very few individuals (>6%) in either cohort received a comorbid Axis I diagnosis or any Axis II–III classification. Higher absolute rates of classification were generally found for Axes IV and V, with increased use of Axis IV apparent over time.
Characteristics and diagnostic rates for patients in pre- and post-training cohorts
| Total (n=336) |
Pre-training cohort (n=212) |
Post-training cohort (n=124) |
Statistics | ||
|---|---|---|---|---|---|
| χ2(d.f.) | P | ||||
| Characteristics | |||||
| Mean age, years: mean (s.d.) | 9.86 (5.12) | 9.42 (4.93) | 10.61 (5.38) | ||
| Male, n (%) | 197 (58.6) | 128 (60.4) | 69 (55.6) | ||
| Axis I diagnosis (ICD-10), n (%) | |||||
| F00–F09: Organic, including symptomatic, mental disorders | 3 (0.9) | 1 (0.5) | 2 (1.6) | ||
| F10–F19: Mental and behavioural disorders due to psychoactive substance use | 3 (0.9) | 1 (0.5) | 2 (1.6) | ||
| F20–F29: Schizophrenia, schizotypal and delusional disorders | 2 (0.6) | 0 (0) | 2 (1.6) | ||
| F30–F39: Mood disorders | 11 (3.3) | 6 (2.8) | 5 (4.0) | ||
| F40–F42: Anxiety disorders | 12 (3.6) | 7 (3.3) | 5 (4.0) | ||
| F43: Stress reactions and adjustment disorders | 9 (2.7) | 5 (2.4) | 4 (3.2) | ||
| F44: Conversion disorders | 3 (0.9) | 1 (0.5) | 2 (1.6) | ||
| F50: Eating disorders | 1 (0.3) | 1 (0.5) | 0 (0) | ||
| F90: Hyperkinetic disorders | 3 (0.9) | 1 (0.5) | 2 (1.6) | ||
| F91: Conduct disorders | 8 (2.4) | 4 (1.9) | 4 (3.2) | ||
| F92: Mixed disorders of conduct and emotion | 10 (3.0) | 5 (2.4) | 5 (4.0) | ||
| F93: Emotional disorders with onset specific to childhood | 5 (1.5) | 2 (0.9) | 3 (2.4) | ||
| F94: Disorders of social functioning with onset specific to childhood and adolescence | 1 (0.3) | 1 (0.5) | 0 (0) | ||
| F98: Other behavioural and emotional disorders with onset usually occurring in childhood and adolescence | 2 (0.6) | 1 (0.5) | 1 (0.8) | ||
| Any Axis I diagnosis | 76 (22.6) | 37 (17.5) | 39 (31.5) | 8.76 (1) | <0.01 |
| No Axis I diagnosis | 46 (13.7) | 39 (18.4) | 7 (5.6) | 10.76 (1) | <0.01 |
| Axis I not specified | 214 (63.7) | 136 (64.2) | 78 (62.9) | 0.053 (1) | 0.818 |
| Any comorbid Axis I diagnosis | 4 (1.2) | 1 (0.5) | 3 (2.4) | ||
| Any Axis II diagnosis | 15 (4.5) | 8 (3.8) | 7 (5.6) | ||
| Any Axis III diagnosis | 6 (1.8) | 2 (0.9) | 4 (3.2) | ||
| Any Axis IV diagnosis | 74 (22.0) | 30 (14.2) | 44 (35.5) | ||
| Any Axis V classification | 100 (29.8) | 63 (29.7) | 37 (29.8) | ||
Implementation processes
The PFGF form was completed at the end of training by 56 participants (88.8%). A thematic content analysis of responses is provided in Table 2. Twenty-four respondents (38.1%) provided follow-up progress scores on the PFGF by email. Overall practitioner progress scores (mean 4.15, s.d. = 0.65) were significantly higher (t(23) = 2.48, P>0.05) than team progress scores (mean 3.71, s.d. = 0.98). Although sample sizes did not permit formal tests of significance on progress scores across different PFGF themes, scrutiny of individual forms suggested that team support most commonly involved providing diagnostic reference books and opportunities for discussing diagnosis in team meetings. Teams were relatively less likely to have provided support by way of protected time for diagnostic formulation and dedicated supervision around diagnostic issues. In comparison, practitioner progress scores were consistently rated as moderate or better across the identified themes.
Themes from practitioners’ Plan For Going Forward and corresponding feedback from managers
| Frequency of identified theme, n (% of 56 forms completed by practitioners at end of training) |
Activities endorsed by team managers, n (% of 4 forms completed at 8-month follow-up) |
|
|---|---|---|
| ‘Three things that I will do in order to put into practice what I have learned today’ | ||
| Take a more systematic approach to diagnostic assessments | 24 (42.9) | – |
| Take a more systematic approach to diagnostic formulation and presentation | 21 (37.5) | – |
| Become more familiar with ICD-10 multi-axial diagnostic criteria | 20 (35.7) | – |
| Discuss case diagnoses with colleagues | 14 (25.0) | – |
| Routinely record ICD-10 multi-axial diagnosis in electronic notes | 11 (19.6) | – |
| Consult diagnostic training handbook and vignettes | 8 (14.3) | – |
| Revisit diagnoses (past and future) to incorporate new information | 7 (12.5) | – |
| Make better use of screening tools | 4 (7.1) | – |
| Ensure team has access to diagnostic reference books | 1 (1.9) | – |
| ‘Three things that my CAMHS team can do to support me in putting into practice what I have learned today’ | ||
| Enable regular opportunities for discussions around diagnosis (e.g. in team meetings/supervision) | 43 (76.8) | 4 (100) |
| Make diagnostic reference books readily available | 9 (16.1) | 4 (100) |
| Enable protected time for preparing diagnostic formulations | 8 (14.3) 2 | (50.0) |
| Provide opportunities for observing/participating in more diagnostic assessments | 4 (7.1) | 2 (50.0) |
| Provide additional/refresher training | 4 (7.1) | 0 (0) |
| Provide support in making better diagnostic use of referral information and assessment feedback from other colleagues/agencies | 3 (5.3) | 3 (75.0) |
| Provide opportunities for discussing rationale for psychiatric diagnosis and alternative formulation models | 3 (5.3) | 3 (75.0) |
| Adapt diagnostic recording systems in electronic patient records to ensure greater relevance to client population | 2 (3.6) | 1 (25.0) |
| Provide support in use of screening measures | 2 (3.6) | 2 (50.0) |
CAMHS, child and adolescent mental health services.
In the main, findings regarding team progress were corroborated by feedback from team managers. All four of the responding managers confirmed that progress had been made in enabling more team discussions about diagnosis, as well as providing more diagnostic reference books. As shown in Table 2, other areas of team support were less consistently endorsed by managers.
Discussion
This study has shown that in-service training in psychiatric diagnosis, delivered across multiple CAMHS teams within a large service locality, is both feasible and acceptable to multidisciplinary practitioners. The one-day training intervention resulted in significant and sustained improvements in practitioners’ self-rated diagnostic competencies, providing further evidence for the utility of interactive, practice-based educational methods in healthcare settings. Reference Janssen-Noordman, Merriënboer, van der Vleuten and Scherpbier7 Training was also associated with significantly increased recording of Axis I disorders, and a reduction in the proportion of cases for which an Axis I diagnosis was explicitly ruled out. However, more than half of all cases in the post-training cohort still failed to receive a specific Axis I code, despite the fact that training participants were advised to assign an Axis I classification (i.e. disorder present or absent) for all assessed patients.
The low absolute rates of Axis I classification and comorbidity suggest that practitioners were continuing to underdiagnose such cases, or at least failing to record diagnosis within the early phases of service engagement. This may be an indication that a one-off training workshop, no matter how well received, does not provide multidisciplinary staff with the requisite knowledge and skills to successfully perform complex diagnostic tasks within a fixed time frame. Indeed, even highly trained and experienced clinicians are susceptible to biases that can affect diagnostic validity, such as prioritising confirmatory evidence at the expense of conflicting data; utilising a limited range of information sources; inconsistent application of standardised diagnostic criteria; and judgements unduly influenced by clients’ gender, ethnicity and age. Reference Garb8–Reference Whaley and Geller10 It may also be relevant to consider previous research, which has shown that awareness of the complexities surrounding diagnosis increases alongside more basic knowledge, with clinicians becoming more sensitive to the potential impacts of both a correct and incorrect diagnostic label on patients. Reference de Lusignan, Wells, Hague and Thiru11 As such, clinicians might have been more likely to ‘hedge their bets’ after training, becoming less likely to rule out an Axis I disorder altogether, yet still hesitating to record/enter one or more specific disorder code categories in the electronic patient record.
More comprehensive and sustained workforce development strategies may therefore be needed in order to achieve desired changes in diagnostic practice. Consistent with this conclusion, a systematic review Reference Oxman, Thomson, Davis and Haynes12 of the wider literature on educational interventions in health settings has suggested that ‘many interventions have modest or negligible practical effects when used alone. However, when coupled with other strategies the effects may be cumulative and significant’ (p. 1427). For example, provision of ongoing supervision following distribution of psychotherapy manuals has been shown to improve staff performance to a greater extent than dissemination of printed materials alone. Reference Sholomskas, Syracuse-Siewert, Rounsaville, Ball, Nuro and Carroll13 Aside from reinforcing trainees’ acquisition of technical knowledge and skills, supervisory feedback can increase awareness of the standards required to carry out complex clinical activities and promote self-reflection of recent learning and performances. Other research Reference Bennett-Levy and Beedie14 focused on mental health staff training has shown that these processes may temper trainees’ self-rated competence during learning, so that ‘the more they know, the more they realise they do not know’ (p. 72). This raises the possibility that a period of supervised practice would have influenced more realistic appraisals of competence within our training sample.
According to the widely researched theory of planned behaviour and related theory of reasoned action, self-efficacy beliefs are just one of three key determinants of behavioural change. Reference Perkins, Jensen, Jaccard, Gollwitzer and Oettingen15 Future training initiatives may also benefit from paying greater attention to the other critical variables that are theorised to influence behavioural intentions, namely ‘expected value’ (i.e. practitioners’ attitudes about the merits of diagnostic assessments in CAMHS) and ‘subjective norms’ (i.e. normative beliefs about diagnosis, and individuals’ motivation to comply with these norms). Anecdotally, a significant minority of participants in our workshop voiced concerns about the value and meaning of assigning psychiatric classifications to children and adolescents. The requirement to assign a diagnostic classification to all CAMHS patients, as mandated by the local trust, was also viewed by some non-medical staff as an unwelcome burden on clinical time. By developing a deeper understanding about these and other potential barriers to changing diagnostic practice, it may be possible to devise educational strategies that can be targeted at particular groups of practitioners (for example, those who are less confident but motivated or more capable yet sceptical). Further work is needed to develop and evaluate educational interventions that systematically address potential barriers to changing diagnostic practice, while examining the downstream effects on service user outcomes and experience.
Limitations
This study has several limitations. First, we audited diagnoses over a relatively short-time period (4 months from date of case referral) rather than the full episode of care. It is likely that a number of individuals whose cases were coded in our database as ‘Axis I not specified’ would have received a diagnosis at a later stage in the care pathway. However, we felt that it was reasonable to expect an individual referred to CAMHS to have received a diagnostic assessment within 17 weeks of referral, especially since participating services had implemented an 11-week maximum waiting time from referral to first appointment. A certain proportion of audited cases would also have been referred but never seen in CAMHS, thus inflating the number of cases expected to have a diagnosis. Second, we did not have a method for linking audited cases to participating clinicians. Although we included over 90% of practitioners working in the targeted service locality, we could not guarantee that audited cases were assessed by one of the training participants, or that participants were responsible for diagnosing a similar number of cases. Third, sampling bias might have occurred because of missing follow-up data from participants, although it is worth noting that our response rates of around 50% are in line with previous email surveys of health professionals. Reference Fischbacher, Chappel, Edwards and Summerton16 Fourth, we did not use an objective measure of practitioners’ diagnostic competencies (for example, structured clinical observations or vignette-based assessments), which would have enabled verification of self-efficacy scores. Fifth, we did not examine whether the training made any difference to the quality of diagnostic assessments. One option, which was beyond the scope of the present project, would have involved measuring agreement between routine and independent diagnostic assessments, such as those generated by researcher- or carer-completed forms of the Development and Well-Being Assessment. Reference Goodman, Ford, Richards, Gatward and Meltzer17 As an alternative, we developed a new tool for rating the quality of diagnostic formulations included in assessment reports. However, on piloting we found that fewer than 20% of audited cases had both an Axis I diagnosis and a corresponding report (typically in the form of a letter back to the referrer) entered into the electronic patient record. Given the lack of available data, we did not explore this further.
Clinical implications
Brief training in psychiatric diagnosis appears to be effective at increasing CAMHS practitioners’ self-rated competencies and may encourage more circumspection about ruling out a diagnosis altogether, while also increasing the positive identification of Axis I disorders. However, a single ‘one-size-fits-all’ workshop for multidisciplinary staff may be insufficient for changing practice so that diagnostic information is systematically gathered and promptly recorded for a majority of patients. Future initiatives aimed at improving routine diagnostic practice should consider supplementary strategies, such as provision of specialist supervision, and more targeted approaches to overcoming attitudinal and motivational barriers within different sections of the CAMHS workforce.
Funding
Southwark CAMHS, South London & Maudsley NHS Foundation Trust, UK.
Acknowledgements
We thank Craig Colling for his assistance in collating the data and George Goldie for his assistance with data collection. We also thank Partha Banerjea and Hena Vijayan for their contributions to the early planning of the training intervention.
Open access
eLetters
No eLetters have been published for this article.