Improving psychiatric diagnosis in multidisciplinary child and adolescent mental health services

Aims and method 
We examined learning outcomes, practice impacts and implementation processes for a training intervention in diagnostic skills delivered to multidisciplinary child and adolescent mental health service practitioners (n = 63). 
 
Results 
Training was viewed positively by most participants and associated with significant increases in practitioner self-efficacy, with the effect sustained at 8-month follow-up. A comparative audit before and after training indicated that clinicians were significantly more likely to assign an Axis I diagnosis following the training intervention. However, absolute rates of Axis I classification remained relatively low (< 40%) both before and after training. Practitioners were moderately successful at following through on personal plans for implementing new learning; inconsistent support for implementation was provided within teams. 
 
Clinical implications 
A brief training workshop may have limited effects in changing practitioners' behaviour so that diagnoses are made more promptly and appropriately recorded. Future workforce development initiatives should consider more comprehensive and diversified strategies, including targeted post-training support, if increased self-efficacy following training is to be translated into sustained changes in diagnostic practice.


Training intervention
The training intervention involved a one-off, full-day workshop that was delivered over consecutive weeks to three separate practitioner cohorts, each comprising two multidisciplinary clinical teams.The aims were to increase competencies in the following areas: structuring a diagnostic assessment; recognising symptom patterns and comorbidity; selecting and interpreting standardised screening measures; coming to an accurate and meaningful diagnostic formulation; presenting a formulation to colleagues; and feeding back to the young person, family and referrer.The workshop was facilitated by two specialist registrars in child and adolescent psychiatry, with the contents and format informed by existing literature on training interventions for health professionals as well as consultations with senior clinicians in local services.A variety of interactive teaching methods were used, including small group work with case vignettes, larger group discussion about challenging issues in diagnosis, and roleplay using simulated patients.Didactic teaching focused on basics of the ICD-10 5 multi-axial system and practicalities of recording a diagnosis in electronic patient records.Participants were also provided with a specially developed handbook that gave an overview of the main emotional, behavioural and developmental disorders, with descriptions of diagnostic criteria, differential diagnoses, common comorbidities, interview prompts and relevant screening measures.

Learning outcomes
Participants' satisfaction was assessed at the end of training using a modified version of the Training Acceptability Rating Scale (TARS). 6The original format (a four-point Likert scale combined with open-ended questions enabling qualitative feedback) was preserved, with individual items modified to ensure relevance (all measures are available from the authors on request).Acquisition of diagnostic competencies was assessed using a newly developed Self-Efficacy Questionnaire (SEQ).Respondents rated their confidence in performing nine specified diagnostic tasks on a six-point Likert scale, with higher scores indicating greater self-efficacy.A repeated-measures design was used to compare pre-training ratings with scores immediately after training, and again at 8-month follow-up.Initial questionnaires were administered in person at training sites; follow-up data were obtained by email.

Practice outcomes
The impacts of training on diagnostic rates were examined in a comparative audit of electronic patient notes across two case cohorts.The pre-training cohort included cases referred to participating teams during a 2-month window starting 6 months prior to the training intervention.The post-training cohort included cases referred during a 3month window starting 1 month after the training intervention.The extended duration of the post-training audit window was chosen so that a larger number of cases would be available for comparison.For both cohorts, we were interested in multi-axial ICD-10 diagnoses recorded in electronic case notes within 4 months of the referral date.This ensured that the two sets of audit data would reflect distinct, non-overlapping time periods with identical followup periods.Figure 1 illustrates how the cases were selected for inclusion in the pre-and post-training audit cohorts.Audit approval was obtained from the local CAMHS Clinical Governance Committee prior to commencement.

Implementation processes
A newly developed Plan For Going Forward (PFGF) form was used to record practitioners' personal action plans for implementing changes in diagnostic practice, as well as eliciting suggestions about how clinical teams could help to sustain changes in practice.This form was completed at the end of the training workshop.Eight months later, PFGF forms were emailed back to individuals in a modified format that included a six-point likert scale for rating the extent to which original plans had been implemented.Separate 'practitioner progress' and 'team progress' scores were obtained from mean ratings on this measure.
An additional ten-item questionnaire was constructed based on a thematic analysis of team-related responses on PFGF forms.This was circulated by email to the six team managers at follow-up.Respondents were asked to provide 'yes' or 'no' answers (and supporting examples) to indicate whether or not their team had provided specified sources of support to practitioners.

Learning outcomes
Thirty-four participants (53.9%) completed the TARS at the end of training.The majority of respondents (67.9%) reported that their knowledge and understanding had improved 'quite a lot' or 'a great deal'.Most participants (78.1%) expected to use their learning in routine practice 'quite a lot' or 'a great deal', and 83.6% were similarly satisfied with the training overall.The most frequently identified strengths of the training were the use of actors as simulated patients, role-play and opportunities for group discussion.Participants recommended that the training could be improved by tailoring clinical vignettes to the diagnostic and age groups seen by different teams, and by individualising the training to suit the previous experience and expertise of participants.

Practice outcomes
Table 1 summarises the sociodemographic and diagnostic characteristics of cases included in the audit.Compared with the pre-training cohort, individuals referred after training were significantly more likely to have a confirmed Axis I diagnosis (31.5% v. 17.5%), and significantly less likely to have an Axis I diagnosis explicitly ruled out (5.6% v. 18.4%).However, there was no difference between the two groups in the proportion of cases with an unspecified Axis I classification, which was more than 60% both before and after training.In addition, there were no discernible differences in relative frequencies of Axis I disorder subtypes, and very few individuals (56%) in either cohort received a comorbid Axis I diagnosis or any Axis II-III classification.Higher absolute rates of classification were generally found for Axes IV and V, with increased use of Axis IV apparent over time.

Implementation processes
The PFGF form was completed at the end of training by 56 participants (88.8%).A thematic content analysis of responses is provided in Table 2. Twenty-four respondents (38.1%) provided follow-up progress scores on the PFGF by email.Overall practitioner progress scores (mean 4.15, s.d.= 0.65) were significantly higher (t(23) = 2.48, P50.05) than team progress scores (mean 3.71, s.d.= 0.98).Although sample sizes did not permit formal tests of significance on progress scores across different PFGF themes, scrutiny of individual forms suggested that team support most commonly involved providing diagnostic reference books and opportunities for discussing diagnosis in team meetings.Teams were relatively less likely to have provided support by way of protected time for diagnostic formulation and dedicated supervision around diagnostic issues.In comparison, practitioner progress scores were consistently rated as moderate or better across the identified themes.
In the main, findings regarding team progress were corroborated by feedback from team managers.All four of the responding managers confirmed that progress had been made in enabling more team discussions about diagnosis, as well as providing more diagnostic reference books.As shown in Table 2, other areas of team support were less consistently endorsed by managers.

Discussion
This study has shown that in-service training in psychiatric diagnosis, delivered across multiple CAMHS teams within a large service locality, is both feasible and acceptable to multidisciplinary practitioners.The one-day training intervention resulted in significant and sustained improvements in practitioners' self-rated diagnostic competencies, providing further evidence for the utility of interactive, practice-based educational methods in healthcare settings. 7raining was also associated with significantly increased  'Three things that I will do in order to put into practice what I have learned today' Take a more systematic approach to diagnostic assessments 24 (42.9)-Take a more systematic approach to diagnostic formulation and presentation 21 (37.recording of Axis I disorders, and a reduction in the proportion of cases for which an Axis I diagnosis was explicitly ruled out.However, more than half of all cases in the post-training cohort still failed to receive a specific Axis I code, despite the fact that training participants were advised to assign an Axis I classification (i.e.disorder present or absent) for all assessed patients.The low absolute rates of Axis I classification and comorbidity suggest that practitioners were continuing to underdiagnose such cases, or at least failing to record diagnosis within the early phases of service engagement.This may be an indication that a one-off training workshop, no matter how well received, does not provide multidisciplinary staff with the requisite knowledge and skills to successfully perform complex diagnostic tasks within a fixed time frame.][10] It may also be relevant to consider previous research, which has shown that awareness of the complexities surrounding diagnosis increases alongside more basic knowledge, with clinicians becoming more sensitive to the potential impacts of both a correct and incorrect diagnostic label on patients. 11As such, clinicians might have been more likely to 'hedge their bets' after training, becoming less likely to rule out an Axis I disorder altogether, yet still hesitating to record/enter one or more specific disorder code categories in the electronic patient record.
More comprehensive and sustained workforce development strategies may therefore be needed in order to achieve desired changes in diagnostic practice.Consistent with this conclusion, a systematic review 12 of the wider literature on educational interventions in health settings has suggested that 'many interventions have modest or negligible practical effects when used alone.However, when coupled with other strategies the effects may be cumulative and significant' (p.1427).For example, provision of ongoing supervision following distribution of psychotherapy manuals has been shown to improve staff performance to a greater extent than dissemination of printed materials alone. 13Aside from reinforcing trainees' acquisition of technical knowledge and skills, supervisory feedback can increase awareness of the standards required to carry out complex clinical activities and promote self-reflection of recent learning and performances.Other research 14 focused on mental health staff training has shown that these processes may temper trainees' self-rated competence during learning, so that 'the more they know, the more they realise they do not know' (p.72).This raises the possibility that a period of supervised practice would have influenced more realistic appraisals of competence within our training sample.
According to the widely researched theory of planned behaviour and related theory of reasoned action, selfefficacy beliefs are just one of three key determinants of behavioural change. 15Future training initiatives may also benefit from paying greater attention to the other critical variables that are theorised to influence behavioural intentions, namely 'expected value' (i.e.practitioners' attitudes about the merits of diagnostic assessments in CAMHS) and 'subjective norms' (i.e.normative beliefs about diagnosis, and individuals' motivation to comply with these norms).Anecdotally, a significant minority of participants in our workshop voiced concerns about the value and meaning of assigning psychiatric classifications to children and adolescents.The requirement to assign a diagnostic classification to all CAMHS patients, as mandated by the local trust, was also viewed by some non-medical staff as an unwelcome burden on clinical time.By developing a deeper understanding about these and other potential barriers to changing diagnostic practice, it may be possible to devise educational strategies that can be targeted at particular groups of practitioners (for example, those who are less confident but motivated or more capable yet sceptical).Further work is needed to develop and evaluate educational interventions that systematically address potential barriers to changing diagnostic practice, while examining the downstream effects on service user outcomes and experience.

Limitations
This study has several limitations.First, we audited diagnoses over a relatively short-time period (4 months from date of case referral) rather than the full episode of care.It is likely that a number of individuals whose cases were coded in our database as 'Axis I not specified' would have received a diagnosis at a later stage in the care pathway.However, we felt that it was reasonable to expect an individual referred to CAMHS to have received a diagnostic assessment within 17 weeks of referral, especially since participating services had implemented an 11-week maximum waiting time from referral to first appointment.A certain proportion of audited cases would also have been referred but never seen in CAMHS, thus inflating the number of cases expected to have a diagnosis.Second, we did not have a method for linking audited cases to participating clinicians.Although we included over 90% of practitioners working in the targeted service locality, we could not guarantee that audited cases were assessed by one of the training participants, or that participants were responsible for diagnosing a similar number of cases.Third, sampling bias might have occurred because of missing follow-up data from participants, although it is worth noting that our response rates of around 50% are in line with previous email surveys of health professionals. 16ourth, we did not use an objective measure of practitioners' diagnostic competencies (for example, structured clinical observations or vignette-based assessments), which would have enabled verification of self-efficacy scores.Fifth, we did not examine whether the training made any difference to the quality of diagnostic assessments.One option, which was beyond the scope of the present project, would have involved measuring agreement between routine and independent diagnostic assessments, such as those generated by researcher-or carer-completed forms of the Development and Well-Being Assessment. 17As an alternative, we developed a new tool for rating the quality of diagnostic formulations included in assessment reports.However, on piloting we found that fewer than 20% of audited cases had both an Axis I diagnosis and a corresponding report (typically in the form of a letter back to the referrer) entered into the electronic patient record.Given the lack of available data, we did not explore this further.

Clinical implications
Brief training in psychiatric diagnosis appears to be effective at increasing CAMHS practitioners' self-rated competencies and may encourage more circumspection about ruling out a diagnosis altogether, while also increasing the positive identification of Axis I disorders.However, a single 'onesize-fits-all' workshop for multidisciplinary staff may be insufficient for changing practice so that diagnostic information is systematically gathered and promptly recorded for a majority of patients.Future initiatives aimed at improving routine diagnostic practice should consider supplementary strategies, such as provision of specialist supervision, and more targeted approaches to overcoming attitudinal and motivational barriers within different sections of the CAMHS workforce.

'
5) -Become more familiar with ICD-10 multi-axial diagnostic criteria 20 (35.7) -Discuss case diagnoses with colleagues 14 (25.0)-Routinely record ICD-10 multi-axial diagnosis in electronic notes 11 (19.6)-Consult diagnostic training handbook and vignettes 8 (14.3) -Revisit diagnoses (past and future) to incorporate new information 7 Three things that my CAMHS team can do to support me in putting into practice what I have learned today' Enable regular opportunities for discussions around diagnosis (e.g. in team meetings/supervision) Provide support in making better diagnostic use of referral information and assessment feedback from other colleagues/agencies 3 (5.3) 3 (75.0)Provide opportunities for discussing rationale for psychiatric diagnosis and alternative formulation models 3 (5.3) 3 (75.0)Adapt diagnostic recording systems in electronic patient records to ensure greater relevance to client population 2 (3.6) 1 (25.0)Provide support in use of screening measures 2 (3.6) 2 (50.0)CAMHS, child and adolescent mental health services.

Table 2
Themes from practitioners' Plan For Going Forward and corresponding feedback from managers

Table 1
Characteristics and diagnostic rates for patients in pre-and post-training cohorts