There is increased focus on the use of outcome data to support improvements in clinical services providing interventions for children and young people with mental health problems.1 In the UK, data are reported nationally and are derived from routine outcome measures (ROMs) completed for cases seen within child and adolescent mental health services (CAMHS) on at least two occasions.Reference Hall, Moldavsky, Baldwin, Marriott, Newell and Taylor2 The benefits of using ROMs include greater clinical responsiveness to the young person's progress, better engagement in therapy and more effective interventions.Reference Bickman, Kelley, Breda, de Andrade and Riemer3–Reference Unsworth, Cowie and Green5 However, although routine outcome monitoring has recognised benefits, widespread uptake within CAMHS has been slow, with paired ROMs typically being obtained for <50% of cases.Reference Baruch and Vrouva6–Reference Hall, Moldavsky, Taylor, Sayal, Marriott and Batty8 A recent evaluation of the caregiver-reported Strengths and Difficulties Questionnaire (SDQ), one of the core recommended outcome measures in the UK, revealed that although 40% were completed during assessment, only 3% were completed at 6-month follow-up.Reference Morris, Macdonald, Moghraby, Stringaris, Hayes and Simonoff9
Routine outcome monitoring
Although the routine use of ROMs is currently limited, even less is known about the outcomes that are achieved for young people receiving treatment as usual from CAMHS.Reference Bear, Edbrooke-Childs, Norton, Krause and Wolpert10 Within everyday CAMHS, outcomes may not be consistent and may be lower than those obtained in well-conducted research studies, where interventions are delivered under more controlled conditions.Reference Weisz, Kuppens, Ng, Eckshtain, Ugueto and Vaughn-Coaxum11 In terms of outcome metrics, research studies typically report between-group (control versus treatment) differences, which is not particularly relevant or meaningful at an individual level.Reference Walkup, Parkhurst and Lavigne12 Instead, attention has turned to establishing reliable change,Reference Jacobson, Follette and Revenstorf13 a metric used to determine whether significant change has occurred at an individual level and is not a result of measurement error.Reference Wolpert, Görzig, Deighton, Fugard, Newman and Ford14 Additionally, reliable recovery explores reliable change for those within the clinical range at the start of treatment who then move to the normal range. It has been suggested that reliable change and recovery are more meaningful outcome metrics for patients attending everyday clinical services.Reference Walkup, Parkhurst and Lavigne12
Reliable change in CAMHS
The adult Improving Access to Psychological Therapies (IAPT) programme aimed to achieve a 50% recovery rate, a target that was predicted to be lower in CAMHS because of the complexity and diversity of problems in young people accessing the service.Reference Wolpert, Jacob, Napoleone, Whale, Calderon and Edbrooke-Childs15 ROM data collected from multiple services during the initial stages of the Children and Young Person IAPT programme have been reported. A national evaluation of paired data (n = 4464), collected between 2011 and 2015 from CAMHS in the UK, explored reliable recovery on the youth self-report Revised Child Anxiety and Depression Scale (RCADS), a commonly used ROM. Reliable recovery was 46% for those with anxiety, 42% for depression and 26% for those with comorbid anxiety and depressive disorders.Reference Edbrooke-Childs, Wolpert, Zamperoni, Napoleone and Bear16 Using a different outcome measure and rater, the parent-report version of the SDQ, a reliable recovery rate of 36% has been reported.Reference Wolpert, Jacob, Napoleone, Whale, Calderon and Edbrooke-Childs15,Reference Fugard, Stapley, Ford, Law, Wolpert and York17 These findings are consistent with a recent meta-analysis that found 40% of young people receiving treatment as usual from CAMHS reliably recovered, as assessed by a range of outcome measures.Reference Bear, Edbrooke-Childs, Norton, Krause and Wolpert10 The review highlighted the dearth of research on the outcomes of treatment as usual and the considerable methodological shortfalls, and raised questions about the effectiveness of everyday CAMHS.Reference Walkup, Parkhurst and Lavigne12 This lack of published outcome data about CAMHS treatment as usual is concerning, particularly given the transformational aim of the Children and Young Person IAPT programme to embed ROMs within everyday services. It is unclear whether this objective has been achieved within individual services, or whether recovery rates have been maintained over time.
Aims
Our aim is to contribute to the evidence base regarding the outcomes for children and adolescents attending everyday CAMHS in the UK. We will investigate reliable change and recovery rates, using paired outcome measures collected over two different time periods.
Method
Study design
We conducted in-depth prospective audits of new, consecutive referrals, accepted into Bath and North East Somerset, Swindon and Wiltshire community CAMHS during two separate time periods: November 2017 to January 2018 and April to September 2019. The time frames were pragmatically determined by the availability of psychology assistants to undertake the audits.
This study is an audit of anonymised collective data, and as such, ethical approval was not required. Patients provided informed consent to complete outcome measures and for this to be analysed.
Outcome data
The service has a local procedure where ROMS are completed at baseline, after three sessions and at discharge. The service has mandated the use of the following core outcome measures, to be completed when considered appropriate:
(a) The Current View is a clinician-completed questionnaire that is conducted either on first contact with the patient or when there is a change in situation or understanding. It is used to provide an overview of presenting problems, problem severity, comorbidity, complexity (e.g. intellectual disability or parental health issues) and contextual factors (e.g. home or community), and impact on education/employment in terms of both attendance and attainment.Reference Jones, Hopkins, Kyrke-Smith, Davies, Vostanis and Wolpert18,Reference Vostanis, Martin, Davies, De Francesco, Jones and Sweeting19
(b) The RCADS is designed for children and young people aged 8–18 years, and is a 47-item questionnaire with subscales for major depression, panic disorder, obsessive–compulsive disorder, generalised anxiety, separation anxiety and social phobia. There is a youth self-report questionnaire and a parent-report version.Reference Chorpita, Yim, Moffitt, Umemoto and Francis20
(c) The SDQ is a 25-item questionnaire with subscales for emotional symptoms, conduct problems, hyperactivity/inattention, peer relationship problems and prosocial behaviour. It has self-report and parent-report versions.Reference Goodman21
Clinicians choose the outcome measure they feel is most appropriate to assess the young person's problems. Essentially, the RCADS is primarily used for problems of anxiety and low mood, and the SDQ is primarily used for conduct or attention-deficit hyperactivity disorder presentations.
Procedure
All accepted referrals during the two audit periods were identified through the Oxford Health National Health Service (NHS) Foundation Trust electronic patient information system. Records were screened to identify those who had completed the above outcome measures at assessment. Cases with baseline ROMs were reviewed to identify those young people who had been seen on more than three occasions. If ROMs were not available, the primary worker/care coordinator was prompted by email on three occasions and requested to complete follow-up ROMs. The second ROMs could therefore be completed at discharge or when still in treatment. These were either completed in paper format or inputted directly onto an online outcome monitoring system (True Colours for Windows).
Data analysis
Baseline and follow-up scores for youth- and parent-report versions of the RCADS and SDQ were extracted and inputted into SPSS version 23 for Windows. The RCADS and SDQ consist of a number of subscales, which are used to inform the focus of the intervention and monitor progress. Subsequently, change may be specific to a particular subscale relating to the primary presenting problem rather than the total score. Therefore, we analysed baseline and follow-up data for each subscale.
Reliable change was calculated for each subscale of the RCADS and SDQ. First, we calculated Cronbach's alpha, using baseline data to assess the internal consistency of each scale. Second, we calculated the standard error of the difference score. Finally, we calculated the reliable change index (RCI) as 1.96 times the standard error of difference between pre- and post-test scores. Any change above or below the RCI is considered reliable at the 95% confidence level. The RCI was used to classify individuals as reliably improved, deteriorated or no change, on each subscale.
Recovery rates were calculated if there were sufficient numbers of paired outcomes and cases demonstrating reliable change. For the RCADS, raw scores were converted to age- and gender-adjusted t-scores. A t-score of ≥65 was used to classify young people as within the clinical range. For the SDQ, those with a total score placing them in the borderline or abnormal range were identified. Reliable recovery was determined if a young person above the clinical cut-off at baseline showed a reliable reduction in post-test scores and moved into the normal range.
Results
Assessment completion and baseline ROMS
The total number of accepted referrals identified in each audit was 1074 in 2018 and 1172 in 2019. For those seen where ROMS were judged to be appropriate, over three-quarters completed baseline measures (2018: n = 672, 78.2%; 2019: n = 744, 77.5%). Of those eligible for a follow-up ROM, 174 (59.2%) completed at least one questionnaire in 2018 and 155 (45.7%) completed at least one questionnaire in 2019. It is unclear why paired ROM completion rates were lower in 2019.
Presentation, problem severity, comorbidity and complexity
Baseline clinician-completed Current View assessments were available for 464 (47.5%) of those assessed in 2018 and 310 (30.0%) of those assessed in 2019. The three most commonly identified problems across both of the years were generalised anxiety (69–70%), depression (65–67%) and social anxiety (59–68%). In 2018, 3506 separate problems were identified, with 39% (n = 1366) being rated as either moderate or severe. Although fewer problems were identified in 2019 (n = 2187,) more were rated as moderate or severe (45.3%, n = 990).
Comorbidity was high, with only 3% (n = 14) of referrals presenting with single problems in 2018 and 7.1% (n = 22) in 2019. The most common number of identified problems per referral was between five and ten (59.1% in 2018; 49.1% in 2019).
Across both audit periods, around a quarter of referrals occurred within a context of poor parental health. In 2019, a fifth of referrals had experienced abuse or neglect, and a sixth presented with pervasive developmental disorders.
Pre- and post-test change
Paired outcome data on total internalising scores for the youth- and parent/carer-completed RCADS and total difficulties scores for the SDQ are summarised in Table 1. Average pre- and post-test scores on all measures completed by both young people and parents show a reduction (improvement) over time.
Pre- and post-test youth- and parent-reported outcome data
RCADS, Revised Child Anxiety and Depression Scale; SDQ, Strengths and Difficulties Questionnaire.
* P < 0.05.
Reliable change: youth-completed outcomes
Paired RCADS data was obtained for 145 young people in 2018 and 123 young people in 2019. Paired SDQ data was available for 71 young people in 2018 and 75 young people in 2019 (Table 2).
Youth-reported reliable change
ROMs, routine outcome measures; RCADS, Revised Child Anxiety and Depression Scale; SDQ, Strengths and Difficulties Questionnaire.
Overall, 69 (47.6%) young people in 2018 and 60 (48.8%) young people in 2019 reported reliable improvement on at least one subscale of the RCADS. Reliable improvement on the SDQ was more modest. Overall, 11 (15.5%) young people in 2018 and 19 (25.3%) young people in 2019 reported a reliable improvement on at least one subscale of the SDQ.
Reliable change: parent/carer-completed outcomes
Paired RCADS were completed by 64 parents/carers in 2018 and by 65 parents/carers in 2019. Paired SDQs were completed by 42 parent/carers in 2018 and 37 parents/carers in 2019 (Table 3).
Parent/carer-reported reliable change
ROMs, routine outcome measures; RCADS, Revised Child Anxiety and Depression Scale; SDQ, Strengths and Difficulties Questionnaire.
In both years, 29 parents/carers (45%) reported that their child had reliably improved on at least one subscale of the RCADS. As with the youth self-reported SDQ, there was comparatively less reliable change over time. In 2018, 11 parents (26.2%) reported a reliable improvement on at least one subscale, with a similar rate in 2019 (n = 10; 27%).
Recovery
Given the comparatively small number of paired SDQs and the number who demonstrated a reliable improvement (range 11–19, depending on year and rater), we did not undertake any further analysis of recovery on this measure. For the RCADS, almost 90% of those who completed baseline assessments scored within the clinical range. Follow-up RCADS assessments highlight that half of the young people assessed reported reliable recovery on at least one subscale of the RCADS, compared with 40% of those assessed by parents (Table 4).
Youth- and parent/carer-reported recovery
RCADS, Revised Child Anxiety and Depression Scale.
Discussion
The aim of this study was to determine the outcomes of treatment as usual provided by one community CAMHS over time. We used different outcome metrics of significant change, reliable change and recovery, and explored young person and parent/carers ratings on two commonly used ROMs.
Our cohorts presented with predominantly anxiety and/or mood disorders, and as such, we had more paired RCADS than SDQs. Data from the Current View assessment support Wolpert et al's assertions that CAMHS referrals are often complex and comorbid, and present within a challenging context.Reference Wolpert, Görzig, Deighton, Fugard, Newman and Ford14 Pre- and post-test total scores showed a reduction (improvement) in youth- and parent-completed ratings over time, which was particularly marked on the RCADS and the impact assessment of the SDQ. Reliable change and recovery on individual subscales revealed improvement rates of up to 45–48% on the RCADS and up to 25–27% on the SDQ, depending on the informant. Given the predominance of emotional disorders in our cohort, the RCADS may be a more sensitive outcome measure. The SDQ was developed as a behavioural screening questionnaire, and although its brevity is a strength, only five items assess emotional symptoms.Reference Goodman, Meltzer and Bailey22 Similarly, agreement between informants on the same measure vary, and correlations are often low.Reference Goodman23 Young people may be more aware of changes in internal mood states or cognitions that are not directly observable or noticed by parents/carers. These findings highlight the importance of ensuring that outcome measures reflect the primary presenting problems, and of carefully choosing the primary informant.
The reliable recovery rates we report in this predominantly comorbid, emotionally disordered cohort are higher than those reported by others, and are more in line with recovery rates reported in the IAPT programme for working-age adults in the UK.Reference Edbrooke-Childs, Wolpert, Zamperoni, Napoleone and Bear16,Reference Bear, Edbrooke-Childs, Norton, Krause and Wolpert10,Reference Clark, Canvin, Green, Layard, Pilling and Janecka24,Reference Gyani, Shafran, Layard and Clark25 This may reflect our analysis of individual subscales rather than total scores. The use of subscales has been advocated as an important aspect of routine clinical practice, where relevant subscales are recommended to be completed session by session, to track progress.Reference Law and Wolpert26 The subscale will reflect the primary problem and the focus of the intervention. The outcome of an intervention for social anxiety, for example, may be better reflected on the social anxiety subscale of the RCADS, and may become diluted if change is assessed solely on total scores. This view is supported by an evaluation of a specialist obsessive–compulsive disorder service that found broad outcome measures seriously underestimated the effectiveness of the service.Reference Lee, Jones, Goodman and Heyman27 Our findings suggest that assessing change on specific subscales, rather than broad total scores, may be a more appropriate way of assessing outcomes. Clinically, this will involve a discussion with the young person to agree what outcomes or subscales are important for them.Reference Bear, Edbrooke-Childs, Norton, Krause and Wolpert10
Although the reliable recovery rates we report are encouraging, it is evident that half of the cohort showed no reliable improvement. There is also the possibility that some may have reliably deteriorated, although rates of deterioration were low (2018: 0.7%; 2019: 5.7%). There is clearly a need to improve the effectiveness of treatment as usual, and to find ways to help the sizeable proportion of young people who are not currently responding to the interventions they are receiving. A greater emphasis on supervised practice may be helpful, and the routine use of outcome monitoring can alert clinicians to patients who are not responding to treatment, which can be discussed during clinical supervision.Reference Walkup, Parkhurst and Lavigne12 It is also important to acknowledge that clinical change and recovery may not be the only important outcomes of a CAMHS intervention. Stabilisation and prevention of deterioration are also important indicators of a successful outcome, and need to be acknowledged.Reference Wallis, Potier, Milson and Beck28
The comparison of outcomes over different audit periods enables trends in presentations and outcomes to be identified. For patients, contemporary, meaningful and relevant local information about recovery and reliable change can help with decision-making. For clinicians the routine provision of this data may help to allay any suspicions or negativity, and promote a more positive attitude toward the routine use of outcomes.Reference Hall, Moldavsky, Baldwin, Marriott, Newell and Taylor2 At a service level, changes in presentations and outcomes can be quickly identified, and any necessary training or supervision implemented. Finally, with the increased national focus on the outcomes of CAMHS, repeated outcome audits provide additional in-depth, local contextual data that can help commissioners understand the outcomes that are achieved in everyday practice.1 This can lead to further exploration of issues such as problem severity, chronicity and diagnosis on outcomes, and how current interventions may need to be modified to become more effective.
Strengths and limitations
These two audits undertaken at different time periods provide consistent and encouraging results. Nonetheless, our study has a number of limitations that need to be acknowledged. First, although we report paired outcome data from almost 400 informants, we recognise that our sample is small. The number of paired SDQs we were able to analyse was particularly small. As such, our evaluation of the SDQ lacks power and our results may not be sensitive to change. Second, we obtained paired outcome data for between 46–60% of eligible young people, and so our findings may not be representative of those who receive treatment from CAMHS. Some cases without paired outcome data may have been dissatisfied or dropped out of the service, leading to our change rates being overinflated. Similarly, our findings may underrepresent recovery. Although we report paired ROMs, we do not know whether the second ROMs were completed at the end of the intervention or whether it was still in progress. Third, although we report reliable change and recovery, we do not know whether this is attributable to the treatment as usual the young people received from CAMHS. Spontaneous remission rates in mental health are high, and improvements may occur without any intervention. Fourth, although we were able to analyse paired outcomes for approximately half of eligible cases, there is still a significant amount of missing data. It is important to increase the number of paired ROMs to fully understand the effects of the service. In this study, we prompted clinicians to undertake the second ROMs, but we do not have data to determine whether such an approach increased completion rates. Finally, this study was undertaken within one mental healthcare trust. Although the service is based on the ithrve model, an integrated, person centred and needs led approach for delivering mental health services for children, young people and their families, recommended in the NHS Long Term Plan (NHS 2019),Reference NHS29 there may be important differences in service organisation or provision that limit the generalisability of our findings
To conclude, the results of these two in-depth audits are consistent, and detail the encouraging outcomes obtained from treatment as usual provided by community CAMHS. Reliable change and recovery rates indicate that up to half of those for whom data was available improved on at least one subscale. Our findings highlight the importance of agreeing the primary outcome and informant at the onset of the intervention. Furthermore, attention should be paid to specific subscale change, which may better reflect the focus of the intervention provided.
Data availability
The data that support the findings of this study are available on request from the corresponding author, P.S. The data are not publicly available due to restrictions.
Acknowledgements
We thank Megan Denne and Abigail Croker, who were involved in the initial audit.
Author contributions
P.S. devised the study idea. N.G. and E.H. collected the data. All authors were involved in the analysis of the data and drafting the paper. P.S. confirms that they had full access to all the data in the study and takes responsibility for the integrity of the data in the study and the accuracy of the data analysis.
Declaration of interest
None.
Open access
eLetters
No eLetters have been published for this article.