Assessing recovery in treatment as usual provided by community child and adolescent mental health services

Background Despite the importance of routinely assessing the outcomes of everyday practice, few studies have reported outcome metrics for child and adolescent mental health services (CAMHS). Aims Our aim is to investigate reliable change and recovery rates for treatment as usual, provided by one community CAMHS over two time periods. Method We prospectively audited accepted consecutive referrals from November 2017 to January 2018, and April to September 2019. Cases with paired outcomes were identified, and reliable change and recovery rates were calculated. Results Baseline outcome data were obtained for 672 (78.2%) and 744 (77.5%) young people in 2018 and 2019, respectively. Of eligible participants, 174 (59.2%) and 155 (45.7%) completed at least one follow-up outcome measure in 2018 and 2019, respectively. Pre- and post-test scores on the Revised Child Anxiety and Depression Scale (RCADS) and Strengths and Difficulties Questionnaire (SDQ) showed a reduction in symptoms. Total RCADS scores showed 21–25% of participants reliably improved, with 44–49% showing reliable improvement on one or more subscale. On the SDQ, 11 (15.5%) and 19 (25.3%) participants reported reliable improvement on at least one subscale in 2018 and 2019, respectively. Reliable recovery rates ranged from 48 to 51% for youth-completed and 40 to 42% for parent-completed RCADS. Conclusions Half of young people receiving treatment as usual from CAMHS reliably improved on at least one routine outcome measure subscale, improvement rates comparable with adult psychological therapies services. Our findings indicate that reliable change and recovery on subscale rather than total scores may be a better indication of outcomes.

There is increased focus on the use of outcome data to support improvements in clinical services providing interventions for children and young people with mental health problems. 1 In the UK, data are reported nationally and are derived from routine outcome measures (ROMs) completed for cases seen within child and adolescent mental health services (CAMHS) on at least two occasions. 2 The benefits of using ROMs include greater clinical responsiveness to the young person's progress, better engagement in therapy and more effective interventions. 3-5 However, although routine outcome monitoring has recognised benefits, widespread uptake within CAMHS has been slow, with paired ROMs typically being obtained for <50% of cases. 6-8 A recent evaluation of the caregiver-reported Strengths and Difficulties Questionnaire (SDQ), one of the core recommended outcome measures in the UK, revealed that although 40% were completed during assessment, only 3% were completed at 6-month follow-up. 9 Routine outcome monitoring Although the routine use of ROMs is currently limited, even less is known about the outcomes that are achieved for young people receiving treatment as usual from CAMHS. 10 Within everyday CAMHS, outcomes may not be consistent and may be lower than those obtained in well-conducted research studies, where interventions are delivered under more controlled conditions. 11 In terms of outcome metrics, research studies typically report between-group (control versus treatment) differences, which is not particularly relevant or meaningful at an individual level. 12 Instead, attention has turned to establishing reliable change, 13 a metric used to determine whether significant change has occurred at an individual level and is not a result of measurement error. 14 Additionally, reliable recovery explores reliable change for those within the clinical range at the start of treatment who then move to the normal range. It has been suggested that reliable change and recovery are more meaningful outcome metrics for patients attending everyday clinical services. 12

Reliable change in CAMHS
The adult Improving Access to Psychological Therapies (IAPT) programme aimed to achieve a 50% recovery rate, a target that was predicted to be lower in CAMHS because of the complexity and diversity of problems in young people accessing the service. 15 ROM data collected from multiple services during the initial stages of the Children and Young Person IAPT programme have been reported. A national evaluation of paired data (n = 4464), collected between 2011 and 2015 from CAMHS in the UK, explored reliable recovery on the youth self-report Revised Child Anxiety and Depression Scale (RCADS), a commonly used ROM. Reliable recovery was 46% for those with anxiety, 42% for depression and 26% for those with comorbid anxiety and depressive disorders. 16 Using a different outcome measure and rater, the parent-report version of the SDQ, a reliable recovery rate of 36% has been reported. 15,17 These findings are consistent with a recent meta-analysis that found 40% of young people receiving treatment as usual from CAMHS reliably recovered, as assessed by a range of outcome measures. 10 The review highlighted the dearth of research on the outcomes of treatment as usual and the considerable methodological shortfalls, and raised questions about the effectiveness of everyday CAMHS. 12 This lack of published outcome data about CAMHS treatment as usual is concerning, particularly given the transformational aim of the Children and Young Person IAPT programme to embed ROMs within everyday services. It is unclear whether this objective has been achieved within individual services, or whether recovery rates have been maintained over time.

Aims
Our aim is to contribute to the evidence base regarding the outcomes for children and adolescents attending everyday CAMHS in the UK. We will investigate reliable change and recovery rates, using paired outcome measures collected over two different time periods.

Study design
We conducted in-depth prospective audits of new, consecutive referrals, accepted into Bath and North East Somerset, Swindon and Wiltshire community CAMHS during two separate time periods: November 2017 to January 2018 and April to September 2019. The time frames were pragmatically determined by the availability of psychology assistants to undertake the audits.
This study is an audit of anonymised collective data, and as such, ethical approval was not required. Patients provided informed consent to complete outcome measures and for this to be analysed.

Outcome data
The service has a local procedure where ROMS are completed at baseline, after three sessions and at discharge. The service has mandated the use of the following core outcome measures, to be completed when considered appropriate: (a) The Current View is a clinician-completed questionnaire that is conducted either on first contact with the patient or when there is a change in situation or understanding. It is used to provide an overview of presenting problems, problem severity, comorbidity, complexity (e.g. intellectual disability or parental health issues) and contextual factors (e.g. home or community), and impact on education/employment in terms of both attendance and attainment. 18,19 (b) The RCADS is designed for children and young people aged 8-18 years, and is a 47-item questionnaire with subscales for major depression, panic disorder, obsessive-compulsive disorder, generalised anxiety, separation anxiety and social phobia. There is a youth self-report questionnaire and a parent-report version. 20 (c) The SDQ is a 25-item questionnaire with subscales for emotional symptoms, conduct problems, hyperactivity/inattention, peer relationship problems and prosocial behaviour. It has selfreport and parent-report versions. 21 Clinicians choose the outcome measure they feel is most appropriate to assess the young person's problems. Essentially, the RCADS is primarily used for problems of anxiety and low mood, and the SDQ is primarily used for conduct or attention-deficit hyperactivity disorder presentations.

Procedure
All accepted referrals during the two audit periods were identified through the Oxford Health National Health Service (NHS) Foundation Trust electronic patient information system. Records were screened to identify those who had completed the above outcome measures at assessment. Cases with baseline ROMs were reviewed to identify those young people who had been seen on more than three occasions. If ROMs were not available, the primary worker/care coordinator was prompted by email on three occasions and requested to complete follow-up ROMs. The second ROMs could therefore be completed at discharge or when still in treatment. These were either completed in paper format or inputted directly onto an online outcome monitoring system (True Colours for Windows).

Data analysis
Baseline and follow-up scores for youth-and parent-report versions of the RCADS and SDQ were extracted and inputted into SPSS version 23 for Windows. The RCADS and SDQ consist of a number of subscales, which are used to inform the focus of the intervention and monitor progress. Subsequently, change may be specific to a particular subscale relating to the primary presenting problem rather than the total score. Therefore, we analysed baseline and follow-up data for each subscale.
Reliable change was calculated for each subscale of the RCADS and SDQ. First, we calculated Cronbach's alpha, using baseline data to assess the internal consistency of each scale. Second, we calculated the standard error of the difference score. Finally, we calculated the reliable change index (RCI) as 1.96 times the standard error of difference between pre-and post-test scores. Any change above or below the RCI is considered reliable at the 95% confidence level. The RCI was used to classify individuals as reliably improved, deteriorated or no change, on each subscale.
Recovery rates were calculated if there were sufficient numbers of paired outcomes and cases demonstrating reliable change. For the RCADS, raw scores were converted to age-and gender-adjusted t-scores. A t-score of ≥65 was used to classify young people as within the clinical range. For the SDQ, those with a total score placing them in the borderline or abnormal range were identified. Reliable recovery was determined if a young person above the clinical cut-off at baseline showed a reliable reduction in post-test scores and moved into the normal range.

Assessment completion and baseline ROMS
The total number of accepted referrals identified in each audit was 1074 in 2018 and 1172 in 2019. For those seen where ROMS were judged to be appropriate, over three-quarters completed baseline measures (2018: n = 672, 78.2%; 2019: n = 744, 77.5%). Of those eligible for a follow-up ROM, 174 (59.2%) completed at least one questionnaire in 2018 and 155 (45.7%) completed at least one questionnaire in 2019. It is unclear why paired ROM completion rates were lower in 2019.
Comorbidity was high, with only 3% (n = 14) of referrals presenting with single problems in 2018 and 7.1% (n = 22) in 2019. The most common number of identified problems per referral was between five and ten (59.1% in 2018; 49.1% in 2019).
Across both audit periods, around a quarter of referrals occurred within a context of poor parental health. In 2019, a fifth of referrals had experienced abuse or neglect, and a sixth presented with pervasive developmental disorders.

Pre-and post-test change
Paired outcome data on total internalising scores for the youth-and parent/carer-completed RCADS and total difficulties scores for the SDQ are summarised in Table 1. Average pre-and post-test scores on all measures completed by both young people and parents show a reduction (improvement) over time.
In both years, 29 parents/carers (45%) reported that their child had reliably improved on at least one subscale of the RCADS. As with the youth self-reported SDQ, there was comparatively less reliable change over time. In 2018, 11 parents (26.2%) reported a  Assessing recovery rates in CAMHS reliable improvement on at least one subscale, with a similar rate in 2019 (n = 10; 27%).

Recovery
Given the comparatively small number of paired SDQs and the number who demonstrated a reliable improvement (range 11-19, depending on year and rater), we did not undertake any further analysis of recovery on this measure. For the RCADS, almost 90% of those who completed baseline assessments scored within the clinical range. Follow-up RCADS assessments highlight that half of the young people assessed reported reliable recovery on at least one subscale of the RCADS, compared with 40% of those assessed by parents (Table 4).

Discussion
The aim of this study was to determine the outcomes of treatment as usual provided by one community CAMHS over time. We used different outcome metrics of significant change, reliable change and recovery, and explored young person and parent/carers ratings on two commonly used ROMs. Our cohorts presented with predominantly anxiety and/or mood disorders, and as such, we had more paired RCADS than SDQs. Data from the Current View assessment support Wolpert et al's assertions that CAMHS referrals are often complex and comorbid, and present within a challenging context. 14 Pre-and post-test total scores showed a reduction (improvement) in youth-and parent-completed ratings over time, which was particularly marked on the RCADS and the impact assessment of the SDQ. Reliable change and recovery on individual subscales revealed improvement rates of up to 45-48% on the RCADS and up to 25-27% on the SDQ, depending on the informant. Given the predominance of emotional disorders in our cohort, the RCADS may be a more sensitive outcome measure. The SDQ was developed as a behavioural screening questionnaire, and although its brevity is a strength, only five items assess emotional symptoms. 22 Similarly, agreement between informants on the same measure vary, and correlations are often low. 23 Young people may be more aware of changes in internal mood states or cognitions that are not directly observable or noticed by parents/carers. These findings highlight the importance of ensuring that outcome measures reflect the primary presenting problems, and of carefully choosing the primary informant.
The reliable recovery rates we report in this predominantly comorbid, emotionally disordered cohort are higher than those reported by others, and are more in line with recovery rates reported in the IAPT programme for working-age adults in the UK. 16,10,24,25 This may reflect our analysis of individual subscales rather than total scores. The use of subscales has been advocated as an important aspect of routine clinical practice, where relevant subscales are recommended to be completed session by session, to track progress. 26 The subscale will reflect the primary problem and the focus of the intervention. The outcome of an intervention for social anxiety, for example, may be better reflected on the social anxiety subscale of the RCADS, and may become diluted if change is assessed solely on total scores. This view is supported by  an evaluation of a specialist obsessive-compulsive disorder service that found broad outcome measures seriously underestimated the effectiveness of the service. 27 Our findings suggest that assessing change on specific subscales, rather than broad total scores, may be a more appropriate way of assessing outcomes. Clinically, this will involve a discussion with the young person to agree what outcomes or subscales are important for them. 10 Although the reliable recovery rates we report are encouraging, it is evident that half of the cohort showed no reliable improvement. There is also the possibility that some may have reliably deteriorated, although rates of deterioration were low (2018: 0.7%; 2019: 5.7%). There is clearly a need to improve the effectiveness of treatment as usual, and to find ways to help the sizeable proportion of young people who are not currently responding to the interventions they are receiving. A greater emphasis on supervised practice may be helpful, and the routine use of outcome monitoring can alert clinicians to patients who are not responding to treatment, which can be discussed during clinical supervision. 12 It is also important to acknowledge that clinical change and recovery may not be the only important outcomes of a CAMHS intervention. Stabilisation and prevention of deterioration are also important indicators of a successful outcome, and need to be acknowledged. 28 The comparison of outcomes over different audit periods enables trends in presentations and outcomes to be identified. For patients, contemporary, meaningful and relevant local information about recovery and reliable change can help with decision-making. For clinicians the routine provision of this data may help to allay any suspicions or negativity, and promote a more positive attitude toward the routine use of outcomes. 2 At a service level, changes in presentations and outcomes can be quickly identified, and any necessary training or supervision implemented. Finally, with the increased national focus on the outcomes of CAMHS, repeated outcome audits provide additional in-depth, local contextual data that can help commissioners understand the outcomes that are achieved in everyday practice. 1 This can lead to further exploration of issues such as problem severity, chronicity and diagnosis on outcomes, and how current interventions may need to be modified to become more effective.

Strengths and limitations
These two audits undertaken at different time periods provide consistent and encouraging results. Nonetheless, our study has a number of limitations that need to be acknowledged. First, although we report paired outcome data from almost 400 informants, we recognise that our sample is small. The number of paired SDQs we were able to analyse was particularly small. As such, our evaluation of the SDQ lacks power and our results may not be sensitive to change. Second, we obtained paired outcome data for between 46-60% of eligible young people, and so our findings may not be representative of those who receive treatment from CAMHS. Some cases without paired outcome data may have been dissatisfied or dropped out of the service, leading to our change rates being overinflated. Similarly, our findings may underrepresent recovery. Although we report paired ROMs, we do not know whether the second ROMs were completed at the end of the intervention or whether it was still in progress. Third, although we report reliable change and recovery, we do not know whether this is attributable to the treatment as usual the young people received from CAMHS. Spontaneous remission rates in mental health are high, and improvements may occur without any intervention. Fourth, although we were able to analyse paired outcomes for approximately half of eligible cases, there is still a significant amount of missing data. It is important to increase the number of paired ROMs to fully understand the effects of the service. In this study, we prompted clinicians to undertake the second ROMs, but we do not have data to determine whether such an approach increased completion rates. Finally, this study was undertaken within one mental healthcare trust. Although the service is based on the ithrve model, an integrated, person centred and needs led approach for delivering mental health services for children, young people and their families, recommended in the NHS Long Term Plan (NHS 2019), 29 there may be important differences in service organisation or provision that limit the generalisability of our findings To conclude, the results of these two in-depth audits are consistent, and detail the encouraging outcomes obtained from treatment as usual provided by community CAMHS. Reliable change and recovery rates indicate that up to half of those for whom data was available improved on at least one subscale. Our findings highlight the importance of agreeing the primary outcome and informant at the onset of the intervention. Furthermore, attention should be paid to specific subscale change, which may better reflect the focus of the intervention provided.

Data availability
The data that support the findings of this study are available on request from the corresponding author, P.S. The data are not publicly available due to restrictions.