Introduction
Anxiety disorders are the most common mental health disorder in children (Polanczyk et al., Reference Polanczyk, Salum, Sugaya, Caye and Rohde2015), with a median age onset of 11 years (Kessler et al., Reference Kessler, Chiu, Demler and Walters2005). Effective treatments for child anxiety exist, in particular cognitive behavioural therapy (CBT) (James et al., Reference James, Reardon, Soler, James and Creswell2020), yet increasing demand for services (Sadler et al., Reference Sadler, Vizard, Ford, Goodman, Goodman and McManus2018) and limited numbers of trained mental health practitioners (Ford, Reference Ford2008) mean that few access evidence-based treatment (Lawrence et al., Reference Lawrence, Johnson, Hafekost, Boterhoven de Haan, Sawyer, Ainley and Zubrick2015; Reardon et al., Reference Reardon, Harvey and Creswell2019). Online treatments offer a potentially efficient, cost-effective (Learmonth et al., Reference Learmonth, Trosh, Rai, Sewell and Cavanagh2008; Spurgeon and Wright, Reference Spurgeon and Wright2010) flexible, and non-stigmatizing means to widen access to evidence-based treatments and meet the increasing demands on services (Jolstedt et al., Reference Jolstedt, Ljótsson, Fredlander, Tedgård, Hallberg, Ekeljung, Högström, Mataix-Cols, Serlachius and Vigerland2018; Sweeney et al., Reference Sweeney, Donovan, March and Forbes2019).
Online CBT treatments for child anxiety disorders have been developed in the USA (Child Anxiety Tales; Khanna et al., Reference Khanna, Carper, Harris and Kendall2017), Australia (BRAVE-ONLINE; Spence et al., Reference Spence, Donovan, March, Gamble, Anderson, Prosser, Kercher and Kenardy2008) and Sweden (Barninternetprojektet – BiP Anxiety: Vigerland et al., Reference Vigerland, Ljótsson, Thulin, Öst, Andersson and Serlachius2016). These are effective compared with waitlist (Khanna et al., Reference Khanna, Carper, Harris and Kendall2017; March et al., Reference March, Spence and Donovan2008; Vigerland et al., Reference Vigerland, Ljótsson, Thulin, Öst, Andersson and Serlachius2016) and face-to-face treatment (Spence et al., Reference Spence, Donovan, March, Gamble, Anderson, Prosser and Kenardy2011). In terms of recovery rates, 20% of children were recovered from their anxiety diagnosis immediately after receiving BiP Anxiety, which rose to 50% at 3-month follow-up (Vigerland et al., Reference Vigerland, Ljótsson, Thulin, Öst, Andersson and Serlachius2016). For BRAVE-ONLINE, 30% of children no longer met criteria for their primary anxiety disorder diagnosis at post-treatment and this further improved at 6-month follow-up with 75% recovered (March et al., Reference March, Spence and Donovan2008). Despite encouraging results for the effectiveness of online treatments for child anxiety, adoption of evidence-based online treatments within routine clinical practice remains limited (Hill et al., Reference Hill, Creswell, Vigerland, Nauta, March, Donovan, Wolters, Spence, Martin and Wozney2018).
Acceptability of online treatments to service users and service providers is key to successful dissemination, and yet user engagement, uptake and adherence to online treatments can be problematic (Fleming et al., Reference Fleming, Merry, Stasiak, Hopkins, Patolo, Ruru, Latu, Shepherd, Christie and Goodyear-Smith2019; Hollis et al., Reference Hollis, Falconer, Martin, Whittington, Stockton, Glazebrook and Davies2017; Pennant et al., Reference Pennant, Loucas, Whittington, Creswell, Fonagy, Fuggle, Kelvin, Naqvi, Stockton and Kendall2015). Involving the intended users in the development of online treatments in a process of co-design is argued to be best practice and crucial to maximizing engagement and adoption within routine practice (Hill et al., Reference Hill, Creswell, Vigerland, Nauta, March, Donovan, Wolters, Spence, Martin and Wozney2018; Pennant et al., Reference Pennant, Loucas, Whittington, Creswell, Fonagy, Fuggle, Kelvin, Naqvi, Stockton and Kendall2015). OSI (Online Support and Intervention for child anxiety) is a new online plus therapist support treatment for anxiety problems in children aged 7 to 12 years old that was developed in the UK using a co-design approach (Hill et al., Reference Hill, Reardon, Taylor and Creswell2022; see Table S1 (Supplementary material) for an overview of the online treatment). The treatment is based on a therapist-supported, parent-led CBT intervention for child anxiety disorders in which parents read an accompanying book (Creswell and Willetts, Reference Creswell and Willetts2012) and apply the CBT strategies with their child with approximately 5 hours of support from a therapist over about 8 weeks. This face-to-face treatment has been shown to be effective compared with waitlist (Thirlwall et al., Reference Thirlwall, Cooper, Karalus, Voysey, Willetts and Creswell2013), cost-effective compared with another brief psychological intervention (solution-focused therapy; Creswell et al., Reference Creswell, Violato, Fairbanks, White, Parkinson, Abitabile, Leidi and Cooper2017), acceptable and feasible for use within routine clinical practice (Creswell et al., Reference Creswell, Hentges, Parkinson, Sheffield, Willetts and Cooper2010), and is now widely used in early intervention services in the UK (Wood, Reference Wood2021). Translating this approach into an online format brings potential to further increase therapist efficiency and reduce waitlists, and ultimately increase access to evidence-based treatment for child anxiety. Furthermore, an online format has other potential advantages over ‘bibliotherapy’ such as audio/visual guides and opportunities for interactive learning, which may improve engagement and effectiveness, plus content can be easily and quickly updated to respond to the latest developments in the field (Marks et al., Reference Marks, Cavanagh and Gega2007).
The current study is an initial evaluation of OSI to assess clinical outcomes, feasibility and acceptability within an NHS clinic in England.
Method
Participants
Participants were children aged between 7 and 12 years old who were offered OSI as treatment for a primary anxiety difficulty in a Research Clinic between March and October 2020. While this is a university-based clinic, it is commissioned to provide treatment on behalf of the National Health Service (NHS), receives all referrals through local NHS mechanisms, and brief interventions are provided by Children’s Wellbeing Practitioners trained to standards governed by the NHS/Health Education England.
Children referred to the Research Clinic underwent a routine clinical assessment and those aged 7–12 years who were assessed to have a primary anxiety problem associated with significant functional impairment were offered OSI as the Clinic’s routine treatment for child anxiety at that time. As a brief intervention service, the Research Clinic does not provide treatment to children with an autism diagnosis, significant learning disability, or where there are substantial risk/safeguarding concerns; beyond these clinic criteria, there were no additional inclusion or exclusion criteria for offering OSI as a treatment beyond parents having access to the internet and the ability to read in English language in order to access the OSI content.
Twenty-four families were offered OSI between March and October 2020. One family declined OSI as they specifically wanted face-to-face support, so this case series reports outcomes on 23 participants. See Table 1 for a summary of sample characteristics.
Sample characteristics (% (n))
1 Child ethnicity is routinely collected from referral information but this was not provided for some participants. OSI, Online Support Intervention.
Study design and procedure
This study is an uncontrolled case series with no comparison group. Routine data for patient feedback and service evaluation were collected pre-treatment, post-treatment, and at a 4-week follow-up session (routinely carried out by the clinic to agree on the next steps for the family).
Measures
Clinical effectiveness
The following measures of clinical effectiveness are collected as Routine Outcome Measures (ROMs) as part of the Children and Young People Improving Access to Psychological Therapies (CYP-IAPT) initiative in NHS CAMHS (Edbrooke-Childs et al., Reference Edbrooke-Childs, Jacob, Law, Deighton and Wolpert2015a), with the exception of the Child Anxiety Impact Scale (CAIS), which is routinely collected in this clinic in line with recent guidelines that recommend measurement of both symptoms and impairment (Krause et al., Reference Krause, Chung, Adewuya, Albano, Babins-Wagner, Birkinshaw and Wolpert2021). All ROMs were built into OSI for compulsory completion at the start of each module.
Revised Child Anxiety and Depression Scale-Parent report (RCADS-P; Chorpita et al., Reference Chorpita, Yim, Moffitt, Umemoto and Francis2000)
The parent-report version of the Revised Child Anxiety and Depression Scale (RCADS-P) consists of 47 items relating to symptoms of separation anxiety disorder, generalized anxiety disorder, social anxiety disorder, obsessive-compulsive disorder, panic disorder and major depressive disorder which are rated on a 4-point scale (never, sometimes, often, always; scored 0–3). t-scores were calculated based on the child’s school year and gender, based on developer norms (Chorpita et al., Reference Chorpita, Moffitt and Gray2005). In line with CYP-IAPT procedures, the full version of the RCADS-P was administered at pre-treatment (module 0), post-treatment (module 6) and follow-up, and one RCADS-P subscale that best reflected the child’s primary difficulty was tracked each week. Scores for the tracked subscale, total anxiety and total score were calculated. The RCADS-P has good reliability and validity (Chorpita et al., Reference Chorpita, Yim, Moffitt, Umemoto and Francis2000; Chorpita et al., Reference Chorpita, Moffitt and Gray2005) and internal consistency was acceptable to excellent in the current study (Cronbach’s alpha for subscales ranged from 0.76 to 0.95).
Child Outcome Rating Scale (CORS; Duncan et al., Reference Duncan, Sparks, Miller, Bohanske and Claud2006)
The CORS is a 4-item scale that provides a measure of the child’s psychosocial functioning appropriate for measuring treatment outcomes. Parents of children under 13 report on their perception of their child’s personal wellbeing, family life, school and general sense of wellbeing/distress by marking a 10 cm visual analogue line anchored with an unhappy face (score of 0) on the left and happy face on the right (score of 10). Each item is scored by measuring the distance from the unhappy face to the nearest centimetre. A total score out of 40 is generated by summing the four items, with higher scores indicating better psychosocial functioning. The clinical cut-off for the parent-report on the CORS is a total score of 28. The CORS has good reliability and validity (Casey et al., Reference Casey, Patalay, Deighton, Miller and Wolpert2020; Duncan et al., Reference Duncan, Sparks, Miller, Bohanske and Claud2006) and internal consistency was good in the present study (Cronbach’s alpha = 0.86).
Goal-based outcomes (GBO; Law and Jacob, Reference Law and Jacob2013)
Parents report on their child’s progress towards up to three collaboratively agreed therapeutic goals using a 10-point Likert scale from 0 (no progress has been made towards goal) to 10 (goal has been reached fully). GBO ratings can provide a useful assessment of treatment impact to support standardized symptom measures. The psychometric properties of GBO ratings are not widely established, although evidence for acceptable internal consistency is emerging (Edbrooke-Childs et al., Reference Edbrooke-Childs, O’Herlihy, Wolpert, Pugh and Fonagy2015b). Due to variation in the number of goals set by each family in the present study, an average GBO rating was calculated for each timepoint to provide a measure of overall progress towards goals (Law and Jacob, Reference Law and Jacob2013). Examples of the therapeutic goals include ‘for X to be able to stay in bed from 10pm, without calling downstairs, after we have said goodnight’, ‘for X to join/re-join an activity/club that they previously enjoyed (e.g. street dance) – without needing a friend to join her’ and ‘for X to be able to get up and go downstairs to play (before mum and dad) at the weekend’.
Child Anxiety Impact Scale – Parent report (CAIS-P; Langley et al., Reference Langley, Bergman, McCracken and Piacentini2004)
The CAIS-P consists of 27 items that measure parental perception of the functional impact of anxiety on activities within school, social, home/family contexts and four items which relate to the global impact of anxiety. Two items not typically relevant for pre-adolescent children were excluded (going on a date, having a boyfriend/girlfriend). Items are rated on a 4-point scale (not at all, just a little, pretty much, very much; scored 0–3) and summed to produce four subscales (school, social, home/family, global) and a total impairment score. The CAISP was administered in full at pre-treatment, post-treatment and follow-up, whilst the global items were collected at each module. The CAIS-P has good internal consistency and convergent validity with other measures of child anxiety (Langley et al., Reference Langley, Falk, Peris, Wiley, Kendall, Ginsburg, Birmaher, March, Albano and Piacentini2014) and can be used to identify recovery from common anxiety disorders (Evans et al., Reference Evans, Thirlwall, Cooper and Creswell2017). Internal consistency was acceptable to excellent in the current study (Cronbach’s alpha was 0.93 for total score and 0.72 for global subscale).
Treatment satisfaction
Session Rating Scale (SRS; Miller et al., Reference Miller, Duncan and Johnson2000)
The SRS measures the key domains of effective therapeutic relationships and is used to gain feedback on the therapeutic alliance immediately after a therapy session. Parents were asked to complete the SRS at the end of each telephone review session with their therapist and the next OSI module was not unlocked until the SRS was completed. Parents rated four items relating to respect and understanding, relevance of goals and topics, client–practitioner fit and overall alliance by placing a mark on a 10 cm visual analogue scale closest to the end which represents their experience of the therapy session. Each item is scored by measuring where the line is placed to the nearest centimetre. A total score (range 0–40) is calculated by summing scores for the four items, with higher scores indicating greater therapeutic alliance. A total score of 36 is considered to be the cut-off for a cause for concern and should be followed up by the therapist. The SRS has been demonstrated to have good reliability and concurrent validity (Campbell and Hemsley, Reference Campbell and Hemsley2009; Duncan et al., Reference Duncan, Miller, Sparks, Claud, Reynolds, Brown and Johnson2003) and internal consistency was excellent in the current study (Cronbach’s alpha = 0.97).
OSI feedback questionnaire
Parents were asked to complete an optional module feedback questionnaire at the end of each module. This was adapted from the Programme Content and Usability Questionnaire (PCUQ) used by Wozney et al. (Reference Wozney, Baxter and Newton2015). The questionnaire consisted of 11 items (e.g. ‘The module was easy to understand’, ‘The module took an appropriate amount of time to complete’, ‘The module was easy to navigate’) which were rated on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree).
Treatment engagement and understanding
Information related to treatment engagement was measured through usage data routinely collected within OSI: (1) the number of times each page of each module was viewed, (2) the time spent on each page of each module (excluding ROMs), (3) the device used to access OSI (desktop/laptop, smartphone or tablet), and (4) time of day OSI was accessed (9am–5pm, 5.01pm–12am, 12.01am–8.59am). Engagement with the interactive elements of OSI (answering the questions in the module, completing the quizFootnote 1 ) were also included as measures of treatment engagement. The percentage of correctly answered quiz questions provided a measure of parental understanding of the treatment material.
OSI treatment
OSI is an online therapist-supported parent-led CBT treatment for pre-adolescent children with anxiety problems that was co-designed by NHS clinicians and service users (parents and children) (C. Hill et al., under review). OSI consists of eight modules which are summarized in Table S1 (Supplementary material). Modules are released weekly and parents have a 20-minute telephone support session with a therapist after each module to help them apply the strategies to their child and problem solve any difficulties, as part of routine clinical practice. Each module follows a similar format in which parents complete routine outcome measures at the start of each module (RCADS-P, CAIS-P, GBOs, CORS), followed by the module content and an optional module quiz and module feedback questionnaire. Parents must complete the module, therapist telephone support session and SRS before the next module is released. Each module takes approximately 30 minutes to complete. Treatment material is presented in short chunks of easy-to-read text with video testimonials and animations to explain how to apply the strategies. Modules are interactive, with questions to answer as the parent works through the treatment material, worksheets and quiz. It is not compulsory for the parent to engage in any of the interactive elements of OSI, with the exception of the ROMs. Parents are strongly encouraged and supported to complete/attempt the between-session tasks (e.g. exploring thoughts with their child, development and implementation of the step-by-step plan to test fears, use of worry boxes/worry time, etc.) before the telephone support session with the therapist so that parents are able to get the most out of the sessions (i.e. to review how they found implementing the techniques and/or troubleshoot any difficulties they have faced). Parents can track their child’s progress throughout treatment with immediate feedback on the ROMs. Parents also receive a written summary of the therapist support session through OSI.
OSI has an accompanying game application (Dekker et al., Reference Dekker, Eek, Flapper, Horstink, Meulenkamp, van der Meulen, Kooij, Snoeijer and Marin2018) for children called ‘Monster’s Journey: Facing Fears’, which is available to download onto smartphones or tablets for registered OSI users on the Apple Store and Google Play. The game is introduced in module 3 (Facing fears) as a tool to help motivate the child to test their fears by providing a reward mechanism. There is a parent portal which parents can use to reward their child with virtual coins that the child can use to unlock various features of the game. The game is optional for families to use, and usage is not monitored.
Data analysis
Repeated measures analysis of variance (ANOVA) were used to analyse clinical outcomes with ‘time’ (pre, post and follow-up) as the repeated factor. Where the main effect of ‘time’ was significant, Bonferroni-corrected pairwise comparisons were consulted comparing pre to post, pre to follow-up, and post to follow-up. These comparisons were bootstrapped using the simple method, with 2000 bootstrap samples and bias corrected and accelerated (BCa) 95% confidence intervals for data that violated the assumptions of parametric testing (RCADS total raw and t-score at post-treatment and follow-up, RCADS total anxiety raw and t-score at follow-up, CORS at follow-up, CAIS at post-treatment and follow-up). All participants with at least paired data (i.e. data were available for at least two modules of OSI) were included in the primary analysis. The last observation carried forward (LOCF) procedure was used where missing data for non-completers (or for the few instances, 11 data points, where data were missing at random due to technical difficulties) were replaced with data from the last observation (as long as paired data were available).
Per-protocol analyses were also carried out and any differences in interpretation with the primary analysis were noted. Participants were included in the per-protocol analyses if they had completed at least five OSI modules (module 0 – module 4) as these participants received the key components of the OSI treatment and the LOCF approach was used for missing data from module 5 onwards. (Note that per-protocol and LOCF analyses were not conducted for the full RCADS and CAIS at the post-treatment assessment as in order to have paired data these had to have been completed measures at both pre- and post-treatment assessments.)
For treatment studies with clinical populations, it is reasonable to expect large effects. In order to detect a large effect from pre- to post-treatment with 80% power (5% alpha), we calculated a priori that we would need data to be available from at least 15 participants. Accordingly, interpretation of findings was based on the p<0.05 significance level alongside effect sizes (partial eta-squared for F-test and Cohen’s d for pairwise comparisons). No formal statistical analyses were undertaken for measures that were taken at each OSI module because there were no a priori hypotheses regarding change between each module; however, plots are presented to illustrate the pattern of change across treatment.
We followed the approach taken by Edbrooke-Childs et al. (Reference Edbrooke-Childs, Wolpert, Zamperoni, Napoleone and Bear2018) to calculate reliable change, recovery and reliable recovery on the primary outcome measures (RCADS-P total anxiety subscale and total score). A reliable change criterion (RCC: SEdiff × 1.96, where
${\rm{SEdiff}} = {\rm{standard\;deviation\;}} \times {\rm{\;}}\sqrt 2 {\rm{x\;}}\sqrt 1 - {\rm{reliability}}$
) was calculated for these subscales using the internal consistency (Cronbach’s alpha) of the subscale and the standard deviation of the final included sample at pre-treatment (Jacobson and Truax, Reference Jacobson and Truax1991). Participants were classified as ‘reliably improved’ if scores had improved more than the RCC, ‘reliably deteriorated’ if scores had deteriorated more than the RCC, and ‘no reliable change’ if scores had not crossed the RCC. For ‘recovery’, those who had RCADS-P scores above the clinical cut-off before treatment and below the clinical cut-off at post-treatment or follow-up were considered ‘recovered’ and those who remained above the clinical cut-off were considered ‘non-recovered’. To be classified as ‘reliably recovered’, scores must be both ‘recovered’ and ‘reliably improved’ after treatment. Accordingly, analyses were run for the whole sample and for the subsample whose RCADS-P scores were above the clinical cut-off at baseline (i.e. t-score of 65 or higher for their school year and gender based on developer norms).
Treatment satisfaction was examined using descriptive statistics for SRS and module feedback as no a priori hypotheses were made regarding change in treatment satisfaction over time. Descriptive statistics are presented on all available engagement data. No formal statistical analyses were conducted as there were no a priori hypotheses on how engagement may differ across the modules.
Results
Adherence to OSI
The primary analysis was conducted on the n = 23 participants that took up OSI for all measures apart from the full version of the RCADS-P and CAIS-P, where the analyses were conducted on n = 18 who had paired data from modules 0 and 6 (i.e. when these measures were taken).
Twenty of the 23 families met criteria for ‘per-protocol’ participation in OSI. The three families who did not meet criteria dropped out after module 1 (n = 1; technical difficulties), module 2 (n = 1; anxiety resolved) and module 3 (n = 1; family disengaged from service). This treatment drop-out rate of 13% compares with an overall average early drop-out rate for the clinic of 11%. All other families completed all the treatment modules apart from one who discontinued treatment after module 4 (difficulty in managing treatment alongside home-schooling and work during lockdown), and one after module 5 (difficulty managing treatment alongside domestic issues and lockdown). Two families did not attend the 4-week follow-up session (reasons unknown). Six families attended the 4-week follow-up session but did not complete the measures on OSI.
Clinical effectiveness of OSI
Primary outcome: RCADS-P
As shown in Table 2, there was a significant main effect of time on mean RCADS-P anxiety and total t-scores with large effect sizes. Bonferroni-corrected pairwise comparisons showed significant reductions in total anxiety and total scores from pre-treatment to post-treatment, and pre-treatment to follow-up with large effect sizes, with mean post-treatment and follow-up scores in the non-clinical range. Post-treatment to follow-up comparisons were not significant, although effect sizes were medium, indicating maintenance or further improvement in the immediate period beyond treatment.
Mean (SD) 1 RCADS-P raw scores and t-scores at pre-treatment, post-treatment and follow-up with comparisons
1 Data presented are from the primary analysis using the LOCF approach for all cases with at least paired data (the per-protocol analysis is the same as paired data is from module 0 and module 6 when the full RCADS-P was administered). RCADS-P, Revised Child Anxiety and Depression Scale – Parent report.
Looking at change across the treatment, most participants tracked either the separation anxiety (39.1%, n = 9) or generalized anxiety (30.4%, n = 7) subscale of the RCADS-P; others tracked the social phobia (13%, n = 3), panic disorder (13%, n = 3) and major depression (4.3%, n = 1) subscales. As shown in Fig. 1A, the mean RCADS-P tracked subscale t-score declined steadily across each OSI module with mean scores moving in to the borderline clinical range from module 2 and the normal range from module 6 (i.e. post-treatment).
Sessional changes in routine outcome measures. (A) Tracked RCADS-P subscale t-score across OSI treatment modules (n = 23); (B) CORS total score across OSI treatment modules (n = 23); (C) CAIS-P global score across OSI treatment modules (n = 23); (D) average GBO rating across OSI treatment modules (n = 22); (E) SRS total score across OSI treatment modules (n = 15).
As shown in Table 3, 44% of children showed reliable improvement at post-treatment which increased to 56 and 61% at follow-up for the anxiety and total score, respectively. However, when looking at just those who were above the RCADS-P clinical threshold at baseline, 62% showed reliable improvement at post-treatment on both scales, which increased to 77 and 85% at follow-up for the anxiety and total scores, respectively. With regard to recovery, 69 and 62% were recovered by follow-up for the anxiety and total scores, respectively, and 62% were reliably recovered on both scales. None of the participants reliably deteriorated.
Reliable and clinically significant change (% (n))
1 Number of participants above the clinical cut-off for this subscale at baseline (i.e. t-score of 65 or higher for their school year and gender based on developer norms).
2 Results presented for those above clinical cut-off for this subscale at baseline. RCADS, Revised Child Anxiety and Depression Scale.
Secondary outcomes: CORS, CAIS-P and GBOs
As shown in Table 4, the repeated measures ANOVA for the CORS had a significant main effect with a large effect size. Pairwise comparisons showed that the CORS total score significantly improved with large effect sizes from pre-treatment to post-treatment and follow-up. The change between post-treatment and follow-up was not significant, with a small effect size, indicating maintenance of (but not improvement in) treatment gains. Figure 1B shows the change in CORS total score across treatment. There was a gradual improvement in overall child functioning over time, which was in the non-clinical range from module 1 onwards. A consistent pattern was seen in the per-protocol analyses.
Mean (SD) CORS total score, GBO average score and CAIS-P total score at pre-treatment, post-treatment and follow-up with comparisons
1 Data presented are from the primary analysis using the LOCF approach.
2 Data presented are from the per-protocol analysis.
3 n = 22 for this analysis because data must be available for three modules (modules 0, 1 and 2) as GBO is rated from module 1 onwards.
4 Data presented are from the primary analysis using the LOCF approach only (the per-protocol sample is the same as paired data are from modules 0 and 6 when the full CAIS-P was administered). CORS, Child Outcome Rating Scale; GBO, goal-based outcomes; CAIS-P, Child Anxiety Impact Scale – Parent report.
The CAIS-P total score decreased significantly over time, with a large effect size. Pairwise comparisons showed a significant decline from pre- to post-treatment and pre-treatment to follow-up with medium effect sizes, whereas the difference between post-treatment and follow-up was not significant with a trivial effect size. Change in the CAIS-P global impact of anxiety scores across treatment is shown in Fig. 1C. There was a steady decline across treatment that was more marked towards the end of treatment.
There were marked improvements in progress towards achieving the goals that were set for treatment in OSI module 1 as indicated by a significant main effect of time with a large effect size. Specifically, there was a significant increase in progress towards treatment goals from pre-treatment (module 1) to post-treatment with large effect sizes. The increase in progress from post-treatment to follow-up was not significant, with a small effect size. Figure 1D shows that steady progress was made across treatment towards meeting treatment goals.
Clinical outcomes
For those who completed OSI (n = 18), the majority of participants (83.3%, n = 15) were discharged from the service based on both the parents’ and clinicians’ view that no further psychological input was required. The remaining three families were either offered further treatment for non-anxiety issues within the clinic (n = 1), referred elsewhere for further treatment for anxiety (n = 1), or referred elsewhere for assessment/treatment for non-anxiety issues (n = 1).
Treatment satisfactionFootnote 2
Participants appeared to be highly satisfied with both the OSI treatment material and the therapist support telephone review sessions. Figure Figure 1e shows that participants consistently rated the telephone review sessions with their therapist very positively and above the cut-off score. Table S2 (Supplementary material) shows that the mean feedback scores were consistently in the ‘agree’ to ‘strongly agree’ range for each module.
Treatment engagement and understanding
Parents used either a desktop computer or laptop (39.1%), smartphone (22.4%) or both (39.1%) to log onto OSI. No one used a tablet. The total number of log-ins to OSI from the sample as a whole (n = 23) over the treatment period was 376, with an average of 16.35±8.38 (range 4–37) log-ins per participant. A higher total number of log-ins per participant (mean 9.70±10.12, range 0–37) were made on a desktop computer or laptop compared with smartphone (mean 6.22±7.83, range 0–30). All parents accessed OSI during 9am and 5pm at least once and this was the most common period for log-ins (mean number of log-ins per participant 10.22±7.82, range 1–32), with 82.6% (n = 19) also accessing OSI between 5pm and 12am (mean number of log-ins per participant 4.26±5.56, range 0–18) and 69.6% (n = 16) between 12am and 9am (mean number of log-ins per participant 1.87±2.12, range 0–8).
As shown in Table S3 (Supplementary material), the mean number of pages viewed per module was substantially higher than the actual number of pages in the module, which indicates that parents tended to revisit and engage with the module material more than once during treatment. The mean total time parents spent on each module varied between 10.17 minutes (module 0: ‘Welcome’) and 52.82 minutes (module 3: ‘Facing fears’). The total time spent on each module reflected the module content, with a greater amount of time spent on modules that contained a lot of material and worksheets to revisit (e.g. module 3: ‘Facing fears’ and module 5: ‘Problem-solving’) compared with those which were lighter on content (e.g. module 0: ‘Welcome’ and module 6: ‘Keeping things going’). This suggests that parents were engaging with the treatment material in a meaningful way rather than simply clicking through the pages. This conclusion was reinforced by the mean percentage of module questions completed (range 76.32–97%) and the mean percentage of quiz questions answered (range 90.48–100%) which were high for each module (see Table S3 in Supplementary material). Parents showed an excellent understanding of the module material across treatment, with the mean percentage of correctly answered quiz questions ranging from 83.57 to 100%.
Discussion
This case series has provided initial evidence to suggest that OSI is associated with good outcomes and is feasible and acceptable to families for use within clinical practice. Average anxiety symptoms significantly improved to below the clinical cut-off at post-treatment with further reduction at follow-up. When reliable change in anxiety symptoms was considered for the sample as a whole, over half had reliably improved by follow-up, with higher rates (over 75%) among those who scored above the clinical threshold at baseline. Significant improvements in overall psychosocial functioning, impairment from anxiety, and progression towards therapeutic goals were found at post-treatment and were consistently maintained at follow-up. The majority of children were discharged without further treatment for anxiety required. However, it is important to note that this study is a preliminary non-controlled case series, and thus firm conclusions regarding the efficacy of this treatment cannot be drawn.
The present study did not include diagnostic assessments of anxiety and so direct comparisons cannot be made, but the results presented here compare favourably with those reported for other online treatments for child anxiety (Spence et al., Reference Spence, Donovan, March, Gamble, Anderson, Prosser and Kenardy2011; Vigerland et al., Reference Vigerland, Ljótsson, Thulin, Öst, Andersson and Serlachius2016), particularly given that the follow-up period in the present study was shorter than in previous studies, and it could be reasonably assumed that the proportion of children with reliable improvement would increase over time (e.g. Thirlwall et al., Reference Thirlwall, Cooper, Karalus, Voysey, Willetts and Creswell2013). Our results also compare favourably with those reported from a large (n = 4564) study of the CYP-IAPT initiative in England (Edbrooke-Childs et al., Reference Edbrooke-Childs, Wolpert, Zamperoni, Napoleone and Bear2018) and a recent meta-analysis (n = 11,739) of outcomes of treatment as usual (TAU) in routine clinical practice by specialist mental health services globally (Bear et al., Reference Bear, Edbrooke-Childs, Norton, Krause and Wolpert2020). Specifically, we found rates of 76.9% for reliable improvement, 69.2% for recovery and 61.5% for reliable recovery, compared with 52, 60.9 and 45.9% across the CYP-IAPT initiative (Edbrooke-Childs et al., Reference Edbrooke-Childs, Wolpert, Zamperoni, Napoleone and Bear2018) and in Bear et al. (Reference Bear, Edbrooke-Childs, Norton, Krause and Wolpert2020) 38% reliably improved, 44% showed no reliable change, and 6% reliably deteriorated; although it should be noted that our sample was younger.
Although our primary outcome was anxiety symptomatology (RCADS-P), it is reassuring that similar benefits were also observed on overall child psychosocial functioning (CORS) and the functional impact of anxiety (CAIS-P). Most treatment outcome studies focus on symptomatology; however, functioning may be equally valued by service users and is key to meeting diagnostic criteria, yet few studies include measures of functional impact (Krause et al., Reference Krause, Bear, Edbrooke-Childs and Wolpert2019). Particularly positive effects were seen on progress towards treatment goals (GBOs), and such idiographic outcome measures have been argued to be potentially more sensitive to change than standardized outcome measures (Edbrooke-Childs et al., Reference Edbrooke-Childs, Jacob, Law, Deighton and Wolpert2015a).
Uptake of OSI was excellent, with only one family declining OSI in favour of face-to-face treatment. Drop-out was low, with 87% of families considered ‘treatment completers’ and only two families disengaged from OSI due to the treatment itself (anxiety had resolved in the other family). These findings also compare favourably with those of other studies, which have found variable uptake of online treatments for child anxiety. For example, nearly a third (32%) chose not to start a therapist-assisted version of BRAVE-ONLINE in a primary care setting in New Zealand (Moor et al., Reference Moor, Williman, Drummond, Fulton, Mayes, Ward, Dovenberg, Whitaker and Stasiak2019). The drop-out rate also compared favourably with the 28–75% of children/young people who drop out of psychological treatments (De Haan et al., Reference De Haan, Boon, de Jong, Hoeve and Vermeiren2013), drop out from face-to-face parent-led treatments for child anxiety (17% in Creswell et al., Reference Creswell, Hentges, Parkinson, Sheffield, Willetts and Cooper2010; 38% in Lyneham and Rapee, Reference Lyneham and Rapee2006; and 23% in Thirlwall et al., Reference Thirlwall, Cooper, Karalus, Voysey, Willetts and Creswell2013), and the average 20% drop-out rate from guided online psychological interventions (Andrews et al., Reference Andrews, Cuijpers, Craske, McEvoy and Titov2010; Van Ballegooijen et al., Reference Van Ballegooijen, Cuijpers, Van Straten, Karyotaki, Andersson, Smit and Riper2014).
Parents engaged meaningfully with the OSI material, with high rates of completion of the interactive elements of OSI and an excellent understanding of the treatment material. This may reflect the high levels of satisfaction parents reported for OSI and the ability for parents to log into OSI on any device at any time of day (which they did), making OSI a convenient treatment option for busy family lives. Unlike other online treatments for child anxiety, OSI was co-designed by the intended end-users and found to be easy and enjoyable to use in usability testing (Hill et al., Reference Hill, Reardon, Taylor and Creswell2022), which may also contribute to the excellent uptake, adherence and engagement.
This initial evaluation of OSI has several strengths. Firstly, despite the study being set in a university-based research clinic, this service is commissioned by the NHS and operates as a typical ‘Getting help/Getting more help’ NHS CAMHS clinic with treatments provided by Children’s Wellbeing Practitioners (Wolpert et al., Reference Wolpert, Jacob, Napoleone, Whale, Calderon and Edbrooke-Childs2016a). Therefore, it is representative of the clinical services, staff and service users that OSI is intended to be routinely used in and so the results presented here are likely to be generalizable. In addition to this, our analysis examined reliable change and recovery. It has been argued that this gives a clearer indication of treatment outcomes at the individual level rather than average outcomes for the whole group (Jensen and Corralejo, Reference Jensen and Corralejo2017; Wolpert et al., Reference Wolpert, Görzig, Deighton, Fugard, Newman and Ford2015b). This is especially important for observational studies without a control group, as it considers remission as the goal for treatment rather than superiority to a control (Walkup et al., Reference Walkup, Parkhurst and Lavigne2020). Presenting results on clinically meaningful change also helps service users and providers to understand the likely impact of OSI in terms of whether the child will reliably improve and no longer require use of mental health services.
Despite the encouraging results, there are several limitations which should be acknowledged. Firstly, there was no control group and so the contribution of spontaneous remission and placebo effects on the outcomes is unknown. Although this was a referred sample who underwent a thorough clinical assessment, nearly half (43.5%, n = 10) did not score above the clinical threshold on the primary outcome measure (RCADS-P), and so the sample size was reduced for the analysis of clinically significant change. Previous evaluations of the RCADS have shown sensitivity and specificity ranging from .59 to .92, indicating that there will be a proportion of cases where their ‘clinical’ status according to the RCADS does not align with their diagnostic status. This may at least in part reflect the fact that the RCADS does not include items on interference, which means children who experience a small number and/or low frequency of symptoms that nevertheless cause big problems may have low scores on the RCADS. Nonetheless, despite its potential lack of sensitivity of the RCADS for this age group (Evans et al., Reference Evans, Thirlwall, Cooper and Creswell2017), it has also been reported to be the most sensitive to change of the CYP-IAPT ROMs (Wolpert et al., Reference Wolpert, Cheng and Deighton2015a). All of the outcome measures used in the present study were parent-report and results may not be replicated in child-reported outcomes, especially as parent–child agreement on anxiety symptoms is low to moderate (Popp et al., Reference Popp, Neuschwander, Mannstadt, In-Albon and Schneider2017). However, given that parent-report is usually more consistent with outcomes from diagnostic assessments (Evans et al., Reference Evans, Thirlwall, Cooper and Creswell2017), it has been recommended that parent-report is prioritized for studies with pre-adolescent children (Creswell et al., Reference Creswell, Nauta, Hudson, March, Reardon, Arendt, Bodden, Cobham, Donovan and Halldorsson2021). ROMs were not completed by 60% of participants (n = 10) who attended the follow-up session, and it is possible that the LOCF approach for the primary analysis is a more conservative indication of the clinical effectiveness of OSI. Incomplete ROMs may represent the lack of incentive to complete ROMs when no new module material is subsequently accessed for the follow-up module on OSI (i.e. this module simply is the ROMs). However, otherwise ROMs are mandatory to complete during the treatment period meaning we had complete data, which is a clear strength given that the collection of complete ROMs data within NHS CAMHS has typically been poor, for example only 25% of cases had pre–post treatment measures in the Child Outcome Research Consortium (CORC) evaluation of routine practice in CAMHS (Wolpert et al., Reference Wolpert, Harris, Hodges, Fuggle, James, Wiener, McKenna, Law, York, Jones, Fonagy, Fleming and Munk2016b). Usage data were not collected by the OSI child game app and therefore the uptake and engagement of the game is unknown. Finally, this study was conducted during the COVID19 pandemic when the UK was in lockdown for several months, and this may have influenced the uptake of OSI as face-to-face treatments were not available throughout the entire study period.
In conclusion, this initial evaluation has provided preliminary evidence that OSI is an effective and acceptable online treatment for child anxiety for use within routine clinical practice. Further systematic evaluation of OSI through randomized controlled trials in routine clinical settings is now warranted.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S1352465822000157
Data availability statement
As this study reports on routinely collected NHS data we do not have permission to make individual data publicly available.
Acknowledgements
We thank the AnDY research clinic clinicians and participating families. We also thank Kaja Warwzak for her help preparing the manuscript.
Authorship contributions
Claire Hill and Cathy Creswell conceived the study. Claire Hill, Chloe Chessell and Ray Percy collated the data. Claire Hill had full access to the data in the study and takes responsibility for the integrity of the data and accuracy of the data analysis. All authors contributed to the writing or editing of the manuscript and share responsibility for the final version of the work submitted and published.
Claire Hill: Conceptualization (equal), Data curation (lead), Formal analysis (lead), Investigation (equal), Methodology (equal), Project administration (equal), Visualization (lead), Writing – original draft (lead), Writing – review & editing (lead); Chloe Chessell: Methodology (supporting), Project administration (supporting), Writing – original draft (supporting), Writing – review & editing (supporting); Ray Percy: Methodology (supporting), Project administration (supporting), Writing – original draft (supporting), Writing – review & editing (supporting); Cathy Creswell: Conceptualization (equal), Formal analysis (supporting), Funding acquisition (lead), Investigation (supporting), Methodology (equal), Project administration (supporting), Writing – original draft (supporting), Writing – review & editing (supporting).
Financial support
Cathy Creswell and Claire Hill were funded by NIHR Research Professorship awarded to Cathy Creswell (RP_2014-04-018). Cathy Creswell is supported by the Oxford and Thames Valley NIHR Applied Research Collaboration. NIHR had no role in the study design, collection, analysis or interpretation of the data, writing the manuscript, or the decision to submit the paper for publication. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.
Conflicts of interest
Cathy Creswell receives royalties for the book on which this online intervention is based; however, none of the authors make any financial gain from the online intervention presented in this paper.
Ethics statements
This paper presents data from a routine service evaluation and therefore did not require ethical approval. All authors abided by the Ethical Principles of Psychologists and Code of Conduct as set out by the British Association for Behavioural and Cognitive Psychotherapists (BABCP) and British Psychological Society (BPS).
Open access
Comments
No Comments have been published for this article.