There are concerns about the wide-ranging difficulties of children now being placed in care, both in the UK and in other countries Reference Warman and Roberts1,Reference Sinclair, Wilson and Gibbs2 and the very limited amount of evaluative intervention research. Reference Pallett, Scott, Blackeby, Yule and Weissman3 The National Institute for Health and Care Excellence (NICE) and the Social Care Institute for Excellence (SCIE) have highlighted this lack of evidence and the need for robust evaluation methodologies. 4 The British Government in 2002 introduced a national implementation of Multidimensional Treatment Foster Care (MTFC), a wrap-around multimodal foster care intervention for children with challenging behaviour, originally developed by the Oregon Social Learning Centre in the USA. Reference Chamberlain5 Previously, the primary focus of MTFC had been on adolescent criminality and evaluation on delinquency outcomes and a Cochrane review Reference Macdonald and Turner6 reported positively on its effectiveness in terms of offending and reduction in days in custody, but expressed concern about generalisability, since all the studies were based in the USA and had involved the programme developers. Given the lack of independent evaluation beyond the centre of origin and of any evaluation of such a programme in a UK social care context, the Care Placement Evaluation (CaPE) was commissioned to be conducted independently of the originators of the model and the UK implementation team (trial registration: ISRCTN 68038570). This paper reports the main outcomes of this evaluation in the context of English looked after care.
Method
Design
This was a pragmatic randomised controlled trial (RCT) embedded within an observational quasi-experimental case-control design to test two parallel groups: Multi-dimensional Treatment Foster Care for Adolescents (MTFC-A) and usual care. Reluctance from some local authorities, despite extensive preparation, to participate in an RCT was anticipated. The inclusion of the second option of an observational study was prespecified in the study design. It allowed these authorities still to take part to maximise representativeness; as well as providing a default design should the RCT prove impractical in any area. Adaptations to the standard RCT model were also made in light of the special circumstances of vulnerable children needing care placement, including the professionals’ need to ‘match’ child characteristics to placements, the need for informed consent of vulnerable young people who were often in crisis, and the need for social workers to be able to manage the placement process flexibly and sensitively. There was a two-stage consent, first into the study as a whole and then to specific individual placements, including MTFC-A. Random allocation was made into a ‘pool’ of cases from which appropriate individual placements could be made, optimising the timing of mutual availability of young people with foster families and allowing for flexibility on specific placement ‘matching’ within the MTFC-A protocol. Reference Dixon, Biehal, Green, Sinclair, Kay and Parry7 (The full trial protocol is available on www.bbmh.manchester.ac.uk/staff/projectprofiles/index.aspx?ID=87993&ResearchProjectId=1429.)
Participants
Twenty-three English local authorities participated in the CaPE evaluation study between June 2005 and December 2008. Eighteen of these had taken part in the national MTFC-A implementation in four waves between 2004 and 2007 (see online Table DS1 for details of participating authorities).
Inclusion criteria were young people aged 10-17 years (a) in a placement that was unstable, at risk of breakdown or not meeting their assessed needs, or at risk of custody or secure care and (b) showing complex or severe emotional difficulties and/or challenging behaviour. Exclusion criteria were severe intellectual difficulties (referred to as learning disabilities by UK health services, this was indexed by specialist school placement) or psychotic illness from medical records.
The experimental intervention
In MTFC-A, specialist foster parents receive training and ongoing support and supervision in the intensive social learning approach pioneered at the Oregon Social Learning Center. Attention is paid to the mental health of foster children through the provision of psychiatry and psychology input, including individual and family therapy, social skills training and support with education. The aim is for a short-term intensive placement, of around 9 months, followed by a short period of aftercare. Key elements Reference Chamberlain5 include: the provision of a consistent reinforcing environment in which young people are mentored and encouraged; a clear structure, with clearly specified boundaries to behaviour and specified consequences that can be delivered in a teaching-oriented manner; close supervision of young people’s activities and whereabouts at all times; diversion from associations with antisocial peers and help to develop positive social skills that will help young people form relationships with more positive peers. Behaviour is closely monitored and positive behaviours are reinforced in a concrete manner using a system of points and levels; moving during the course of the programme from early restrictions through a series of ‘levels,’ each of which brings increased privileges and enhanced incentives.
Specialist foster carers are paid a full-time salary, provided with continuously available intensive support, have daily telephone interviews with MTFC-A staff for support and to complete a Parent Daily Report (PDR), a checklist enabling the team to monitor intervention adherence, and identify problems, progress and carer stress. Foster carers have weekly face-to-face group meetings with the intervention team. Participating intervention teams received initial training from the UK national implementation group and the programme developers in the USA to prespecified levels of fidelity. Following this, ongoing fidelity to the model throughout the programme was monitored through weekly supervision telephone calls with the programme developers in the USA, including evaluation of individual PDR data. In each local team there were two additions to the US model: (a) an education worker; and (b) a part-time programme manager to liaise with the Social Services department.
Usual care
Usual care consisted of care placements routinely in use in local authorities at the time. These included existing (non-MTFC-A) family foster care, residential care, residential schools and other placements. Details of the use of these placements and of other mental health services were gathered at carer interview.
Measures
Primary outcome
The diverse impacts from the multifaceted MTFC-A intervention were evaluated with two standard summative measures of mental health and social and physical functioning: the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) Reference Gowers, Harrington, Whitton, Lelliott, Beevor and Wing8 and the Children’s Global Assessment Scale (CGAS). Reference Shaffer, Gould and Brasic9 A two-stage process for rating the 13 HoNOSCA domains was developed specifically for this study in order, systematically, to integrate the diverse range of complex data sources and to enable a masked overall rating for outcome estimation. A researcher used all available multi-informant sources relevant to each HoNOSCA domain, covering a period of 6 months prior to index placement at baseline and between 3 months before and 6 months after the end-point evaluation date. Sources included structured interviews with the young person and carers, the standard carer-rated Child Behaviour Checklist (CBCL) and self-rated Youth Self Report (YSR), Reference Achenbach and Edelbrock10 along with collated reports and records directly accessed from education, health and social services. This information was integrated, transcribed, fully anonymised and then located within each relevant HoNOSCA domain before being rated. A second researcher, masked to all other case data including the first rating, independently rated this anonymised information within each domain. Outcome CGAS scoring was derived from the same information. The CGAS score has been widely used within child mental health settings and epidemiology and intervention studies; scores below 60 distinguish clinical from non-clinical ‘cases’ and scores above 70 are considered to be in the normal range. The CGAS scores correlate with total problem scores from the CBCL, IQ, family dysfunction and suicide attempts, and predict service use. Reference Winters, Collett and Myers11 Within the study sample there was excellent interrater reliability for HoNOSCA and CGAS total scores at both baseline (intraclass coefficient (ICC) 0.75, 95% CI 0.68-0.80) and end-point (ICC = 0.81, 95% CI 0.75-0.85). The HoNOSCA domain scores showed ICCs varying from 0.53 to 0.89 at baseline and 0.51-0.89 at end-point, with only one domain at end-point showing an ICC<0.6.
Secondary outcomes
Education
Education outcomes were assessed using masked ratings on the two education-related HoNOSCA domains (scholastic/language skills and education attendance) using the procedure above.
Offending
Data on specific incidents of offending (reprimand, caution or charged with offence) during the previous 6 months were gathered from the social worker at baseline and from carer and social worker at end-point covering the previous 3 months.
Procedures
Baseline data were collected in relation to the 6-month period before children moved to their index placements in MTFC-A or usual care. End-point data were collected 12 months after the baseline date. Young people consenting to the RCT were randomly allocated to an offer of MTFC-A or to usual care. Remote randomisation by telephone was provided independently by the University of Manchester Biostatistics Group according to a predefined randomisation schema. On most occasions just two young people were available for a single placement, and so one was randomly selected for MTFC-A with the other allocated to usual care. On two occasions three young people were available for two places, and two were randomly selected for MTFC-A. The trial manager communicated allocation by telephone and email to the liaison person in the local authority team responsible for undertaking placements. Young people consenting into the observational study had placement decisions allocated in the way usual for the local authorities involved, usually through a regular ‘placement panel’ of senior social workers. Criteria for allocation were the balance of the young person’s needs in relation to placements available, including MTFC-A.
All primary outcome and educational secondary outcomes were coded masked to group allocation (see outcomes above). Within this method, pooling of data from triangulated reports and records as well as direct and telephone interviews served to minimise any effect from reporting bias. Presence of offending was identified directly from carer and social worker report.
Analysis
Sample size
A target sample size of 220 (130 for the RCT and 90 for the observational study) was estimated to yield an approximately 80% chance in the RCT of finding a significant difference where the ‘true difference’ is 0.5 of a standard deviation, and an approximately 95% chance of detecting the same difference in the case-control study. Statistical analyses of the primary outcome were carried out using Stata Release 11 for Windows. All other statistical analyses used the software package PASW 18. Frequency distributions were used to describe the sample and parametric and non-parametric tests to compare those eligible and not eligible for MTFC-A at allocation, along with those randomised to RCT and observational cohorts. Statistical analysis of the RCT cohort was by intention to treat (ITT), subject to the availability of outcome data. The benefit of MTFC-A as compared with usual care at end-point was estimated using an analysis of covariance adjusting for the baseline value of the outcome (either CGAS or HoNOSCA). In the observational cohort, the benefit of MTFC-A as compared with usual care at end-point was estimated using a propensity score method, details of which are given in the results section.
Missing data
Total scores for primary outcome HoNOSCA at baseline and end-point were computed using the pro-rating methodology in which values for missing scale items are imputed with the mean of other items prior to calculating a total score for the scale. This procedure was used provided at least half of the scale items were not missing for a participant. If more than half of the scale items were missing, the total score for that participant was set to missing.
Subgroup analysis
Since the characteristics of young people in care included in this study were broader than the predominantly antisocial samples in which the intervention was originally evaluated, we specified a subgroup analysis to investigate any differential effect of the intervention on antisocial characteristics in the sample. Ethical approval was obtained from University of York Research Ethics committee (ref PK/LG) and approval from the UK Association of Directors of Social Services (ref DW/NK).
Results
Participant flow and representativeness
The flow of participants through the trial is summarised in Fig. 1 (modified CONSORT). The young people included in the MTFC-A group of the CaPE study (RCT: n = 20, observational study: n = 92, total n = 112) represented 67% of the total 166 young people who had been placed in the national implementation during the time period of the study. Comparison of key baseline data with audit data from the National Implementation Team on all 193 children who had entered the programme by mid-2010 showed a similar profile - suggesting the study group are representative of the MTFC-A implementation as a whole (online Table DS2).
Implementation of the intervention
The MTFC-A protocol was delivered in 18 authorities. The mean duration of a MTFC-A placement was 263 days. At end-point, 50/112 (45%) participants allocated into MTFC-A were still in this placement; of those who had moved on, 23% (14/62) were in other foster placements, 8% (5/62) back with parents or relatives, 19% (12/62) in local children’s homes and 26% (16/62) in other residential placements or semi-independent living. These non-MTFC-A end-point placements were similar in their proportions to the usual care arm of the study.
Use of the various forms of usual care within the different cohorts of the study is detailed in online Table DS3; main use was equally spread between other forms of foster care, local authority residential care and private residential care homes and schools. Detailed data on characteristics of intervention targets and management strategies across usual care and MTFC-A placements are shown in online Table DS4.
Baseline data
Baseline group data are summarised in Table DS3. The RCT cohort and MTFC-A arm of the observational cohort show close similarity, whereas the usual care arm in the observational cohort differs from these in being older and having higher (less severe) CGAS scores. Post hoc multiple comparison tests showed differences between observational sample arms in age (P<0.0001), HoNOSCA score (P = 0.048) and CGAS (P<0.0001). The RCT sample were more likely to be in residential care at entry (P = 0.029) than the observational sample. In the observational sample, the MTFC-A group were less likely to be in residential care than the usual care group (P = 0.023).
In order to adjust for baseline imbalance in the observational cohort data, a propensity score procedure was undertaken, using methods described by Rubin: Reference Rubin12 (a) a propensity score was derived using logistic regression models fitted to the binary variable ‘Receipt of MTFC-A’ and baseline covariates of gender, age, prior placement, CGAS and HoNOSCA score; (b) predicted probability of receiving MTFC-A was calculated for each participant in the cohort, following standard practice all variables were used when determining the normalised propensity score and probability of receipt of MTFC-A for each participant regardless of statistical significance as non-significant variables may improve the prediction; (c) participants with probabilities of receiving MTFC-A above 95% or below 5% were trimmed from the data-set; (d) the propensity score models in (a) were refitted on the trimmed data-set and probability of receiving MTFC-A calculated; (e) inverse probability weights were calculated for each participant by taking the inverse of the predicted probability of their assigned treatment; (f) weighted analyses corresponding to those for the randomised cohort were carried out applying the inverse probability weights to the data. Participants with a high probability of receiving their assigned treatment are down-weighted in the analysis and participants with a low probability up-weighted so improving the balance between samples. When this was done and participants with probabilities of receiving MTFC-A above 0.95 or below 0.05 were dropped from the data-set, a ‘trimmed’ data-set containing 25 fewer cases (n = 153) was generated. In the trimmed sample, there was still evidence that baseline age (P<0.0001), CGAS score (P = 0.007) and to a lesser extent placement in residential care prior to the study (P = 0.081) were imbalanced between the groups but, nevertheless, balance had been improved. The trimmed observational data-set was used in subsequent analysis. Data from the randomised cohort were not included in the propensity analysis as they were balanced by randomisation.
Primary outcome
Table 1 shows that participants in all arms of the study show an improvement in functioning, on average, over time during the study. Table 2 summarises the key differences between the groups at end-point for the randomised study and the observational comparison. In the randomised cohort, young people in the MTFC-A group had a marginally better outcome for CGAS at end-point than the usual care group (adjusted mean difference 1.3, 95% CI –7.1 to 9.7), but this was not statistically significant (P = 0.75). A similar effect was observed in the observational comparison (adjusted mean difference 0.95, 95% CI –2.38 to 4.29, P = 0.57). In summary, these data suggest no evidence that MTFC-A gives significant overall benefit compared with usual care; either in the randomised or observational cohort. The study sample size had been determined to detect a standardised effect size of 0.5, which corresponds to approximately five units on the CGAS scale and three units on the HoNOSCA total score.
Study flow.
CaPE, Care Placement Evaluation; RCT, randomised controlled trial; MTFC-A, Multidimensional Treatment Foster Care for Adolescents.
Secondary outcomes
For education attendance and school exclusion, those who received MTFC-A did not do any better than those in usual care placements (Table 3). When analysed as ordered quintiles, the estimates of odds ratios from proportional odds ordinal logistic regression were for attendance, 2.5 (95% CI 0.48-13.13, P= 0.28) in the RCT and 0.57 (95% CI 0.28-1.15, P = 0.118) in the observational sample; and for scholastic skill, 0.6 (95% CI 0.15-2.4, P = 0.48) in the RCT and 0.85 (95% CI 0.39-1.9; P = 0.7) for the observational sample.
No differential effect of intervention on offending behaviours was found between MTFC-A and usual care (Table 4). Odds ratio for offending behaviours adjusted for age, gender, baseline offending and antisocial behaviour with inverse probability weighting by propensity score was 1.24 (95% CI 0.22-7.38, P = 0.80) for the RCT and 1.07 (95% CI 0.43-2.64, P = 0.89) for the observational sample.
Moderator analyses
Subgroup analysis investigated whether there was a relationship between treatment effect and initial level of antisocial behaviour for the primary outcomes (Table 5). This suggested a differential effect of the intervention on antisocial young people (predefined as high scorers (score 3, 4) on the antisocial scale 1 of HoNOSCA). In the absence of an overall treatment effect in the analysis, this suggests that the intervention may be beneficial for some young people (showing high levels of antisocial behaviour at baseline) but less beneficial than usual care for others (those showing low levels (scores 0, 1, 2, on HoNOSCA scale 1) at baseline).
No adverse events were reported.
Discussion
This study applied a RCT within a rigorous mixed-methods design in order to evaluate a complex intervention in social care in England. The study overall achieved its target recruitment (219 compared with planned 220), and since the original intention to recruit the majority of participants within the RCT could not be accomplished because of the smaller than planned number of local authorities opting into the RCT design and small referral flows within these authorities, the mixed-method approach proved its pragmatic value. Results from analyses of both the RCT and observational cohorts were convergent. Both MTFC-A and usual care groups tended towards improved functioning over time, but there was no evidence that use of MTFC-A resulted in better overall outcomes than usual care on the primary outcome of adaptive functioning or on secondary education or offending outcomes. In the absence of matched controls for young people not in care at all, it is not possible to say whether the overall improvement in both cohorts is the result of statistical or clinical regression to the mean, or a therapeutic effect in both arms; but as an equivalence trial, MTFC-A does not show overall advantage over current care.
Scores on the Child Global Assessment Scale (CGAS) and the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) at baseline and end-point in the randomised controlled trial (RCT) and trimmed observational samplesFootnote a
| Usual care group | MTFC-A group | |||
|---|---|---|---|---|
| Mean (s.d.) | n | Mean (s.d.) | n | |
| CGASFootnote b | ||||
| RCT sample Baseline | 48.31 (9.05) | 13 | 47.65 (9.39) | 20 |
| End-point | 55.25 (12.56) | 12 | 56.00 (10.06) | 17 |
| Observational sample (trimmed) | ||||
| Baseline | 49.69 (9.24) | 74 | 46.29 (6.83) | 80 |
| End-point | 53.78 (10.82) | 69 | 53.54 (9.70) | 80 |
| HoNOSCAFootnote c | ||||
| RCT sample Baseline | 18.47 (4.45) | 13 | 18.89 (5.44) | 20 |
| End-point | 14.93 (7.99) | 12 | 14.04 (5.57) | 17 |
| Observational sample (trimmed) Baseline | 18.20 (6.50) | 74 | 19.49 (5.45) | 80 |
| End-point | 16.88 (6.74) | 68 | 16.98 (6.46) | 80 |
MTFC-A, Multidimensional Treatment Foster Care for Adolescents.
a. Missing data (n = 4) for end-point primary outcome in both usual care and MTFC-A arms.
b. Higher scores represent less impairment.
c. Lower scores represent less impairment.
Summary differences in outcome between Multidimensional Treatment Foster Care for Adolescents (MTFC-A) and usual care groups for both randomised controlled trial (RCT) and observational samples (linear regression estimates)
| Adjusted mean differenceFootnote a (95% CI) | P | n | |
|---|---|---|---|
| Child Global Assessment Scale | |||
| RCT | 1.30 (–7.14 to 9.74) | 0.75 | 29 |
| Observational (trimmed) | 0.95Footnote b (–2.38 to 4.29) | 0.57 | 149 |
| Nation Outcome Scales for Children and Adolescents | |||
| RCT | –1.04 (–6.21 to 4.13) | 0.68 | 29 |
| Observational (trimmed) | –1.09Footnote b (–3.64 to 1.46) | 0.40 | 148 |
a. Effect of MTFC-A compared with usual care adjusted for baseline score.
b. Weighted estimate.
Secondary educational outcomes at baseline and end-point: randomised controlled trial (RCT) and trimmed observational samples
| RCT cohort | Observational cohort | |||||||
|---|---|---|---|---|---|---|---|---|
| HoNOSCA | Usual care, n (%) | MTFC-A, n (%) | OR (95% CI) s.e. | P | Usual care, n (%) | MTFC-A, n (%) | OR (95% CI) s.e. | P |
| Scholastic/language | ||||||||
| Baseline score | 13 | 20 | 78 | 78 | ||||
| 0 | 2 (15) | 1 (5) | 0.6 (0.15-2.4) 0.43 | 0.479 | 25 (32) | 18 (23) | 0.85 (0.39-1.9) 0.34 | 0.709 |
| 1 | 4 (31) | 7 (35) | 24 (31) | 19 (24) | ||||
| 2 | 4 (31) | 2 (10) | 20 (26) | 20 (26) | ||||
| 3 | 3 (23) | 10 (50) | 8 (10) | 17 (22) | ||||
| 4 | 0 (0) | 0 (0) | 1 (1) | 4 (5) | ||||
| End-point score | 12 | 16 | 76 | 84 | ||||
| 0 | 4 (33) | 6 (37.5) | 22 (29) | 16 (19) | ||||
| 1 | 3 (25) | 5 (31) | 23 (30) | 27 (32) | ||||
| 2 | 1 (8) | 3 (19) | 22 (29) | 26 (31) | ||||
| 3 | 3 (25) | 1 (6) | 9 (12) | 12 (14) | ||||
| 4 | 1 (8) | 1 (6) | 0 (0) | 2 (2) | ||||
| School attendance | ||||||||
| Baseline score | 12 | 20 | 85 | 82 | ||||
| 0 | 2 (17) | 8 (40) | 2.5 (0.48-13.1) 2.1 | 0.276 | 33 (39) | 43 (52) | 0.57 (0.28-1.15) 0.20 | 0.118 |
| 1 | 1 (8) | 2 (10) | 5 (6) | 1 (1) | ||||
| 2 | 2 (17) | 1 (5) | 10 (12) | 8 (10) | ||||
| 3 | 1 (8) | 2 (10) | 4 (5) | 2 (2) | ||||
| 4 | 6 (50) | 7 (35) | 33 (39) | 28 (34) | ||||
| End-point score | 10 | 17 | 76 | 85 | ||||
| 0 | 6 (60) | 6 (35) | 34 (45) | 40 (47) | ||||
| 1 | 0 (0) | 2 (12) | 7 (9) | 5 (6) | ||||
| 2 | 1 (10) | 3 (18) | 10 (13) | 9 (10) | ||||
| 3 | 1 (10) | 2 (12) | 8 (11) | 4 (5) | ||||
| 4 | 2 (20) | 4 (24) | 16 (21) | 31 (36) | ||||
HoNOSCA, Health of the Nation Outcome Scales for Children and Adolescents; MTFC-A, Multidimensional Treatment Foster Care for Adolescents.
Offending behaviours at end-point in relation to baseline
| Usual care group, n (%) | MTFC-A group, n (%) | |||||
|---|---|---|---|---|---|---|
| No | Yes | No | Yes | OR (95% CI) | P | |
| Randomised controlled trial cohort | ||||||
| Baseline offending | 9 | 4 | 14 | 6 | ||
| Offending at follow-up | 3 (33) | 1 (25) | 4 (29) | 3 (50) | 1.24 (0.22-7.38) | 0.80Footnote a |
| Observational cohort | ||||||
| Baseline offending | 51 | 34 | 63 | 29 | ||
| Offending at follow-up | 4 (8) | 14 (41) | 14 (22) | 8 (28) | 1.07Footnote b (0.43-2.64) | 0.89Footnote c |
MTFC-A, Multidimensional Treatment Foster Care for Adolescents.
a. P-value using Fisher’s exact test.
b. Odds ratio adjusted for baseline offending age, gender, baseline offending and antisocial behaviour with inverse probability weighting by propensity score.
c. P-value using Wald test with robust standard errors.
Interaction between treatment effect and baseline scores in antisocial behaviourFootnote a
| Adjusted mean differenceFootnote b (95% CI) | P | n | |
|---|---|---|---|
| CGAS | |||
| RCT | 10.05 (–8.44 to 28.54) | 0.273 | 29 |
| Observational | 9.64 (2.95 to 16.33) | 0.005 | 149 |
| HoNOSCA | |||
| RCT | –5.61 (–17.23 to 6.00) | 0.329 | 29 |
| Observational | –5.01 (–10.32 to 0.29) | 0.064 | 148 |
CGAS, Child Global Assessment Scale; RCT, randomised controlled trial; HoNOSCA, Health of the Nation Outcome Scales for Children and Adolescents.
a. Predefined as high scorers (3, 4) on the antisocial scale 1 of HoNOSCA.
b. Effect of Multidimensional Treatment Foster Care for Adolescents compared with usual care adjusted for baseline score.
This result of the study is different to previous reports of effectiveness of the MTFC-A model. However, the context of intervention in the UK differs significantly from the originating studies from the USA, since these were focused on convicted delinquent youth where the alternative was incarceration. Reference Macdonald and Turner6 This UK study is focused on looked after adolescents in public care, a high proportion of whom showed some antisocial behaviour, but only about a quarter had recent offences at baseline. Our usual care comparison included the range of care provision in the UK, and the fact that the control condition in the US studies approximated, by contrast, to juvenile custody (which research suggests worsens youth functioning Reference Dmitrieva, Monahan, Cauffman and Steinberg13 ) may therefore go some way to explaining the different results in this UK study. The usual care placements in CaPE vary in their explicit focus on ‘treatment’. Reports of the actual management strategies used in placements suggest some similarity in practice, for instance between MTFC-A and residential care, in terms of a behavioural management focus in promoting social skills and peer relationships (Table DS4). Foster care showed less intervention focus of this kind, typically being seen as ‘care’ rather than ‘treatment’. This points to the theoretical relevance of interventions like MTFC-A as a therapeutically oriented adjunct to conventional foster care in complex situations.
A high proportion of the index usual care placements in both arms were residential, compared with rates of only 9% of looked after children in England generally; these high rates reflect their age and severe disturbance. The potential adverse effects of residential care are often emphasised, Reference Berridge, Biehal and Henry14 but residential treatment in the different context of child and adolescent mental health can be effective Reference Green, Jacobs, Beecham, Dunn, Kroll and Tobias15,Reference Green, Kroll, Imre, Frances, Begum and Gannon16 and this is relevant given the severity and complexity of the mental health and social problems tackled in this MTFC-A programme, reflecting the inclusion criteria (placements at risk of breakdown and young people with psychopathology), and shown in their baseline characteristics (Table DS3) and comparison with general national data on looked after children 17 (Table DS2). The study cohort were more likely, compared with this national data, to be in (non-secure) residential care or secure accommodation, more likely to have clinically significant emotional and behavioural difficulties, to have recent convictions, problems of substance misuse and to have been assessed as having special educational needs. Furthermore, baseline severity on CGAS (ranging from 46.29 to 49.69, standard deviations from 6.83 to 9.39 in different arms) compares with a mean admission CGAS of 44.0 (s.e. = 1.1) in the largest study to date of residential child mental health (n = 145). Reference Green, Jacobs, Beecham, Dunn, Kroll and Tobias15 In other words the participants included in this project are not dissimilar in severity of the most severe mental health problems treated in children. At 1-year end-point, CGAS across arms in this study ranged from 53.54 to 56 (standard deviations from 9.7 to 12.5); compared with a mean in the psychiatric unit cohort of 58.3 (s.e. = 1.5). Given the level of these difficulties, then, it is rational that an intensive therapeutic approach is needed.
Secondary analysis (Table 5) showed an interaction in both RCT and observational cohorts between baseline antisocial scores and treatment for CGAS scores: MTFC-A showed improved results over usual care for a group of children who were highly antisocial at baseline, but results less good than usual care for those who were not highly antisocial. The measure used here to index ‘antisocial’ (HoNSCA scale 1) has been shown to have good external validity with independent diagnosis of conduct disorder and externalising symptoms. Reference Brann, Coleman and Luk18,Reference Hanssen-Bauer, Langsrud, Kvernmo and Heyerdahl19 Since this effect was only found in the MTFC-A and not usual care groups, a simple regression to the mean seems unlikely. A chance (type 1 error) finding in secondary analysis is possible, or the finding could point to a differential effect of MTFC-A on the subgroup of young people showing predominant antisocial behaviour at baseline. In favour of this explanation could be the predominantly social learning framework behind MTFC-A, something particularly adapted for and shown to be successful with antisocial behaviour in other studies. By contrast, this style of intervention may be less appropriate for the group of young people in care with primarily attachment, interpersonal or emotional needs. Further analysis is needed to explore whether this explanation may apply or whether the finding reflects no more than chance.
Strengths and limitations
The strengths of the study lie in the independence of evaluation from implementation, the sample size, very careful attention to triangulating and masked rating of the primary outcome data in a complex assessment context, and the complementary use of mixed methods with a low rate of attrition to end-point. The study recruited from a geographically and demographically representative sample of local authorities across England. There are limitations to analysis in each cohort (sample size in the RCT and baseline imbalance in the observational cohort). The imbalance in the observational cohort was mitigated using propensity score analysis, and the convergence of findings from the two methods used and the confidence intervals of outcome estimations, gives some confidence to inferences from the results.
The MTFC-A is a relatively brief intervention model with the aim of improving the young person’s behaviour and functioning so that future fostering breakdown would be less likely. This study tested change at the end of the planned short-term intervention period but was not able to test any effect on future fostering breakdown. There may be theoretical limitations in using a brief intervention of this kind, however skilfully implemented, because of its disruptive effect in the context of a long-term need for care. The lack of overall measured short-term effect in this study could reinforce such concern, but longer-term evaluation of rates of placement breakdown subsequent to MTFC would be necessary to fully judge the strategy.
The design of the study reflects the desire in the government of the time, and NICE guidance, 4 to introduce experimental designs into practice evaluation - an important strategic necessity for further development of the evidence-base in the field. Randomised controlled trial designs have undergone successive modifications to new evaluation contexts Reference Green20,Reference Everitt, Wessely, Everitt and Wessely21 and great attention at the design stage of this study was paid to appropriate and feasible adaptations of the RCT method to an evaluation in social care: extensive prior consultation with social service managers and teams discussed the practicality and ethics of applying RCT methods, and the design was refined in the light of feedback. Reference Dixon, Biehal, Green, Sinclair, Kay and Parry7 The final design had the support of the trial steering committee, social service directors and a government policy advisor. Thus this study has shown the feasibility in principle of applying RCT methodology to the complex environment of social care; but with only 6 out of 18 local authorities finally agreeing to participate in the RCT, further sustained work is clearly needed to establish this kind of evaluation in a widespread way. This experience reflects other successive government initiatives to integrate systematic evaluation into large complex implementation projects. For instance, it has been argued that evaluation of the National Sure Start programme from 1999 was handicapped by the initial decision not to include an RCT evaluation, 22 whereas the recent national targeted mental health for schools (TaMHS) project successfully included both an observational and an RCT evaluation into the funding contract and achieved approximately two-thirds acceptance rate from schools into a wait list RCT. 23 Such projects have shown that systematic evaluation of complex implementations with random allocation trials can be designed to be acceptable to users and successfully provide valuable outcome data to guide future practice. However, previous experience, including our own, Reference Dixon, Biehal, Green, Sinclair, Kay and Parry7 suggests that careful attention needs to be paid to design adaptations and that it is essential to include the evaluation design within the funding contract with providers at the outset.
Implications
This trial shows both the feasibility, as well as the challenges, of implementing a robust randomised design into social care evaluation. The results from both the RCT and observational cohorts are convergent in suggesting no significant overall added value of the MTFC-A model compared with usual care on global functioning, educational placement or offending behaviours in this complex at-risk population. However, there is evidence of possible treatment effect heterogeneity in relation to baseline antisocial behaviour, and this justifies further investigation and analysis, particularly since previous studies of the MTFC-A programme have largely focused on antisocial and offending populations and that the rationale of the intervention is specifically focused on such behavioural problems. Further qualitative study is also indicated to analyse similarities and differences from current care practice, and to suggest possible intervention modifications for the future. The results represent the first substantive fully independent evaluation of MTFC-A in a European care population and suggest that the positive results from largely offender populations in trials of MTFC-A in the USA are not replicated in this context.
Funding
The project was funded by a grant from the UK Department for Children, Schools and Families to the Institute of Psychiatry (reference: PACLBMC). It was sponsored by the University of Manchester.
Acknowledgements
The authors are very grateful to all the young people, their families, social workers and carers who took part in the project. Helen Jones and Dr Caroline Thomas at the government Department for Children, Schools and Families gave generous and sustained assistance to the project throughout its work. We are grateful for the hard work and constant support of the trial steering committee (Alan Rushton and Sharon Witherspoon (chairs), Helen Jones, Steve Miley, Rosemary Roberts, Stephen Scott, Gillian Schofield, Julie Selwyn, Caroline Thomas).
eLetters
No eLetters have been published for this article.