What is the research question?

Using a sophisticated search strategy, the authors identified 1200 citations from which they winnowed five methodologically adequate studies that compared the ‘CMHT approach’ with ‘standard care’ (Reference Tyrer, Coid and SimmondsTyrer et al, 2000). This paucity of evidence is highly disappointing for professionals and policymakers alike, given the centrality of the CMHT to contemporary mental health care (Reference TansellaTansella, 1998).

The casual reader would be forgiven for assuming that Simmonds et al are reporting studies of an active agent (analogous to a pharmaceutical compound) versus placebo, the active agent in this case being ‘the’ CMHT. This is far from the case. Indeed, only recently have attempts been made to provide a typology of CMHTs (Reference Thornicroft, Becker and HollowayThornicroft et al, 1999). Not surprising, therefore, is the massive heterogeneity of the five chosen studies of ‘CMHT management’, which long predate such attempts at systematising. Two of the studies date back to 1970s Canada (Reference Fenton, Tessier and StreuningFenton et al, 1979) and Australia (Reference Hoult, Reynolds and Charbonneau-PowisHoult et al, 1981), both of them comparing an innovative community team with hospital-based care for people presenting for admission. As such, they figure in the Cochrane Collaboration review of ‘Crisis intervention for those with severe mental illnesses’ (Reference Joy, Adams and RiceJoy et al, 1998). These two early studies focused on admission diversion and address an important but distinct research question about the utility of dedicated community teams in avoiding in-patient admission and saving bed days. They contribute heavily to Simmonds et al's finding of a significant overall saving of bed days with the CMHT condition.

The other three studies are of London in the 1990s. One of the studies (Reference Merson, Tyrer and OnyettMerson et al, 1992) compared over the short term the value of a dedicated community team against traditional hospital-based care for patients presenting in psychiatric crisis to the accident and emergency department of an inner-London teaching hospital. The second study (Reference Burns, Beadsmore and BhatBurns et al, 1993) reports an intriguing comparison of two models of response to routine and emergency referrals to catchment area services, within which the active condition was a home-based multi-disciplinary assessment. A generally clinic-based uni-disciplinary assessment provided the comparator. The third study (Reference Tyrer, Evans and GandhiTyrer et al, 1998) acquired cases at the time of discharge from acute in-patient units and compared standard care with CMHTs. Each is addressing a different research question with a different service model applied to patients acquired at different points in their illness career. Crucially, the control condition also has varied significantly between the studies.

Only two of the five studies included in the meta-analysis (Reference Burns, Beadsmore and BhatBurns et al, 1993; Reference Tyrer, Evans and GandhiTyrer et al, 1998) approximate to the investigation of the functioning of a comprehensive CMHT with responsibility for the total service to a defined catchment area. The research question as posed by Tyrer et al (Reference Tyrer, Coid and Simmonds2000) is, in any case, of rather little relevance to contemporary mental health services in the UK. Here the CMHT, mandated to provide follow-up to all vulnerable patients under the Care Programme Approach (Reference KingdonKingdon, 1994), is the standard. The key policy-relevant issue is the value of adding to this now standard pattern assertive outreach, crisis and early psychosis services (Department of Health, 1999).

What studies to include and exclude?

A key issue in any meta-analysis is how to select relevant research studies. This introduces surprising subjectivity into the process, which cannot be overcome by subsequent statistical rigour. In their search strategy Simmonds et al report that they excluded studies of assertive community treatment and case management, which have been reported on in separate Cochrane Collaboration reviews (Marshall, 1998, 1999). However, they included the Hoult et al (Reference Hoult, Reynolds and Charbonneau-Powis1981) study, which not only forms an important element in the admission-diversion literature (Reference Joy, Adams and RiceJoy et al, 1998) but is generally seen as belonging to the literature on assertive community treatment (Reference Mueser, Bond and DrakeMueser et al, 1998).

Some studies that provide evidence on the efficacy of the CMHT approach must be excluded from a meta-analysis for methodological reasons because they are not RCTs. When they existed in the UK, psychiatric case registers could be used to track changes in service utilisation over time within an epidemiological framework. Tyrer et al (Reference Tyrer, Turner and Johnson1989) presented data on admissions in Nottingham over a 10-year period, finding a marked decrease that coincided with the introduction of sectorised CMHTs to the city. No such decrease occurred in England as a whole. Gater et al (Reference Gater, Goldberg and Jackson1997) used cluster randomisation, where the unit of analysis was not the individual patient but a general practice. They found that a primary-care-based CMHT provided better quality of care than a traditional hospital-based service. Both studies provide evidence that favours models of CMHT against historical ‘standard’ care. In contrast, the PRiSM study compared two models of community mental health care serving similar catchment areas by following up all known patients with a psychosis from the two areas (Reference Thornicroft, Wykes and HollowayThornicroft et al, 1998). It found little or no evidence in favour of a service model that included two well-resourced community teams (focused on crisis intervention and long-term care) compared with a less well-resourced generic CMHT.

Other studies are excluded from the meta-analysis, despite being RCTs. An example is the seminal UK700 study (Reference Burns, Creed and FahyBurns et al, 1999), which assessed the additional value of low case-load in services that already espoused the sorts of best practice inherent in the CMHT approach. Process measurement showed increased patient contact in the intensive case management condition (Reference Burns, Fiander and KentBurns et al, 2000) without resultant improved clinical or social outcomes (Reference Burns, Creed and FahyBurns et al, 1999). In routine clinical practice within well-organised CMHTs, small, at least below a limit, is not necessarily beautiful for case-load size (Reference TyrerTyrer, 2000).

What does this meta-analysis tell us?

This heterogeneous set of well-conducted studies of community care provides some consistent findings, notably that these arrangements result in fewer cases lost to follow-up. This is fully consistent with the case management literature (Marshall et al, Reference Marshall, Gray and Lockwood1998, Reference Marshall and Lockwood1999; Reference Mueser, Bond and DrakeMueser et al, 1998) and now would be seen to be something of a truism. If you are keen to follow people up, you tend to succeed. The paper does not, however, inform us in any meaningful way about the appropriate staffing, skill-mix, training, policies, procedures, practices and management of the CMHT.

Metaphorically Simmonds et al have carried out a careful meta-analysis of fruit growing. This has compared four baskets of two old varieties of apple (Reference Fenton, Tessier and StreuningFenton et al, 1979; Reference Hoult, Reynolds and Charbonneau-PowisHoult et al, 1981), two baskets of grapes (Reference Merson, Tyrer and OnyettMerson et al, 1992) and four baskets of oranges (Reference Burns, Beadsmore and BhatBurns et al, 1993; Reference Tyrer, Evans and GandhiTyrer et al, 1998) blended into two fruit salads. They have found that a fruit salad prepared from fruit grown by enthusiastic, well-capitalised and careful horticulturalists tastes rather better than one prepared from fruit grown in a variety of comparative conditions that include poor soil, aphids and dreadful weather. Possibly these results provide useful information about fruit-growing in general that could not be found by studying the culture of the individual fruit in more detail. More plausibly, important lessons about fruit-growing have been lost in the mixture.

Simmonds et al provide little information of direct relevance to the practitioner, service manager or purchaser tasked with improving community mental health care. Indeed, the apparatus of the meta-analysis potentially obscures the really important findings that are not amenable to the RCT methodology. These relate to such issues as the acceptability of interventions, applicability in non-experimental situations, generalisability to long-term routine clinical practice, long-term sustainability, the impact of service changes at an epidemiological level and the causes of variance between treatment sites. A good example of variance occurs in the details of the Tyrer et al (Reference Tyrer, Evans and Gandhi1998) study, which found significant, if modest, cost advantages of the CMHT approach over standard care in the follow-up of discharged in-patients. These advantages were dwarfed by differences between the two experimental sites, with the CMHT having little or no impact in the site lacking adequate in-patient resources, where costs were very much higher owing to inefficient overspill of patients into out-of-area and private-sector beds.

Conclusions

Applying the apparatus of a meta-analysis to such an ill-defined concept as the CMHT is perhaps premature. Arguably it is unnecessary because the generic comprehensive CMHT is set to become extinct in the UK in favour of the North Birmingham model of a plethora of specialist teams (Reference PeckPeck, 1999). The monopoly buyer of health services in the UK, the Department of Health, has chosen more exotic fruit than Simmonds et al included in their meta-analysis. There are cogent reasons for concern over the apparent abandonment of the generic CMHT (see Reference Burns, Thornicroft and SzmuklerBurns, 2001, for a thoughtful discussion of the issues). Sadly, Simmonds et al contribute little to this important policy debate.

There is an enormous research agenda for mental health services, an agenda that can be pursued only in part by means of the RCT, which is suitable for addressing only a subset of the questions for which answers are urgently required (Reference Richardson, Baker and BurnsRichardson et al, 2000). The next generation of RCTs in mental health services research should be more focused about the questions being asked and be supplemented by a range of other empirical studies using a whole variety of methodologies that are relevant to the complex questions that service commissioners and providers face.