Mental health provision in most of the industrialised world seems to be in a state of continual revolution. In the UK the National Health Service (NHS) Plan (Department of Health, 2000) has mandated a fundamental reorganisation of care, with the establishment of 260 assertive outreach teams, 360 crisis resolution/home treatment teams, and 50 early intervention services (each comprising two to four teams) for first-episode psychosis. This enormous upheaval is an ‘evidence-based policy’ drawing predominantly on mental health services research. How secure are its foundations?
The past 20 years have witnessed an explosion in the volume of health services research and in community mental health services research in particular (Reference Mueser, Bond and DrakeMueser et al, 1998; Reference Catty, Burns and KnappCatty et al, 2002). Most of this research has looked at ‘innovations’ – often when a new service is compared against the prevailing standard care. The impression is of a relentless march of progress; ‘outdated’ and ‘unscientific’ services are replaced by more effective successors, which persist until something is proved to be even better.
This is not the entire picture. For decades now we have been trying to develop alternatives to in-patient care. Despite this, the acute in-patient ward has survived, essentially unchanged, and many of these alternative services have disappeared. A Health Technology Assessment review of home-based care (Reference Burns, Knapp and CattyBurns et al, 2001) found (by following up the authors of the included studies by questionnaire) that virtually all of the experimental services had changed their practice since the studies were conducted, often departing significantly from the model described. Over half had ceased to operate at all, and – even more strikingly – 12% had ceased to exist even before the index study was published. This was the case even where significant advantages had been demonstrated for the experimental service.
With no formal evidence to support it,the acute in-patient ward has survived all these attempts to replace it. Its survival cannot be attributed simply to the level of investment in it. These wards have survived despite the wholesale closure of the large mental hospitals which used to house them. New wards have been built with radical improvements in the bricks and mortar, but with little change in clinical practice. In the UK the generic sector community mental health team also seems to have survived repeated attempts to replace it. In the USA and in most European countries, office-based practice stubbornly endures.
PROBLEMS WITH THE CURRENT RESEARCH BASE
Researchers often bemoan the failure of their findings to translate into practice and imply murky political forces. However, there are problems with our current approach to researching services. The National Institute for Clinical Excellence has voiced reservations about the generalisability of current studies, encouraging more effectiveness studies in mental health where services are assessed in ‘real world’ conditions rather than in highly atypical research settings (Reference Thornicroft, Wykes and HollowayThornicroft et al, 1998). The effect size in effectiveness studies is almost invariably smaller, for predictable reasons (Reference CoidCoid, 1994).
Evaluations of innovation
An exclusive focus on evaluating newly initiated services gives a one-sided picture. Most service developments inevitably require the demise of other services and yet we rarely attempt to measure the effects of this. A notable exception is a New Hampshire study evaluating the impact of closing day hospitals to liberate staff for a new vocational rehabilitation service (Reference Drake, Becker and BiesanzDrake et al, 1996). A similar carefully conducted case–control study of a day hospital closure in the UK (Reference Goddard, Burns and CattyGoddard et al, 2004) was rejected without methodological criticism by three journals – simply deemed by the editors to be insufficiently interesting. Finding out whether patients are significantly disadvantaged by the loss of a closed service or whether their needs were equally met by other parts of the system (and at what cost) should be of importance.
The case of day hospitals
Psychiatric day hospitals provide a particularly intriguing case study. Acute psychiatric day hospitals have been proposed as an evidence-based approach that has failed to translate into routine practice (Reference MarshallMarshall, 2003; Reference Briscoe, McCabe and PriebeBriscoe et al, 2004). There is some evidence that acute day hospitals can work as an alternative to in-patient care for a substantial proportion of patients, yet it is very difficult to maintain this focus on acute care over time.
Creed's group in Manchester demonstrated the ability to manage successfully a significant proportion of patients with acute illness without any in-patient care and to reduce duration of care for others (Reference Creed, Black and AnthonyCreed et al, 1990). However, a comparative study of the same approach using a second day hospital less than 30 km distant found much less success, particularly in managing the patients with more severe illness (Reference Creed, Black and AnthonyCreed et al, 1991). The authors concluded that the findings of their study needed to be interpreted in the light of local service issues, such as staffing and investment. This highlights the problems with studies where the service is essentially the subject of research but the unit of outcome analysed is the individual patient. It is not possible with any confidence to conclude which differences in local service issues contributed to the differences found. A randomised controlled trial with several day hospitals themselves as the unit of analysis would address this problem, but would be prohibitively costly and impractical.
An alternative approach would be to focus research specifically on the aspects of services that make them sustainable. These might not necessarily be the same aspects that make a service effective. For instance, clinicians from many day hospitals in Europe emphasise the importance of a fairly rigid programme, with clear rules about who can be admitted and what parts of the programme they must attend. Such factors may have little to do with effectiveness but are supposed to affect the sustainability of a given model in the real world and, thus, the longevity of the service. Yet these assumptions are based on personal experience and anecdotal evidence. No systematic study has ever investigated the factors that, for at least 25 years, have made acute day hospitals survive, change or disappear.
Researchers might be more useful partners in the dialogue with policy-makers if they provided evidence that addressed not only which services are effective, but also what components of the service model and the context influence sustainability over time.
RESEARCH IMPLICATIONS
Careful attention to – and structured research into – the survival of services and their changes over time could be one way of obtaining a better understanding of the determinants of clinical effectiveness and of the reasons why some innovations become and remain established and others do not. Such an approach might help distinguish the core features of durable services, in the same way that the Health Technology Assessment review mentioned above (Reference Burns, Knapp and CattyBurns et al, 2001) tested for common components of successful services.
Methodological quality in community psychiatry studies has improved enormously since Coid's criticisms, but there is still some way to go (Reference Priebe and SladePriebe & Slade, 2002). Studies have become larger and are often multicentred, with more attention to selection of outcome measures and statistical rigour. Despite this, naturalistic long-term service observations and studies deserve renewed consideration. The current raft of new community services in the UK should provide both enough features in common and enough differences to identify by careful study those that are associated with sustainability. Such studies would need to assess service components and context which, without a widely accepted taxonomy of services, would have to be comprehensive and detailed. To gain credibility and to develop research skills and capacity, more rigour must be brought to this exercise than is currently the case with many service-level audits. In particular, it requires the formulating of hypotheses to be tested prospectively.
Such an approach to naturalistic long-term studies will be challenging. As relevant factors for the sustainability and survival of services may be both internal to the service and contextual (Reference Burns and PriebeBurns & Priebe, 1996), studies would need sociological research expertise. They require at least the same careful attention to detail as experimental studies to warrant a place in respectable scientific health service research. Without such a place, findings would have no influence and researchers of sufficient calibre would not be attracted to the field, with the result that, despite a patina of evidence-base, service development and practice would continue to be largely driven by anecdote, individual prejudices and political expedience.
DECLARATION OF INTEREST
None.
eLetters
No eLetters have been published for this article.