An understanding of the limits of completeness, reliability and validity of data collected in routine practice

Trained assessors who use validated diagnostic interview schedules have acceptable levels of interrater reliability for determining psychiatric diagnoses. The assignment of diagnosis in routine clinical settings is far from this ideal (Reference Glover, Knight and MelzerGlover et al, 1997). The task may be left to very junior medical staff and is sometimes carried out by unqualified coding staff. The problem is compounded by the high proportion of diagnostic codings which are missing altogether from central returns (Reference Elphick, Anthony and LinesElphick et al, 1997). The same limits apply to other data that require a clinical judgement during collection, for example data relating to symptoms, problems, needs or outcomes. It has been argued that even the use of brief and simple measurement instruments requires training of raters and ongoing quality control if data are to be aggregated and/or used for comparisons (Reference Dunn, Thornicroft and TansellaDunn, 1996).

The need to agree data-recording standards

The training of psychiatrists imposes some uniformity on how they elicit and record clinical information. However, this does not extend to the aggregation of data derived from free-text clinical histories and mental states. For example, there is no nationally accepted set of standards stating which information items should comprise a personal history or a past psychiatric history, or of the terms used to describe these items. The problem becomes even greater when it is considered that nurses, social workers, psychologists and occupational therapists also assess people as part of their contribution to the work of a community mental health team. However, the NHS Executive's Clinical Headings Project has begun to address this problem by working towards an agreed set of headings for a clinical summary (Reference LewisLewis, 2002).

Concepts related to case-mix

When data about groups of patients are compared, it is important to ensure that like is being compared with like or that the effect of differences between the patient groups is taken into account. This applies whether the comparison is made to balance case-loads across a community mental health team, or to compare the activities or outcomes of different teams or services. Both require an understanding of the effect of case-mix on service use.

Two major factors should determine the needs for, and outcomes of, care of people with a mental illness. The first is the nature of the problem and the second is its degree. The nature of a problem can be described in a number of ways, most obviously in terms of a diagnosis. The degree of a problem has a number of possible dimensions, including severity, risk and level of dysfunction. To complicate matters further, people with a severe and enduring mental illness often have a complex array of comorbid conditions (such as substance misuse or personality disorder), social disabilities and social handicaps.

The extent of the challenge facing those working on case-mix in mental health is illustrated by the difficulty of even reaching agreement on a reliable definition of a concept as broad and ‘high level’ as severe mental illness (Reference Ruggeri, Leese and ThornicroftRuggeri et al, 2000). Despite this, early English work on case-mix showed that diagnosis, severity and complexity do predict use of resources as determined by use of hospitals beds or community services (Reference Elphick, Anthony and LinesElphick et al, 1997). In that study, severity and complexity were determined by three simple measures: score on the HoNOS; whether the person was subject to a section of the Mental Health Act 1983; and whether he or she was a ‘new’ or an ‘old’ patient (with less than or more than 6 months’ contact).

More-recent work has tested ‘health benefit groups’ and ‘health care resource groups’ (Reference Carthew and PageCarthew & Page, 2000). Health benefit groups are ‘groups of people with similar healthcare needs who, if given appropriate treatment, have similar outcomes’. Draft health benefit groups are based on a combination of problem, severity and complexity (based on HoNOS scores) and other factors such as risk and physical disability. Health care resource groups ‘are groups of treatments that are more-or-less iso-resource and clinically homogeneous’. It is proposed that a health care resource group be composed of three parts: the setting, principally whether residential or not; the contact rate and type; and the intervention type.

The experience in both England and Australia has been that case-mix variables relating to the nature and severity of disorder are only weak predictors of resource use (Reference Buckingham, Burgess and SolomonBuckingham et al, 1998). It is probable that their effect is swamped by a third factor, the idiosyncratic behaviour of individual practitioners. It is likely that the first major contribution of case-mix analysis in mental health will be to elucidate this.

The need to ensure the security of clinical data

The new NHS information strategy (NHS Executive, 1998) insists that the ‘implementation of the policies and procedures necessary to protect the privacy of patients must become an overriding priority’ (p. 50). Complex issues of data security, confidentiality and ethics arise when patient information is put to secondary uses. This is particularly true when the information is in a form that allows the identification of patients and/or is shared with people from outside the immediate team providing direct patient care.

Health and social care workers are still waiting for clear guidance from the Department of Health and the professional bodies regarding the framework that will guide them through these complex issues. Until this is given, mental health services will have to rely on local protocols for data security. Those developing these protocols will need a good working knowledge of the Data Protection Acts 1984 and 1998 (the latter bringing the principles of the European Directive on Data Protection into national law), the Subject Access Amendment Order 1987; the Caldicott report (NHS Executive, 1997); the General Medical Council's guidance on confidentiality (General Medical Council, 2000); and the Royal College of Psychiatrists’ good-practice guidelines on confidentiality (Royal College of Psychiatrists, 2000).

Individual practitioner or clinical team level

Clinical audit is perhaps the most common secondary use of patient information at this level. Unfortunately, the patchy quality and lack of standardisation of clinical information recorded in case notes limits the extent to which meaningful audit can be conducted using only data captured during routine clinical practice. Perhaps the major exception to this is information about drug treatments, because the rules governing the writing and dispensing of prescriptions impose the standardisation that is lacking elsewhere in the clinical record. This is particularly true for in-patients for whom drugs are dispensed by hospital pharmacists.

The potential value of prescription data is illustrated by the results of a multi-centre audit of prescribing of antipsychotic medication for in-patients (Reference Harrington, Lelliott and PatonHarrington et al, 2002a ). The information used in this large-scale project was collated by local staff using data that were already recorded in case notes and on prescription charts. The audit standards were derived from guidelines and consensus statements published by the professional bodies of a number of English-speaking countries. The standards were about the prescription of high doses of antipsychotic medication and the concurrent use of more than one antipsychotic drug (polypharmacy).

The main results were unequivocal: high-dose medication and polypharmacy are commonly prescribed and, when they are, practice often fails to adhere to the recommended standards. In particular, less than 10% of patients had been given an electrocardiograph before being prescribed a high dose and less than 20% of case notes recorded the fact that the patient had been informed if the dose of medication exceeded recommended limits.

Local provider or commissioner organisation level

Surprisingly little routine, secondary use of patient information is made at this level. Most data collected for routine monitoring and performance management purposes are obtained through separate channels parallel to those involving information used for clinical purposes. As a result, many mental health services, or even community mental health teams, are still unable to describe accurately the number of patients receiving care from their service and the intensity of that care, let alone its outcomes.

There are good examples, however, of ad hoc use of patient information that have influenced service development. One is the local review of serious incidents, including deaths of people in contact with services. When information about these events is pooled and discussed by those who have had contact with the patient, important lessons can be learned that lead to better care and services (Reference RoseRose, 2000).

Regional or national level

The National Confidential Inquiry into Suicide and Homicide by People with Mental Illness extends and complements local reviews of serious incidents by using patient information held by local clinicians. Here the large number of incidents reviewed makes possible the identification and dissemination of important common themes (Reference Appleby, Shaw and AmosAppleby et al, 1999; Reference Amos and ShawAmos & Shaw, 2000). If it succeeds in establishing its national reporting system, the National Patient Safety Agency would extend this to other types of incident and to ‘near misses’.

The multi-centre audit of prescribing described above illustrates the potential for using patient information to make comparisons between services (Reference Harrington, Lelliott and PatonHarrington et al, 2002b ). Acute psychiatric wards in 44 NHS trusts participated in the audit. The variation between wards in the proportion of in-patients who were prescribed a high dose ranged from 0% to 50% and, in the proportion prescribed polypharmacy, from 12% to 71%. Simple measures of differences between the patient groups in the wards in different services could explain only a small percentage of this variance. Presumably, at least some of the remaining variance is explained by differences in the practice of clinicians. Such results provide the basis for discussion by local practitioners.

Another potential use of patient information at the national or regional level is in the derivation of indicators of the effectiveness of mental health services. The ultimate aim would be indicators that reflect meaningful outcomes for groups of patients and that could both comment on the performance of mental health services nationally and allow comparisons of the performance of individual services. A working group of the Department of Health has proposed a set of candidate outcome indicators for severe mental illness (Box 4; Reference Charlwood, Mason and GoldacreCharlwood et al, 1999). A number of these are based on information that is collected routinely in clinical practice.

The National Service Framework (Department of Health, 1999) is committed to identifying and applying such indicators, and strenuous efforts have been made to introduce performance indicators by March 2003.