Discrepancies and inaccuracies in statistics for detained patients

While conducting a study on the nature and extent of compulsory admissions to mental hospitals within the four London regional health authorities, discrepancies and Inaccuracies were revealed in many of those mental hospital In-patient statistics. The nature and extent of some of these are examined. It is suggested that the value of such statisticsfor government planning must be questioned as Is their value for research. It Is recommended that a centralised system of collecting and collating such data be Introduced as a matter of priority and that such a system be operated by the Mental Health Act Commission.

Increasingly questions are being asked about the quality and value of some of the data provided by the National Health Service (NHS) mental hospitals. For example, research workers using such national (or local) data have often noted their deficiencies and when conducting research have often preferred to collect their own (Barnes et oÃ-, 1990;DoH Service Indicators Group 1988;Hirsch et al, 1979;Glover, 1987;Glover et cd, 1990). Yet clearly the national published data are collected and produced at some cost, and presumably are used as a basis for government planning. (DoH Statistical Bulletin, 1992). If, however, they are defective, as some have claimed, then In the interests of promoting an efficient and effective National Health Service some attention needs to be given to this.
The question is how defective are they -If at all? And if defects can be Identified what would be the appropriate response? In this paper, attempts have been made to answer some of these questions using our research on The nature and extent of compulsory admissions to mental hospitals within the four London regions.* The methods for that research involved examining the nature of some This research was funded by the King Edward's Hospital Fund for London to whom acknowledgement Is given. mental health hospitals' data and their method of collection and dissemination. Three areas of Interest have been identified: first, a description of the quality of the NHS mental hospital data, second, an assessment of their value, and finally suggestions about likely changes.

Data collecting and processing in the hospitals under study
The compulsory admissions research, which was the main research study, was not specifically aimed at examining the methods of collecting and recording hospital data. The information presented here Is therefore a by product of that research. In that research records of certain London mental hospitals over a three year period from 1987 to 1990 were studied. These hospitals were selected on the basis of their compulsory admission rates as presented in the official returns to the Department of Health Statistics and Management Information (Fylde)-to be called the Fylde data from now on. Moreover, that compulsory admission research was only concerned with compulsory admissions, i.e. not all admissions to mental hospitals. Hospitals were therefore selected on the basis of the aggregate and proportion of their compulsory admissions.
The first point to note is that the methods of collecting and recording the data in the mental hospitals studied varied. Each hospital seemed to have Its own system. The system that was used seemed to be related to the interests and qualifications of the staff involved, and of the technical systems used to produce the data. In very general terms the system seemed to work something like this.

Stage 1 -Hospital unit level
This Is the basic level of data recording where the Mental Health Act administrator or the patient services manager records the day to day entries In a section book -usually handwritten.
The Mental Health Act administrator keeps this record in order to ensure that detained patients' rights under the 1983 Mental Health Act are adhered to. The Mental Health Act administrator might use these data for hospital manager's reports etc. Some Mental Health Act administrators compile KÃ ¶rnerreturns from these data but the majority do not-in fact most Mental Health Act administrators do not know who compiles the KÃ ¶rnerreturns.

Stage 2 -Hospital medical records level
Data obtained from 'Patient on Admission' records are fed into the Patients' Administrative System (PAS), one of the computer systems developed for the NHS. It is often difficult to identify who does thisfrequently the responsibility belongs to a clerk in the medical records department. In many hospitals the Mental Health Act administrator does not work alongside or near medical records. It was found that there is considerable variation about who collates data for KÃ ¶rnerreturns at the hospital level. Sometimes the PAS system is used alongside the hospital's own software package, sometimes not. From the Stage 2 level the data are sent to the next stage, which for the purposes of this paper is called the district information level.

Stage 3 -District information level
At this level the aim is to translate the data about compulsory admissions from PAS into the KÃ ¶rnerreturns. fThese returns were first used in April 1987 as a result of the KÃ ¶rner review on Health Service Information. This review led to procedures for data collection of Mental Health Statistics on detained patients admitted to NHS hospitals). The compulsory admissions data are collated on forms called KHIBs and changes in legal status are collated on forms called KH16s. Both returns cover the financial year 1 April to 31 March.

Stage 4 -Regional level
This stage, the regional level, is added only for completeness1 sake. The statistical data pass through the regional level on their way from district to Fylde but are not handled or processed by region -at least this is what happened in the London areas but it may not be true elsewhere.

Stage 5 -Department of Statistics Division (Fylde)
The final stage is where KÃ ¶rnerreturns are collated and official data on mental health are produced from Fylde in Blackpool. The ultimate responsibility, however, is to the Department of Health in London.

The data produced
Statistical data were obtained from all these levels including those from the KÃ ¶merreturns from Fylde. Certain discrepancies were found when a comparison was made of the information from the various hospitals comparing KÃ ¶merreturns with the unit level data. Some examples follow fTables 1 and 2). These are taken from two hospitals in the South West Thames Regional Health Authority for the year 1 April 1989 and 31 March 1990. The Fylde and district data are the Official Statistics. Below are two further examples from different hospitals in the same region taken over the same period.  Tables 1 and 2 given above are selective but they show nonetheless that discrepancies exist. The range of discrepancies was not always as large as this; we found, however, that in ten out of the 23 hospitals the differences were greater than 30%. Agreement exists between the district and Fylde data -as would be expected because the Fylde data are taken from the district data. Clearly then the basic problem lies at the unit/ district level. These examples illustrate the range of some of the differences. It needs to be stressed that these were not isolated examples: such differences were common. In only one hospital out of 23 was there an agreement between the unit level and the Fylde data.
A closer examination of the data revealed other inaccuracies and defects, some relating to presentation, others about interpretation.
A few examples will illustrate the point. These have been grouped under a number of headings.
(a) These are described as listings under Incorrect headings. As an example, in one hospital 38 patients were listed as having been admitted under section 5(2) of the 1983 Act. In fact section 5(2) is not an admission order: it provides powers to hold existing informal patients for up to 72 hours. Again, this particular hospital was not alone in so recording section 5(2)s as an admission order: this section was often wrongly interpreted. Moreover, section 5(2) was also recorded as compulsory admission on the KÃ ¶mer Aggregate Form (KH15) when it should be collated on KH16 as a "change in legal status".
(b) These errors are referred to as computer distortions. They include a large group of errors -although they should more accurately be called defects. As one example of such a defect, all PAS systems work to a minimum data set. They will not accept an admission when a postal code is missing, or contains new addresses the system does not recognise. This is particularly important where patients are homeless (i.e. having no postal code) or when they are housed perhaps locally in bed and breakfast accommodation.
Or take another example; some PAS systems do not recognise the existence of community mental health teams or community mental health centres. In one hospital about 50% of referrals came from CMHTs. These were then recorded as GP referrals.
Or again, other PAS systems allow only section 2 or 3 patients (the assessment and treatment orders) to be shown on admissionother patients such as those on section 4, i.e. those on an emergency order, have to be shown as being admitted informally. They are then regraded to section 2 or 3 when the appropriate time occurs.
(These hospitals would, of course, be seen not to have psychiatric emergencies).
Finally, some PAS systems do not differentiate between short-term and long-term leave. The KÃ ¶mer returns require that leave spells must not exceed 28 days but the 1983 Mental Health Act allows leave up to six months. Some PAS systems have to 'return' patients to the hospital from leave and 'send them out again' on paper and repeat this paper transaction every 28 days until the period of leave allowed ends.
(Incidentally this could have accounted for the high compulsory admission figures in some of the data). (c) These errors are described as consultant managerial errors and cover those in which rates are distorted due to errors involving the transfer of patients between the systems in the NHS. For example, a long-stay patient in one of the old asylums, who might have been an in-patient for a number of years, might spend one night in a general hospital. This patient would then reappear as a new admission on return to the hospital and so in technical terms be regarded as a new consultant spell. What seems to be happening is that the systems are often linked to consultant episodes rather than patients. Similarly, in another hospital patients transferred from one annexe to another were listed as being new admissions. (d) This group of errors are not so much errors of data as limitations as to their use. These are described as restrictions.
For example, on most AUDIT systems once a consultant episode is over that episode is concluded. This makes corrections of previous errors difficult or impossible to undertake. Or when attempts are made to retrieve information only the current consultant episode is shown. If earlier episodes are required a separate search is necessary. This means, of course, that any form of analysis, even a modest one aimed at determining trends, becomes almost impossible. (e) Finally, there are errors which are simply caused by incompetence, sometimes due to inexperience or simply by lack of training in the legal requirements of the 1983 Mental Health Act of those required to produce the data. As stated above, it was apparent that there was no one method of data collecting in the individual hospitals. This means that often the responsibility for collating mental health data seemed to belong to no clearly identifiable person.
It is not possible at this stage to be clear about the full extent of the discrepancies. Nearly all the personnel spoken to who were responsible for the data collection and presentation in the hospitals openly stated that the system produced errors. Yet only those trained in the legal requirements of the Mental Health Act, and in the details of the PAS system, including KÃ ¶rner returns, seemed to know the extent of the problem.

The value of the data
With differences as great as those found in this research it is difficult to believe the official data have much value. (To emphasise the point further: one hospital in South East London stated 153 compulsory admissions for 1989/90, ofwhich 119 were under section 2; the unit data showed 301 of which 219 were section 2). Consistently the unit figures were higher than the Fylde figures, although not always so and only very occasionally (2 out of 18) did they correspond.
One obvious feature of those differences, as far as the main study was concerned, was that the scale of the discrepancies makes it difficult to use such data for research, especially as a sampling frame. Hospitals thought to have low admission rates turned out to have rather more when the data were examined closely, whereas some thought to have high rates turned out to have fewer than expected. Clearly, any research relying on such hospital data must be suspect if those data are used as a research instrument.
Questions must be asked too about the effect on the work of agencies such as the Mental Health Act Commission whose task it is to inspect records. Which set of data was given to them: that from the unit or from Fylde? And how could they ever be certain that they, the Commissioners, were able to protect the rights of detained patients when they were unclear how many detained patients there were in the first place. And what of the financial cost of producing such data when so few are of value?
Consider the resource implications. Admissions under part 3 of the 1983 Mental Health Act, i.e. admissions from courts or prisons, are a small proportion of the total formal admissions.
Yet, being small, distortions or inaccuracies are all the more significant. The use of these sections carries large resource implications and reflects recent policy changes regarding mentally disordered offenders. So too for other matters such as homelessness where there are also implications for resource allocation and capitalisation weighting. It is in the interests of mental health providers and district purchasers to have accurate informationtogether with all who are involved in policy formation.

Suggestions about change
It was shown earlier that there seemed to be many reasons why deficiencies and inaccuracies occurred: some were due to lack of training, some due to poor management and some due to a lack of understanding about information systems generally. (Incidentally the systems were rarely compatible between hospitals and sometimes incompatible within hospitals where two systems in the same hospital had incompatible software packages). In addition, some defects were due to the low priority given to data collection and some due to the low status of the Mental Health Act administrator -the latter is important because with the prescription for change suggested here, it is suggested that the Mental Health Act administrators be upgraded in status and position. Too often it was noted how the title 'Mental Health Act administrator' has disappeared to be replaced by 'patients' Inaccuracies in statistics for detained patients AUDIT services' or 'site services manager'. And In some hospitals no-one knew who was the Mental Health Act administrator! It seems important that the term 'Mental Health Act' be incorporated In the administrator's title if the appropriate emphasis is to be given to the post.
As far as recommendations for the future are concerned, and these recommendations are only within the context of compulsory admissions, it seems the Scottish system has advantages over that used in England and Wales. In Scotland the equivalent to the Mental Health Act Commission, the Scottish Mental Welfare Commission, receives all information on compulsory admissions. Those data come direct from all the hospitals. It is suggested that this should take place in England and Wales too. The advantages of such a system are numerous. Aside from improving the quality of the data and reducing the errors, that method of data collection allows trends to be detected more speedily, allows more sophisticated analysis to be conducted (the system in England and Wales, for example, does not permit an examination of readmissions except when the patient was readmitted to the same hospital) and encourages data uniformity. It also provides a Justification other than bureaucratic expediency in that a centralised system gives a more certain way to protect patients' rights, especially If the Mental Health Act Commission is to receive the data in the first place.
This proposal would not be difficult to Implement, nor would it be expensive, and may work out cheaper than the method existing in the psychiatric hospitals now. It would certainly help improve matters. But most of all it Is difficult to see how the existing procedures can be retained given that they produce such discrepancies and inaccuracies.