HoNOS: a cautionary tale of their use in a rehabilitation service

Aims and method Health of the Nation Outcome Scales (HoNOS) were piloted in 248 patients with serious persistent mental health problems. Staff were trained by a cascade approach and rated patient outcome after sixmonths. Results There was poor interraterreliabilitybetween nurses and psychiatrists/psychologists. Outcome measured by HoNOS often contrasted with clinical experience. Clinical implications HoNOS were not a reliable or valid measure of outcome. The scales are now completed only under supervision.

The first mental health target in the Health of the Nation (Department of Health, 1992) is "to improve the health and social functioning of mentally ill people". The Health of the Nation Outcome Scales (HoNOS) have been developed by the Research Unit of the Royal College of Psychiatrists (Wing et al 1996) to measure the changes in symptoms and social outcome of all patients with mental disorders. Health author ities have been encouraged to introduce them to routinely collect data on patient outcome (NHS Executive, 1996). This study addressed two concerns. First, their applicability and sensitivity to change in a population that comprised exclusively of seriously disabled people. Second, based on the service's experience of using other standardised measures of functioning, there was concern about discrepancies in the styles of ratings between disciplines.
training day organised by the Royal College of Psychiatrists.
They then trained four other senior professionals who in turn trained all other qualified staff in groups. Training con sisted of an explanation of the need to measure outcome, an introduction to the scales and how to score them, and group ratings of patients who were familiar to all raters. There was opportunity for feedback and discussion and further supervision was offered at the second rating. To measure the six-month clinical out come, HoNOS forms (version 3) were completed on two occasions: July/August 1995 (248 patients) and January/February 1996 (243 patients). Forms were completed by each pa tient's keyworker unless another member of staff was more closely involved with them. They were directed to consult other professionals as necessary.
An inter-disciplinary reliability exercise was performed. For 32 patients, ratings were per formed independently by the nurse keyworker and the psychologists (17 patients) or psychiatrist (15 patients) involved in their care. For another eight, ratings were made indepen dently by two nurses. Data were entered and analysed using the program SPSS (Norusis. 1986) and the t-test/analysis of variance used to measure significant differences in mean HoNOS scores.

The study
The study was conducted in the rehabilitation and continuing care department of a psychiatric service in south-west London. Patients accepted by the service resided in a range of settings from highly staffed in-patient wards and hospital hostels in the community to staffed group homes, supported lodgings and family homes. They typically had a diagnosis of schizophrenia and were treatment unresponsive and difficult to place.
A cascade approach was adopted to staff training in the use of HoNOS. Two senior staff (a psychiatrist and a senior nurse) attended a

Findings
Overall results obtained Males comprised 59% of the sample, 83% had a primary diagnosis of schizophrenia and mean length of service contact was 30 years. HoNOS rated half with hallucinations and delusions, 25% were aggressive and 25% had physical health problems but a third had no significant clinical problems and half had no significant social problems (HoNOS rating of <2). Fourteen (6%) were rated with a total HoNOS score of 0 (no problems at all).

ORIGINAL PAPERS
HoNOS scores by place of residence There were significant differences between the mean total HoNOS scores of people living in different types of accommodation at the first (F=10.18. P<0.0001) and second (F=10.2. P< 0.0001) assessments.
In-patients scored as more disabled than those living outside hospital. However, there were two notable anomalies: an unusually high decrease in scores occurred in patients living in supported accommodation, and the scores of people living in one community hospital hostel were strikingly low.

Inter-and intra-disciplinary reliability of ratings
The disciplines of raters were on the first occasion nurses (73%), psychiatrists (21%) and psychologists (1%). The mean score (s.d.) of the sample was 6.4 (5.8) rated by nurses and 15.6 (6.8) rated by psychologists/psychiatrists. This difference was highly statistically significant (i=6.86. P< 0.001). Highly significant rating differences occurred for 'memory/orientation', 'aggression', 'occupation recreation and finance', 'accommodation', 'social relationships' and 'over all disability'. For those patients rated indepen dently by two nurses, there was a high level of agreement (mean total HoNOS scores 13.75 and 14.25, respectively).

Sensitivity to change
The total mean HoNOS scores decreased by 2.5% over six months from a mean of 10.8 to 9.5 (f=2.64, P<0.01, n=214). A closer look at the individual items revealed some large changes, for example the proportion with disabilities in relation to daily living fell from 77 to 56%. This finding suggests that a substantial proportion of patients could be discharged. In-patients who had moved to accommodation with less support, the mean HoNOS score decreased from 11.6 to 8.6 reflecting clinical improvement.

Comment
The staff training programme was straight forward to implement and had joint nursing leadership. After an initial reluctance, it was accepted by most staff as it did not increase workload (replacing a more complex standard ised outcome measure already in use). The main finding of this study was a low inter-disciplinary reliability of an instrument designed for use by any qualified mental health professional who has received training. The nursing staff had good interrater reliability but under-rated symptoms and disability when compared with psychiatrists and psychologists. In contrast with our findings, nurses and psychiatrists in the HoNOS field trials (Wing et al, 1996) did not differ in their rating style. Additionally, when measuring sixmonth outcome, HoNOS scores often appeared to be subject to an upward drift in patients who did not appear clinically to make any gains.
Our study has not been able to replicate the reliability of the HoNOS field trials and nursing staff appear to have had difficulty in using the scales. Although this study is not able to directly address why such a discrepancy has occurred, some explanations could be offered. This could be seen to reflect a poor general level of training or lack of enthusiasm for the pilot study from nursing staff. However, they are all familiar with using structured instruments, a nurse part icipated in the leadership of the training pro gramme, educational opportunities are widely available and multi-disciplinary working is firmly established. Alternatively, the training process may be at fault. As we adhered to the protocol suggested by the Royal College of Psychiatrists, it is unlikely that deficiencies in our training process occurred. It is more likely that the suggested training process has failed to train our staff despite a correct implementation. Finally, there may be problems of reliability and validity of the HoNOS scales themselves in this patient group with considerable psychiatric disability.
HoNOS assessments are now completed at care planning meetings under the supervision of senior clinical staff. Services for those with severe and enduring mental health problems planning to introduce HoNOS need to give special attention to the training and supervision of raters in order to achieve valid outcome data.