Routine measurement of mental health service outcomes : Health of the Nation Outcome Scales in Nova Scotia

AIMS AND METHOD Although the Health of the Nation Outcome Scales (HoNOS) were designed for routine clinical use, completion rates in clinical settings rarely exceed 60%.We evaluated two initiatives to increase completion rates: timely feedback to clinicians, useful at individual, team and service levels, and improved supporting materials (tabulated glossaries for all versions covering the lifespan).


AIMS AND METHOD
Although the Health of the Nation Outcome Scales (HoNOS) were designed for routine clinical use, completion rates in clinical settings rarely exceed 60%.We evaluated two initiatives to increase completion rates: timely feedback to clinicians, useful at individual, team and service levels, and improved supporting materials (tabulated glossaries for all versions covering the lifespan).

RESULTS
Clinicians in South Shore Health district provided ratings on all outpatient referrals over 12 months as part of routine care.Data were captured using the Nova Scotia routine administrative data system.Completion rates rose from 61% to 86% (n=1190).Clinicians' ratings of the instrument's ease of use were significantly improved.

CLINICAL IMPLICATIONS
Use of a tabulated glossary and enhanced feedback of clinically useful information improved clinician support for the routine measurement of health outcomes with HoNOS.
Routine measurement of mental health outcomes is difficult as they are long-term, complex and multidimensional.Designed for routine clinical use, the Health of the Nation Outcome Scales (HoNOS) are comprehensive of clinical and social functioning, acceptable to clinicians, sensitive to change, valid and generally reliable (Wing et al, 1998;Amin et al, 1999;Burns et al, 1999a;Orrell et al, 1999;Lauzon et al, 2001;Pirkis et al, 2005b).They cover all ages and are used in Australia, New Zealand and the UK (Wing et al, 1996;Trauer et al, 1999;Pirkis et al, 2005aPirkis et al, , 2005b)).However, completion rates in routine clinical use, as opposed to research settings, rarely exceed 60% (Fonagy et al, 2005), as was also reflected in previous work from Nova Scotia (Kisely et al, 2007).We therefore evaluated the effect of two initiatives to increase completion rates: feedback to clinicians in the form of reports that are timely, relevant and useful at a range of levels (i.e.individual, team and service levels), and improved and simplified supporting materials in the form of tabulated glossaries for all versions of HoNOS covering the lifespan.

Method
The study was carried out in South Shore District Health Authority, Nova Scotia, the site of an earlier pilot project evaluating the psychometric properties of HoNOS in a North American setting (Kisely et al, 2007).All clinical staff were trained in the use of the HoNOS version for working age adults and those aged over 65 years old; clinicians specialising in youth services received further training in the child and adolescent version, Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA; Gowers et al, 1999aGowers et al, , 1999b)).Training was conducted by a person who had completed HoNOS training at the Royal College of Psychiatrists.Intraclass correlation coefficients (ICC) for the overall global score at the end of training were good; ICC scores for the subsections were lower but still significant (Kisely et al, 2007).
Study data were collected from October 2005 to October 2006.All new out-patient encounters were eligible for inclusion.Assessments were completed for each new referral at the beginning and end of each care episode.Data were entered using the routine administrative data system of the Nova Scotia province.The Mental Health Out-Patient Information System records demographics, ICD-10 diagnoses and care episodes for all contacts.Clinicians attached HoNOS scores to the returns submitted on each patient visit.These reports were also used to give participating clinicians feedback in the form of summary statistics on their patients' scores.
We assessed the effect of the following two initiatives to increase completion rates.First, we provided feedback to clinicians using reports tailored to their needs, aiming to make these timely, relevant and useful at a range of levels (e.g.scores for individual patients and aggregates for their caseload).Second, in consultation with clinicians we improved and simplified supporting materials by developing tabulated glossaries for the child/ adolescent and adult versions, based on the tabulated glossary that had been developed by the 65+ HoNOS working party (Burns et al, 1999a(Burns et al, , 1999b)).There were two glossaries: one combining the working age adults and 65+ versions of the HoNOS, the other covering the child and adolescent version.These glossaries included each aspect of individual HoNOS items in rows (e.g.aggression, non-accidental self-injury, etc.), with a definition of the severity rating in columns.For instance, the HoNOS item on non-accidental self-injury contained rows for ideation, planning and acts.We used the description from the original HoNOS glossaries to preserve the psychometric properties of the instrument.
We assessed changes in clinician acceptance by using percentage of patient contacts for which HoNOS ratings had been completed, as well as five-point Likert scales completed by clinicians at the end of the project period.All administrative and survey data supplied to the researchers were anonymised, with no patient or clinician identification.We did not identify individual clinicians' completion rates.The relevant research ethics board approved the protocol.
We assessed changes in the clinicians' views (based on Likert ratings) using Wilcoxon signed rank tests for global and individual item scores, as we could not assume with certainty that the scores had an underlying interval scale.Lack of symmetry and constant variance can also undermine the validity of parametric tests, especially when sample sizes are small.

Baseline data
Over 12 months, clinicians provided ratings on 1190 (out of 1377 eligible) out-patient referrals as part of routine care.The completion rate was therefore 86% compared with 61% prior to the changes introduced in the current project (Kisely et al, 2007).Of the 1190 patients who had had at least one HoNOS rating, 606 (51%) were female.The average age was 36 years old (s.d.=21.2);74% were rated using the adult version (n=876), the remaining 314 using the child and adolescent version.

Clinicians' viewpoints
Twelve participating clinicians were asked about their experiences with the instrument using five-point Likert scales (0=strongly disagree, 1=disagree, 2=neutral, 3=agree, 4=strongly agree).We present medians given the ordinal nature of the scale and non-normal distribution of the data.The median score for ease of HoNOS use and usefulness of the glossary was 4 ('strongly agree').We analysed differences in clinician viewpoints before and after the changes made in this project.There were significant improvements in terms of ease of use (medians of 3 v. 4, z=73.65,P50.001) and utility of the glossary (medians of 3 v. 4, z=73.23,P=0.001).In terms of the feedback reports provided to clinicians on their patients' scores, ratings of usefulness (medians of 1 v. 3, z=73.10,P=0.002) and relevance (medians of 2 v. 3, z=72.39,P=0.02) of the information were much higher after the study.

Discussion
This is the first project to evaluate the introduction of HoNOS across all age groups in a way that resembled normal clinical practice as much as possible.No research workers were involved in data collection.All relevant staff had attended training and participated in an assessment of interrater reliability.Importantly, all the data were captured using existing routine administrative collection.Baseline HoNOS scores for out-patients in this study were similar to those reported in other studies of adults (Trauer et al, 1999) and children (Gowers et al, 1999a), although higher than in some other adult populations (Orrell et al, 1999).This may be partly because of the fact that our study was of new referrals rather than of patients in treatment (Orrell et al, 1999).We included clinicians' comments in the planning of the pilot project and provided them with feedback on their own patients.
International experience suggests that the success of routine outcome measurement depends, to a critical degree, on the engagement of clinicians who actually complete the outcome measures (Pirkis et al, 2005a).Engagement is most likely to occur when clinicians are given meaningful information that enables reflective practice and improves patient care (Fonagy et al, 2004;  Pirkis et al, 2005a).Acceptance of HoNOS appears to have improved by using a tabulated glossary.
Improved feedback of clinically useful information also seems to have been associated with greater clinician support for the routine measurement of health outcomes with HoNOS.Our data do not support the view that clinicians necessarily see the completion of rating scales as an intrusion that will not be tolerated for any length of time (Stein, 1999).

AIMS AND METHODS
We looked retrospectively at referrals over 1 year and determined the length of time from receipt of the referral to the date when authorisation to see the patient was given, and the time from authorisation to first contact.For in-patients we determined the time from the recommendation for admission to the time authorisation was granted, and the time from authorisation to admission.

RESULTS
Of the 108 referrals, 80 (74%) were seen within 13 weeks (mean=53 days); 7 patients (6%) had to wait over 13 weeks, usually because of delays in authorisation;10 patients were admitted;14 were never admitted (although we had recommended admission).The time from recommendation to admit to authorisation of admission, 27 days, was the same as the time they waited for a bed once authorisation had been given.

CLINICAL IMPLICATIONS
Waiting for authorisation seemed in some patients to delay their access to tertiary services.Any advantage of using panels to authorise referrals, like ensuring better use of local resources, may be outweighed by patients taking longer to get the care that best meets their needs.
The Lishman Brain Injury Unit has expertise in the diagnosis and management of adults with cognitive, behavioural and psychiatric sequelae of acquired brain injury.There is an out-patient service and a small in-patient unit for the medically and surgically stable.Patients may have acquired their injuries in a variety of ways, often after road traffic accidents or hypoxic brain injury.
The unit acts at a tertiary level with the majority of referrals coming from consultants in district general hospitals and mental health professionals.When a referral is received a decision is made as to what further action is required.This may take the form of an out-patient assessment.If an outreach assessment is needed a member of the team will assess the patient, usually with a view to determining their suitability for in-patient admission to the unit.
Those who commission the service need to be confident that what is being recommended by the specialist unit is appropriate.Over the last few years more and more authorisation panels have been set up by local primary care trusts to oversee specialist referrals.In some cases these panels seem to have led to additional delay for the patient before they can be seen.It is important to understand any delays in the system, partly because of the National Health Service's (NHS) target for all patients to be seen within 13 weeks of referral (Department of Health, 2000).
The aim of the present audit was to investigate the influence of any delay getting authorisation for funding on the overall waiting time from receipt of referral to the first out-patient appointment or date of admission.

Method
We reviewed referrals to the service between 23 November 2004 and 24 November 2005.Procedures for authorisation of out-patient and outreach assessments