Managing outcome performance in mental health using HoNOS : experience at St Andrew ’ s Healthcare

AIMS AND METHOD Modern mental healthcare providers must demonstrate service-level clinical effectiveness to key stakeholders.We introduced two performance indicators of clinical effectiveness based on outcome data from routinely collected Health of the Nation Outcome Scales (HoNOS) and HoNOS-secure assessments across St Andrew’s Healthcare, a charity providing in-patient services in Northampton and Essex.We present outcome data from the period 2004-2007.

The indicators showed consistent 90-day improvement rates and increasing stability over time.The validity of results is supported by levels of change along predicted lines among different patient cohorts.

CLINICAL IMPLICATIONS
It is possible and beneficial to use routine outcome measures to demonstrate clinical effectiveness at service level.The future of managing mental health outcomes will be in electronic health records systems.
Providers of modern mental healthcare are increasingly required to demonstrate the effectiveness of their services to a range of stakeholders including purchasers, service users and regulators.Measures of service effectiveness must of course cover value across a range of cost and quality indicators.The 'balanced scorecard' 1 is a tool to bring a mixed bag of key performance indicators into sharp focus, aiming to provide managers with clear, guiding indicators of progress against key strategic aims.It has been likened to the dials in an aeroplane cockpit or dashboard.In healthcare management the scorecard should cover domains such as clinical quality and effectiveness, staffing, service demand, activity, financial performance and organisational learning and growth. 2he implementation of a balanced scorecard in psychiatric services has previously been described 3,4 but measures of clinical effectiveness have been problematical 4 and have thus far been absent from these accounts.Below, we describe the integration into a balanced scorecard of two performance indicators using aggregated data from routinely performed outcome measures, and present service-level data demonstrating the usefulness of the key performance indicators for service providers and other key stakeholders.
Although the routine use of outcome measures has not been shown to bring about improvements in outcomes such as patient-rated unmet need and quality of life, there is promising evidence that in-patient days and thus service cost are reduced. 5There is consensus that outcomes measurement needs to be multidimensional 6,7 both in terms of the outcomes measured and the viewpoints or perspectives considered, including those of service user, clinician or other stakeholders.Numerous items have been used or nominated to support outcome-focused clinical key performance indicators including frequency of unplanned discharges and readmissions, incidence of injurious behaviour and self-harm, transfers, use of restraint, seclusion and number of falls, 8 provision of pre-admission care, 9 medication use and insurance claims. 10For those with severe mental illness, proposed outcome indicators include routine analysis of needs assessment measures 11 and Health of the Nation Outcome Scales (HoNOS) 12 data at both service and individual level. 7ealth of the Nation Outcome Scales for workingage adults and the related family of variants includes outcome measures for children, for older people, and for those with a learning disability or acquired brain injury.All of the HoNOS variants aim to measure mental health and behavioural functioning in order to provide data to judge the effectiveness of mental health services.The tools perform adequately or better in terms of reliability, validity, sensitivity to change and utility. 13In the UK, use is widespread and increasing: 14 49% of all English mental health service providers were using HoNOS in more than one setting by 2002 and recently all members of the HoNOS family have been taken up into the National Health Service (NHS) Connecting for Health IT Programme.The HoNOS-secure 15,16 has been developed for users of secure psychiatric services, prisons and forensic community services, where diagnostic heterogeneity, multiple treatment needs and additional offence-related treatment needs adds further complexity to outcomes measurement. 6The HoNOS-secure contains modified versions of the original twelve HoNOS scales, and a separate 'security scale' comprising seven additional ratings to track change in the outcome of ongoing clinical risk assessment, and in the need for physical, relational and procedural security measures (Box 1).In short, HoNOS-secure is aimed at all users of secure and forensic services, and is designed to be applicable to these service users in the community also, allowing continuity in tracking outcomes.

Method
Setting St Andrew's Healthcare is a charity operating in the UK independent healthcare sector, providing a range of specialist mental health services for NHS patients with many extended care pathways: men's, women's and adolescent services covering mental health and learning disability; Kemsley, the National Centre for Brain Injury Rehabilitation; Townsend services for older adults; and also the St Andrew's low secure adult mental health service in Essex.The six services, totalling approximately 500 beds, all provide a level of secure care.

Measures
Since 2004, all six services have used HoNOS outcome measures, with HoNOS assessments routinely conducted for all individuals at the end of a pre-admission assessment, within 7 days of admission, and thereafter at 3 monthly intervals until a final rating is made at discharge.Rating is undertaken by the individual's care team as part of the care programme approach process.Individuals are rated using the most appropriate HoNOS tool (HoNOSsecure, HoNOS-LD (for people with learning difficulties), HoNOS 65+ (for people aged 65 years and older), HoNOS-CA (for use with children and adolescents) or HoNOS-ABI (for people with acquired brain injury), collectively referred to as HoNOS specialty scales).Because all services provide a level of secure care, all individuals are rated on the seven-item HoNOS-secure 'security scale'.

Aims
Two key performances indicators of clinical outcome, covering overall progress and need for secure care, were added to an existing balanced scorecard at St Andrew's Healthcare in January 2004.We present aggregated outcome data across clinical services at St Andrew's Healthcare from 2004 to 2007 in order to demonstrate the value to service managers.Although the performance indicators are based on aggregated data, and not amenable to inferential statistics, we hypothesised that face validity of the service-level measures would be demonstrated if: . change within different patient cohorts were to differ along reasonably predictable lines, with less change among more intractable patient groups such as those with dementia, and more among those with most potential to change such as those in the adolescent service; .change over time, both within and across services, were to be relatively stable, and for this stability to increase as quality of data collection improved.

Key performance indicator calculations
Two key performance indicators, each indicating an average 90-day improvement rate, are calculated quarterly from individual patient HoNOS specialty and HoNOS-secure 'security scale' assessments.The method of calculation for the HoNOS-specialty scales and for the HoNOS-secure 'security scale' is the same, being the change in score as a proportion of the total possible score.This captures the direction and size of change in a cohort, and allows a broad comparison across services using different HoNOS specialty scales.Each quarterly data collection therefore captures the current patient cohort, and picks up at ward-and service-level any change in the patient group under treatment.Because the indicator is calculated to identify change rather than overall levels of well-being, it is not overly sensitive to fluctuations brought about by additions to and subtractions from the cohort through admission and discharge.

Results
Data for 2004-2007 were collated and 90-day change rates calculated as outlined above for HoNOS specialty and HoNOS-secure 'security scale' ratings.The mean successful completion rate of ratings within 3 months for the entire 1-year period for 2004 was 65% rising to 83.0% in 2007.Eachitemis scored ona 0-4 scale using a glossary provided. 15t around or just over 1% for the HoNOS-secure 'security scale'.Figures 3 and 4 show average annual 90-day change rate by service for the HoNOS specialty and HoNOS-secure 'security scale' key performance indicators respectively.As predicted, most change is noted among people using the adolescent service and least among elderly people.The six services were ranked in order of 'security scale' total scores ('need for security measure') and in order of behavioural and psychiatric symptomatology totals (from HoNOS specialty scales).These rankings were positively correlated (Spearman's rank correlation coefficient 0.6).Positive change over time in symptomatology is demonstrated for the low secure mental health cohort, but not in the need for security measures (Fig. 4).This may be because this smaller service is integrated more with external care pathways, with less opportunity for individuals to progress through levels of security within the service.However, it should be noted that the small negative 90-day change rate on the HoNOS-secure 'security scale' key performance indicator for this service is largely because of a negative outcome in year 1 possibly as a result of data collection and calculation issues.Excluding year 1, the low secure service actually achieved modest improvements across the period 2005-7.

Discussion
Two performance indicators based on routinely collected data from HoNOS specialty and the HoNOS-secure 'security scale' provided key information about the performance of clinical services and were an important addition to a balanced scorecard of indicators of business performance.It was possible to detect relatively small quarterly changes to overall health rating and security needs, suggesting HoNOS-secure is sensitive to change among a group of individuals referred to our specialist services from the NHS.As would be expected, greater change in both health and security status was found among people using the adolescent services where symptomatology and functioning problems are less entrenched.

Limitations
Initial significant problems with data collection and completion have been identified, including return within appropriate time scales.Additionally, some errors in data processing have been identified and addressed.The solutions in these areas are for all HoNOS ratings to be 'owned' by the clinical teams and for easy-to-use electronic systems with good management.The now complete introduction across the charity of electronic patient records promises to further improve outcome data collection.There has been considerable recent debate about patients' own subjective rating of outcome

Figures 1 and 2
show 90-day change rates calculated as annual averages for the HoNOS specialty scales and the HoNOS-secure 'security scale' respectively.Relatively low completion rates in 2004 are reflected in the wide range of change across services, and this range generally narrows over time for both key performance indicators.The average 90-day change rate across the study period for the entire St Andrew's Healthcare service hovers around or just under 1% for HoNOS specialty scores and Sugarman et al Managing outcome performance in mental health original papers Box 1. HoNOS-secure seven-item 'security scale' A Risk of harm to adults or children B Risk of self-harm C Need for building security to prevent escape D Need for a safely staffed living environment E Need for escort on leave (beyond secure perimeter) F Risk to individual from others G Need for risk management procedures