Providers of modern mental healthcare are increasingly required to demonstrate the effectiveness of their services to a range of stakeholders including purchasers, service users and regulators. Measures of service effectiveness must of course cover value across a range of cost and quality indicators. The ‘balanced scorecard’ Reference Kaplan and Norton1 is a tool to bring a mixed bag of key performance indicators into sharp focus, aiming to provide managers with clear, guiding indicators of progress against key strategic aims. It has been likened to the dials in an aeroplane cockpit or dashboard. In healthcare management the scorecard should cover domains such as clinical quality and effectiveness, staffing, service demand, activity, financial performance and organisational learning and growth. Reference Sugarman2 The implementation of a balanced scorecard in psychiatric services has previously been described Reference Sugarman and Watkins3,Reference Coop4 but measures of clinical effectiveness have been problematical Reference Coop4 and have thus far been absent from these accounts. Below, we describe the integration into a balanced scorecard of two performance indicators using aggregated data from routinely performed outcome measures, and present service-level data demonstrating the usefulness of the key performance indicators for service providers and other key stakeholders.
Although the routine use of outcome measures has not been shown to bring about improvements in outcomes such as patient-rated unmet need and quality of life, there is promising evidence that in-patient days and thus service cost are reduced. Reference Slade, McCrone, Kuipers, Leese, Cahill and Parabiaghi5 There is consensus that outcomes measurement needs to be multidimensional Reference Cohen and Eastman6,7 both in terms of the outcomes measured and the viewpoints or perspectives considered, including those of service user, clinician or other stakeholders. Numerous items have been used or nominated to support outcome-focused clinical key performance indicators including frequency of unplanned discharges and readmissions, incidence of injurious behaviour and self-harm, transfers, use of restraint, seclusion and number of falls, Reference Thomson, Taber, Lally and Kazandjian8 provision of pre-admission care, Reference Dausey, Rosenheck and Lehman9 medication use and insurance claims. Reference Gilbody, House and Sheldon10 For those with severe mental illness, proposed outcome indicators include routine analysis of needs assessment measures Reference Phelan, Slade, Thornicroft, Dunn, Holloway and Wykes11 and Health of the Nation Outcome Scales (HoNOS) Reference Wing, Beevor, Curtis, Park, Hadden and Burns12 data at both service and individual level. 7
Health of the Nation Outcome Scales for working-age adults and the related family of variants includes outcome measures for children, for older people, and for those with a learning disability or acquired brain injury. All of the HoNOS variants aim to measure mental health and behavioural functioning in order to provide data to judge the effectiveness of mental health services. The tools perform adequately or better in terms of reliability, validity, sensitivity to change and utility. Reference Pirkis, Burgess, Kirk, Dodson, Coombs and Williamson13 In the UK, use is widespread and increasing: Reference James14 49% of all English mental health service providers were using HoNOS in more than one setting by 2002 and recently all members of the HoNOS family have been taken up into the National Health Service (NHS) Connecting for Health IT Programme. The HoNOS–secure Reference Sugarman and Walker15,Reference Dickens, Sugarman and Walker16 has been developed for users of secure psychiatric services, prisons and forensic community services, where diagnostic heterogeneity, multiple treatment needs and additional offence-related treatment needs adds further complexity to outcomes measurement. Reference Cohen and Eastman6 The HoNOS–secure contains modified versions of the original twelve HoNOS scales, and a separate ‘security scale’ comprising seven additional ratings to track change in the outcome of ongoing clinical risk assessment, and in the need for physical, relational and procedural security measures (Box 1). In short, HoNOS–secure is aimed at all users of secure and forensic services, and is designed to be applicable to these service users in the community also, allowing continuity in tracking outcomes.
Method
Setting
St Andrew's Healthcare is a charity operating in the UK independent healthcare sector, providing a range of specialist mental health services for NHS patients with many extended care pathways: men's, women's and adolescent services covering mental health and learning disability; Kemsley, the National Centre for Brain Injury Rehabilitation; Townsend services for older adults; and also the St Andrew's low secure adult mental health service in Essex. The six services, totalling approximately 500 beds, all provide a level of secure care.
Measures
Since 2004, all six services have used HoNOS outcome measures, with HoNOS assessments routinely conducted for all individuals at the end of a pre-admission assessment, within 7 days of admission, and thereafter at 3 monthly intervals until a final rating is made at discharge. Rating is undertaken by the individual's care team as part of the care programme approach process. Individuals are rated using the most appropriate HoNOS tool (HoNOS–secure, HoNOS–LD (for people with learning difficulties), HoNOS 65+ (for people aged 65 years and older), HoNOS–CA (for use with children and adolescents) or HoNOS–ABI (for people with acquired brain injury), collectively referred to as HoNOS specialty scales). Because all services provide a level of secure care, all individuals are rated on the seven-item HoNOS–secure ‘security scale’.
Box 1. HoNOS–secure seven-item ‘security scale’
-
A Risk of harm to adults or children
-
B Risk of self-harm
-
C Need for building security to prevent escape
-
D Need for a safely staffed living environment
-
E Need for escort on leave (beyond secure perimeter)
-
F Risk to individual from others
-
G Need for risk management procedures
Each item is scored on a 0–4 scale using a glossary provided. Reference Sugarman and Walker15
Aims
Two key performances indicators of clinical outcome, covering overall progress and need for secure care, were added to an existing balanced scorecard at St Andrew's Healthcare in January 2004. We present aggregated outcome data across clinical services at St Andrew's Healthcare from 2004 to 2007 in order to demonstrate the value to service managers. Although the performance indicators are based on aggregated data, and not amenable to inferential statistics, we hypothesised that face validity of the service-level measures would be demonstrated if:
-
• change within different patient cohorts were to differ along reasonably predictable lines, with less change among more intractable patient groups such as those with dementia, and more among those with most potential to change such as those in the adolescent service;
-
• change over time, both within and across services, were to be relatively stable, and for this stability to increase as quality of data collection improved.
Key performance indicator calculations
Two key performance indicators, each indicating an average 90-day improvement rate, are calculated quarterly from individual patient HoNOS specialty and HoNOS–secure ‘security scale’ assessments. The method of calculation for the HoNOS–specialty scales and for the HoNOS–secure ‘security scale’ is the same, being the change in score as a proportion of the total possible score. This captures the direction and size of change in a cohort, and allows a broad comparison across services using different HoNOS specialty scales. Each quarterly data collection therefore captures the current patient cohort, and picks up at ward- and service-level any change in the patient group under treatment. Because the indicator is calculated to identify change rather than overall levels of well-being, it is not overly sensitive to fluctuations brought about by additions to and subtractions from the cohort through admission and discharge.
Results
Data for 2004–2007 were collated and 90-day change rates calculated as outlined above for HoNOS specialty and HoNOS–secure ‘security scale’ ratings. The mean successful completion rate of ratings within 3 months for the entire 1-year period for 2004 was 65% rising to 83.0% in 2007.
Figures 1 and 2 show 90-day change rates calculated as annual averages for the HoNOS specialty scales and the HoNOS–secure ‘security scale’ respectively. Relatively low completion rates in 2004 are reflected in the wide range of change across services, and this range generally narrows over time for both key performance indicators. The average 90-day change rate across the study period for the entire St Andrew's Healthcare service hovers around or just under 1% for HoNOS specialty scores and at around or just over 1% for the HoNOS–secure ‘security scale’. Figures 3 and 4 show average annual 90-day change rate by service for the HoNOS specialty and HoNOS–secure ‘security scale’ key performance indicators respectively. As predicted, most change is noted among people using the adolescent service and least among elderly people. The six services were ranked in order of ‘security scale’ total scores (‘need for security measure’) and in order of behavioural and psychiatric symptomatology totals (from HoNOS specialty scales). These rankings were positively correlated (Spearman's rank correlation coefficient 0.6). Positive change over time in symptomatology is demonstrated for the low secure mental health cohort, but not in the need for security measures (Fig. 4). This may be because this smaller service is integrated more with external care pathways, with less opportunity for individuals to progress through levels of security within the service. However, it should be noted that the small negative 90-day change rate on the HoNOS–secure ‘security scale’ key performance indicator for this service is largely because of a negative outcome in year 1 possibly as a result of data collection and calculation issues. Excluding year 1, the low secure service actually achieved modest improvements across the period 2005–7.
Health of the Nation Outcome Scales (HoNOS) specialty scales: 90-day change scores.
Health of the Nation Outcome Scales (HoNOS)–secure ‘security scales’: 90-day change scores.
Discussion
Two performance indicators based on routinely collected data from HoNOS specialty and the HoNOS–secure ‘security scale’ provided key information about the performance of clinical services and were an important addition to a balanced scorecard of indicators of business performance. It was possible to detect relatively small quarterly changes to overall health rating and security needs, suggesting HoNOS–secure is sensitive to change among a group of individuals referred to our specialist services from the NHS. As would be expected, greater change in both health and security status was found among people using the adolescent services where symptomatology and functioning problems are less entrenched.
Health of the Nation Outcome Scales (HoNOS) specialty scales: 90-day change scores average by service.
Health of the Nation Outcome Scales (HoNOS)–secure ‘security scales’: 90-day change scores average by service.
Limitations
Initial significant problems with data collection and completion have been identified, including return within appropriate time scales. Additionally, some errors in data processing have been identified and addressed. The solutions in these areas are for all HoNOS ratings to be ‘owned’ by the clinical teams and for easy-to-use electronic systems with good management. The now complete introduction across the charity of electronic patient records promises to further improve outcome data collection. There has been considerable recent debate about patients’ own subjective rating of outcome and its ability to effect change, and measures such as the ‘Recovery Star’ 17 could be included alongside existing performance indicators in a balanced scorecard using calculations demonstrated here.
Implications
The use of key performance indicators based on routinely collected outcome data promotes a transparent, open-book culture about outcomes. It allows healthcare providers to monitor performance at the same time as giving clinicians the freedom to develop therapeutic programmes that can best deliver real health improvement. More widespread use of similar performance indicators among other providers of secure and forensic services would facilitate benchmarking and further improvement.
Declaration of interest
P.S., L.W. and G.D. are employees of St Andrew's Healthcare.
Acknowledgements
We are grateful to all consultants, care coordinators and other clinical staff at St Andrew's Healthcare who have made this work possible by contributing to routine outcome measures collection using HoNOS specialty measures and the HoNOS–secure ‘security scale’ since 2004.
Open access
eLetters
No eLetters have been published for this article.