The importance of outcome measurement has been increasingly acknowledged in demonstrating progress on an individual patient level. In addition, outcome measurements play a significant role in service evaluation and policy effectiveness. The UK Government White Paper The Health of the Nation (Department of Health 1992) identified mental health as one of five key areas in which strategies to reduce morbidity and mortality were to be prioritised. One of the aims was to ‘improve the health and social functioning of mentally ill people’. This was echoed in the National Service Framework for Mental Health (Department of Health 1999). A sensitive and reliable instrument that covered common clinical problems and social functioning was required to quantify progress and improvement in mental health. The Health of the Nation Outcome Scales (HoNOS; www.rcpsych.ac.uk/training/honos.aspx) were developed to meet these needs (Reference Wing, Curtis and BeevorWing 1996). The scales were created as a simple-to-use, brief measure that could be completed as a routine clinical indicator of change in mental health outcomes within a minimum data-set containing other relevant information. The most obvious clinical use of HoNOS, according to the authors, was as a current record of patient progress – a ‘present state profile’. The administrative performance of HoNOS as a single measure was to be borne out in the real world. The authors also saw the potential for use of the scale for epidemiological and administrative purposes in the absence of another simple measure. A variety of mental health professionals have received training in the clinical or research use of HoNOS, including psychiatrists, nurses, psychologists, social workers and occupational therapists.
Method
The aim of this review is to describe the development and subsequent evolution of the HoNOS and to provide a narrative account of its current use in psychiatric practice as evidenced in published literature. To preserve its comprehensive nature, we searched multiple databases for all articles involving the use of any version of the HoNOS in English, resulting in 585 potential articles. (The full search strategy is available on request.) The search resulted in a heterogeneous group of articles that were grouped under the following subheadings for the purposes of this review:
-
• development and evolution
-
• development into other languages
-
• psychometric properties
-
• qualitative aspects
-
• use of HoNOS as an outcome measurement for service evaluation and in interventional studies.
Brief description of HoNOS
The HoNOS is completed after a comprehensive assessment or review of the patient by the clinician. Reported completion times for the HoNOS form vary from 2–3 min for HoNOS to 8–15 min for other versions, such as HoNOS for Children and Adolescents (HoNOSCA; Reference Gowers, Harrington and WhittonGowers 1999) and HoNOS65+, for older adults (Reference Burns, Beevor and LelliottBurns 1999). The variation is likely to be attributable to individual differences between clinicians, including familiarity with the forms and availability of background information. The clinician rates a patient on 12 items (Box 1), each scored on a scale of 0 (no problem) to 4 (severe or very severe problem), with a maximum total score of 48 and ratings that cover a 2-week period. It is designed to measure change in response to an intervention. A software application (HoNOSSoft) is available to analyse and print results.
Items against which patients are scored in HoNOS
-
1 Aggression
-
2 Self-harm
-
3 Drug/alcohol problems
-
4 Cognitive impairment
-
5 Physical problems
-
6 Hallucinations and/or delusions
-
7 Depression
-
8 Other psychological symptoms
-
9 Social relationships
-
10 Activities of daily living
-
11 Accommodation problems
-
12 Employment and leisure problems
Development and evolution
The Royal College of Psychiatrists’ Research Unit (now the College Centre for Quality Improvement) was commissioned to develop a set of scales that was to be used in routine clinical practice. It created HoNOS-1, a 20-item instrument covering four key areas of functioning: behaviour, impairment, symptoms and social functioning (Reference Wing, Beevor and CurtisWing 1998). This was refined during four stages of testing over 3 years, including a field trial on 2706 patients, resulting in the development of the final 12-item HoNOS for use in general psychiatry.
HoNOS65+ and HoNOSCA were later developed for use with older adults and children and adolescents respectively. HoNOS-secure version 2b was developed in 2007 for use in forensic services (Reference Sugarman and WalkerSugarman 2007). The unpublished HoNOS-ABI (acquired brain injury) targets psychiatric and other sequelae of brain injury, and HoNOS-LD Footnote ‡ (Reference Roy, Matthews and CliffordRoy 2002) is used for measuring outcomes in people with intellectual (learning) disabilities and additional mental health needs. The HoNOS has also been modified to produce a self-report and other versions of HoNOS that assume significance particularly within the context of recovery and well-being (Reference StewartStewart 2009). More recently, HoNOS payment by results (HoNOS-PbR) is being developed (Department of Health 2010), and this is likely to play a role in mental health funding in the National Health Service in England.
Development into other languages
The HoNOS has been translated into many (mostly European) languages, including German, French, Spanish and Italian, and some authors have taken into account cultural influences in adapting the scales for use in other countries. There are mixed reports of the validity and reliability of HoNOS when used in other languages and further studies are required (Reference Andreas, Harfst and DirmaierAndreas 2007; Reference Phuaphanprasert, Srisurapanont and SilpakitPhuaphanprasert 2007).
Comparing HoNOS with other scales
Several studies have compared HoNOS with other outcome measurements, including the Clinical Outcomes in Routine Evaluation – Outcome Measure (CORE-OM; Reference Barkham, Evans and MargisonBarkham 1998) and the Camberwell Assessment of Need Short Appraisal Schedule (CANSAS; Reference Slade, Beck and BindmanSlade 1999). Comparisons of HoNOS with measures of psychiatric symptoms and social functioning using the Schedules for Clinical Assessment in Neuropsychiatry (SCAN), Brief Psychiatric Rating Scale (BPRS), Symptom Checklist 90 – Revised (SCL-90-R) and the Social Behaviour Schedule (SBS) have also been reported in the literature (Reference Bebbington, Brugha and HillBebbington 1999; Reference Adams, Palmer and O'BrienAdams 2000). Varying levels of correlation have been found between HoNOS and other outcome measurements; correlation is frequently low, often because of the lack of directly comparable items. Some studies have, therefore, found improved correlations when evaluating subscales, rather than the complete instrument.
Psychometric properties
An ideal outcome measurement should have high validity, the ability to discriminate between groups and sensitivity to change. It should be comprehensive, user friendly and allow comparison between services.
Limitations within HoNOS
Although HoNOS is considered to be well designed and generally appropriate in the main, concerns have been voiced about restriction on the rater to indicate only one problem in item 8 (Other psychological symptoms) and questions raised about the ability of item 6 (Hallucinations and/or delusions) to accurately describe symptoms in a patient with schizophrenia (Reference Pirkis, Burgess and KirkPirkis 2005a). The social items were considered problematic because of the complexity of information needed to rate them. Difficulty in knowing which item to use for rating some symptoms, such as elated mood, was also reported.
Internal consistency
Studies report the internal consistency of HoNOS to be moderately high (Cronbach’s α=0.59–0.76; Reference Pirkis, Burgess and KirkPirkis 2005a). In terms of relative weight to overall score, items 7 (Depression), 8 (Other psychological symptoms) and 9 (Social relationships) contribute 14–19%, whereas items 11 (Accommodation problems) and 12 (Employment and leisure problems) contribute to a lesser degree.
Test–retest reliability
Studies examining the test–retest reliability of HoNOS report fair to moderate scores (Reference Orrell, Yard and HandysidesOrrell 1999). However, low reliability has been reported for the following items: 1 (Aggression); 3 (Drug/ alcohol problems); 7 (Depression); and 10 (Activities of daily living).
Interrater reliability
Reference Amin, Singh and CroudaceAmin et al (1999) explored the interrater reliability of HoNOS, demonstrating high correlation between trained researchers and key workers who had had only brief training. Poor reliability has been demonstrated for items 4 (Cognitive impairment), 7 (Depression), 8 (Other psychological symptoms) and 9 (Social relationships) (Reference Bebbington, Brugha and HillBebbington 1999; Reference Orrell, Yard and HandysidesOrrell 1999). Interrater reliability is influenced by the clinician’s interpretation of the glossary as well as their experience, training and enthusiasm (Reference MacDonaldMacDonald 2002).
Sources of information
Variations in the source of information used by clinicians can affect reliability on items 11 (Accommodation problems) and 12 (Employment and leisure problems). Information used by clinicians when completing HoNOS is most often obtained from interviewing the patient or through direct observation. The main secondary sources include medical records and consultation with family, carers and other staff.
Examination of HoNOS has shown it to be sensitive to change, with the greatest degree of change reflected in in-patient settings compared with community settings (Reference Audin, Margison and ClarkAudin 2001). A decrease in HoNOS score over time corresponds with patients’ self-ratings that their goals have been met. The total HoNOS score is a relatively crude measure; there can be dramatic reductions in some of the scale scores with compensatory increases in others, reflecting major clinical change even though the total score remains the same (Reference MacDonaldMacDonald 2002).
Training in the use of HoNOS
The importance of training in the performance of HoNOS and the resource implications of large-scale training were raised in the early days following its development. Trained researchers achieved greater reliability than trained key workers with related clinical inexperience. However, adequately trained key workers performed better than key workers with less rigorous training (Reference Bebbington, Brugha and HillBebbington 1999). Training continues to remain a key issue from the rater perspective, given that it may contribute to a more positive shift in attitudes towards HoNOS and perception of ease of use (Reference Trauer, Callaly and HerrmanTrauer 2009). Both initial training and regular refresher training has been recognised to be important in integrating outcome measurement into routine practice (Reference Meehan, McCombes and HatzipetrouMeehan 2006).
Overview of HoNOS use
This section focuses on the clinical and service utilisation of HoNOS. Despite misgivings about the reliability of HoNOS (Reference Bebbington, Brugha and HillBebbington 1999; Reference Orrell, Yard and HandysidesOrrell 1999), it is widely used in both the UK and Australia. The HoNOS is one of several instruments that form the Mental Health National Outcomes and Casemix Collection, through the Australian National Mental Health Strategy (Reference Pirkis, Burgess and CoombsPirkis 2005b). Our literature search reflects the impression that the majority of studies using HoNOS have been carried out in Australia.
Table 1 illustrates the use of HoNOS globally, in largely observational and service evaluation studies. It highlights the diverse range of settings, sample sizes and study designs in which it has been used. Studies based in the community (43), in-patient (33) and combined community and in-patient settings (7) have all used HoNOS. A variety of professionals, predominantly nurses, psychiatrists, psychologists and social workers, administered HoNOS in these studies. In the main, HoNOS and its adapted versions have been used to evaluate both new and existing services. It is interesting to note that comparisons of models of care or services account for only 9% (7) of studies, given the future prominence that HoNOS may play in benchmarking service outcomes.
Summary of published literature describing HoNOS use by various descriptors
Use in routine clinical practice
Most clinicians do not use standardised measures routinely in their clinical practice (Reference Gilbody, House and SheldonGilbody 2002). Evidence of HoNOS being used in the development of individual care plans is relatively sparse and clinicians were initially cautious about its role in care-planning (Reference Sharma, Wilkinson and FearSharma 1999). The lack of studies on the direct clinical application of HoNOS data in the care of individual patients is borne out in our review.
Several authors have considered issues regarding the routine clinical use of HoNOS and its value. Reference SteinStein (1999) envisaged lack of clinical utility, considering the scales to be of no direct benefit to the patient during the consultation and unable to inform the diagnostic process. He also cited ‘intrusion’ into valuable clinician–patient time as a potential hurdle. Reference Slade, Beck and BindmanSlade (1999) and colleagues argued that assessing morbidity and identifying specific areas for ongoing intervention in an existing caseload and thus differentiating between met needs and no needs would not be possible with HoNOS alone. They recommend using HoNOS with another scale for ongoing interventions to assess unmet needs.
In an illustrative example, Reference Prowse and CoombsProwse & Coombs (2009) describe the implementation of a flag system based on HoNOS scores to prompt decisions regarding discharge/transfer of patients from a community mental health team to general practitioner (GP) care or another appropriate team. Cut-off scores were calculated on the basis of previously published literature describing mean HoNOS scores of patients in GP and shared care settings. There have been attempts to establish and test clinically significant cut-off scores on HoNOS. These studies represent the first steps in the translation of aggregated data to benefit individual patients – a perhaps somewhat surprising finding given that more than a decade has passed since HoNOS was originally developed.
Breadth of use in service evaluation
Since its development, HoNOS has commonly been used to demonstrate clinical and functional changes in patient populations over time. It has found a role particularly in service evaluation research. This may be a result of its relative ease of administration and ability to measure change across diagnostic boundaries. Mandatory collection of HoNOS in some regions and sensitivity to change has most likely influenced the widespread use in service evaluation.
Use has largely taken the form of observational or cross-sectional approaches, often in combination with other outcome measurements. There are examples of comparisons across team settings or models of working. Reference Burgess, Pirkis and CoombsBurgess (2006) advises caution against implying causality. For example, a particular intervention, hospital admission or service reconfiguration can lead to a reduction in scores when HoNOS is used in an observational capacity to measure change.
HoNOS has been used to evaluate diverse and multiple aspects of mental health service provision. Pathways to care, clinical and unmet service needs assessment in specialist psychological services and studies discriminating between levels of disability and severity of mental illness have all utilised HoNOS. It has also been utilised to anticipate differences in use of services (Reference BroadbentBroadbent 2001). HoNOS has been used particularly to demonstrate outcomes in newly established services, for example, early intervention in psychosis services. It has also been used to inform service quality (Reference Burgess, Pirkis and CoombsBurgess 2006) and as a basis for quantifying service costs.
There are a few examples of HoNOS being utilised in large patient populations. Reference Burgess, Pirkis and CoombsBurgess’ (2006) service evaluation study of in-patient and community mental health services in Australia captured 14 000 acute in-patient episodes and more than 23 000 community episodes.
Translating data to inform service provision
Despite HoNOS being used to evaluate diverse aspects of mental health services, many of which it was not originally intended for, there are fewer examples of HoNOS being used in a more meaningful way to inform service provision and reconfiguration on a broader level, despite often significant investments being made in collecting data.
Reference McKay and McDonaldMcKay (2008) reviewed the use of HoNOS65+ in an older adults’ in-patient and community team in Australia. They used HoNOS65+ scores to inform individual care, individual clinical teams and service management such as benchmarking. This led to better understanding of changes in the flow of patient numbers across services.
Evaluation and comparisons of HoNOS data
The evaluation of HoNOS data has gradually grown more sophisticated and statistical analysis has now enabled HoNOS data to be given more clinical and service context, even in observational studies. Reference Burgess, Pirkis and CoombsBurgess et al (2009) highlight the importance of homogeneity of data analysis in enabling comparisons of HoNOS data across services and geographical areas. They examined three indicators of change for statistical analysis, finding that the degree of effectiveness demonstrated depended on the statistical measure used.
HoNOS data have been aggregated to reflect broader changes in scores at a patient and service level to enhance the meaningfulness of the information for clinicians. The level of change in HoNOS scores varies according to clinical setting, with in-patient settings tending to show greater changes compared with community-based services. Degrees of change in HoNOS scores may also depend on other factors, such as diagnosis (Reference Burgess, Pirkis and CoombsBurgess 2009).
Qualitative aspects
Clinicians have largely been pessimistic about what outcome measurement might achieve, both for themselves and for their patients, and about the ability of tools such as HoNOS to measure change. A majority of clinicians do not use outcome measurement in treatment planning or monitoring. Therefore, qualitative studies can highlight potential hurdles to the successful implementation of HoNOS.
Clinician attitude
Reference Meehan, McCombes and HatzipetrouMeehan (2006) conducted focus groups with 324 mental health staff and found that 40% supported the collection and use of outcome measurements, including HoNOS. Whereas 30% expressed ambivalence, another 30% were openly critical and concerned about motives, including reduction of staff numbers. Competing work priorities and lack of support from senior medical staff were other important concerns. Reference Callaly, Hyland and CoombsCallaly (2006) echoed issues of mistrust and clinicians also reported that HoNOS did not provide additional information.
Organisational climate, attitudes of senior staff and stable, consistent leadership are important factors in acceptance and utilisation of outcome data (Reference Meehan, McCombes and HatzipetrouMeehan 2006). Implementing routine use of HoNOS has been found to be challenging, despite clear leadership, training strategies and the endorsement of management teams, with staff remaining sceptical.
Resource issues
Reference Walter, Cleary and ReyWalter (1998) reports that a majority of survey respondents were disinclined to implement routine outcome measurement, despite the possibility of leading to improved service provision, mainly because of time constraints. This sentiment is echoed in other surveys in which staff felt that outcome measurements, including HoNOS, were a ‘waste of time’ and additional paperwork.
Feedback of outcome data is crucial, particularly as staff that have had feedback tend to value outcome measurement more, and feedback also aids clinician engagement (Reference Callaly, Hyland and CoombsCallaly 2006; Reference Trauer, Callaly and HerrmanTrauer 2009). Attention has also been drawn to the importance of presenting data to staff in an easy-to-understand format. It is also important that the data are set within clinical and service contexts; all of this necessitates supporting hardware and software.
Health economic aspects
There is a clear lack of studies examining cost implications of implementing HoNOS across the UK. Reference Slade, McCrone and KuipersSlade (2006) estimated the costs of their use, which included HoNOS as a secondary measure, at £400 per person. Further well-designed studies to assess the cost-effectiveness of implementing HoNOS are needed.
Discussion and future directions
Outcome measurements in mental health have some value in monitoring the progress of individual patients, auditing and determining effectiveness of policy (Reference Slade, Beck and BindmanSlade 1999). There has therefore been a major drive to introduce outcome measurement in healthcare over the past two decades. This is in spite of the long-term debate about the relative merits of outcome v. process evaluation in relation to quality of healthcare. It has been argued that outcome measurements have little objective value outside the context of randomised controlled trials (Reference Tansella and ThornicroftTansella 2001). HoNOS became mandatory as part of the Mental Health Minimum Dataset (MHMDS; NHS Information Centre 2011) in England in 2003, although there have been problems with data quality. As illustrated in this article, it has been adopted for use globally and HoNOS data collection is mandatory in Australia. The current wide-scale use of HoNOS at a service level will undoubtedly increase and diversify further.
Clinicians’ attitudes towards outcome measurement in general and HoNOS in particular include ambivalence and fear of hidden agendas. It is also seen to be burdensome and lacking in clinical usefulness in everyday practice. This probably contributes to the problem of incomplete data-sets (Reference Audin, Margison and ClarkAudin 2001). The lack of a robust evidence base to support the use of routine outcome measurement (Reference Slade, McCrone and KuipersSlade 2006) is another possible hurdle to widespread acceptance and use. The need for supporting information systems that can provide meaningful feedback to the clinician would be crucial to widespread staff acceptance.
There is a complicated interplay between mental health needs and service provision. Clinicians may feel that an outcome measurement like HoNOS does not comprehensively capture the subtleties of both clinical and social function. There are also concerns regarding suitability in some service settings, such as those providing psychological therapies (Reference Audin, Margison and ClarkAudin 2001). All of these factors could lead to clinician reluctance in routine outcome data collection. This may lead to inevitable tensions with managers who are driven by the need to demonstrate service effectiveness.
The Darzi report emphasises quality and clinical effectiveness (Department of Health 2008) and outcome measurements are, therefore, likely to play an integral role in their demonstration. In addition to being an outcome measurement, HoNOS also underpins the planned system to develop mental health payment by results in the form of HoNOS-PbR. It incorporates a further six scales to assign service users to care clusters, designed to indicate level of need. The overall aim is to enable the comparison and benchmarking of mental health services. These classifications will also be used to underpin a national tariff for mental health funding, thus promoting transparency.
Therefore, there can be no doubts as to the importance that will continue to be placed on HoNOS data collection and analysis because of the consequences for funding in the future. However, the challenge remains in creating an outcome-oriented culture, which is necessary to ensure successful implementation.
MCQs
Select the single best option for each question stem
-
1 The HoNOS is one of several instruments that form the Mental Health National Outcomes and Casemix Collection in:
-
a the UK
-
b Australia
-
c the USA
-
d Germany
-
e Canada.
-
-
2 HoNOS is:
-
a used only in hospital settings
-
b clinician-rated
-
c scored over a 6-week period
-
d completed before and after patient review
-
e used only in the UK.
-
-
3 HoNOS does not include the following item:
-
a physical problems
-
b aggression
-
c family history of mental illness
-
d delusions
-
e cognitive impairment.
-
-
4 HoNOS can be of use:
-
a in assessing clinical and functional change following an intervention
-
b in diagnosis
-
c in assessing need
-
d as a patient self-assessment instrument
-
e as a psychotherapeutic intervention.
-
-
5 Barriers to using HoNOS in routine clinical practice identified in qualitative studies include all except:
-
a competing work priorities
-
b lack of support from senior medical staff
-
c lack of information technology resources
-
d fear of a hidden agenda behind data collection
-
e preference for alternative measures to the HoNOS.
-
MCQ answers
| 1 | b | 2 | b | 3 | c | 4 | a | 5 | e |
eLetters
No eLetters have been published for this article.