The use of outcome measures is important to assess and improve the quality of service delivery. There are many well-established and validated outcome measures for use in psychiatric disorders; 1 however, there is no substantial evidence supporting the value of such scales in routine psychiatric practice. Reference Gilbody, House and Sheldon2
The Health of the Nation Outcome Scales (HoNOS) instrument was developed for use in routine clinical practice. 3 It is a clinician-rated scale which measures 12 dimensions on a 5-point (0-4) scale of severity. It is mandated within the National Health Service (NHS) in England as part of the minimum mental health data-set.
The Department of Health in England has set a clear focus on the measurement of patient-reported outcome measures. 4 This study sought to determine the feasibility of introducing patient-reported outcome measures into routine psychiatric community practice and to determine how HoNOS scores compared with three patient-reported measures:
-
• patient-identified problems (PIPs)
-
• EuroQol Quality of Life – 5 dimension (EQ-5D) 5
-
• EuroQol Visual Analogue Scale (EQ VAS). 5
We chose the HoNOS due to its widespread use in the UK and the two quality-of-life measures were selected for brevity and simplicity. A patient-identified problem was used as an individualised measure of patient outcome.
Method
Patients
The sample consisted of patients aged 18 and above attending a mental health community hospital out-patient service in Poole, Dorset, over a period of 3 months. Patients with a diagnosis of dementia were excluded from the study because of potential difficulty in rating the questionnaires.
To detect a correlation of 0.3 between HoNOS and the patient-reported measures with an 80% power, assuming a two-sided 5% significance level and a loss to follow-up of 20%, a total of 105 patients were required for the study.
Questionnaires or outcome measuring scales
For all patients, four outcome measures were collected.
-
• EQ-5D questionnaire, a quality-of-life measure; patients rate functioning on a 5-point scale in five areas: mobility, self-care, usual activities, pain/discomfort and anxiety/depression.
-
• EQ VAS, a quality-of-life measure; patients rate their quality of life by marking a visual analogue scale.
-
• PIP, rated on a 7-point scale (0-6) based on the Personal Questionnaire Rapid Scaling Technique (PQRST); Reference Chalkley and Mulhall6 patients identified up to three current problems and rated each on a 7-point scale, from 0 = no problem at all, to 6 = an overwhelming problem.
-
• HoNOS, a clinician-rated outcome tool measuring symptoms and social functioning.
All patients attending for a first out-patient appointment were requested to identify up to three main problems or difficulties for which they were seeking help (PIPs) and asked to rate each on a scale of 0-6. Patients formulated their problems with their clinician at the end of the assessment and were assisted in developing goals that were SMART (specific, measureable, achievable, relevant and timed). Patients were also requested to complete both EuroQol questionnaires. Clinicians completed a HoNOS at the same time.
A total of 12 psychiatrists and 10 other clinicians provided data for the study. All the professionals had received formal training in completing the scales before the project began.
Simple brief information leaflets about the project were handed to all patients before completing the questionnaires. Further information about the project was available on request from out-patient reception staff and clinicians.
All four measures were completed by the patients and clinicians on every further assessment until 3 months or discharge, whichever occurred first. Clinicians were not masked to the previous ratings.
The initial baseline assessment of the problems was conducted from January to April 2009. The follow-up data were collected until August 2009. It was agreed by the trust clinical research lead that ethical approval was not necessary, as the project was evaluating outcomes and was not a research project.
Analysis
Data were analysed using the Stata7 software version 10 for Windows. The initial first appointment and a total of three follow-up appointment comparison data were analysed. Spearman's correlation coefficients were calculated to determine the bivariate correlations at different time points. The correlations were primarily established for HoNOS against the PIP, EQ-5D and EQ VAS, and at each individual different time point. For the PIP, of the three problems identified and rated, only the one rated highest (i.e. the most severe problem as perceived by the patient) was used in the analysis; this helped simplify the analysis.
Results
In total, 145 patients were eligible for the project (30 patients declined or were not asked to enter the study). Twelve PIP scores had to be discounted, as the patients identified and rated different problems at further appointments. Data were available for 103 patients. The median age of the participants was 44 years (interquartile range 30-58); 63 participants (61%) were female. The median duration for follow-up of appointments was 4 weeks. The number of assessments completed for each measure at each follow-up is shown in Table 1. The top four PIPs could be broadly categorised as:
-
• psychological symptoms (65%)
TABLE 1Number of outcome measures recorded at each assessment
Assessment Outcome measure 1
n = 1032
n = 883
n = 654
n = 40HoNOS 103 85 55 26 Patient-identified problems 98 61 51 19 EuroQol-5D 98 63 51 19 EuroQol VAS 102 68 53 20 n, number of patients attending each appointment; EuroQol-5D, Quality of Life—5 dimension questionnaire; EuroQol VAS, EuroQol Visual Analogue Scale; HoNOS, Health of the Nation Outcome Scales.
-
• social issues (12%)
-
• relationship problems (9%)
-
• physical problems (5%).
Figure 1 shows the mean scores for each scale as a function of follow-up time with 95% confidence intervals. There is a general improvement in the health status from the initial assessment to the first follow-up; afterwards the scores remain stable.
Means for each of the scale over time (the capped spikes represent 95% CI). EQ-5D, EuroQol Quality of Life-5 dimension questionnaire; EQ VAS, EuroQol Visual Analogue Scale; HoNOS, Health of the Nation Outcome Scales; PIP, patient-identified problem.
The correlations between the HoNOS scores and the three patient-recorded measures at initial assessment and follow-up are shown in Table 2.
Spearman correlation coefficients and P values for each time point
| Variables compared | Initial assessment | First follow-up | Second follow-up | Third follow-up |
|---|---|---|---|---|
| HoNOS v. EQ-5D index | n = 98, r = —0.238, P = 0.018 | n = 62, r = —0.405, P = 0.001 | n = 50, r = —0.720, P < 0.001 | n = 18, r = —0.47, P = 0.047 |
| HoNOS v. EQ VAS | n = 102, r = —0.273, P = 0.006 | n = 67, r = —0.473, P < 0.001 | n = 52, r = —0.668, P < 0.001 | n = 19, r = —0.29, P = 0.228 |
| HoNOS v. PIP | n = 98, r = 0.059, P = 0.563 | n = 60, r = 0.197, P = 0.132 | n = 50, r = 0.665, P < 0.001 | n = 19, r = 0.324, P = 0.176 |
EQ-5D, EuroQol Quality of Life—5 dimension questionnaire; EQ VAS, EuroQol Visual Analogue Scale; HoNOS, Health of the Nation Outcome Scales; PIP, patient-identified problem.
The results in Table 2 show that the HoNOS and EQ-5D index correlate at 5% significant levels at each time point, whereas the HoNOS and EQ VAS correlated at the initial assessment and first and second follow-up, but not at the third follow-up. The HoNOS and PIP did not correlate except for the second follow-up. At the third follow-up appointment the number of participants significantly reduced to only around 20% of the initial sample.
Discussion
This study demonstrated that outcomes can be routinely measured in clinical practice but that even with focused clinical leadership there will be missing data, with some patients being unwilling to rate their symptoms/problems and some clinicians forgetting to record or enter the follow-up data.
The good correlation between the HoNOS and the quality-of-life measures suggest that both do not need to be assessed routinely. The EQ-5D has been widely used in physical healthcare. The advantage of using it in mental healthcare would be to allow comparisons of quality-of-life improvement to be made between physical and psychological interventions. Thus, for example, it would be possible to compare the benefits of a hip replacement for arthritis with cognitive-behavioural therapy for depression. In routine clinical practice, however, this study suggests that a specific quality-of-life measure does not provide additional information over and above the HoNOS scores.
Not all patients received follow-up at all points. In part, this reflected patients dropping out of treatment and not returning for follow-up appointments, and in part it was due to a failure by the clinician to complete questionnaires and to ask the patients to repeat the questionnaires. This study was not intended to be a research study demonstrating the effectiveness of the interventions used but rather to answer the question of whether outcome measures can be applied in routine clinical practice. The results demonstrate that indeed they can and that they are of value, but there are likely to be difficulties in ensuring full compliance with completion.
The PIP rating did not reveal statistically significant correlations with HoNOS. The reason for this may be that either the PIP measures a different concern to measurements on the HoNOS, or that it may be more difficult to define and rate reliably compared with the well-validated HoNOS score. The PIP scale was acceptable to both clinicians and patients. Although it was not a focus of this study, asking patients to identify and rate problems provides a focus for clinicians on the key issues rated as important by the patients. This is likely to improve adherence and patient satisfaction.
Clinicians were in principle broadly supportive of the use of outcome measures. Concerns were raised as to how bureaucratic the process would be and to ensure that the results were fed back to both individuals and teams in a meaningful way. The implementation of routine outcome measures will be a cultural change for many organisations and will need clinical leadership to ensure clinician support. Organisations will need to invest appropriately in information technology so that the outcome measures collected can inform the individual patient/clinician interaction as well as being used for monitoring individual and team performance.
The use of PIPs helps ensure that clinicians remain focused on what is important for patients in service delivery. This is at the heart of recovery-based models of care. 7 It may be that a more positive measure of recovery, patient-identified goals, should be used rather than the patient's problem. This will be a focus of further investigation.
This study has led to the routine measure of a patient-focused outcome measure (alongside HoNOS ratings) in Dorset Healthcare University NHS Foundation Trust.
Acknowledgements
Professor Peter Thomas (Professor of Healthcare Statistics and Epidemiology, Bournemouth University) assisted the project with the calculation of the sample size and advice on outcome measure tools.
Open access
eLetters
No eLetters have been published for this article.