From April 2009 it has become mandatory for National Health Service (NHS) acute trusts to collect patient-reported outcome measures (PROMs) before and after certain elective procedures (unilateral hip replacements, knee replacements, groin hernia and varicose vein surgery). 1 Both the preoperative and postoperative PROM questionnaires comprise a generic health status measure (EQ-5D) and a condition-specific health status measure (e.g. the Oxford Hip Score). The aggregate PROM data can potentially be used by managers, regulators and commissioners to benchmark providers' performance, and in theory the disaggregated PROM data could be used by clinicians to help improve patient care, although whether mechanisms for the latter have been put in place is unclear. It is anticipated that the scope of PROMs will be widened significantly by the Department of Health in 2010 and 2011 and the roll-out to mental health services has been explored. Reference Jacobs2,3
Although there is consensus that outcomes should be routinely measured in mental health services, there is in fact limited evidence that routine outcome measurement can deliver improvements in local service delivery and patient-level care. Reference Gilbody, House and Sheldon4-Reference Gilbody, House and Sheldon6 The evidence from randomised controlled trials suggests that one-off or infrequent outcome measurements have little effect on improving quality of life or other subjective secondary outcome measures. Reference Gilbody, House and Sheldon7 However, outcome measurement that is done more regularly (more than once or twice) can significantly improve patient quality of life and reduce psychiatric admissions. Reference Ashaye, Livingston and Orrell8,Reference Priebe, McCabe, Bullenkamp, Hansson, Rossler and Torres-Gonzales9 One outcome measure, the Health of the Nation Outcome Scales (HoNOS), has been mandated for use by mental healthcare providers in England. Reference Slade, McCrone, Kuipers, Leese, Cahill and Parabiaghi10 This 12-item measure was developed by the Royal College of Psychiatrists' Research Unit for people with severe and enduring mental illness. It forms part of the Mental Health Minimum Dataset (MHMDS) introduced in April 2003, the collection of which is mandatory. As a minimum, HoNOS is recommended for use for all patients with more complex needs, with at least one measurement taken per year. Despite this, not all providers complete HoNOS and coverage varies widely among those that do.
This work aimed to investigate the barriers and incentives to the collection of outcome measures generally, and in particular HoNOS, using both qualitative and quantitative methods. We examined the main advantages and disadvantages of HoNOS, as well as the characteristics of providers that routinely collect HoNOS.
Method
Qualitative
We conducted 28 semi-structured interviews using two NHS providers as interview sites, one which had a relatively high HoNOS coverage and the other a lower HoNOS return. We conducted four to six interviews at each site. In addition, we interviewed policy makers and experts involved in outcome measurement, including the College Research Unit, the Department of Health, the National Institute for Mental Health in England (NIMHE), the Care Services Improvement Partnership and the Service User Research Group in England. We interviewed managers, commissioners, psychiatrists and other clinical staff involved in the collection of outcomes data.
Quantitative
We investigated the variation in HoNOS scores across mental health providers in order to determine the factors associated with good coverage of HoNOS, using as our dependent variable the percentage of HoNOS in properly integrated records from the MHMDS for 2004/5. The unit of analysis was the 84 mental health trusts and primary care trusts that provide mental health services. We constructed a database containing a variety of explanatory variables from the MHMDS, the Healthcare Commission Mental Health Survey, Hospital Episodes Statistics and Hospital Activity Statistics. The MHMDS data covered variables on care level, ethnic coding and care prevalence. The Healthcare Commission survey data covered variables on patient care and treatment, medications, contact with health professionals, support in the community and crisis care. Hospital Episodes Statistics data covered number of admissions and the proportions of these that are emergency and day cases, as well as data about waiting times and length of stay. The age profile of patients, bed occupancy and number of bed-days were extracted from the Hospital Activity Statistics data.
From the interviews it emerged that Foundation Trust status was a key factor in the uptake of routine outcome measurement. This status is the voluntary application for greater autonomy from central control given to high-performing trusts. In the period leading up to application for Foundation Trust status, hospitals have to provide evidence of value for money, and outcome measurement was seen as having a key role. Mental health trusts only became Foundation Trusts from 2005/6 onwards, but would have been engaged in the preparation for application in 2004/5. Hence we included a variable to take account of trusts that would apply for Foundation status in 2005/6. This data came from the regulator Monitor.
The model was also run with and without a set of dummy variables representing the strategic health authority codes of the providers included. The strategic health authority effects might pick up higher-level factors such as data quality and performance management. The final database contained a total of 236 variables.
Statistical analysis
A linear probability model was estimated using ordinary least squares analysis. The coefficients on the explanatory variables indicate how the probability of a mental healthcare provider having HoNOS scores in properly integrated records in the MHMDS changes with changes in the characteristics of the provider. We tested several explanatory models using variance inflation factors (VIF) of each, the Ramsey Regression Equation Specification Error Test (RESET) and the R Reference Jacobs2 -test, in order to determine the most appropriate model. The VIF is a measure of the multicollinearity in a regression (whether variables tend to measure the same concept), whereas the RESET is a general test for specification. Reference Fonagy, Matthews and Pilling11 The R Reference Jacobs2 -test is a measure of goodness-of-fit or how much of the variation in the model is accounted for by the explanatory variables in the model. Table 1 gives a description of the explanatory variables included in our final model as well as the descriptive statistics.
Description, source and descriptive statistics of variables included in model
| Variable | Description and source | Obs. | Mean | s.d. | Min. | Max. |
|---|---|---|---|---|---|---|
| HoNOS | Percentage HoNOS in properly integrated records (MHMDS 2004/5) | 84 | 5 | 10 | 0 | 56 |
| Foundation Trust status | Dummy variable of mental healthcare providers who applied for Foundation Trust status (Monitor – Independent Regulator of NHS Foundation Trusts) | 85 | 0.15 | 0.36 | 0 | 1 |
| Age | Mean age, years (HES 2004/5) | 83 | 49 | 8 | 18 | 79 |
| Admissions | Total number of admissions (HES 2004/5) | 83 | 3249 | 4928 | 559 | 40 233 |
| Day-case rate | Day cases as a proportion of total admissions (HES 2004/5) | 83 | 0.02 | 0.06 | 0 | 0.41 |
| Waiting time | Mean waiting time, days (HES 2004/5) | 83 | 38 | 201 | 0 | 1833 |
| Mental health beds occupancy | Percentage of mental health beds that are occupied (HES 2004/5) | 83 | 87 | 7 | 66 | 100 |
| Available mental health beds | Number of available beds for mental health (HES 2004/5) | 83 | 366 | 235 | 44 | 1115 |
| Length of stay | Mean length of stay, days (HES 2004/5) | 83 | 62 | 25 | 6 | 124 |
| CPN contact | Average number of contacts per patient with a CPN (MHMDS 2004/5) | 74 | 9 | 4 | 1 | 21 |
| Ethnic coding overview | Percentage of records in the MHMDS with an ethnic coding (MHMDS 2004/5) | 84 | 63 | 21 | 7 | 100 |
| CPA review | Percentage of patients with CPA review (MHMDS 2004/5) | 70 | 31 | 23 | 0 | 81 |
| Detained under the Mental Health Act in previous 12 months | Percentage of patients in the survey who were detained under the Mental Health Act in the previous 12 months (HCMHS 2005) | 81 | 7 | 3 | 2 | 16 |
| Patient had a say in care | Percentage of patients who felt they had a say in their care and treatment decisions (HCMHS 2005) | 81 | 40 | 5 | 27 | 52 |
| Seen CPN in previous 12 months | Percentage of patients in the survey who saw a CPN in the previous 12 months (HCMHS 2005) | 81 | 58 | 13 | 32 | 87 |
| Contact with service for less than 1 year | Patients who had contact with mental health services for 1 year or less (HCMHS 2005) | 81 | 16 | 6 | 6 | 51 |
| Seen psychiatrist in previous 12 months | Patients who saw a psychiatrist in the previous 12 months (HCMHS 2005) | 81 | 84 | 7 | 62 | 96 |
Results
Qualitative
A number of themes emerged from the interviews, most notably concerning the barriers and incentives to the introduction of outcome measurement, and the advantages and disadvantages of HoNOS. Specific anonymised quotes from interviewees are included.
Barriers
Probably the most crucial barrier to the introduction of outcome measures is that clinicians are unable to see the clinical relevance of such measures. This is partly because they have not been given a clear rationale for their use, partly because they are simply told to complete scores, but primarily because they never receive any feedback on them. There needs to be clear communication about the clinical benefits, but these would best be appreciated if feedback were received in a clinically useful and timely manner.
‘[Clinicians] don't fully understand what outcome measurement can potentially deliver.’
One-off measurements simply provide a snapshot of case mix or severity, thus undermining any useful clinical feedback as a proper outcome measure. Only with repeat ratings at times T 1 and T 2 can scores properly be used as an outcome measure. In one of the trusts with the highest HoNOS collection rates, only 6% of all episodes had a paired HoNOS rating (at two time points).
If feedback is not received, even on one-off measurements, completion rates will be poor and the benefits will not be appreciated. Often clinicians produce data that are then used only at aggregate level; hence it becomes extremely difficult to maintain enthusiasm for what is seen as a paper-filling exercise for managers. Many view the completion of forms as akin to ‘pouring valuable clinical information into a black hole’. There were examples of where feedback mechanisms had been introduced in services with varying degrees of success. However, for clinicians to derive the most clinical value from outcomes data, they were often required to ‘design’ their own computing systems. This was often done on clinicians' own initiative with no trust support. The lack of feedback is associated with the inability of information technology (IT) systems to provide appropriate, useful and timely clinical feedback. Many IT systems crash often, are unreliable and require substantial investment.
‘The networks are down around 20-30% of the time. People don't see IT as the solution, rather the problem.’
Clinical teams often felt that informatics teams did not appreciate the clinical requirements that were deemed important and did not necessarily see the value of getting the most out of the HoNOS scores within the MHMDS.
‘One can't even get one's caseload on the electronic system, so there is no belief that one could get nicely labelled graphs of routine outcome measures.’
There was also concern that many clinical staff lack the necessary IT skills to interact effectively with informatics systems.
In terms of other barriers to the uptake of HoNOS, most clinicians did not see the time element to completing HoNOS as a major barrier. It was felt that for the most part it was quick and easy to complete.
‘Once you get used to HoNOS it only takes a couple of minutes. Once you've built it into what you do, it's a doddle.’
In addition, clinicians did not see training in the use of HoNOS as a major barrier. However, it became clear during the course of the interviews that many staff were not formally trained in the use of this measure. This can have serious implications for the reliability of ratings, the validity of scores and their interpretation. However, with the many competing demands on resources, the cost of training staff to use HoNOS was not always seen as a high priority.
Incentives
Several possible incentive mechanisms to increase the uptake of outcomes were explored with interviewees. Ideally, commissioners should purchase healthcare according to the health outcomes achieved by providers and drive the process of getting routine outcome measures embedded in practice. However, comments around commissioners included the following:
‘Extremely weak in mental health… nobody trusts them, therefore nobody will listen to them.’
Although commissioners said that they were becoming more proactive in encouraging providers to use HoNOS, this was not the impression obtained from the providers and it was felt that until HoNOS was seen as a ‘must-do’ by commissioners, they were unlikely to help increase coverage. In contrast, the Healthcare Commission - now the Care Quality Commission (CQC) - was seen as driving the process of outcome collection, particularly if it became a biting target in the Commission's performance management regime. The general view was that a poor set of ratings would reduce the ability to apply for Foundation status and have serious repercussions.
On the back of making an Foundation Trust application, many trust boards were strongly promoting the use of HoNOS because of its mandatory collection as part of the MHMDS and their desire to show compliance. One interviewee argued that their trust board had not been interested in outcome measurement prior to the application, but that it had suddenly become a priority. However, once Foundation status had been achieved, it was felt that trust boards had become less interested in continuing to push for outcome measurement implementation. If anything, the Foundation Trust status had weakened the incentives for mandatory routine outcome measurement in trusts.
Payment by results is a prospective method of financing providers being introduced in the NHS whereby a fixed payment is associated with a particular set of case-mix adjusted activities. If payment by results is in place, healthcare could potentially be purchased by commissioners at a fixed tariff according to the outcomes being achieved. The extension of this method of financing to mental health services remains a key objective for the government. Interviewees felt that if HoNOS were attached to the payment by results tariff as an incentive to increase coverage, completion rates would be extremely high. Those enthusiastic about the widespread introduction of routine outcome measurement said that they would ‘surf this wave [payment by results], however evil it might be’. However, others argued it might lead to perverse incentives in collecting accurate outcomes data, particularly given that HoNOS is a clinician-rated tool rather than a patient report measure. Therefore, outcome measurement must be introduced with the aim of improving clinical work and not just as an aid to a financial tool. Finally, one of the words that emerged often when asking about barriers and incentives to outcome measurement was ‘culture’.
‘People are afraid of others viewing their outcome measures - you want a culture where people aren't afraid. We're a long way off that.’
One trust was encouraging all their clinicians to publish their outcome measures on their website, although there was considerable resistance. It was suggested that peer pressure could also be an effective means of increasing uptake.
‘When we make it mandatory for all clinicians to publish their outcome measures on the web, then we will see the ratings being used more consistently and more widely.’
Incorporating outcome measurement into the consultant appraisal was seen as another way to encourage ratings. However, many clinicians had concerns over outcome measurement being used for performance management.
‘Using outcome measures for constructive purposes rather than for finger-pointing will be very, very important.’
One of the key challenges is to establish an outcome-oriented clinician culture.
Advantages and disadvantages of HoNOS
Although HoNOS was viewed as one of the best-supported measures available, there were mixed views about its usefulness. Clinicians who used HoNOS were either told to do so by their trust and did not favour it as their instrument of choice, or alternatively were enthusiastic about it. They felt HoNOS was the best validated, tested and socially relevant outcome measure available, rooted in a medical model based on what clinicians regard as important aspects of care, covering symptoms but also other important domains such as occupation, relationships, accommodation and social inclusion. They felt that HoNOS picks up the patient's condition at an acute phase as well as at discharge and is reasonably sensitive. Others felt that HoNOS is a blunt measure and that it underrepresents the user's perspective. In addition, the growth in specialist services in mental health was seen to be at odds with the drive to use broad non-specific measures such as HoNOS, which were deemed to be too general and not sensitive enough to detect change.
Ultimately, clinicians would always differ in their views on the instruments they liked or did not like, and it was argued by some that since no consensus could ever be reached, one might as well choose a reasonably well-validated instrument such as HoNOS.
‘I don't see why we shouldn't draw a line in the sand, and get on with it.’
Quantitative
The MHMDS provided data on HoNOS completion rates for 2004/5 and 2005/6. Table 2 sets out descriptive statistics for providers who completed HoNOS. This shows that 44 providers returned HoNOS scores in 2004/5 but that this had dropped to 37 providers the following year. Coverage, for those who completed HoNOS, also dropped slightly, from 9.6 to 9.4%.
Descriptive statistics for providers who completed Health of the Nation Outcome Scales (HoNOS), pooled for both years and individually for 2004/5 and 2005/6
| Variable | Obs. | Mean (s.d.) | Minimum | Maximum |
|---|---|---|---|---|
| Percentage HoNOS in properly integrated records in MHMDS completed 2004/5–2005/6 | 81 | 9.53 (12.25) | 0.01 | 55.92 |
| Percentage HoNOS in properly integrated records in MHMDS completed 2004/5 | 44 | 9.61 (13.11) | 0.01 | 55.92 |
| Percentage HoNOS in properly integrated records in MHMDS completed 2005/6 | 37 | 9.44 (11.32) | 0.02 | 45.55 |
Table 3 gives the results of the linear regression model. Results were robust to various specifications tested. At a 5% level of significance only three variables are significant: admissions, ethnic coding overview and community psychiatric nurse (CPN) seen in previous 12 months. Providers with higher numbers of admissions have lower rates of collection of HoNOS data, suggesting that perhaps clinicians are too busy to complete this task. Higher admission rates may also coincide with larger trusts where size and coordination issues make for lower outcome collection rates. Ethnic coding overview is a variable that essentially measures data quality. It is one of the key targets in the CQC annual health check for mental health providers. The result suggests that providers who meet this performance standard also have a higher number of HoNOS returns. As we might expect, high data coverage on one item of the MHMDS is associated with high data coverage on another. Thus setting a performance target on HoNOS completion in the MHMDS as part of the CQC annual health check might be one way of increasing completion rates. This result concurs with the qualitative findings.
Linear regression model of Health of the Nation Outcome Scales (HoNOS) 2004/5a
| Dependent variable – percentage HoNOS in properly integrated records in MHMDS | Coefficient | Robust s.e. |
|---|---|---|
| Foundation Trust status | 0.0608 | 0.0480 |
| Age | 0.0023 | 0.0024 |
| Admissions | −0.0047* | 0.0022 |
| Day-case rate | 0.1397 | 0.1589 |
| CPN contact | 0.0030 | 0.0037 |
| Detained under the Mental Health Act in previous 12 months | −0.0044 | 0.0048 |
| Ethnic coding overview | 0.1339* | 0.0626 |
| Waiting time | −0.0063 | 0.0245 |
| Mental health bed occupancy | −0.1686 | 0.1644 |
| Patient had a say in care | −0.0009 | 0.0033 |
| Seen CPN in previous 12 months | 0.0023* | 0.0011 |
| Available mental health beds | 0.0055 | 0.0436 |
| Length of stay | −0.3769 | 0.5323 |
| Contact with service for less than 1 year | 0.0019 | 0.0022 |
| Seen psychiatrist in previous 12 months | 0.0004 | 0.0014 |
| Patients with CPA review | 0.0013 | 0.0009 |
| Constant | −0.1701 | 0.2738 |
The variable ‘seen CPN in previous 12 months’ indicates that if a patient has been in contact with a community psychiatric nurse in the previous 12 months, that provider will have a higher HoNOS coverage. This result could suggest that trusts with a stronger ability to follow up patients in the community have higher HoNOS coverage.
The results of the RESET test indicate that the model is correctly specified and the mean VIF for the model is 1.57, which suggests low levels of multicollinearity. The strategic health authority effects did not alter the results and so are not included.
Discussion
The Department of Health and the Care Services Improvement Partnership, as part of NIMHE, have run a national outcomes measures project. Reference Jones12-14 They have surveyed all mental health trusts in England to establish a compendium of outcome measures in mental health services, which outlines the available measures, their advantages and disadvantages, their psychometric properties and copyright issues, although no particular suite of measures has been mandated. Reference Selbie15 The only measure that has been mandated is HoNOS as part of the MHMDS, although collection rates are poor. Ironically, there was still some confusion among providers we spoke to as to whether HoNOS was mandatory or not. There is a lack of clarity within the service about what the minimum requirements actually are. Some trusts were using this perceived lack of clarity as a reason not to push for implementation. The guidance from the Department of Health on HoNOS within the MHMDS needs to be clearer.
From the Department of Health's point of view the use of HoNOS is not as widespread as it would like. If indeed PROMs is to expand to mental health services, it is likely that HoNOS would receive a greater priority from the Department in the future, given that the measure is currently the most widely used tool in the service. Although HoNOS is clinician-rated rather than a patient report measure, it is conceivable that it will be the most likely contender for any mandatory expansion of outcome measurement; however, it is possible that other condition-specific and possibly self-report measures might be encouraged for use alongside. If this were the case, HoNOS completion within the MHMDS as a CQC target would be one of the most effective ways to drive forward this process, as suggested by both our qualitative and quantitative results. With the introduction of PROMs and payment by results in acute services, there is a serious concern that commissioners might slowly divest themselves of mental health services because they can more readily see what value for money they are getting in other areas of healthcare. Reference Persaud and Pearson16 Mental health commissioning for outcome measurement needs to become a priority to ensure continued investment.
Information technology systems in many trusts are poorly developed and cannot support any kind of routine outcome measurement. 17 If the drive for routine outcome measurement is to be pursued in earnest, this needs to be addressed urgently, with additional resources and clear guidance. If trusts are expected to make investments in IT systems in weak local health economies with overspent primary care trusts and retracting budgets, no real progress will be made. At provider level, feedback mechanisms need to be found that can supply timely and useful clinical feedback. A dialogue needs to take place about clinical requirements and informatics capabilities. Management support is imperative to help provide the resources needed to embed outcome measurement into routine practice to support clinical decision-making in the first instance, rather than solely to support management decisions. Top-down drives to enforce routine collection will rarely be effective. From the clinician's point of view this would be seen as a form-filling waste of time, and it would be important to strike a balance by generating clinically useful patient-level feedback for clinicians along-side aggregate data for managers. Similar conclusions can be drawn from the body of literature investigating barriers to implementation of new models of working in healthcare, 18,Reference Appleby19 which indicate the need for a shift in the work practices of both clinicians and managers.
Finally, and most challengingly, an outcome-oriented culture needs to be developed, driven by the desire to learn about improving service quality rather than by the urge to benchmark, league-table and remove ‘failing’ services. This culture might, however, incorporate some aspects of mild coercion and peer pressure, provided this takes place in a developmental and learning culture.
Funding
This research was commissioned and funded by the Office for Health Economics Commission on National Health Service Productivity. R.J. also holds a fellowship from the Department of Health's Research and Development Programme examining performance measurement in mental health services.
Acknowledgements
We thank all the interviewees who participated in this study. Ethical approval was obtained from Leeds (East) Research Ethics Committee. Veena Raleigh and Giovanna Polato from the Care Quality Commission and Gyles Glover from the North East Public Health Observatory kindly provided data. We are grateful to participants at the Ninth Workshop on Costs and Assessment in Psychiatry: Quality and Outcomes in Mental Health Policy and Economics (Venice, Italy, March 2009) for useful comments as well as to two anonymous referees. The opinions in this article are those of the authors alone, not those of the OECD nor of its member countries.
Open access
eLetters
No eLetters have been published for this article.