The Commission for Health Improvement (CHI) ceased to function at the end of March 2004. This provides the opportunity to review its contribution and achievements as a new body, the Commission for Healthcare Audit and Inspection (CHAI), takes over its functionsFootnote *. CHI recently published its assessment of mental health services (http://www.chi.nhs.uk/eng/news/2003/dec/11.shtml). The report is based on the 35 clinical governance reviews, in England and Wales, published between July 2001 and October 2003; two investigations into serious service failures; and a report on safeguarding arrangements for children in England and a self-audit of child protection arrangements. CHI concluded that mental health services lag behind acute health services in developing clinical governance systems and processes that promote high-quality care and continuous improvement. It specifically highlighted the shortages of psychiatrists and in-patient nurses, and the reliance on agency nurses and locum staff; the unsuitability of buildings and facilities; the pressures on in-patient beds; the lack of management capacity and poor information systems; and the low priority given to services for children and older people.

The Commission for Health Improvement recognised that mental health services have undergone unprecedented change in recent years. It set its report in the context of the changes from institutional to community care, the publication of National Service Frameworks for mental health and the elderly, respectively, and current discussions regarding reform of the Mental Health Act 1983. Furthermore, it recognised the structural and organisational changes in the provision of services including the creation of specialist mental health trusts, the integration of health and social care and the provision of mental health services by primary care trusts.

In this paper, we will examine how CHI fulfilled its role during its existence, the method it used in conducting clinical governance reviews, the problems and limitations of this methodology, the findings of mental health service reviews and the room for change as CHI's role is taken over by CHAI.

CHI's role and methods

The Commission for Health Improvement was established in the Health Act 1999, following the abolition of the Clinical Standards Advisory Group. It described its overarching aim as follows: ‘to help NHS [National Health Service] organisations improve the quality of their services to the benefit of patients’ (Reference Bawden and LugonBawden & Lugon, 2002). On its formation, CHI had four functions. These were: conducting reviews of NHS organisations’ arrangements for clinical governance; investigating specific cases of serious service failures; conducting national service reviews; and providing advice and guidance on clinical governance arrangements. Many have already commented on CHI's self-inventing energy. Its budget doubled from £12 million in 2001-2002 to £25 million in 2002-2003. Its staff in 2002 was over 300, excluding the 500 part-time reviewers from which inspection teams are drawn (Reference Day and KleinDay & Klein, 2002).

CHI's methods were a compromise between an inspectorate system providing judgmental reports in the style of Ofsted (Office for Standards in Education) and a developmental system that stressed the positive in its reports, citing examples of good practice worthy of imitation (Reference Day and KleinDay & Klein, 2002). In essence, CHI collected and analysed data about a trust's activities and performance. Informed by these data, CHI then reviewed in depth a number of clinical areas. It also sought the reviews of staff, patients, and external individuals and bodies such as general practitioners and others. It is significant that CHI did not specify the standards and criteria by which trusts were being judged. It simply set out a 4-point scale that reviewers used to assess each component of clinical governance (Box 1). CHI itself described this method as a combination of quantitative and qualitative techniques. It asserted that all CHI's evidence is triangulated, that is, confirmed from at least three different sources. However, there remains considerable doubt regarding the reliability and validity of CHI's method.

Box 1.

CHI: Definitions of scores

• I Little or no progress at strategic and planning level, or at operational level • II (a) Worthwhile progress and development at strategic and planning levels, but not at operational level • II (b) Worthwhile progress and development at operational level, but not at strategic and planning levels • II (c) Worthwhile progress and developments at strategic and planning levels and at operational level, but not across the whole organisation • III Good strategic grasp and substantial implementation. Alignment across the strategic and planning level, and the operational level of the trust • IV Excellence: coordinated activity and development across the organisation and with partner organisations in the local health economy that is demonstrably leading to improvement. Clarity about the next stage of clinical governance

Problems and limitations of CHI's methods

CHI was part of the increasing regulatory arm of government. As Walshe (Reference Walshe2002) has argued, regulation is the sustained and focused control exercised by a public agency over activities that are valued by a community. It is also a means by which politicians distance themselves from difficult issues or unpleasant decisions. According to Walshe (Reference Walshe2002), in general, regulatory agencies subscribe to one of two models of regulation, deterrence or compliance. The deterrence model assumes that the organisations being regulated need to be forced to behave well by strict regulation, demanding standards and tough enforcement. The compliance model assumes that the organisations are fundamentally well-meaning, and regulators provide support and advice. On the face of things, CHI's approach was more like the latter. It never published explicit, demanding standards, but rather took a developmental, supportive approach. How far this stance will survive the imminent changes will only become clear in the future. What is clear is that the process and method that CHI used for clinical governance reviews did not appear to produce reports that were consistent in tone or in the degree that they made their criticisms overt (Reference Day and KleinDay & Klein, 2001).

Day & Klein (Reference Day and Klein2002) reviewed 25 reports published in the last 2 months of 2001 and the first month of 2002. They concluded there was no consistency in the basic data provided. In some reports, data were broken down by speciality or even by individual surgeons. In others they were not. There were also inconsistencies in how patient experience was captured. Reports included quotes from patients about the quality of food or staff attitudes, yet without any comments about whether these comments were representative of patients in general. This point is also made and reinforced by Burns (Reference Burns2002) when he asks ‘are quoted opinions representative or illustrative?’ Day & Klein (Reference Day and Klein2002) also commented on the failure to put data into context and critically analyse the data. They concluded that the reviews lacked precision and authority.

There have been criticisms of the inexperience and lack of full training of reviewers (Reference Day and KleinDay & Klein, 2002; Reference BurnsBurns, 2002). In addition, there was demand for large data sets by CHI, which were often not meaningfully utilised in the review, and reports came under criticism. Quite often, the data sets were misunderstood or ignored (Reference ShapiroShapiro, 2003). But, perhaps the most important criticism is the absence of explicit measures. As Burns (Reference Burns2002) says: ‘If the questions that are relevant to most trusts cannot be stated simply, then we must question the value of an inspectorate. If the same questions are not asked and their answers not reported in a consistent manner, then we seriously mislead ourselves in comparing across reports and trusts’.

Assessing health care institutions

CHI report on mental health services showed that except for the key areas of ‘patient involvement’ and ‘education and training’, mental health services performed more poorly than other trusts (Table 1). So in areas such as risk management, clinical audit, clinical effectiveness, staffing and management, and the use of information, mental health trust’ performances were poor. The question is whether this summary of the state of mental health trusts accurately reflects the facts. Even if the conclusions drawn by CHI seem instinctively correct, are they based on techniques and measures that are reliable and valid? There is every reason to think not.

Table 1.

Percentage of all Trusts scoring each Commission for Health Improvement (CHI) score for each key area compared with the percentage of Mental Health Trusts scoring each CHI score for each key area

	Patient involvement (%)		Risk management (%)		Clinical audit (%)		Clinical effectiveness (%)		Staffing and Management (%)		Education and Training (%)		Use of Information (%)
CHI key area	All Trusts	Mental Health	All Trusts	Mental Health	All Trusts	Mental Health	All Trusts	Mental Health	All Trusts	Mental Health	All Trusts	Mental Health	All Trusts	Mental Health
I	21	22	18	28	23	38	24	41	16	31	6	9	31	56
II	71	69	72	72	66	56	61	53	70	63	72	81	60	41
III	7	9	10	0	10	6	15	3	14	6	21	9	10	3
IV	0	0	0	0	0	0	1	3	0	0	1	0	0	0

There is already a substantial body of literature on how to assess health care institutions. The literature sets out clearly the definitions of terms such as ‘standards’, ‘indicators’, ‘sentinel events’ and ‘criteria’ (Reference Oyebode, Brown and ParryOyebode et al, 1999). Furthermore, there is consensus that the attributes of a performance measure should at least include: targeting improvements in the health of populations; be amenable to precise definition and specification; be reliable; be valid; be amenable to being easily interpreted; be amenable to being risk-adjusted or stratified; the data must be available and accessible; be useful in making judgements about an institution; be under the control of the provider institution; and must be comprehensible to the public (www.jcaho.org/pms/core+measures/). There is little doubt that CHI's methods and instruments did not meet many of these criteria. Many of the conclusions that CHI came to could have been reached without their expensive and intrusive method. We are not arguing against visits or inspections, or indeed against the idea of ensuring that trusts deliver efficiently and effectively to the populations served. Rather, we are arguing that tried and tested assessment characteristics be adhered to.

Commission for Healthcare Audit and Inspection

CHAI took over from CHI in April 2004. It took on the functions of CHI as well as that of the National Care Standards Commission. It also subsumes the Audit Commission's responsibility in respect of national studies of the efficiency, effectiveness and economy of health care, and in due course the role of the Mental Health Act Commission. In addition to the roles described above, it is also responsible for providing an independent assessment of complaints, assessing the arrangements in place to promote public health, and acting as the leading inspectorate in relation to health care. CHAI vision documents are available at www.chi.nhs.uk/eng/about/chai/about.shtml.

CHAI aims to provide patients and users of services with clear statements of the safety, quality, efficiency and effectiveness of services that they received; maximise the contribution of regulation to the improvement of health and social services; and reduce the burden of regulation on providers of health and social services, commensurate with improvements achieved in the provision of services. CHAI intends to develop assessment tools that address the quality of care received by patients, the quality of patients’ experiences, and the quality of organisations and their capacity to produce improvements in services. The aim is to achieve this by the use of ‘ intelligent information’; not the intelligent use of information. The characteristics of this intelligent information are yet to be defined. The hope is that what is already known about performance measures will inform the development of these new measures. No matter what the weaknesses of CHI, it is a pity that its life was so short. It is self-evident that an evolutionary process in which knowledge and expertise accumulates and develops over time, a substantial period as well, measured in decades rather 3-5 years, is necessary for confidence in any system to take root. CHAI's vision is laudable and it needs all the support it can get. A regulatory body only succeeds to the degree that it has the confidence of the wider population that it serves, including those who are patients and those who work in health care institutions. And trust is always hard-earned, never legislated for.

Conclusions

CHI was influential in shaping the quality agenda within the NHS. It had immediate impact, blighting several managerial careers (Reference Day and KleinDay & Klein, 2002). However, whether it directly contributed to improvements in clinical care is doubtful. There is no doubt, though, that the quality of clinical care is now more central to how managers and politicians judge the NHS. This is at least consolation for those who regard regulation and inspection as unwieldy tools for producing improvements in the NHS. CHAI, the new organisation, must learn from the errors of CHI. We can only hope that the Government will allow CHAI to exist for long enough for its influence on clinical care to be measurable.