Internal benchmarking

Internal benchmarking should be the starting point of any benchmarking process. It is usually necessary to document internal working processes first and internal benchmarking is the most straightforward way of accomplishing this. The partner should be within another part of the service, not too geographically distant and should share similar functions and processes. For example, it would be relatively straightforward to compare appointment booking for out-patients in different hospital departments. The contrasting procedures may reveal some best practices within the organisation that can be modified and replicated elsewhere. Previous attempts at internal benchmarking have shown that practice varies greatly from area to area. This can be as simple as innovations in one ward being unappreciated on an adjacent ward (Reference PattersonPatterson, 1993).

The advantages of internal benchmarking are numerous. As well as ease of data collection resulting from greater internal consistency, it is also relatively inexpensive and use of an internal partner can bring to light potential problems in working with external organisations. Another advantage is that it should not be difficult to find partners within the NHS: internal benchmarking is a natural choice for hospital systems, as there are multiple sites within a hospital and even multiple hospitals within a trust for comparison. It also allows sharing of comparative data and internal trends with departments within the service, allowing the exploration and integration of multidisciplinary approaches to optimise processes and outcomes. However, a limitation of internal benchmarking is that the level of the best performer within the organisation usually determines the level achieved by the rest.

More specialised services may consider their work to be too sophisticated for benchmarking because of the futility of finding a suitable partner. This is often a mistake, as it should be possible to concentrate on benchmarking of more general processes. For instance, most psychiatric services, including specialised services, have strategies for managing substance misuse. Benchmarking these shows what a partner is doing differently and perhaps more successfully.

Competitive benchmarking

The principal aim of competitive benchmarking is to compare a specific process with that of the best competitor in the same industry and to identify performance levels to be surpassed. This is the stage after baseline attempts at internal benchmarking. Competitive benchmarking is important because a progressive organisation, in order to assess its strengths and weaknesses, must at some stage assess the gap between its own operations and the competition. Occasionally, this process can be hindered if the competition is revealed to be performing less well than you are.

Functional benchmarking

Functions of an organisation that are performed in other industries as well as in healthcare form the basis of functional benchmarking. If the best partner operates within a different industry, the advantage of this type of comparison is that a mental health service attaining this level of benchmarking has the opportunity to improve functioning beyond the best NHS or non-NHS competitor. For example, a mental health service should find worth in benchmarking against institutions that have a reputation for consistent delivery of accurate and important information to staff, for example an airline service. Such an exercise could advance performance in shared similar functions, including providing customer satisfaction, information processing and risk management strategies. A disadvantage of functional benchmarking is that it does not focus on the processes of the partners. The lessons learnt might therefore be harder to implement because even though one is able to learn from the information processing function of an airline, transferring this knowledge to a mental health service requires considerable integration.

Generic process benchmarking

Generic process benchmarking allows benchmarking of specific processes across different industries to find the best practices wherever they may exist. For example, an ammunition manufacturer was able to produce smoother, shinier shells following consultation with a lipstick company. Senior hospital managers seeking to improve income might concentrate on increasing revenue from their hospital's assets by comparing against any company that is performing this process well. Reference SussexSussex (1999) suggests that benchmarking against water and electricity companies should be enlightening even though at first glance they seem incompatible with health services. He highlights their similarities: for example they are local monopolies delivering essential services and they are affected by significant inherited inflexibilities such as location, capacity and aged equipment. Their many similar generic processes offer the hospital opportunity for a large improvement in performance.

Relationship to clinical governance

Faced with yet another quality initiative the sceptical clinician may be wondering what advantages benchmarking offers. The Department of Health has challenged managers with the responsibility of commissioning and providing efficient service delivery through quality improvement activities (Department of Health, 1997). It has recommended benchmarking as an activity that, like audit, well-considered guidelines and other systematic reviews of practice, can inform the process of clinical governance (Department of Health, 1999). Clinical audit by itself has been recognised as having a poor record in improving practice (Reference HopkinsHopkins, 1996). There is growing concern that the wealth of guidelines impinges on creativity, fails to solve the problems of poor care and may in some circumstances even be harmful (Reference McDonaldMcDonald, 2003). Traditional measures of service use such as hospital readmission rates and length of stay often reflect service policy and provision in a self-fulfilling manner rather than giving true information about the impact of treatment on patients (Reference ShooterShooter, 1997). Psychiatrists also encounter difficulties in making sense of experimental evidence to inform clinical decision-making because of the small scale and short-term nature of most psychiatric trials (Reference Wykes and MarshallWykes & Marshall, 2004).

Benchmarking demonstrates several refinements over these other activities (Box 1), mainly because its doctrine of continuous improvement is a central tenet of clinical governance. It is a means by which the practices needed to reach new goals are discovered and understood. Benchmarking can be seen as a direction-setting procedure that helps to manage the relationship between systematic policy developments, inefficient processes, identified clinical pathways and evidence-based outcomes. It moves away from the traditional method of establishing targets, the extrapolation of internal past practices and trends. Benchmarking does not restrict an organisation to the limited supply of internal ideas and performance assessments advanced by initiatives such as quality circles (Reference ColeCole, 1999). It also allows for potentially boundless continuing improvement by comparison against a wealth of medical and business organisations anywhere in the world. Thus, a mental health service can keep up with the rapidly changing external environment and reduce staleness associated with conventional goal-setting.

It is important to recognise that effective benchmarking is not a one-off exercise. It is a continuing process of improvement with the expectation that as one exercise stops another should start. Box 2 illustrates some of the mechanisms by which benchmarking may improve practice. Naturally, there are limits to the improvements that can be achieved and, potentially, to the ability of an organisation to use benchmarking effectively.

Limitations of benchmarking in mental health practice

It might be argued that benchmarking is a valid exercise in mental health practice only if it produces an improvement in patient care. Use of a benchmarking team to improve an administrative or other support procedure is unsatisfcatory if there is no benefit for patient care. For example, significant expenditure on an industry-leading information technology system does not automatically lead to improved patient care (although that potential obviously exists). Worse, it may appropriate money from an existing service that was providing useful and appreciated support.

There are other factors that limit the ability of a mental healthcare organisation to use benchmarking effectively. First is the lack of good outcome measures for mental health services. Industrial companies can measure improvements in terms of reduced or improved profits, whereas mental health benchmarking outcomes are likely to be more qualitative and may require more careful deliberation. If it takes a long time to decide measurable outcomes before setting up a project or too long (more than about 6 months) to complete, a project a team can lose enthusiasm and support from within the organisation. The same can happen if healthcare services spend a lot of time in search of ideally compatible facilities against which to compare themselves rather than taking a broader approach and learning from wherever they can. Finally, initiating a project for the sake of undertaking benchmarking is usually much more difficult than starting one in an area where a specific breakthrough improvement has been long required (Reference Mosel and GiftMosel & Gift, 1994).

Box 3 highlights some inappropriate uses of benchmarking.

What assets do healthcare services have for benchmarking?

Adapting benchmarking to the healthcare sector is no different in principle from adapting it to any other sector in which many different professionals with specialised functions work closely together. Reference Camp and TweetCamp & Tweet (1994) suggest that most hospitals are relatively small organisations when compared with large industrial enterprises. This should make it easier for them to communicate internally, make rapid decisions, question a breadth of staff and access extensive records that contain much potentially valuable information (Reference LelliottLelliott, 2003). This has to be balanced against the likelihood that they will have fewer full-time clinical staff free to carry out a benchmarking project.

Internal and competitive benchmarking require a relatively open exchange of information. Although this may well benefit both parties in the long term, the benchmarking partner may be more reluctant than the ‘petitioner’ to reveal its processes. Fortunately, the health service involves organisations that are more ready to share information. For example, there is a wealth of readily available information about clinical risk management from audits, complaints, incident forms and inquiries within each hospital which is often limited to within a trust. Benchmarking of the processes by which other trusts use information about errors and complaints to augment performance should be relatively straightforward owing to the degree of candidness that health services cultivate. Benchmarking relies heavily on an atmosphere of trust, and such easy access to information may not be manifest when mental health services start benchmarking against non-medical companies, which may wish to protect their industrial secrets.

The following case example is fictional but faithfully reflects clinical reality. Any resemblance to an actual case is purely coincidental.

Case example

Dr S, a general adult psychiatrist working in St Cuthbert's district general hospital is dismayed by a complaint from the mother of a patient. Her son has been on a waiting list for cognitive–behavioural therapy (CBT) for psychosis for 10 months. The clinical director of the service, who has an interest in benchmarking, convinces Dr S to review why the waiting list is unacceptably long.

Dr S forms a team to map provision of CBT for psychosis (Fig. 2). It identifies a number of delays that generate an average waiting time of 13 months. For example, a questionnaire reveals that that there is only one psychologist available to provide one session a week, whereas the waiting list suggests a need for eight sessions a week. It also demonstrates that patients receive CBT for positive symptoms of schizophrenia but do not receive treatment for negative symptoms or poor social skills. The team members decide that, should they not find a better solution, they might use junior doctors and employ a part-time supervising psychologist to reduce the waiting list.

Fig. 2

Process map of the pathway for an in-patient to receive CBT for psychosis

A serendipitous meeting with a manager in the primary care trust highlights a service recently set up in a large forensic psychiatric unit 10 miles away. It is within the same trust and appears to be an ideal internal benchmarking partner.

The team visit the unit, and find that the forensic service is using weekly group therapy to treat 31 patients divided between three groups. Treatment continues for 4 months, during which time it addresses social skills, and positive and negative symptoms. Six trained nurses and four trained psychologists run the groups. The six nurses also work on the wards and supervise other nurses. The groups are repeated three times per year and the average waiting time to join a group is 2 months.

The team members realise that they would never have devised such a scheme themselves. They adopt the practice in their own hosptial, providing similar groups on a smaller scale, and surpass their earlier goal.

The benchmarking exercise resolved the two most significant problems that had resulted in the previous sluggishness: the limited number of free rooms and a non-evidence-based belief that CBT had to be delivered individually.

Analysis of internal practices

Having established the process to be reviewed, the team should embark on an analysis of the current practices within it, using the tools of process mapping (Reference Lenz, Myers and NorlundLenz et al, 1994). This groundwork allows the team to recognise steps crucial to meeting objectives, in addition to inefficient or redundant steps. Mapping is particularly useful in identifying blockages (e.g. the delays in Fig. 2), as well as augmenting any future comparison against external benchmarking partners when determining how they manage inefficiencies and impediments.

Box 4 Suggested key players in a benchmarking project

The team leader	Plans the development of projects and relays information to the whole organisation about the team's advancement
The benchmarking team	Team members undertake the majority of the project work and should ideally work with or be familiar with the process being benchmarked
Functional experts	Valued for their appreciation of how the service functions and ability to predict how change to one part of the process will ultimately affect the rest of the service
Mental health users and carers	Add the customers’ perspective, ensuring that the project does not result in ineffective change
The executive champion	Preferably a senior manager, not only taking responsibility for the project but also ensuring recognition of the team's work
The process sponsor	A manager such as a clinical director with a direct interest in the results of the project and who has a closer involvement in the process than the executive champion
Stakeholders	Although not involved in the benchmarking itself, their dissatisfaction with a service may help initiate the process; they are important in setting limits for the scope of the study and ensuring that the objectives of the study are met

It is advisable to conduct a pilot study before proceeding to the full study. This allows team members to familiarise themselves with the tools, modify their aims if necessary and, of paramount importance, encourages senior managers to support later work on the basis of positive results.

Choosing what to measure within a specific process requires identification of the essential quantifiable stages of the process that are aligned most closely with the service's duties, aims and objectives. The choice might be based on counting patient complaints about a process as a gauge of its success. If particular data cannot be used in achieving the specified improvement it is unlikely that they need to be gathered. For instance, user satisfaction is often regarded as a basic measure of service delivery. However, it may not be much use as a measure if the objective is to reduce out-patient waiting times, because it does not directly reflect any improvement in waiting times and is likely to be confounded by other variables.

Having identified the process and outcome measures, the next step is to confirm current levels of success, using a well-designed questionnaire. The first stage in designing the questionnaire is to agree an objective for it to test. The questionnaire should also be able to extract data from the benchmarking partner to discover the mechanisms behind their best practices. Questionnaire development is a complicated process, and an internal pilot questionnaire can help to rectify vagueness, eliminate jargon, ensure that the questionnaire asks the right questions and add to the data already gathered about the process in the organisation. Having as few questions as possible reduces the chance of alienating a benchmarking partner. Every question should contribute independently to realising the objectives of the study and not be repetitive or overlapping. Analysing the answers of the pilot will indicate how well the questionnaire meets these objectives.

Identifying potential benchmarking partners and best practice

In parallel with the above is the search for partners, often based on reputation for excellence, similarities of service and willingness to share data. The presentation to the partner needs to be intelligible, convincing and tested. Some of the key questions that help extract data from a partner are summarised in Box 5. Most of a team's time will be spent on collecting data rather than meeting with the benchmarking partner, and team members should not underestimate the amount of time required to grasp the intricacies of their information systems, extract information manually from records and pursue missing data. The procedure is simplified if there is a user-friendly system for retrieving data, some assurance that the data are accurate and the capability to amalgamate the best practices of one internal department with another.

Best practice is determined to a large extent by how much of an impact the practice has on the business, the degree to which the results are related to specific goals and whether the practice dovetails with other programmes and operations already being undertaken. For example, a system for dealing with service user complaints could be subjected to functional benchmarking against the customer complaints procedure of a successful manufacturer that receives relatively few complaints and manages these efficiently. The low level of complaints may be related to the manufacture of efficient and reliable products. Nevertheless, excellent integration of the manufacturing process with the practice of quick, satisfactory settlement of grievances leads to far superior performance, evidenced by even better sales.

Analysing data and modifying practice

Analysis of data should lead the team to ask whether there are pointless or inconsistent practices causing loss of direction or sluggish movement and whether each contributor to the process clearly understands the other participants’ information requirements. For example, in a study optimising out-patient room use, it might be found that, compared with the partner, staff managing patient records in one part of the building do not know when clinicians require updated records in the clinic, or that the transport department underestimates the lead time necessary to convey patients punctually for their appointments. Knowing the cause of the discrepant practices between partners is only the beginning of the process because one then has to identify the factors that propel those differences, such as variations in the use of protocols between departments, unreliability of suppliers, antiquated facility design or uneasy interdepartmental dealings.

Modification of practice is likely to be beneficial and lead to better performance when differences and the reasons for those differences between partners have been identified.

Careful consideration is required in implementation of changes, as healthcare benchmarking is fundamentally different from its industrial equivalent. It is important to ensure that potential improvement does not take on a purely competitive nature. If it does, quality of patient care may be sacrificed for speed of service or cost savings. It is advisable to employ a technique such as the Plan, Do, Study, Act (PDSA) cycle (Reference Langley, Nolan and NolanLangley et al, 1996), which provides a framework for testing whether planned changes actually make the desired improvements before they are fully implemented.

A mastery of benchmarking is related to an understanding of its key elements (Box 6). It also requires the team to understand the relationships between all individuals involved in the process and to promote a cohesive and creative way of working. Without regard to the needs of these individuals, any changes that are made may lead to increased tensions, fragmentation and discord. To avoid this, all findings must be communicated on a regular basis to the parties that will have to implement the changes. This regular feed of information often induces more commitment to change, as resistance typically stems from the fear of loss of control. Box 7 outlines some of the major aspects involved in managing change. Finally, any variation to processes should be made in collaboration with senior managers. Commitment from managers is vital to ensuring continuity, coordination, the involvement of relevant practitioners, removal of obstacles and access to sensitive information (Reference EllisEllis, 1995). Without sufficient support or authority, the changes suggested are unlikely to be implemented or to be effective.