The ‘ability to benefit’ from health care interventions is an essential component of the Department of Health model of need (National Health Service Management Executive, 1991). The model implies that need only exists if there are interventions with proven efficacy and effectiveness available to address that need, and that needs for which there are no such interventions should not attract resources. This provides a crude justification for rationing; only those individuals who have health problems that can benefit from interventions should receive treatment. This paper evaluates definitions of outcome and approaches to measuring outcome both generally in mental health care and specifically in relation to mentally disordered offenders (MDOs). It also examines the relationship between types of outcome measure and how outcome fits into a broader framework of service evaluation. Finally, it draws together the strands of need, outcome and service evaluation in an attempt to create a coherent framework.

DEFINITION AND MEASUREMENT OF OUTCOME

The concept of ‘ability to benefit’ is inextricably linked with the measurement of outcome, because establishing the ability to benefit requires that health care interventions are clearly defined and linked with specific benefits or outcomes for specific patient groups. It can be argued that knowledge about outcome is a prerequisite for establishing the ability to benefit, because selection of the right patients for the right interventions cannot occur without evidence about likely outcome. Both outcome and ability to benefit are also highly politicised, because they relate closely not only to evidence-based practice but also to clinical governance. Yet, the definition and measurement of ‘ability to benefit’ and outcome have long presented theoretical and empirical difficulties in all health and related services research, particularly in relation to mental health.

Ovretveit (Reference Ovretveit1995) defines health service outcomes as the effect on a person or population that can be attributed to a health treatment, service or intervention. However, establishing outcome in relation to mental health interventions, even in general mental health, is poorly developed. Psychiatric disorders and their associated social disabilities are complex and multi-factorial in their aetiology and manifestation (Reference Wing, Brewin, Thornicroft, Thornicroft, Brewin and WingWing et al, 1992). Baseline information is limited or non-existent, and outcomes are multi-dimensional and difficult both to define and measure. There are also difficulties in defining operationally many of the treatments and interventions available. Different treatments may be delivered to the same patient by different professionals, and multi-agency involvement adds further complexity. Consequently, demonstrating valid and reliable causal relationships between specific interventions and outcomes is problematic. Ovretveit (Reference Ovretveit1995) notes that outcome measurement frequently focuses on end-points rather than health gains made during the treatment process, and is critical of the tendency for outcome measurements to fail adequately to include the effects of other services and environments, or other factors that affect health. He also bemoans the tendency of outcome studies to overlook patient views and the quality of service delivery. However, he notes how costly and methodologically difficult it is routinely to measure outcome effectively, and that commissioners who require providers to measure outcome will pay in higher prices.

PRINCIPLES OF OUTCOME MEASUREMENT

Atkisson et al (Reference Atkisson, Cook and Karno1992) suggest that outcome research should adhere to the following seven principles:

1. (a) Outcome measurement should be multi-dimensional and should cover clinical, rehabilitation, humanitarian and public safety domains. The clinical domain relates to various aspects of psychopathology and the course of illness over time, and the rehabilitation domain focuses on adaptation and functional capacity. The humanitarian domain is concerned with subjective well-being, consumer satisfaction and quality of life, and the public safety domain is concerned with setting a balance between liberty and paternalism that will maximise individual and societal rights to physical safety and well-being. The clinical, rehabilitation and humanitarian domains are reflected in well-established ‘research industries’ in general mental health, but there is a dearth of empirical evidence regarding each of these domains specifically in relation to MDOs. Indeed, Robertson (Reference Robertson1997) bemoans the lack of attention paid to mental health outcome measures in forensic psychiatric research. The majority of outcome research in relation to MDOs has focused on the public safety domain. For example, many studies have focused on recidivism (mainly re-arrest and reconviction rates), especially within the special hospital population (e.g. Reference Bailey and MaccullochBailey & MacCulloch, 1992; Reference BuchananBuchanan, 1998), and on the validity and reliability of risk assessment (cf. Reference Blumenthal and LavenderBlumenthal & Lavender, 2000). Although this domain is obviously crucial in relation to MDOs, future research should endeavour to include other domains.

2. (b) Outcome should be measured from multiple perspectives (e.g. patient, carers, clinicians). Much of the outcome research adopts a clinical perspective of what constitutes a positive outcome and neglects the views of other stakeholders.

3. (c) Outcome measurement should take into account the fact that different individuals and groups may perceive the usefulness (or utility) of mental health outcomes differently. These ‘individual utility differences’ are a source of variability that should be measured and accounted for in outcome studies.

4. (d) Cross-sectional and longitudinal studies should be conducted. Longitudinal studies are especially important in mental health, given the chronic nature of many mental disorders.

5. (e) Standardisation of research design and measurement should be worked towards, in order to facilitate comparison between studies. However, a balance must be established between standardisation and specificity.

6. (f) Costs should be incorporated into outcome measurement, including costs to the patient, family and society of the absence (or refusal) of services.

7. (g) Relevance and impact of outcome research should be considered in relation to clinical practice, policy, legislation and science.

These principles provide a comprehensive framework for outcome measurement. Although it is self-evident that most researchers will be unable to measure all of these areas or adhere strictly to all the principles advocated, the framework is still useful. It forces us to adopt a broad perspective about outcome measurement and to recognise the limitations on what is achievable. It also forces us to acknowledge that we may sometimes be prioritising only one dimension of outcome (and a small one at that), neglecting other domains in its favour. Public policy, values and resources will partly drive what aspect of outcome is prioritised and measured, as well as the methods used to achieve this. Hence, there must be explicit acknowledgement of what is not being measured and why — we should be able to justify, for example, why it is more important to measure recidivism as an outcome rather than symptom reduction or quality of life.

PROBLEMS WITH MEASURING OUTCOME IN RELATION TO MDOs

The measurement of outcome for MDOs presents some specific problems. The term MDO is itself difficult to define, and MDOs form a heterogeneous group that may fall into any diagnostic category (Reference Cohen and EastmanCohen & Eastman, 1997). They are likely, therefore, to have many needs for treatment and care that are similar to general psychiatric patients. However, they may have additional needs that relate to their offending behaviour. Consequently, measuring ‘ability to benefit’ and outcome in relation to MDOs must cover a wide range of interventions for a wide variety of problems, including problems going beyond health outcome narrowly defined. It is not possible, therefore, to provide a single model of ‘what works for MDOs’. The additional component of offending that is specifically relevant to MDOs adds at least two further complicating dimensions to the measurement of outcome. First, offending can arise from factors unrelated, or only partially related, to an individual's mental disorder. That is, offending is not necessarily causally related to mental disorder and a wide range of ordinary criminological explanations of offending, both individual to the offender and more broadly societal, may be relevant to an MDO's offending behaviour. This introduces not merely one or two additional factors to a clinical model but superimposes upon it a criminological model that is largely unrelated to mental health services narrowly defined. Second, ‘ability to benefit’ relates, in the specific social policy context being considered, not only to the patient's ability to benefit but also to the benefits to society of detaining, and hopefully successfully treating, individuals who pose a threat to public safety. Indeed, the government's proposals for the preventive detention of ‘dangerous’ individuals with a ‘severe personality disorder’ (Home Office/Department of Health, 1999) particularly emphasises the point. Given the profound uncertainty about the ability of mental health professionals reliably and validly to identify such a policy-defined group, or to be able to offer any interventions that are beneficial to the individual, the distinction between ethically valid ‘public health psychiatry’ and mere crime prevention looks fragile (Reference EastmanEastman, 1999).

OUTCOME, QUALITY AND SERVICE EVALUATION

The measurement of outcome must be placed within a broader framework that relates to service quality. According to Glover & Kamis-Gould (Reference Glover, Kamis-Gould, Thornicroft and Strathdee1996), outcome is just one type of performance indicator that fits into a more general model of service evaluation. Jenkins (Reference Jenkins1990) argues that, in order to evaluate any health care system, it is necessary, in general terms, first to measure the baseline health of the population and then to measure the impact of health care upon that baseline. She suggests that this can be achieved in a valid and reproducible manner only if specific health indicators are established that apply not only to general ‘well-being’ but also to specific categories of illness, and if these categories are then related to specific strategies of treatment and prevention. Jenkins defines an indicator as “a measure that summarises information relevant to a particular phenomenon or a reasonable proxy for such a measure” (Reference JenkinsJenkins, 1990, p. 501). She accepts that indicators should be valid and reliable, but argues that this is difficult to achieve. ‘Health indicators’ are variables that can be measured directly and that reflect aspects of the state of health of a community, and ‘health care indicators’ are variables that reflect aspects of the state of health care in a community (World Health Organization, 1981, cited in Reference JenkinsJenkins, 1990). According to Jenkins, health care indicators can be categorised into ‘input’, ‘process’ and ‘outcome’ (albeit, outcome indicators will also be health indicators). Tansella & Thornicroft (Reference Tansella and Thornicroft1998) refer to this approach as the ‘temporal dimension’, because it is concerned with the chronological steps involved in the delivery of health care.Footnote ¹ ‘Input’ refers to resources that are put into the mental health care system (Reference Tansella and ThornicroftTansella & Thornicroft, 1998). Input variables include type and size of facilities, human resources and characteristics of physical facilities (Reference Biugha, Lindsay, Thornicroft and TansellaBrugha & Lindsay, 1996). ‘Process’ refers to activities that take place to deliver mental health services (Reference Tansella and ThornicroftTansella & Thornicroft, 1998). ‘Process’ variables include the technical or interpersonal elements that occur during a health care intervention, including diagnostic and therapeutic procedures and features of the clinician—patient relationship (Reference Biugha, Lindsay, Thornicroft and TansellaBrugha & Lindsay, 1996).

A similar conceptualisation is offered by Donabedian (Reference Donabedian1980). He divides research about the quality of health care into studies that address structures (e.g. provider systems, organisation of systems, characteristics of treating facilities), process (specific clinical interventions) and outcome.

Berwick (Reference Berwick1989) outlines four types of health services research that relate to quality of care:

1. (a) effectiveness of care (what works for whom);

2. (b) appropriateness of care (using what works);

3. (c) execution of care (doing well what works);

4. (d) examination of the purpose of care (values that underlie action).

According to Atkisson et al (Reference Atkisson, Cook and Karno1992), progress with the paradigms presented by Donabedian (Reference Donabedian1980) and Berwick (Reference Berwick1989) is required in order to advance research about quality of care.

Glover & Kamis-Gould (Reference Glover, Kamis-Gould, Thornicroft and Strathdee1996) propose a model of service evaluation that covers two broad aspects of an organisation/system. The first relates to the capacity of the system. Capacity variables include human and financial resources, the range and quality of clinical facilities and the technical capacity to operate, coordinate and monitor all aspects of organisational functioning. The second relates to the performance of the system. This is concerned with responsiveness and accessibility (e.g. congruence with local needs, cultural sensitivity, promptness and sensitivity of response to clients). Performance is also measured in terms of efficient use of resources (i.e. levels of productivity, cost containment, occupancy rates) and effectiveness.

According to Jenkins (Reference Jenkins1990), aspects of service provision that can be most easily measured at present tend to be those that relate to service input and resources rather than to service outcome. She notes that input is relatively straightforward to measure, and that process tends to be measured in terms of ‘performance’ or ‘activity’ indicators (e.g. occupied bed-days). Process indicators related to delivery of specific interventions or the nature of therapeutic relationships are more difficult to measure and are unlikely to be available routinely. Jenkins also points out that process indicators are frequently selected on the basis of what is collectable (or already available), rather than being derived from previously specified key aspects of performance. Indeed, although there may be good ad hoc studies relevant to some desirable process measures, there is, in fact, a profound lack of ongoing data that could be of use in the monitoring process and, in particular, in monitoring the meeting of mental health needs. Jenkins notes that the measurement of outcomes is more complex than the measurement of input and process. She points out that input and process indicators are often used as proxy measures of outcome, which she suggests is based on faulty logic — that is, that service utilisation (process indicator) is equal to ‘improvement’ (outcome). So, just as service utilisation is a poor proxy for need (Reference Cohen and EastmanCohen & Eastman, 1997), so too is it a poor proxy for outcome.

INPUT, PROCESS AND OUTCOME INDICATORS IN FORENSIC MENTAL HEALTH

Jenkins (Reference Jenkins1990) provides a system of input, process and outcome indicators related specifically to forensic psychiatry. Her approach requires that reference be made to ‘ordinary’ mental health outcome measures in relation to MDOs with mental illness and learning disabilities. She presents ‘special’ indicators only in relation to personality disorder, although her rationale for this is unclear.

As a policy starting point, Jenkins offers a series of health objectives specific to MDOs. These are essentially policy objectives and are clearly influenced by Health of the Nation (Department of Health, 1992) targets. Examples include:

1. (a) reducing the incidence of MDOs;

2. (b) reducing the incidence of personality disorder;

3. (c) reducing suicide rates;

4. (d) preventing entry and re-entry into the criminal justice system;

5. (e) reducing homelessness.

She goes on to suggest a range of input, process and outcome indicators that relate to her proposed objectives. Input indicators include:

1. (a) systems to provide psychiatric services for assessment and advice to agencies of the criminal justice system (e.g. courts) and to provide early diversion from the criminal justice system;

2. (b) systems to provide psychiatric services to prisons and to aid the transfer of MDOs from prison to hospital;

3. (c) access to housing.

She then argues that process indicators should be established that reflect activity on all the above input indicators.

Finally, Jenkins identifies a number of outcome indicators:

1. (a) numbers of patients detained under Part III of the Mental Health Act 1983, and their admission and readmission rates;

2. (b) prevalence of treatable MDOs in the prison population;

3. (c) numbers of patients diverted from the criminal justice system;

4. (d) suicide rates in prison;

5. (e) standardised mortality ratios.

Jenkins' lists of objectives and indicators may no longer accurately reflect policy priorities of the current government, and it is important to note their historical limitations. They were formulated when diversion from the criminal justice system was particularly high on the political agenda and before the publication of the Reed Committee Report (Department of Health/Home Office, 1992). Although the Reed Report itself then reinforced the need for diversion from the criminal justice system, it also suggested an additional range of objectives and indicators, such as systems to identify and treat patients who no longer require particular levels of security. Similarly, the recent Ashworth Inquiry (Reference Fallon, Bluglass and EdwardsFallon et al, 1999), and the wealth of national and local inquiries following homicide by people with mental illness, have subsequently suggested a wide range of other potential objectives and indicators (e.g. Reference SheppardSheppard, 1996; National Confidential Inquiry into Suicide and Homicide by People with Mental Illness, 1999). Hence, a current set of input, process and outcome indicators might now be drawn significantly differently. It is important to recognise, therefore, that appropriate objectives and indicators will change over time, according to altered policy considerations, as well as in response to changes in service structure and advances in the ability to measure need and outcome.

There are, in any event, a number of criticisms that can be levelled at Jenkins' earlier choice of objectives and indicators. One of their disadvantages is that they reflect a public health stance that tends to neglect outcome at the individual level. Jenkins' system also includes some objectives that are difficult to conceptualise as being legitimate objectives of psychiatric services and are dependent upon many factors that are arguably beyond the remit of MDO health or even social service interventions (e.g. to reduce homelessness). Further, the majority of indicators suggested have no adequate baselines specific to MDOs and are not routinely measured at a local, regional or national level, and it is difficult to envisage how many of them could be measured reliably and validly in the future, particularly at a level that would be useful to commissioners and service providers. It is also noteworthy that many of Jenkins' suggested outcome indicators fall well short of being direct measures of outcome. For example, both the number of patients detained under Part III of the Mental Health Act and readmission rates represent indirect or proxy measures of outcome, with an assumption that, in relation to the achievement of goals relating to each, ‘good will follow’ (e.g. that service utilisation equates with a positive outcome). Indeed, of Jenkins' outcome indicators, only the suicide rate in prison and standardised mortality ratios can be seen as direct outcome variables, although these indicators are not currently statistically available specifically in relation to MDOs, and the extent to which they directly reflect mental health outcome is also debatable.

The criticisms levelled at Jenkins' system, which at face value appear entirely reasonable, illustrate just how much of a challenge it is to attempt to formulate any system. It is very difficult to select objectives and indicators that are both reasonable and realistic (e.g. measurable in relation to baselines, input, process and outcome) and that take into account broader policy objectives, as well as clinical and system realities.

CONCEPTUAL FRAMEWORK FOR EVALUATING FORENSIC MENTAL HEALTH SERVICES

Table 1 provides a conceptual framework for the measurement of input, process and outcome for MDOs that integrates the different models presented thus far. This framework uses Tansella & Thornicroft's (Reference Tansella and Thornicroft1998) ‘temporal dimension’ of input, process and outcome as the foundation of the model. It integrates the conceptualisations of Atkisson et al (Reference Atkisson, Cook and Karno1992), Donabedian (Reference Donabedian1980), Berwick (Reference Berwick1989) and Glover & Kamis-Gould (Reference Glover, Kamis-Gould, Thornicroft and Strathdee1996) into the relevant temporal dimensions, while providing examples of the types of variables that may be measured within each dimension. It then superimposes factors that can be measured at each of the temporal dimensions (e.g. values, costs). The model acknowledges that variables within each temporal dimension can be measured at different geographical levels (Reference Tansella and ThornicroftTansella & Thornicroft, 1998) and at different levels of the mental health care system (Reference Beecham, Chisholm and WingBeecham & Chisholm, 1995). The essential value of this framework is that it forces us to acknowledge the inherent complexity of what we are attempting to measure. It also helps us to recognise the interrelatedness of the concepts that we are measuring and makes us acknowledge, and justify, what we are unable, or choose not, to measure. Within the framework, prioritisation will be determined by both national and local policy, by locally assessed need and by what is practically achievable. Of course, it should be possible to justify why a particular element has been prioritised.

Table 1

A comprehensive conceptual framework for the measurement of outcome, quality and service evaluation for mentally disordered offenders¹

Temporal dimension
Input	Process	Outcome
Structure of care	Specific clinical interventions	Effectiveness and efficacy of care (what works for whom)
Provider systems	Somatic therapy	Seven principles (Reference Atkisson, Cook and KarnoAtkisson et al, 1992)
Organisation of system	Psychological therapies/counselling	1. Outcome is multi-dimensional
Characteristics of facilities	Sociotherapy	Clinical (e.g. symptom reduction)
Number of facilities	Assessment	Rehabilitation (social and instrumental functioning)
Capacity of services	Rehabilitation (e.g. occupational therapy)	Humanitarian (quality of life, patient satisfaction)
Financial resources	Relationship between clinician and patient	Public safety (risk to self and others, recidivism, security, risk assessment)
Human resources	Movement between tiers of services	2. Take account of multiple perspectives
Human resource development	Responsiveness and accessibility	3. Take account of individual utility differences
Service protocols	Waiting lists and bed-blocking	4. Strive for standardisation of measures and designs
Access criteria	Bed utilisation (inflow, length of treatment, outflow)	5. Use cross-sectional and longitudinal designs
Good practice guidelines	Pathways to and through care	6. Include measures of costs
Information systems	Frequency and duration of treatment	7. Consider relevance and impact
Government policy and legislation	Patterns of service use
	Care Programme Approach monitoring
	Continuity of care
	Coercion
	Execution of care (doing well what works)
	Efficiency
	Appropriateness of care (using what works)

The conceptual model of outcome measurement and service evaluation presented in Table 1 poses considerable challenges in both methodological and practical terms. It is therefore unlikely that we shall see anything that approaches the degree of comprehensiveness suggested by the model in the near future, although this should be the gold standard towards which to strive. The existing mechanisms and measures available both for estimating outcome and for evaluating services in relation to MDOs verge on being hopelessly inadequate. At the root of this problem is a lack of knowledge about how particular clinical interventions and services influence outcome. Indeed, there are not even any generally agreed upon service designs and protocols that might be measured in their effects. This inhibits not only the determination of appropriate outcome measures but also the definition of ‘need’ itself. How can we define ‘need’ if there is little agreement over the details of effective service response to need? Until we are able adequately to answer questions about input, process and outcome, we shall not be able properly to answer questions about ability to benefit and, hence, about need.

Clinical Implications and Limitations