It is common in linguistics to contrast “theoretical” and “experimental” research. Researchers who pursue experimental research are often asked about the theoretical consequences of their work. Such questions generally equate “theoretical” with theories at a specific high level of abstraction, guided by the questions of traditional linguistic theory. These theories focus on the structural representation of sentences in terms of discrete units, without regard to order, time, finer-grained memory encoding, or the neural circuitry that supports linguistic computation. But there is little need for the high-level descriptions to have privileged status. There are interesting theoretical questions at all levels of analysis. A common experience is that we embark on a project guided by its apparent relevance to high-level theoretical debates. And then we discover new theoretical questions at lower levels of analysis that we had not been aware of previously. We illustrate this using examples from many different lines of experimental research.

In previous chapters there has been a good deal of discussion about ‘theory-driven’ and ‘theory-based’ evaluation. This approach, coupled with a degree of despondency among several evaluators about the apparent non-utilisation of evaluation research findings, has led to a fairly recent attempt to enhance the status of evaluation as an academic/professional activity. Ironically, although the present ‘generation’ of evaluators and writers on evaluation have almost as one discarded the experimental research design as something of a ‘gold standard’ to which all designs should aspire, the quest for evaluations to be taken more seriously by decision makers has turned to a predominantly quantitative system of verification.

To what extent this approach has moved or is likely to move evaluation into a new ‘generation’ or higher level of academic and professional endeavour will be examined later in this chapter. To assert that one of these modes of enquiry – meta-evaluation – is recent is somewhat deceptive since it is generally agreed that the term was coined by Scriven in an article published in 1969 (Scriven, 1969). Interest in meta-evaluation has appeared only spasmodically in evaluation texts and articles until a revival of interest in the topic during the last two decades. Yet, in the book by Rossi et al (1999), out of the 500 pages the topic of ‘meta-analysis’ takes up the equivalent of just half a page. Similarly, the voluminous text by Shaw et al (2006) devotes little more than one page to meta-evaluation or meta-analysis in all its 608 pages, while the general text on evaluation by Green and South (2006) does not mention either meta-analysis or meta-evaluation.

Interestingly, the paper by Scriven (1969) preceded the seminal work by Cochrane (1972) in which he pioneered the development of evidence-based medicine through his advocacy of the randomised controlled trial (RCT) as the most reliable means of producing valid data. Although, as noted in Chapter One, there was some initial opposition to what a few in the medical profession regarded as an unacceptably mechanistic foundation for clinical decision making, Cochrane's stimulus towards increasing efficiency and effectiveness within the health care system is now central to decisions about expenditure on new drugs and other clinical interventions.

The primary purpose of this book is to examine the development of evaluation and its impact on public policy. This will involve a critical analysis of much of the evaluation literature generated mainly in the USA and the UK from the 1950s onwards. In particular, we shall be seeking published accounts of empirical research which has reported on attempts to identify clear evidence of a link between formal evaluation results and policy formulation. The text will also focus on the application of theory-driven evaluation to policy making.

Although discussions of data collection methods, research designs and the selection of criteria for evaluating policies, programmes and projects are included (Chapters Three, Four and Five), this book is not intended to be something akin to an evaluation handbook. These chapters contribute to the key theme of the book, which centres on the interface between the academic and political worlds.

We intend the book to be of interest and relevance to students engaged on a variety of courses at both graduate and undergraduate levels; to public sector personnel involved with implementing and/or evaluating the impact of policies, programmes and projects on intended beneficiaries; and to various stakeholders, notably politicians, community leaders and non-governmental organisations.

Within the broad aim set out above as the primary purpose, the objectives of this book are to:

• • trace the emergence of evaluation in the political and academic worlds;

• • explore the concept of evaluation and its various interpretations;

• • identify the potential opportunities and problems in a range of data collection methods and in the analysis of data;

• • explain the development of evaluation frameworks;

• • assess critically a range of evaluation criteria commonly used in evaluations;

• • examine the role of economic evaluations;

• • consider the extent to which academics have influenced the practice of evaluation;

• • examine the nature of evidence and its use/non-use by decision makers;

Introduction

It is a curious fact that economic evaluation does not feature at all prominently in the evaluation literature although it has to be a major consideration in any official financing of internally or externally commissioned evaluation research. This chapter will develop the discussion introduced in Chapter Four relating to some of the criteria available to evaluators of public programmes and policies, in particular economy, efficiency and equity. These criteria are of particular significance given that this book has been written at a time when virtually all developed economies are grappling with the aftermath of the crisis in the financial sector that has led to one of the most severe economic recessions of recent times. The bailout of economies by international financial agencies and governments has been a common feature of news bulletins and media reports, and in many countries has led to already stretched public service budgets being faced with unparalleled reductions in financial allocations. As a result, public service professionals have become increasingly vociferous in their claims that levels of service provision are unsustainable in the prevailing economic climate.

However, the need to ensure that services are provided as efficiently as possible, for example, is not something confined to times of fierce economic conditions. Demands that scarce resources are used to the best possible effect have become commonplace.

This comes against a background of increasing expectations and demands from their resident populations, which has resulted in a situation in which the provision and funding of public services are clearly one of the most contentious political issues of the day. This has been captured within the context of health care in terms of the health care dilemma (Phillips and Prowle, 1992), but this concept, of ever increasing demand for services against a background of restrictive supply of resources, can be applied across many areas of public service provision.

When we say that a policy is working well (or badly) or that a service is getting better (or worse) what do we mean? Do we mean that some or all members of the public think that it is so? Or that managers or politicians think so? Or that ‘performance indicators’ suggest that it is so? The words ‘well’, ‘badly’, ‘better’ and ‘worse’ are, to say the least, ambiguous.

For evaluations to be useful we need to reduce such ambiguities as far as possible and to be explicit about what we mean. This is where the notion of a criterion comes in. A criterion is a principle or standard by which something may be judged or decided and derives from the Greek for a ‘means for judging’ (New Oxford Dictionary of English). In evaluating a policy or service it is important to be explicit about the criterion or criteria being used. Otherwise a claim that, for example, a service is ‘good’ or ‘getting better’ would be rather meaningless. Would it be getting better in relation to efficiency, effectiveness, equity, reducing costs, or what?

There are a number of criteria which can be used in making such judgements. The criteria are not mutually exclusive, and sometimes the boundaries between them might be blurred; sometimes the criteria might overlap; for example, the ‘acceptability’ of a service might depend on its ‘responsiveness’ to users’ needs and wishes. We shall examine a range of criteria commonly used in evaluations (in no particular order) and set out what each is usually taken to mean. We also critically comment on the limitations of each criterion, demonstrating that in real-world evaluations judgements will always be problematic and contestable.

It is important to recognise that criteria are not always set or ‘pre-ordained’. Sometimes they ‘emerge from the specific social contexts of various stakeholders’ (Abma, 1997, p 35). For example, the ‘acceptability’ of a spending programme, as far as the electorate is concerned, is clearly a largely political question.