Hostname: page-component-7d684dbfc8-hsbzg Total loading time: 0 Render date: 2023-09-23T08:54:41.891Z Has data issue: false Feature Flags: { "corePageComponentGetUserInfoFromSharedSession": true, "coreDisableEcommerce": false, "coreDisableSocialShare": false, "coreDisableEcommerceForArticlePurchase": false, "coreDisableEcommerceForBookPurchase": false, "coreDisableEcommerceForElementPurchase": false, "coreUseNewShare": true, "useRatesEcommerce": true } hasContentIssue false

The Multiple Purposes of Policy Piloting and Their Consequences: Three Examples from National Health and Social Care Policy in England

Published online by Cambridge University Press:  14 November 2014

Department of Health Services Research and Policy, Faculty of Public Health and Policy, London School of Hygiene and Tropical Medicine, UK email:
Department of Health Services Research and Policy, Faculty of Public Health and Policy, London School of Hygiene and Tropical Medicine, UK email:
Department of Health Services Research and Policy, Faculty of Public Health and Policy, London School of Hygiene and Tropical Medicine, UK email:


In England, policy piloting has become firmly established in almost all areas of public policy and is seen as good practice in establishing ‘what works’. However, equating piloting with evaluation can risk oversimplifying the relationship between piloting and policy-making.

Using three case studies from health and social care – the Partnerships for Older People Projects (POPP) pilots, the Individual Budgets pilots and the Whole System Demonstrators (WSD) – the paper identifies multiple purposes of piloting, of which piloting for generating evidence of effectiveness was only one. Importantly, piloting was also aimed at promoting policy change and driving implementation, both in pilot sites and nationally. Indeed, policy makers appeared to be using pilots mainly to promote government policy, using evaluation as a strategy to strengthen the legitimacy of their decisions and to convince critical audiences. These findings highlight the ambiguous nature of piloting and thus question the extent to which piloting contributes to the agenda of evidence-based policy-making.

Copyright © Cambridge University Press 2014 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)


Bate, P. and Robert, G. (2003), ‘Where next for policy evaluation? Insights from researching National Health Service modernisation’, Policy and Politics, 31: 2, 249262.CrossRefGoogle Scholar
Berwick, D. M. (2008), ‘The science of improvement’, Journal of the American Medical Association, 299: 1182–4.CrossRefGoogle ScholarPubMed
Bonell, C., Fletcher, A., Morton, M., Lorenc, T. and Moore, L. (2012), ‘Realist randomised controlled trials: a new approach to evaluating complex public health interventions’, Social Science and Medicine, 75: 2299–306.CrossRefGoogle ScholarPubMed
Bower, P., Cartwright, M., Hirani, S. P., Barlow, J., Hendy, J., Knapp, M., Henderson, C., Rogers, A., Sanders, C. and Bardsley, M. (2011), ‘A comprehensive evaluation of the impact of telemonitoring in patients with long-term conditions and social care needs: protocol for the whole systems demonstrator cluster randomised trial’, BMC Health Services Research, 11: 184.CrossRefGoogle ScholarPubMed
Brindle, D. (2006), ‘“If health can have it, why can't we?ˮ Care services minister Ivan Lewis is ready to review his pans for social care – and they’re not unambitious, finds David Brindle’, The Guardian, 18 October 2006.Google Scholar
Cabinet Office (2000), Adding It Up: Improving Analysis and Modelling in Central Government, London: Performance and Innovation Unit.Google Scholar
Cabinet Office (2003), Trying It Out: The Role of ‘Pilots’ in Policy-Making, Report of a Review of Government Pilots, London: Cabinet Office, Strategy Unit.Google Scholar
Cabinet Office (2014), ‘What is open policy making’,, accessed 2 July 2014.Google Scholar
Cameron, A., Salisbury, C., Lart, R., Stewart, K., Peckham, S., Calnan, M., Purdy, S. and Thorp, H. (2011), ‘Policy makers’ perceptions on the use of evidence from evaluations’, Evidence and Policy: A Journal of Research, Debate and Practice, 7: 429–47.CrossRefGoogle Scholar
Campbell, D. T. 1969), ‘Reforms as experiments’, American Psychologist, 24: 409–29.CrossRefGoogle Scholar
Cartwright, N. and Hardie, J. (2012), Evidence-Based Policy: Doing It Better. A Practical Guide to Predicting If a Policy Will Work for You, Oxford: Oxford University Press.CrossRefGoogle Scholar
Common, R. (2004), ‘Organisational learning in a political environment: improving policy-making in UK government’, Policy Studies, 25: 3549.CrossRefGoogle Scholar
Department of Health (DH) (2005), Independence, Well-Being and Choice: Our Vision for the Future of Social Care for Adults in England, London: Department of Health.Google Scholar
Department of Health (DH) (2006), Our Health, Our Care, Our Say: A New Direction for Community Services, London: Department of Health.Google ScholarPubMed
Exworthy, M. and Powell, M. (2004), ‘Big windows and little windows: implementation in the “congested stateˮ’, Public Administration, 82: 263–81.CrossRefGoogle Scholar
Freeman, R. (2006), ‘Learning in public policy’, in Moran, M., Rein, M. and Goodin, E. (eds.), The Oxford Handbook of Public Policy, Oxford: Oxford University Press.Google Scholar
Glendinning, C., Challis, D., Fernández, J.-L., Jacobs, S., Jones, K., Knapp, M., Manthorpe, J., Moran, N., Netten, A., Stevens, M. and Wilberforce, M. (2008), Evaluation of the Individual Budgets Pilot Programme, Final report, York: IBSEN.Google Scholar
Greenhalgh, T. and Russell, J. (2007), ‘Reframing evidence synthesis as rhetorical action in the policy making drama’, Politiques de Santé, 1: 3442.CrossRefGoogle ScholarPubMed
Haynes, L., Goldacre, B. and Torgerson, D. (2012), Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials, London: Cabinet Office.Google Scholar
Hendy, J., Chrysanthaki, T., Barlow, J., Knapp, M., Rogers, A., Sanders, C., Bower, P., Bowen, R., Fitzpatrick, R. and Bardsley, M. (2012), ‘An organisational analysis of the implementation of telecare and telehealth: the whole systems demonstrator’, BMC Health Services Research, 12: 403.CrossRefGoogle ScholarPubMed
Henry, G. T. and Mark, M. M. (2003), ‘Beyond use: understanding evaluation's influence on attitudes and actions’, American Journal of Evaluation, 24: 293314.Google Scholar
HM Government 1999), Modernising Government, London: The Stationery Office.Google ScholarPubMed
HM Treasury (2011), The Magenta Book: Guidance for Evaluation, London: HM Treasury.Google Scholar
Innvær, S., Vist, G., Trommald, M. and Oxman, A. (2002), ‘Health policy-makers’ perceptions of their use of evidence: a systematic review’, Journal of Health Services Research and Policy, 7: 239–44.CrossRefGoogle ScholarPubMed
Johnson, K., Greenseid, L. O., Toal, S. A., King, J. A., Lawrenz, F. and Volkov, B. (2009), ‘Research on evaluation use a review of the empirical literature from 1986 to 2005’, American Journal of Evaluation, 30: 377410.CrossRefGoogle Scholar
Jowell, R. (2004), ‘“Trying it out”: review of effectiveness of government pilots’, Presentation with Annette King, Government Chief Social Researcher's Office, Cabinet Office, Centre for Comparative Social Surveys City University, London.Google Scholar
Klein, R. (2000), ‘From evidence-based medicine to evidence-based policy?’, Journal of Health Services Research and Policy, 5: 65.CrossRefGoogle ScholarPubMed
Lowndes, V. and Pratchett, L. (2012), ‘Local governance under the coalition government: austerity, localism and the “Big Societyˮ’, Local Government Studies, 38: 2140.CrossRefGoogle Scholar
Majone, G. (1989), Evidence, Argument, and Persuasion in the Policy Process, New Haven, CT: Yale University Press.Google Scholar
Majone, G. and Wildavsky, A. B. (1978), ‘Implementation as evolution’, in Freeman, E. H. (ed.), Policy Studies Review Annual, Vol. II, Beverly Hills, CA: Sage Publications.Google Scholar
Martin, S. and Sanderson, I. 1999), ‘Evaluating public policy experiments measuring outcomes, monitoring processes or managing pilots?’, Evaluation, 5: 245–58.CrossRefGoogle Scholar
Matland, R. E. (1995), ‘Synthesizing the implementation literature: the ambiguity-conflict model of policy implementation’, Journal of Public Administration Research and Theory, 5: 145–74.Google Scholar
May, P. J. 1992), ‘Policy learning and failure’, Journal of Public Policy, 331–54.Google Scholar
McNulty, J. (2012), ‘Symbolic uses of evaluation in the international aid sector: arguments for critical reflection’, Evidence and Policy: A Journal of Research, Debate and Practice, 8: 495509.CrossRefGoogle Scholar
Moran, N., Glendinning, C., Stevens, M., Manthorpe, J., Jacobs, S., Wilberforce, M., Knapp, M., Challis, D., Fernández, J.-L. and Jones, K. (2011), ‘Joining up government by integrating funding streams? The experiences of the individual budget pilot projects for older and disabled people in England’, International Journal of Public Administration, 34: 232–43.CrossRefGoogle Scholar
Newman, S. (2011), ‘The Whole System Demonstrator project (presentation)’, City University, London.Google Scholar
Nutley, S. M., Walter, I. and Davies, H. T. (2007), Using Evidence: How Research Can Inform Public Services, Bristol: The Policy Press.CrossRefGoogle Scholar
Parsons, W. (2002), ‘From muddling through to muddling up-evidence based policy making and the modernisation of British Government’, Public Policy and Administration, 17: 4360.CrossRefGoogle Scholar
Patton, M. Q. (2008), Utilization-Focused Evaluation, Los Angeles, CA: Sage Publications.Google Scholar
Pawson, R. and Tilley, N. (1997), Realistic Evaluation, London: Sage Publications.Google Scholar
Petticrew, M., Mckee, M., Lock, K., Green, J. and Phillips, G. (2013), ‘In search of social equipoise’, British Medical Journal, 347: 1820.CrossRefGoogle ScholarPubMed
Pressman, J. L. and Wildavsky, A. 1973), Implementation: How Great Expectations in Washington Are Dashed in Oakland, Berkeley, CA: University of California Press.Google Scholar
Rittel, H. W. J. and Webber, M. M. (1973), ‘Dilemmas in a general theory of planning’, Policy Sciences, 4: 155–69.CrossRefGoogle Scholar
Salisbury, C., Steward, K., Cameron, A., Peckham, S., Calnan, M., Lart, R., Purdy, S. and Watson, H. (2009), Making the Most of Policy Evaluations: Overview and Synthesis of Evaluations of the White Paper ‘Our Health, Our Care, Our Say, Bristol: University of Bristol.Google Scholar
Sanderson, I. (2002), ‘Evaluation, policy learning and evidence-based policy making’, Public Administration, 80: 122.CrossRefGoogle Scholar
Schofield, J. (2001), ‘Time for a revival? Public policy implementation: a review of the literature and an agenda for future research’, International Journal of Management Reviews, 3: 245–63.CrossRefGoogle Scholar
Shulha, L. M. and Cousins, J. B. (1997), ‘Evaluation use: theory, research, and practice since 1986’, American Journal of Evaluation, 18: 195208.CrossRefGoogle Scholar
Stevens, A. (2011), ‘Telling policy stories: an ethnographic study of the use of evidence in policy-making in the UK’, Journal of Social Policy, 40: 237–55.CrossRefGoogle Scholar
Weiss, C. H. (1979), ‘The many meanings of research utilization’, Public Administration Review, 39: 426–31.CrossRefGoogle Scholar
Weiss, C. H. (1998), ‘Have we learned anything new about the use of evaluation?’, American Journal of Evaluation, 19: 2133.CrossRefGoogle Scholar
Windle, K., Wagland, R., Forder, J., D’amico, F., Janssen, D. and Wistow, G. (2009), National Evaluation of Partnerships for Older People Projects, Final report, Kent: Personal Social Services Research Unit (PSSRU).Google Scholar