Hostname: page-component-8448b6f56d-mp689 Total loading time: 0 Render date: 2024-04-24T08:34:08.069Z Has data issue: false hasContentIssue false
  • English
  • Français

The Multiple Purposes of Policy Piloting and Their Consequences: Three Examples from National Health and Social Care Policy in England

Published online by Cambridge University Press:  14 November 2014

STEFANIE ETTELT ,
NICHOLAS MAYS  and
PAULINE ALLEN
STEFANIE ETTELT
Affiliation:
Department of Health Services Research and Policy, Faculty of Public Health and Policy, London School of Hygiene and Tropical Medicine, UK email: stefanie.ettelt@lshtm.ac.uk
NICHOLAS MAYS
Affiliation:
Department of Health Services Research and Policy, Faculty of Public Health and Policy, London School of Hygiene and Tropical Medicine, UK email: nicholas.mays@lshtm.ac.uk
PAULINE ALLEN
Affiliation:
Department of Health Services Research and Policy, Faculty of Public Health and Policy, London School of Hygiene and Tropical Medicine, UK email: pauline.allen@lshtm.ac.uk
Get access Rights & Permissions [Opens in a new window]

Abstract

In England, policy piloting has become firmly established in almost all areas of public policy and is seen as good practice in establishing ‘what works’. However, equating piloting with evaluation can risk oversimplifying the relationship between piloting and policy-making.

Using three case studies from health and social care – the Partnerships for Older People Projects (POPP) pilots, the Individual Budgets pilots and the Whole System Demonstrators (WSD) – the paper identifies multiple purposes of piloting, of which piloting for generating evidence of effectiveness was only one. Importantly, piloting was also aimed at promoting policy change and driving implementation, both in pilot sites and nationally. Indeed, policy makers appeared to be using pilots mainly to promote government policy, using evaluation as a strategy to strengthen the legitimacy of their decisions and to convince critical audiences. These findings highlight the ambiguous nature of piloting and thus question the extent to which piloting contributes to the agenda of evidence-based policy-making.

Type
Articles
Information
Journal of Social Policy , Volume 44 , Issue 2 , April 2015 , pp. 319 - 337
Copyright
Copyright © Cambridge University Press 2014 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Bate, P. and Robert, G. (2003), ‘Where next for policy evaluation? Insights from researching National Health Service modernisation’, Policy and Politics, 31: 2, 249–262.Google Scholar
Berwick, D. M. (2008), ‘The science of improvement’, Journal of the American Medical Association, 299: 1182–4.Google Scholar
Bonell, C., Fletcher, A., Morton, M., Lorenc, T. and Moore, L. (2012), ‘Realist randomised controlled trials: a new approach to evaluating complex public health interventions’, Social Science and Medicine, 75: 2299–306.Google Scholar
Bower, P., Cartwright, M., Hirani, S. P., Barlow, J., Hendy, J., Knapp, M., Henderson, C., Rogers, A., Sanders, C. and Bardsley, M. (2011), ‘A comprehensive evaluation of the impact of telemonitoring in patients with long-term conditions and social care needs: protocol for the whole systems demonstrator cluster randomised trial’, BMC Health Services Research, 11: 184.Google Scholar
Brindle, D. (2006), ‘“If health can have it, why can't we?ˮ Care services minister Ivan Lewis is ready to review his pans for social care – and they’re not unambitious, finds David Brindle’, The Guardian, 18 October 2006.Google Scholar
Cabinet Office (2000), Adding It Up: Improving Analysis and Modelling in Central Government, London: Performance and Innovation Unit.Google Scholar
Cabinet Office (2003), Trying It Out: The Role of ‘Pilots’ in Policy-Making, Report of a Review of Government Pilots, London: Cabinet Office, Strategy Unit.Google Scholar
Cabinet Office (2014), ‘What is open policy making’, http://my.civilservice.gov.uk/policy/what/, accessed 2 July 2014.Google Scholar
Cameron, A., Salisbury, C., Lart, R., Stewart, K., Peckham, S., Calnan, M., Purdy, S. and Thorp, H. (2011), ‘Policy makers’ perceptions on the use of evidence from evaluations’, Evidence and Policy: A Journal of Research, Debate and Practice, 7: 429–47.Google Scholar
Campbell, D. T. 1969), ‘Reforms as experiments’, American Psychologist, 24: 409–29.Google Scholar
Cartwright, N. and Hardie, J. (2012), Evidence-Based Policy: Doing It Better. A Practical Guide to Predicting If a Policy Will Work for You, Oxford: Oxford University Press.Google Scholar
Common, R. (2004), ‘Organisational learning in a political environment: improving policy-making in UK government’, Policy Studies, 25: 35–49.Google Scholar
Department of Health (DH) (2005), Independence, Well-Being and Choice: Our Vision for the Future of Social Care for Adults in England, London: Department of Health.Google Scholar
Department of Health (DH) (2006), Our Health, Our Care, Our Say: A New Direction for Community Services, London: Department of Health.Google Scholar
Exworthy, M. and Powell, M. (2004), ‘Big windows and little windows: implementation in the “congested stateˮ’, Public Administration, 82: 263–81.Google Scholar
Freeman, R. (2006), ‘Learning in public policy’, in Moran, M., Rein, M. and Goodin, E. (eds.), The Oxford Handbook of Public Policy, Oxford: Oxford University Press.Google Scholar
Glendinning, C., Challis, D., Fernández, J.-L., Jacobs, S., Jones, K., Knapp, M., Manthorpe, J., Moran, N., Netten, A., Stevens, M. and Wilberforce, M. (2008), Evaluation of the Individual Budgets Pilot Programme, Final report, York: IBSEN.Google Scholar
Greenhalgh, T. and Russell, J. (2007), ‘Reframing evidence synthesis as rhetorical action in the policy making drama’, Politiques de Santé, 1: 34–42.Google Scholar
Haynes, L., Goldacre, B. and Torgerson, D. (2012), Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials, London: Cabinet Office.Google Scholar
Hendy, J., Chrysanthaki, T., Barlow, J., Knapp, M., Rogers, A., Sanders, C., Bower, P., Bowen, R., Fitzpatrick, R. and Bardsley, M. (2012), ‘An organisational analysis of the implementation of telecare and telehealth: the whole systems demonstrator’, BMC Health Services Research, 12: 403.Google Scholar
Henry, G. T. and Mark, M. M. (2003), ‘Beyond use: understanding evaluation's influence on attitudes and actions’, American Journal of Evaluation, 24: 293–314.Google Scholar
HM Government 1999), Modernising Government, London: The Stationery Office.Google Scholar
HM Treasury (2011), The Magenta Book: Guidance for Evaluation, London: HM Treasury.Google Scholar
Innvær, S., Vist, G., Trommald, M. and Oxman, A. (2002), ‘Health policy-makers’ perceptions of their use of evidence: a systematic review’, Journal of Health Services Research and Policy, 7: 239–44.Google Scholar
Johnson, K., Greenseid, L. O., Toal, S. A., King, J. A., Lawrenz, F. and Volkov, B. (2009), ‘Research on evaluation use a review of the empirical literature from 1986 to 2005’, American Journal of Evaluation, 30: 377–410.Google Scholar
Jowell, R. (2004), ‘“Trying it out”: review of effectiveness of government pilots’, Presentation with Annette King, Government Chief Social Researcher's Office, Cabinet Office, Centre for Comparative Social Surveys City University, London.Google Scholar
Klein, R. (2000), ‘From evidence-based medicine to evidence-based policy?’, Journal of Health Services Research and Policy, 5: 65.Google Scholar
Lowndes, V. and Pratchett, L. (2012), ‘Local governance under the coalition government: austerity, localism and the “Big Societyˮ’, Local Government Studies, 38: 21–40.CrossRefGoogle Scholar
Majone, G. (1989), Evidence, Argument, and Persuasion in the Policy Process, New Haven, CT: Yale University Press.Google Scholar
Majone, G. and Wildavsky, A. B. (1978), ‘Implementation as evolution’, in Freeman, E. H. (ed.), Policy Studies Review Annual, Vol. II, Beverly Hills, CA: Sage Publications.Google Scholar
Martin, S. and Sanderson, I. 1999), ‘Evaluating public policy experiments measuring outcomes, monitoring processes or managing pilots?’, Evaluation, 5: 245–58.Google Scholar
Matland, R. E. (1995), ‘Synthesizing the implementation literature: the ambiguity-conflict model of policy implementation’, Journal of Public Administration Research and Theory, 5: 145–74.Google Scholar
May, P. J. 1992), ‘Policy learning and failure’, Journal of Public Policy, 331–54.Google Scholar
McNulty, J. (2012), ‘Symbolic uses of evaluation in the international aid sector: arguments for critical reflection’, Evidence and Policy: A Journal of Research, Debate and Practice, 8: 495–509.Google Scholar
Moran, N., Glendinning, C., Stevens, M., Manthorpe, J., Jacobs, S., Wilberforce, M., Knapp, M., Challis, D., Fernández, J.-L. and Jones, K. (2011), ‘Joining up government by integrating funding streams? The experiences of the individual budget pilot projects for older and disabled people in England’, International Journal of Public Administration, 34: 232–43.Google Scholar
Newman, S. (2011), ‘The Whole System Demonstrator project (presentation)’, City University, London.Google Scholar
Nutley, S. M., Walter, I. and Davies, H. T. (2007), Using Evidence: How Research Can Inform Public Services, Bristol: The Policy Press.Google Scholar
Parsons, W. (2002), ‘From muddling through to muddling up-evidence based policy making and the modernisation of British Government’, Public Policy and Administration, 17: 43–60.Google Scholar
Patton, M. Q. (2008), Utilization-Focused Evaluation, Los Angeles, CA: Sage Publications.Google Scholar
Pawson, R. and Tilley, N. (1997), Realistic Evaluation, London: Sage Publications.Google Scholar
Petticrew, M., Mckee, M., Lock, K., Green, J. and Phillips, G. (2013), ‘In search of social equipoise’, British Medical Journal, 347: 18–20.Google Scholar
Pressman, J. L. and Wildavsky, A. 1973), Implementation: How Great Expectations in Washington Are Dashed in Oakland, Berkeley, CA: University of California Press.Google Scholar
Rittel, H. W. J. and Webber, M. M. (1973), ‘Dilemmas in a general theory of planning’, Policy Sciences, 4: 155–69.Google Scholar
Salisbury, C., Steward, K., Cameron, A., Peckham, S., Calnan, M., Lart, R., Purdy, S. and Watson, H. (2009), Making the Most of Policy Evaluations: Overview and Synthesis of Evaluations of the White Paper ‘Our Health, Our Care, Our Say, Bristol: University of Bristol.Google Scholar
Sanderson, I. (2002), ‘Evaluation, policy learning and evidence-based policy making’, Public Administration, 80: 1–22.Google Scholar
Schofield, J. (2001), ‘Time for a revival? Public policy implementation: a review of the literature and an agenda for future research’, International Journal of Management Reviews, 3: 245–63.Google Scholar
Shulha, L. M. and Cousins, J. B. (1997), ‘Evaluation use: theory, research, and practice since 1986’, American Journal of Evaluation, 18: 195–208.Google Scholar
Stevens, A. (2011), ‘Telling policy stories: an ethnographic study of the use of evidence in policy-making in the UK’, Journal of Social Policy, 40: 237–55.Google Scholar
Weiss, C. H. (1979), ‘The many meanings of research utilization’, Public Administration Review, 39: 426–31.Google Scholar
Weiss, C. H. (1998), ‘Have we learned anything new about the use of evaluation?’, American Journal of Evaluation, 19: 21–33.Google Scholar
Windle, K., Wagland, R., Forder, J., D’amico, F., Janssen, D. and Wistow, G. (2009), National Evaluation of Partnerships for Older People Projects, Final report, Kent: Personal Social Services Research Unit (PSSRU).Google Scholar