Skip to main content

The Multiple Purposes of Policy Piloting and Their Consequences: Three Examples from National Health and Social Care Policy in England


In England, policy piloting has become firmly established in almost all areas of public policy and is seen as good practice in establishing ‘what works’. However, equating piloting with evaluation can risk oversimplifying the relationship between piloting and policy-making.

Using three case studies from health and social care – the Partnerships for Older People Projects (POPP) pilots, the Individual Budgets pilots and the Whole System Demonstrators (WSD) – the paper identifies multiple purposes of piloting, of which piloting for generating evidence of effectiveness was only one. Importantly, piloting was also aimed at promoting policy change and driving implementation, both in pilot sites and nationally. Indeed, policy makers appeared to be using pilots mainly to promote government policy, using evaluation as a strategy to strengthen the legitimacy of their decisions and to convince critical audiences. These findings highlight the ambiguous nature of piloting and thus question the extent to which piloting contributes to the agenda of evidence-based policy-making.

Hide All
Bate P. and Robert G. (2003), ‘Where next for policy evaluation? Insights from researching National Health Service modernisation’, Policy and Politics, 31: 2, 249262.
Berwick D. M. (2008), ‘The science of improvement’, Journal of the American Medical Association, 299: 1182–4.
Bonell C., Fletcher A., Morton M., Lorenc T. and Moore L. (2012), ‘Realist randomised controlled trials: a new approach to evaluating complex public health interventions’, Social Science and Medicine, 75: 2299–306.
Bower P., Cartwright M., Hirani S. P., Barlow J., Hendy J., Knapp M., Henderson C., Rogers A., Sanders C. and Bardsley M. (2011), ‘A comprehensive evaluation of the impact of telemonitoring in patients with long-term conditions and social care needs: protocol for the whole systems demonstrator cluster randomised trial’, BMC Health Services Research, 11: 184.
Brindle D. (2006), ‘“If health can have it, why can't we?ˮ Care services minister Ivan Lewis is ready to review his pans for social care – and they’re not unambitious, finds David Brindle’, The Guardian, 18 October 2006.
Cabinet Office (2000), Adding It Up: Improving Analysis and Modelling in Central Government, London: Performance and Innovation Unit.
Cabinet Office (2003), Trying It Out: The Role of ‘Pilots’ in Policy-Making, Report of a Review of Government Pilots, London: Cabinet Office, Strategy Unit.
Cabinet Office (2014), ‘What is open policy making’,, accessed 2 July 2014.
Cameron A., Salisbury C., Lart R., Stewart K., Peckham S., Calnan M., Purdy S. and Thorp H. (2011), ‘Policy makers’ perceptions on the use of evidence from evaluations’, Evidence and Policy: A Journal of Research, Debate and Practice, 7: 429–47.
Campbell D. T. 1969), ‘Reforms as experiments’, American Psychologist, 24: 409–29.
Cartwright N. and Hardie J. (2012), Evidence-Based Policy: Doing It Better. A Practical Guide to Predicting If a Policy Will Work for You, Oxford: Oxford University Press.
Common R. (2004), ‘Organisational learning in a political environment: improving policy-making in UK government’, Policy Studies, 25: 3549.
Department of Health (DH) (2005), Independence, Well-Being and Choice: Our Vision for the Future of Social Care for Adults in England, London: Department of Health.
Department of Health (DH) (2006), Our Health, Our Care, Our Say: A New Direction for Community Services, London: Department of Health.
Exworthy M. and Powell M. (2004), ‘Big windows and little windows: implementation in the “congested stateˮ’, Public Administration, 82: 263–81.
Freeman R. (2006), ‘Learning in public policy’, in Moran M., Rein M. and Goodin E. (eds.), The Oxford Handbook of Public Policy, Oxford: Oxford University Press.
Glendinning C., Challis D., Fernández J.-L., Jacobs S., Jones K., Knapp M., Manthorpe J., Moran N., Netten A., Stevens M. and Wilberforce M. (2008), Evaluation of the Individual Budgets Pilot Programme, Final report, York: IBSEN.
Greenhalgh T. and Russell J. (2007), ‘Reframing evidence synthesis as rhetorical action in the policy making drama’, Politiques de Santé, 1: 3442.
Haynes L., Goldacre B. and Torgerson D. (2012), Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials, London: Cabinet Office.
Hendy J., Chrysanthaki T., Barlow J., Knapp M., Rogers A., Sanders C., Bower P., Bowen R., Fitzpatrick R. and Bardsley M. (2012), ‘An organisational analysis of the implementation of telecare and telehealth: the whole systems demonstrator’, BMC Health Services Research, 12: 403.
Henry G. T. and Mark M. M. (2003), ‘Beyond use: understanding evaluation's influence on attitudes and actions’, American Journal of Evaluation, 24: 293314.
HM Government 1999), Modernising Government, London: The Stationery Office.
HM Treasury (2011), The Magenta Book: Guidance for Evaluation, London: HM Treasury.
Innvær S., Vist G., Trommald M. and Oxman A. (2002), ‘Health policy-makers’ perceptions of their use of evidence: a systematic review’, Journal of Health Services Research and Policy, 7: 239–44.
Johnson K., Greenseid L. O., Toal S. A., King J. A., Lawrenz F. and Volkov B. (2009), ‘Research on evaluation use a review of the empirical literature from 1986 to 2005’, American Journal of Evaluation, 30: 377410.
Jowell R. (2004), ‘“Trying it out”: review of effectiveness of government pilots’, Presentation with Annette King, Government Chief Social Researcher's Office, Cabinet Office, Centre for Comparative Social Surveys City University, London.
Klein R. (2000), ‘From evidence-based medicine to evidence-based policy?’, Journal of Health Services Research and Policy, 5: 65.
Lowndes V. and Pratchett L. (2012), ‘Local governance under the coalition government: austerity, localism and the “Big Societyˮ’, Local Government Studies, 38: 2140.
Majone G. (1989), Evidence, Argument, and Persuasion in the Policy Process, New Haven, CT: Yale University Press.
Majone G. and Wildavsky A. B. (1978), ‘Implementation as evolution’, in Freeman E. H. (ed.), Policy Studies Review Annual, Vol. II, Beverly Hills, CA: Sage Publications.
Martin S. and Sanderson I. 1999), ‘Evaluating public policy experiments measuring outcomes, monitoring processes or managing pilots?’, Evaluation, 5: 245–58.
Matland R. E. (1995), ‘Synthesizing the implementation literature: the ambiguity-conflict model of policy implementation’, Journal of Public Administration Research and Theory, 5: 145–74.
May P. J. 1992), ‘Policy learning and failure’, Journal of Public Policy, 331–54.
McNulty J. (2012), ‘Symbolic uses of evaluation in the international aid sector: arguments for critical reflection’, Evidence and Policy: A Journal of Research, Debate and Practice, 8: 495509.
Moran N., Glendinning C., Stevens M., Manthorpe J., Jacobs S., Wilberforce M., Knapp M., Challis D., Fernández J.-L. and Jones K. (2011), ‘Joining up government by integrating funding streams? The experiences of the individual budget pilot projects for older and disabled people in England’, International Journal of Public Administration, 34: 232–43.
Newman S. (2011), ‘The Whole System Demonstrator project (presentation)’, City University, London.
Nutley S. M., Walter I. and Davies H. T. (2007), Using Evidence: How Research Can Inform Public Services, Bristol: The Policy Press.
Parsons W. (2002), ‘From muddling through to muddling up-evidence based policy making and the modernisation of British Government’, Public Policy and Administration, 17: 4360.
Patton M. Q. (2008), Utilization-Focused Evaluation, Los Angeles, CA: Sage Publications.
Pawson R. and Tilley N. (1997), Realistic Evaluation, London: Sage Publications.
Petticrew M., Mckee M., Lock K., Green J. and Phillips G. (2013), ‘In search of social equipoise’, British Medical Journal, 347: 1820.
Pressman J. L. and Wildavsky A. 1973), Implementation: How Great Expectations in Washington Are Dashed in Oakland, Berkeley, CA: University of California Press.
Rittel H. W. J. and Webber M. M. (1973), ‘Dilemmas in a general theory of planning’, Policy Sciences, 4: 155–69.
Salisbury C., Steward K., Cameron A., Peckham S., Calnan M., Lart R., Purdy S. and Watson H. (2009), Making the Most of Policy Evaluations: Overview and Synthesis of Evaluations of the White Paper ‘Our Health, Our Care, Our Say, Bristol: University of Bristol.
Sanderson I. (2002), ‘Evaluation, policy learning and evidence-based policy making’, Public Administration, 80: 122.
Schofield J. (2001), ‘Time for a revival? Public policy implementation: a review of the literature and an agenda for future research’, International Journal of Management Reviews, 3: 245–63.
Shulha L. M. and Cousins J. B. (1997), ‘Evaluation use: theory, research, and practice since 1986’, American Journal of Evaluation, 18: 195208.
Stevens A. (2011), ‘Telling policy stories: an ethnographic study of the use of evidence in policy-making in the UK’, Journal of Social Policy, 40: 237–55.
Weiss C. H. (1979), ‘The many meanings of research utilization’, Public Administration Review, 39: 426–31.
Weiss C. H. (1998), ‘Have we learned anything new about the use of evaluation?’, American Journal of Evaluation, 19: 2133.
Windle K., Wagland R., Forder J., D’amico F., Janssen D. and Wistow G. (2009), National Evaluation of Partnerships for Older People Projects, Final report, Kent: Personal Social Services Research Unit (PSSRU).
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of Social Policy
  • ISSN: 0047-2794
  • EISSN: 1469-7823
  • URL: /core/journals/journal-of-social-policy
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Altmetric attention score

Full text views

Total number of HTML views: 4
Total number of PDF views: 79 *
Loading metrics...

Abstract views

Total abstract views: 602 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 17th January 2018. This data will be updated every 24 hours.