Introduction

In this chapter we review published evaluations of UK PbR and SIBs and US SIBs to assess the current state of evidence on what works in outcomes-based commissioning. We have used elements of a systematic review methodology to structure our search for evaluations, our assessment of their quality and the synthesis of results.

For every PbR and SIB programme identified in the UK (Tables 3.1 and 3.2) and the SIBs identified in the US (Table 4.1) a thorough search was undertaken for any published evaluation associated with each programme. This included searching websites associated with the programmes, their funders, investors and service providers. In addition, we undertook a structured search of two databases: ASSIA (Applied Social Sciences Index and Abstracts) and Web of Science. In total, 811 papers were sifted for relevance based on reading titles and abstracts.

In the US there are only three published evaluations, making our search relatively straightforward.

In the UK there was more material: we retained 46 empirical evaluations of UK PbR and SIB programmes for detailed analysis. No papers were excluded based on methodological rigour; however, the methodologies of these papers were assessed. Qualitative evaluations were assessed using the set of quality standards for qualitative evaluation that was drawn up by the UK government's Cabinet Office (Spencer et al, 2003). The design of impact evaluations was assessed using Sherman et al's (1998) Scale of Scientific Methods (the Maryland Scale). Some UK papers were not designed primarily as evaluations, but were nevertheless included because they had some evaluative elements. In these cases, and where the methodological standards set out above were relevant, they were applied. Where they were less relevant, professional judgement was used to assess the overall methodological rigour of the paper.

For the UK, synthesis of findings was undertaken in two stages. Some initial themes were taken from previous reviews, including Tan et al (2015), Fraser et al (2016) and NAO (2015), and relevant data from the 46 papers was extracted. Additional themes were also identified during the analytical process, to reflect themes emerging from the data. When synthesising results, findings from papers assessed as methodologically weaker were given less weight.

The bulk of this chapter focuses on the UK programmes, with a short summary of the US evidence at the end. This reflects the relatively greater UK evidence base.