Randomised controlled trials are the most rigorous means of assessing the outcomes of interventions. A control group enables the ‘signal’ of the intervention effect to be distinguished from the ‘noise’ of the other influences in the absence of intervention. Randomised allocation to intervention and control groups allow as fair a comparison as possible. However, most trials and systematic reviews are not as useful as they could be. Trials generally focus on estimating effects, often failing to explore the mechanisms through which these occur or how these interact with context to generate different outcomes in different settings or populations. Systematic reviews concentrate on pooling effect estimates from multiple trials from different contexts, as though there were one underlying effect that can be uncovered by pooling. They often, like most trials, fail to examine mechanisms and how these might interact with context to generate different outcomes in different settings and populations. These omissions hinder trials and systematic reviews in their role of providing useful evidence for understanding which interventions are likely to be the most promising candidates for transfer to other settings and with other populations.

Most systematic reviews concentrate on pooling effect estimates from multiple trials from different contexts, as though there were one underlying effect that can be uncovered by pooling. They often fail to examine mechanisms and how these might interact with context to generate different outcomes in different settings and populations. Realist reviews do focus on questions of what works for whom under what conditions but do not use rigorous methods to search for, appraise the quality of and synthesise evidence to answer these questions. We show how systematic reviews can explore more nuanced questions informed by realism while retaining rigour. Using the example of a systematic review of school-based interventions to prevent dating and other gender-based violence, we first examine how systematic reviews can define context–mechanism–outcome configurations. This can occur through synthesis of intervention descriptions, theories of change and process evaluations.

Realist evaluators argue that evaluations need to ask not just what works but also what works for whom under what conditions. They argue interventions need to be evaluated in terms of the mechanisms they trigger and how these interact with context to generate different outcomes in different settings or populations. Hypotheses should be worded as context–mechanism–outcome configurations (CMOCs). Many realist evaluators argue that randomised trials are not a proper scientific design, do not encompass sufficient variation in contexts to test CMOCs and are inappropriately positivist in orientation. They argue that it is better to test CMOCs using observational designs which do not use randomisation. We welcome the focus on CMOCs but disagree with the view that trials cannot be used for realist evaluation. Trials are an appropriate scientific design when it is impossible for experimenters to control all the factors which have an influence on the result of an experiment. Trials can include sufficient variety of contexts to test CMOCs. Trials need not embody a positivist approach to the science of complex health interventions if they are oriented towards testing hypotheses, draw on theory which engages with deeper mechanisms of causation and use distinctly social science approaches such as qualitative research.

Once context–mechanism–outcome configurations (CMOCs) have been refined through qualitative research, they can be tested using quantitative data. A variety of different analyses can be used to assess the validity of CMOCs. Overall, analyses will not assess CMOCs but are nonetheless still useful in determining overall effects. Mediation analyses assess whether any intervention effect on an outcome is explained by intervention effects on intermediate outcomes, and so can shed light on mechanisms. Moderation analyses see how intervention effects vary between subgroups defined in terms of baseline context (settings or populations) and so shed light on contextual differences. Moderated mediation analyses assess whether mediation is apparent in some context but not others, and so can shed light on which mechanisms appear to generate outcomes in which contexts. Qualitative comparative analyses can examine whether more complex combinations of markers of context and mechanism co-occur with markers of outcome. Together, this set of analyses can provide nuanced and rigorous information on which CMOCs appear most usefully to explain how intervention mechanisms interact with context to generate outcomes.

This chapter examines how context–mechanism–outcome configurations (CMOCs) can be assessed within systematic reviews, again using the example of a review of school-based prevention of dating and other gender-based violence. Rather than testing CMOCs by assessing whether these align with the narratives reported by included studies, realistic systematic reviews assess and refine CMOCs by assessing how they compare with the statistical regularities reported by included studies. Overall meta-analyses indicate overall effects. Network meta-analyses shed light on how different intervention components might enable generation of outcomes. Narrative syntheses of mediation and moderation analyses and meta-regression suggest how mechanisms might work and how these may generate different outcomes in different contexts. Qualitative comparative analyses examine whether more complex combinations of markers of context and mechanism co-occur with markers of outcome. These analyses can provide nuanced and rigorous information on which CMOCs appear to explain how intervention mechanisms interact with context to generate outcomes. A limitation of assessing CMOCs in systematic reviews rather than primary intervention studies is that the analyst has less control over what empirical analyses are possible so analyses tend to be more inductive.

It is important to limit statistical testing of context–mechanism–outcome configurations (CMOCs) to those which are most plausible. This is because testing too many hypotheses will lead to some false positive conclusions. Qualitative research conducted within process evaluations is a useful way to inform refinement of CMOCs before they are tested using quantitative data. Process evaluations aim to examine intervention implementation and the mechanisms that arise from this. They involve a mixture of quantitative (for example, logbooks completed by intervention providers) and qualitative (for example, interviews or focus groups with recipients) research. Qualitative research can be useful in assessing and refining CMOCs because intervention providers and recipients will have insights into how intervention mechanisms might interact with context to generate outcomes. These insights might be explored directly (for example, by asking participants how they think the interventions works) or indirectly (for example, by asking participants about their experiences of an interventions, and the conditions and consequences of this). Sampling for such qualitative research should ensure that a diversity of different participant accounts is explored. Analyses of these accounts can draw on grounded theory approaches which aim to build or refine theory based on qualitative data.

This conclusion summarises how realistic trials and realistic systematic reviews offer a method by which evaluation and evidence synthesis can become more useful by becoming more scientific and moving beyond being a form of sophisticated descriptive monitoring of ‘what works’. Evaluation can become more scientific both by continuing to use the most scientifically rigorous methods, as well as by being focused on the testing and refining of scientific theory. Critical realism and realist evaluation approaches offer a useful framework for constructing such theory because they offer the most plausible account of how causality operates in the complex social world. We hope that, by making these arguments, we have persuaded readers that trials and systematic reviews need to be reoriented and reformed rather than thrown away altogether. The incorporation of realist enquiry methods into randomised trials and systematic reviews offers us the best hope of evaluation and evidence synthesis that generate evidence which is both more scientific and more useful.

Health interventions are purposeful activities intended to improve health. They may involve treatment or care of the ill, or health promotion to prevent disease and illness. Complex health interventions have multiple components interacting with each other and with the context of delivery. Evaluation is important to ensure that complex health interventions are effective in achieving their intended outcomes, represent good value for money and cause minimal harm. Evaluation is also important to detect if interventions reduce or increase health inequalities. Intervention effects are not always obvious. They can easily be confused with other changes that occur. Hence, there is a need for evaluation to use rigorous methods to distinguish the ‘signal’ of intervention effects from the ‘noise’ of other effects in the absence of intervention. Evaluation should provide evidence to inform policy. If not based on evidence, there is a risk that policies may not achieve their intended effects or may create harms.

This chapter reflects on how evidence from realist trials and systematic reviews might be of value, not only in drawing conclusions about specific interventions and their theories of change but also in testing and refining the middle range theories which inform these and other interventions. While evaluation evidence should be of most immediate use in informing decisions about the implementation of the specific interventions being evaluated, a broader and more enduring use for evaluation could be in suggesting refinements to middle range theory. Such refinements might then be used to inform and influence the next generation of complex health interventions. In order to be useful in assessing the validity of middle range theory, evaluations will need to assess interventions informed by a limited number of middle range theories comprising a limited number of well-defined constructs. There may be value in conducting proof of principle studies separately from more pragmatic evaluations in order to test and refine middle range theory.

This chapter explores how policymakers and practitioners in settings beyond the sites of evaluation might make use of evidence from realist trials and systematic reviews, plus local needs assessment, to identify the best candidate interventions for their local contexts. To do this, local decision-makers need to assess how likely are interventions to achieve benefits in their contexts. This is partly a matter of assessing whether interventions are likely to be feasible, accessible and acceptable in their settings, which will be influenced by local capacity and norms. It is also a matter of assessing whether intervention mechanisms will be triggered and whether these are likely to generate beneficial outcomes. This will be influenced by what aetiological mechanisms are generating adverse outcomes in the context and hence what vulnerabilities exist which the intervention may be able to address. It will also be influenced by whether the local context provides affordances so that potential beneficiaries may be able to benefit from the intervention. Thinking through these issues should enable local policymakers and practitioners to decide whether such interventions could be delivered immediately at scale, be implementing but only within evaluated pilot studies or be rejected in favour of other interventions.

Theories of change propose how intervention resources and activities might lead to the generation of outcomes. They are sometimes presented diagrammatically as logic models. Realist evaluators and others have suggested that interventions should be theorised in terms of how intervention mechanisms interact with context to generate outcomes. Our own trial of the Learning Together whole-school intervention to prevent bullying set out to define, refine and test such theories in the form of context–mechanism–outcome configurations (CMOCs). We drew on several sources to define our starting CMOCs. These included existing middle range theory. This is scientific theory about the general mechanisms (i.e. not necessarily concerning an intervention) that generate outcomes. This should be analytically general enough to apply to a range of settings, populations and/or outcomes, but specific enough to be useful in a given application. We also used previous research and public consultation to inform our CMOCs.