1. Drawing together and presenting knowledge within a disciplinary field

The process of drawing together and presenting others’ knowledge requires skill. In a purely academic context, all researchers are already trained in the basics of synthesis, as they must have a robust overview of the knowledge in their own field that is relevant to their own research interests, in order to make their own new contributions to it. Evidence synthesis in the policy context requires two additional elements.Footnote ³

The first additional element is the ability to grasp the meaning and scope of the policy question. As with many instances of evidence provision, this element requires the author(s) of the synthesis to be sufficiently embedded in the systems of exchange between research and policy to understand what matters to the policy-makers.Footnote ⁴ It is rare for a researcher to know what the policy-makers want to know, rather than what the researcher thinks they ought to know. Sometimes both could or should be included in any dialogue, but it is important to acknowledge the distinction between them. For researchers, therefore, creating synthesis requires a commitment to the intellectual challenge and purpose that supersedes their desire either to create wholly new knowledge within their field or to promote their own preference or interests in setting out existing knowledge.

The second additional element is to be able to draw on relevant knowledge through understanding the whole field and then, with the policy purpose in mind, to make judgements about what to include and exclude and how to set out those choices. Creating the synthesis will require judgements on how to weigh different findings and how to present consensus, controversy, variations in interpretation, and known gaps in evidence within the field (Figure 1).

Figure 1.

The creation of evidence synthesis within a single discipline or field.

For many humanities scholars, these activities are partly instinctive but at the same time pose a deep challenge. Essentially, they require the author to put aside their personal academic interests and ambitions, and any personal political preferences, and to try to inhabit the mind of the intended policy user. In the humanities, this poses an additional challenge since personal political views can be mirrored in chosen theoretical approaches to the material under study (e.g., a Marxist approach or a feminist approach). In such circumstances, synthesis must either (1) acknowledge that it is presenting evidence from only one area of theoretical approach, synthesise the most relevant knowledge within that approach, but recognise that such a synthesis is limited, or (2) synthesise from across different theoretical approaches to a given subject matter, as impartially as possible, contrasting and interpreting the findings of different approaches. This latter approach to synthesis might be more challenging to the individual academic, but it will produce more robust and useful evidence.

In making this move from fulfilling the desire to promote a personally preferred line of academic argument or policy outcome, to fulfilling the desire to inform public reasoning through the provision of synthesised evidence, the notion of the “Honest Broker,” which derives largely from work with natural and physical scientists, may be useful.Footnote ⁵ The aspiration to be an Honest Broker can guide and challenge the construction of the synthesis and expand the range of possible orientations towards evidence and policy for an individual researcher to take at any point in their career.

The choice of whether to engage with today’s institutions for public reasoning, and especially whether to engage directly with governments, can appear to be simultaneously a choice about whether to surrender the ability to oppose or critique today’s public goals and aims. However, there are many ways in which researchers may take part in public reasoning, and it is not always necessary to stay outside to retain the capacity to critique; indeed, in some cases, the “difficult” message may be heard more clearly if it comes from an internal or unexpected source. Relationships can enable collaboration in the provision and use of evidence while keeping space for opposition or confrontation when the evidence requires it, and different researchers may choose to play various roles across academic and public domains.Footnote ⁶

A major systemic challenge is that synthesis is not widely rewarded within academic frameworks, in part because it is not seen as creating new academic knowledge. The UK government’s 2021 Research Excellence Framework exercise included originality as one of its three criteria in the assessment of research outputs (the other two being significance and rigour), but the experience of the provision of evidence in policy-making and the media shows that too great a focus on individual pieces of work can distort debate and be counterproductive. The pressure to herald each new paper or book as ground-breaking results in messaging that can be received by those outside academia as meaning that the previous evidence was in some way not robust. That inference, in turn, weakens confidence in current knowledge. Unmediated or non-contextualised controversies and disagreements may also leave policy-makers or the public disinclined to consider any of the evidence, rather than try to find the time and skill to develop a better understanding of why findings change or what causes apparent differences between them.

A further systemic challenge directly affecting the humanities is that many humanities disciplines have deeply embedded notions of the brilliant lone scholar or the influential single public intellectual. The value and complexities of this role are illustrated nicely by Jonathan Koshy Varghese in his article on the public intellectual, Edward Said.Footnote ⁷ Like charismatic stories, charismatic scholars (and scientists) have an important part to play in drawing attention towards the issues they promote. However, reliance on individuals, however comprehensive their knowledge, works against the notion that no person or academic report can say everything that might need to be said for the public good in the context of a major public policy issue. (Too great a focus on the individual may also increase the risk to the well-being of an academic who has a high profile at the time of significant public controversy.) Additionally, except as a pragmatic preference for the alternatives during extremely short-term emergencies, there is neither time nor scope within the formal and informal mechanisms of decision-making to bring all the leading researchers together. Nor is it possible deeply to immerse the policy-makers directly in all the arguments of a field by exposing them personally to all the relevant researchers. Even if there were, in the realm of policy-making and public reasoning, there is additional value in being a record of the knowledge available at the time. For a policy-maker, having a synthesis of the evidence in a form that is available to others, preferably online and free of charge, helps them rely on it in their arguments, whether across government departments, political parties, or the public.

2. Making the knowledge accessible outside the discipline

The evidence synthesis must be not only authoritative (in the sense of being sufficiently robust to challenge from relevant experts that decision-makers can cite it with confidence) but accessible. Here, the challenge is the same as that for any researcher seeking to engage beyond their discipline: how to convey the essence of the academic knowledge in a way that the intended audience can absorb and use it for their purposes, without losing so much credibility that the author’s reputation with their academic cohort is damaged. To quote a practitioner: any statement that is useful is not perfectly accurate and any statement that is perfectly accurate is not useful.Footnote ⁸ Each discipline and sub-discipline has its own terms, and, even more challengingly, many different disciplines use the same term to mean different things (think “model” or “memory” for example). The good news is that the challenge of communicating across disciplines or between humanities and sciences is fundamentally the same as that of communicating with interested public or decision-making audiences, so there is usually only one step to take rather than two.

Two recent short projects – one on the application of narrative evidence to the policy areas of Nuclear War and one on the Future Uses of Space – illustrate how humanities scholars initially unfamiliar with the practice of synthesis can be commissioned to produce useful results. Both projects incorporated the storylistening framework for the theory and practice of applying narrative evidence to policy questions.Footnote ⁹ The projects’ reports, as well as supporting material relating to the creation of the papers and workshops used to carry out the synthesis that informs the reports, are all available online.Footnote ¹⁰ The projects were designed in part to test new ways of synthesising knowledge in the humanities, and with the aim of publishing templates and tools for others working in the humanities and policy, to adapt as needed. A specific illustration of the kind of shift narrative evidence can create was the extended discussion of different ways of understanding networked governance in current and anticipated arrangements for policy towards space in a wide range of physical, technological, and political environments.

Illustrating another systemic challenge to greater engagement in expert advisory systems for all types of researchers is the fact that these project reports are inevitably a form of grey literature: the requirement to be intelligible outside a discipline is usually incompatible with the requirements of peer review within a discipline. Therefore, although the project materials are peer-reviewed in the sense of being quality-assured for the relevant audiences, it requires further steps (not funded as part of the original project grant) to convert all or part of the work into papers suitable for academic journals.

For the Nuclear project, the tight timescale and small budget led the project’s Advisory Board to commission only 10 papers, each synthesising the available knowledge from an area of literature and each no more than 1,500 words long. The authors were selected by the Board for being respected in their disciplines and able to write on topics that were both academically significant and had been identified as potentially policy-relevant. They were given written and oral briefings on length, style, and approach, along with a summary of seven semi-structured expert practitioner interviews that highlighted areas of particular interest to the practitioners. The resulting reports, designated as “stimulus” papers, were used to inform a workshop attended by representatives of a range of disciplines and a mix of practitioners and researchers.

3. Synthesising humanities evidence – some examples of narrative evidence synthesis

Good synthesis follows four general principles.Footnote ¹¹ First, it is inclusive, involving decision-makers so that the findings are relevant and useful to them, and considering as wide a range of evidence sources as needed. Second, it is rigorous, using the most comprehensive feasible body of evidence, recognising and minimising bias and, where at all possible given constraints of time, it is independently reviewed as part of a quality assurance process. Third, it is accessible, being written in language suitable for the intended audiences, and available in a relevant time frame. Except in rare circumstances such as some connected with national security, the material should be freely available online, so that it can be interrogated by anyone. Fourth, it is transparent, in the sense of clearly describing its methods, sources, QA processes, and the personal, political, and organisational interests of those involved in its creation. It is also transparent in the sense of taking care clearly to communicate complexities, areas of contention, limitations and uncertainties, and known evidence gaps.

The exploratory Future Uses of Space project was funded by Research England and Capabilities in Academic Policy Engagement. The project’s purpose was to demonstrate how narrative evidence can inform decision-making. The project report, Reading the Stars: Narrative Evidence for Space Strategy, identifies, synthesises, and presents narrative evidence on space to drive the development of better space policies and strategies.Footnote ¹² It is presented on the website alongside other key project documents: a Project Summary, outlining the activities and phases of the project, including a conceptual map of the project as a whole, and of the workshop; a Key Policy Concerns document; Synthesis Guidance for the academics commissioned by the project, and the Synthesis Papers they produced; all the materials for the Workshop; and Storylistening Methods Charts, which provide a graphic representation of a highly generalised project to provide evidence for policy, showing how narrative evidence can be included within such a structure. We present these materials with the intention that they might provide insights and guidance for the design of future projects concerned with public reasoning, the humanities, and narrative evidence.

The Advisory Board commissioned seven synthesis papers, each of 7,000–8,000 words. The authors were selected to cover areas of academic and policy interest identified by the Advisory Board, and their papers informed a workshop with researchers and practitioners, and were drawn on directly in the project’s final findings. For most academic participants, at the start of the project, evidence synthesis was a novel concept and, as well as written guidance, authors were provided a training session and support throughout. In light of the key policy concerns identified, the commissioned synthesis authors produced syntheses of, for example, narratives of community, participation, and belonging in Space Science; science fiction as evidence to inform terraforming policy; non-mainstream and counter-narratives of outer space; fictional, scientific, and artistic narratives of space domain awareness and dark sky protection; and classic and contemporary narrative framings of space exploration, including the frontier, the maritime, and Afro- and Indigenous futurism. The final report was then itself a synthesis of these synthesis papers, together with knowledge produced at the project workshop and through close readings of specific narratives.

Some differences in synthesising in relation to narrative evidence in particular emerged, such that a distinction had to be made between the synthesis of critical literature about stories, and the synthesis of stories. Often the latter was more productive in producing synthesised knowledge of relevance to a policy question. In this case, synthesis in literary studies, for instance, is perhaps not as far removed from what is traditionally considered “original” research as it may be in other disciplines. Other disciplines within the humanities will need to consider the specificities, challenges, and opportunities of synthesis in their context, which might often mean an assessment of where the location of knowledge in their discipline lies.

The space synthesis papers also revealed the challenge of synthesising without prioritising personal political persuasions. Where the personal values of the researcher are explicit and openly inform the choices made in synthesising relevant evidence, the resulting synthesis is still useful, albeit restricted to a certain set of evidence of relevance to the policy issue. But it may yet be challenging for a literary scholar, for instance, if commissioned to produce a synthesis of valuable economic thinking in literary texts, to produce a synthesis that represents equally and without personal political or economic judgement Edward Bellamy’s socialist utopian thinking in Looking Backward: 2000–1887 (1888), and the stories of Ayn Rand, such as The Fountainhead (1943) and Atlas Shrugged (1957), which played a fundamental role in shaping and sustaining American conservatism and libertarianism.

As for the nuclear storylistening project, the synthesis papers and project report were quality-assured by the Advisory Board but not peer-reviewed. Given more time and resources, it is always desirable to have individual synthesis papers peer-reviewed independently by others within the discipline.Footnote ¹³ However, in practice, the nature of the process for quality assurance or peer review of the outputs of evidence synthesis within or across disciplines depends on the scale of the project, discussed in Section 4. The more time or funds available, the more it is possible to build in review within and across disciplines. This is especially important because the challenges of reviewing multidisciplinary work, discussed in the next section, even for the purposes of academic publication or appointment, are well established.Footnote ¹⁴ In the best instances, it requires a relevant group to learn enough about the language, mental models, and norms of all of the fields concerned, so that they can make thoroughly informed judgements. This requires extended investment in people’s time, with repeat engagement, such as that available in the UK system through Royal Commissions.Footnote ¹⁵ Time brings its own challenges, as delaying the provision of the final advice increases the risk of decoupling the work from the policy need, which may move on more rapidly.

4. Combining knowledge across disciplines

No significant public policy issue can be sufficiently informed by a single discipline. Therefore, evidence synthesis usually also includes a stage of expertly combining knowledge across disciplines. The specific methods depend on the context, particularly the time and funds available, as discussed in the next section. The same types of judgements apply at this stage as within the single-discipline syntheses: what to include, how to weigh material, and how to explain consensus, contention, and gaps (Figure 2). In addition, at this stage, it may be necessary to explain enough about the construction of the different forms of knowledge, based on different disciplinary methods and practices, to enable the intended audiences to internalise and deploy them.

Figure 2.

The creation of evidence synthesis drawing on multiple disciplines or fields.

To deliver on these principles, there need to be in place arrangements to enable judgements about rigour and quality in multiple disciplines and to account for them to the policy-makers, if required. This may be one of the hardest problems for humanities scholars and social scientists engaging in policy-making. Put baldly, if randomised control trials and statistics are considered, as they can sometimes appear to be in discussions of applied research, the best achievable indicators of rigour for policy-making, then qualitative research starts and ends at a disadvantage. This challenge is further compounded by the notion that using rigour as the key concept determining research quality may in itself be another compromise necessary to facilitate the humanities–policy interface as rigour is just one category, and not the predominant one, that humanities scholars use to describe and assess research quality.Footnote ¹⁶

Appealing to the shorthand of the phrase “the scientific method” in some contexts in public reasoning may have value, suggesting that it is a disadvantage that there is no equivalent “humanities method,” but, as Naomi Oreskes points out, there is, in reality, no single scientific method, just as there are many methods within the humanities.Footnote ¹⁷ Indeed, part of the value of evidence synthesis is helping policy-makers understand enough about different methods to help them interpret and make judgements on the findings rather than simply to compare them.Footnote ¹⁸ These complexities and potential confusions mean that it is also necessary for those inside a discipline to go further than they habitually do, to find ways to express to those outside it what constitutes excellence or rigour. Such a process of explaining can feel either too simplistic or irritatingly unnecessary because it seems to imply a lack of trust in or respect for the discipline or an individual(s) working in it. However, it is often not sufficient to rely on the signifiers of excellence (such as academic institution or status of a person or publication), especially in the context of public reasoning where such signifiers carry different levels of salience to different audiences.

Despite the absence of a “humanities method,” the generalised description of the tools or methods whose correct application ensures rigour in humanities work does, perhaps unsurprisingly, sound very similar to that for scientific work. To illustrate from literary studies:

Complementing such an account of method in a humanities discipline, which includes much of what would popularly be considered scientific methodology, is an account of what constitutes excellence in science recently given by a former Chief Scientific Advisor in the UK government and which uses what might popularly be considered subjective or emotional terms:

5. Synthesis at every scale

The principles of good synthesis can be applied at almost any scale along the dimensions of time, number of people, disciplinary spread, and cost. The UN’s Intergovernmental Panel on Climate Change’s many reports represent the work of thousands of experts (although despite its large scale it has not significantly included the humanities) over many years.Footnote ²¹ By contrast, the UK’s Science Advisory Group in Emergencies has given advice in a period of hours and days in response to, for example, the emergency at the Fukushima nuclear power plant.Footnote ²² A single expert in a room with policy-makers at the point they are making a decision may be left with the professional dilemma of offering an on-the-spot form of synthesis, knowing it to be very imperfect, or staying silent and letting the decision be uninformed by any relevant knowledge at all.

In fields with strong and consistent demand for evidence synthesis, it is necessary to ensure that reports are sufficiently up-to-date with new knowledge emerging in the relevant disciplines. So, for example, the Cochrane Database of Systematic Reviews in healthcare encourages authors to update their published reviews to reflect the findings of new evidence when it becomes available.Footnote ²³

6. Plural, interdisciplinary, and novel evidence

The case studies described here were based on the storylistening framework for creating narrative evidence, but the methods are more widely applicable across the humanities. Storylistening draws on theories of the functions of stories across disciplines, including English, Psychology, Sociology, Cultural Studies, Anthropology, Philosophy, History, History and Philosophy of Science, and Futures Studies.Footnote ²⁴ Future projects will further extend and test the range of humanities disciplines being included in evidence synthesis.

Meanwhile, this article, and the Nuclear and Space case studies and their associated materials, act as guidance and tools for those interested in designing their own projects and engaging in evidence synthesis. For evidence synthesis to be more widely created and used, and more frequently to include the humanities, further systemic change is also needed. Those designing expert advisory systems to support policy-makers need to consider how to embed humanities; those funding and rewarding research need to do more to recognise the academic value of high-quality synthesis in its own right, as well as its value in terms of impact. And humanities researchers need to be better at and more comfortable with explaining how they create knowledge, and working in collaboration with those in power, as well as in opposition to them.