Highlights
The agricultural research for development (AR4D) domain is becoming increasingly complex, and theory of change (ToC) approaches can provide critical guidance through the transformation maze concerning engagement, partnership, and research.
Most of the major benefits that accrue to the use of ToCs relate to internal learning within project teams.
Finding the balance between applying a ToC that is both useful and time- and resource-smart is challenging and may need iteration to get right.
Quantitative impact assessment methods must be blended with qualitative methods in ToC-based AR4D so that evaluation becomes about both process and numbers, and new methods are needed for blended evaluation.
The evidence base concerning the efficiency, efficacy, and failings of ToC-based AR4D urgently requires further development and synthesis.
15.1 Introduction
Despite the substantial improvements in human well-being that have occurred over the last thirty years, there is broad agreement that our food systems are not on track to reach the Sustainable Development Goals by 2030. Several different reports published since 2020 address the sustainability, economic, and policy aspects of food systems. These reports converge on one core message: we need to transform our food systems.
Agriculture is generally recognised as a key entry point for effective poverty-reduction strategies, and the adoption of improved practices, technologies, and policies has had strong, positive impacts (Alston, Reference Alston, Pardey and Rao2020; Christiaensen et al., Reference Christiaensen, Demery and Kuhl2006). Even so, food insecurity and rural poverty are persistent challenges (FAO et al., 2021). The reasons are many and complex but can be encapsulated in the observation that the rate of change in many socio-economic and Earth system trends appears to be accelerating to the point where the past is no longer a good indicator of the future (Steffen et al., Reference Steffen, Broadgate and Deutsch2015). Agricultural research for development (AR4D) faces big challenges in prioritising, targeting, and implementing activities of a type and scale that can make the best use of the many billions of dollars required to ensure food and nutrition security for all in the face of economic and zoonotic shocks and of a warming and increasingly variable climate (GCA, 2019).
The CGIAR research program on Climate Change, Agriculture and Food Security (CCAFS) was a relatively early adopter of theory of change (ToC) and impact pathway thinking as a way to orientate research, engagement, and capacity-development activities, in the quest for a food-secure future and ‘best use’ of financial resources. Robust evaluation of the effectiveness of a ToC approach is still some way off; CCAFS’s experience of it has, however, generated lessons that could enhance its effectiveness at scale. In the next section, we provide some background on the theory of change and briefly discuss its implementation in CCAFS, including how the approach was modified through time, driven by the need for pragmatism and nimbleness in reacting to change. We draw out some of the lessons learnt regarding monitoring, evaluation, and institutional and behavioural change. Examining some implications for partnerships, engagement, research, and institutional structures, we conclude with a discussion of the future implementation of ToCs in AR4D food-system transformation programmes.
15.2 Theory of Change Background
The process of AR4D is a set of research activities that produce the outputs used to contribute to behavioural change, or outcomes, via changes in the knowledge, attitudes, skills, and practices of development practitioners, extension services, farmers, and policymakers. These behavioural changes lead to impacts such as increased food security and reduced poverty. The processes that link inputs, outputs, outcomes, and impact are usually much more complicated and iterative than this (Thornton et al., Reference Thornton, Schuetz and Förch2017). How this process has been framed has changed through time, driven mostly by development and funding agencies seeking to heighten accountability to their constituents, attribute impacts, aggregate results, and establish incentives and processes to stimulate the use of performance information in management decision-making (Binnendijk, Reference Binnendijk2000).
With roots in program theory of the 1960s, ToCs have expanded to encompass a range of evaluation approaches introduced to explain why some development interventions created impact while others did not. The call for ToC-informed intervention design was triggered by the needs of evaluation practitioners who sought to understand how outcomes arise (Maru et al., Reference Maru, Sparrow and Butler2018). Another contribution to the evolution of ToCs was the rise of participatory approaches that catalyse positive development outcomes via social learning (Kristjanson et al., Reference Kristjanson, Harvey and Van Epp2014). There is no single definition of a ToC and no set methodology; rather, the approach allows flexibility according to the needs of the user or implementer (Vogel, Reference Vogel2012). A ToC provides a detailed narrative description of a hypothesised impact pathway – the logical causal chain from input to impact – and how changes are anticipated to happen, based on assumptions as to how the world works. The process of developing a ToC ideally involves a range of stakeholders who try to articulate the linkages and assumptions between inputs and outcomes (Figure 15.1). Progress is continuously monitored and the ToC modified in light of unexpected or unforeseen changes. Approaches based on ToCs hold out considerable promise, even if more robust evidence for their effectiveness in delivering the desired outcomes is needed (Alvarez et al., Reference Alvarez, Jost and Schuetz2014; Thornton et al., Reference Thornton, Schuetz and Förch2017). A ToC is no panacea, but it can facilitate broad commitment to learning from individuals and organisations, widely seen as an essential element of sustainable development.
Theory of change cycle (Omore et al., Reference Omore, Kidoido and Twine2019, from O’Flynn & Sonderskov, Reference O’Flynn and Sonderskov2015)
There are several implications of implementing ToC approaches in practice. First, there is the need to formalise a project’s ToC by involving a wide range of stakeholders in its design. Second, the assumptions that underlie the ToC should be examined regularly, and adjustments made if needed – for example, new or different partners may be added, or a particular assumption may simply not hold. Third, an effective and efficient monitoring system must be established, which may be qualitative as well as quantitative. Fourth, space must be provided for project reflection and learning. We revisit such practical issues below.
15.3 Application of Theories of Change in the CGIAR Research Program on Climate Change, Agriculture and Food Security
At different stages in CCAFS’s evolution as a research program, from its design in 2008 to its end in 2021, it utilised various program theories (Figure 15.2 and Table 15.1). At the start, a log frame approach1 was used to plan and monitor project activities across the portfolio. Projects’ annual plans and reports were collected, harmonised, and consolidated manually (Figure 15.2). Because project teams could make individual adjustments to shared templates, submissions lacked standardisation across the project portfolio. It became clear that another way of planning for and capturing outcomes was required, including engagement and capacity enhancement as key strategic elements of the work. Partners started experimenting with learning-based approaches within AR4D, recognising the need to include mechanisms that challenge ‘business as usual’ and support institutional learning and innovation, to ensure research contributes to development outcomes.
In 2013–14, the program piloted ToC approaches in one thematic area, involving six new multi-annual projects selected via a competitive process in two regions for gender-focused research (Jost et al., Reference Jost, Kristjanson and Ferdous2015; Thornton et al., Reference Steffen, Broadgate and Deutsch2014). These activities helped support a stronger focus on outcomes, especially behavioural changes in people, and made partners, engagement, people, and actors for change central to implementation following the ‘three thirds’ principle: a third of the effort spent on working with next-users to build relationships and define their needs, a third spent on the research itself, and a third spent on enhancing next-users’ capacity to take up research outputs (Vermeulen & Campbell, Reference Vermeulen and Campbell2015). A key element was encouraging the program’s researchers to consider and plan for the use of their research results by partners and stakeholders and to take responsibility for the findings being used. Reporting on project progress and results was designed regarding the key elements of a ToC: outcomes, outputs, compliance with program core values such as gender and social inclusion, and partnerships. Reporting was complemented with ‘outcome stories’ in which projects could explain the behavioural changes they observed and present evidence of their contribution to these changes. The program invested in building capacity to train project, thematic, and regional portfolio managers in using the ToC approach. The approach was extended to the entire project portfolio after one year of the pilot.
To strengthen the institutionalisation of ToC approaches, the program combined them with an explicit results-based, adaptive management monitoring, evaluation, and learning framework (Schuetz et al., Reference Schuetz, Förch, Thornton, Uitto, Puri and van den Berg2017). This allowed the program to continuously refine both the ToC and project plans, to react in an agile manner to lessons and opportunities as they arose. ToCs were essentially nested at multiple scales: global, regional, country, and project scales. The program invested in the development of an online system to collect planning and reporting data and information. This system enabled implementation of standards, aggregation of results, and guidance of processes in planning and reporting for accountability, learning, and decision-making. Allied with this functionality was a process of annual project performance evaluation. Projects were scored based on their contributions to outcomes – a heavily weighted variable – delivery of outputs, use of inputs, and compliance with key program values such as gender and social inclusion. The evaluation drew on internal leadership and external reviewers.
Several of these elements were modified and expanded in later years, by which time the ToC and adaptive-management approaches had been adopted throughout the program (Box 15.1). Annual planning and reporting were carried out using the online system, which was also adopted by several other research programs as well as CCAFS. The online system was further developed to provide some analysis and business intelligence information for project and program leaders, which could be evaluated against plans on a regular basis. Outcome stories became a key component of annual reporting, and planning and reporting templates stabilised, resulting in a largely consistent set of plans and reports in several programs’ latter years, covering a substantial proportion of the CGIAR system’s work. Project evaluations informed the stage-gating of investment decisions, that is, whether projects were to be stopped or continued.
The program leader for West Africa guided the development of a ToC that brought together many different elements along the impact pathway. It encompassed projects that were implemented in the region across several different thematic areas and scales. The projects were funded by different donors, but through the ToC they were brought into a coherent programmatic portfolio, complementing each other to address a common challenge within the same geography. Through the ToC approach, the program leader and projects together built synergies and identified gaps and ways to fill them:
In our TOC it was clearly mentioned that for the uptake of climate services we would work with AGRHYMET as a regional climate centre and with national meteorological agencies to improve climate information services, to ensure that they innovate in terms of partnerships with other stakeholders for dissemination. This was part of the ToC. And it is through this that we built new partnerships – for example, with mobile phone companies and rural radio across Senegal – to disseminate climate information. So definitely it was very useful.
15.4 Lessons for Success
Regarding ToCs within CCAFS, several factors contributed to their relatively successful use. Mistakes were certainly made, but the program embraced ToCs, including mainstreaming research that focused on linking knowledge with action, leaders who championed the approach, identification of key leverage points in program-impact pathways, and inclusion of local partners to ground the theories in reality and challenge the assumptions. The successful creation and use of ToCs require mindful and intentional implementation by a program’s leadership (Chapter 18). They must lead by example and demonstrate the necessary internal knowledge, skills, and attitudes towards processes for the creation and use of ToCs (Box 15.1).
In terms of using ToCs to lead transformation, identifying transformational leverage points is critical (Chapters 4–14), along with the partners who can deliver change. As such, ToCs that are more specific in pinpointing partners and desired changes will be likelier to lead to success. Strategic partners and targeted tactics to influence the behaviour of key persons and institutions, that is, ‘leverage points’, are essential, and ToC-process teams must consider these. For example, better sourcing transparency along food supply chains will require engaging with the private sector, but a ToC that specifies ‘the private sector’ as a partner will inherently be less successful than one that clearly names a major supplier in a specific chain, and possibly even identifies an individual or unit within that company as the leverage point on which to focus. In the South Asia regional program, for example, efforts to increase the uptake of index-based crop insurance were successful following direct engagement with one of the major insurance companies and its team. This process helped develop new thresholds for determining when policy payouts should occur. Different types of partners will play a role at different stages along impact pathways; therefore, the leverage points will also differ. For example, establishing local relations for site-based participatory research requires a very different leverage point than convincing a ministry of agriculture to include specific climate actions in its upcoming strategic plans. A ToC is useful to help teams think through these processes as clearly and intentionally as possible.
In these planning processes, ToCs can be generated using a top-down approach, involving only key project team members. However, undertaking a bottom-up participatory, inclusive approach that achieves critical buy-in from key stakeholders for implementation can be more rewarding, despite its challenges. In practice, both top-down and bottom-up approaches are needed. The bottom-up approach can provide insights from a broader group of stakeholders that highlight realities not apparent to those who first envisioned the project and that may have hampered success if not understood from the planning stage onwards. The top-down approach keeps sights set on the overall goals and targets that donors expect to see met. The ToC established at the beginning of a program or project may need to be reviewed at intervals, allowing for course corrections with partners before actions lag too far behind or veer off track. At the same time, keeping accountability at the level of outcomes is key for success; taking accountability to a more granular level may compromise the ability of projects to be agile and take advantage of opportunities as they arise, leading to potentially cumbersome administrative adjustments.
Setting up ambitious science-based goals is important for measuring targets with metrics that cover diverse outcomes – for example, productivity, nutrition, gender, and climate – but are also relevant to specific contexts. For a large research program, this may mean setting up a broad, inclusive results framework.
Adherence to a ToC approach requires investments of time and financial resources. It also necessitates persistence and purposeful revisiting of the ToC on an annual or regular basis to adjust as needed, given that it may be difficult to get right the first time. This can mean getting heavily involved in what can be a complex process for relatively short-term projects. For longer-term projects, the investment can pay off when there is enough buy-in from senior management, the process remains flexible, and critical leverage points and partners are identified from the beginning.
Having ToCs at multiple levels with numerous assumptions, ideally co-produced with partners, can be burdensome; some partners, such as the private sector, may prefer extreme streamlining of the ToC approach. In fact, ToC language may never be used when working with such partners but is nonetheless inherent in the collaboration. The bottom line is that a ToC approach must be as simple as possible while still adding value.
15.5 Theories of Change Looking Forward
Recent experience highlights the need for transformation, both of our food systems and how AR4D is done. Below we pose six questions related to key gaps and the next steps in optimising these approaches to foster transformation.
1. How Broadly Do Theory of Change Approaches Need to Be Designed?
For any large, complex AR4D program with multiple activities and partners, a ‘portfolio’ approach to ToC is appropriate, meaning not all activity areas in such a program need a ToC. For example, while upstream genomics research on organisms may not benefit much from a ToC, downstream plant breeding for traits such as drought or heat resistance would benefit greatly. In that case, a ToC could match the key characteristics of new varieties with the needs of a diverse range of small-scale farmers, to hasten and widen uptake.
2. Is There Adequate Organisational and Institutional Support for Utilising Theory of Change Approaches?
In designing and implementing new projects and programs, further investment in capacity development will likely be needed. The benefits for researchers and research partners seem clear, though institutional culture itself may need to be transformed for advantages to be realised. The funders of AR4D seem fully on board with the ToC approach and the benefits it can provide (Box 15.2).
3. How Can We Build Theories of Change for Multiple Interventions at the Same Time?
Transformation of food systems will likely involve bundling, including the bundling of technical interventions such as climate information services plus climate-smart agriculture (CSA), socio-technical bundles, or technology interventions plus their enablers (Barrett et al., Reference Barrett, Benton and Cooper2020; Herrero et al., Reference Herrero, Thornton and Mason-D’Croz2020) (Chapter 8). Examples from the literature of ToCs that address bundling are currently limited. Good examples that could be shared widely and modified for similar challenges and contexts could save research teams considerable time.
4. How Ready Are Agricultural Research for Development Organisations for Transformational Change?
Historically, AR4D organisations have been very effective in fostering incremental change. Such organisations may need to consider how best to work for transformational change and create enabling environments, including ToCs, that allow it. One challenge for large AR4D organisations working in geographically and politically targeted environments is linking the different initiatives and projects operating in the same environment to maximise synergies among them; pragmatic ToC approaches can help achieve this objective (Box 15.3).
5. What Are Suitable Monitoring and Evaluation Methods?
We need new tools for the monitoring and evaluation of transformation beyond the so-called gold standard econometric approaches, which may miss many of the complex impacts of the work of AR4D organisations. Examples of such tools are provided by Carneiro et al. (Reference Carneiro, Resce and Ruscica2020) using web analytics and high-level syntheses of outcome stories (Nowak et al., Reference Nowak, Poulos and Chan2021), and there is considerable potential for other big-data-assisted methods. Regarding the power of outcome stories, as noted above, these have developed over time, and their standards have improved. These do not replace impact assessments but are complementary, broadening the evidence base of impacts and increasing inclusivity. One example is the Kenya CSA work, where impacts were assessed using a mixture of soft and hard approaches (Okumu, Reference Okumu2021).
6. What Is the Value Added by Using Theory of Change Approaches?
We must continue strengthening the evidence base for the effectiveness of ToC approaches. Transformation takes time if it is not to be utterly disruptive, and rapid transformation without all enablers in place may be relatively ineffective. On the other hand, doing all the groundwork with well-facilitated, inclusive participatory processes can take considerable time, yet also contributes to the enabling conditions needed for interventions to succeed. There is little current evidence concerning the trade-off between the degree of use of a ToC, its utility, and its costs; this evidence would be very useful in future food-system transformation projects and program design.
We interviewed a diverse range of funding partners, and all indicated that their own institutions are using ToC approaches, although for some they are quite new. Some, including the Australian Centre for International Agricultural Research, stressed their strategic and team-building value, and how a ToC approach can dampen top-down efforts to control the science agenda, which runs the risk of supporting existing injustices and behaviours that need to change. Others, for example the World Bank, had a more practical focus on how it strengthens projects’ results frameworks and the monitoring of results. Some donors, such as the Dutch Research Council and the British Foreign, Commonwealth, and Development Office, used the CCAFS approach with ToCs as an example of good practice, including the use of outcome and impact stories. There was also a recognition that project funders and implementers were able to draw on wide networks of policymakers and negotiators and build on the many partnerships based on close engagement. A ToC was thought to aid linking high-quality quantitative work with an understanding of how to motivate and change behaviour; CCAFS was appreciated for taking a holistic, outcome-focused approach that embedded the uptake of results.
Looking forward, issues to address include how to foster the consistent application of ToCs across large research systems; continued resistance to ToC application by those that want to focus only on the science and do not want to engage in bottom-up processes or be responsible for uptake; and fostering regional and country-level consultations with intended users and beneficiaries. Funders also mentioned that some ToCs get too complicated and cover areas beyond the immediate control of researchers, that is, the outcome-to-impact level is not very rigorously considered. Some funders expressed a desire to see assumptions more meticulously tracked during project life, from outputs to outcomes. Pursuing opportunities for incorporating social or triple loop learning (integrating diverse knowledge and value systems through a sequence of learning cycles), going beyond outcome stories, and using ToCs to reflect on learnings and adapt approaches in response needs to be encouraged. As one funding partner put it, ‘We need the discipline of thinking through how we think the world works and how you actually create change. Science has in many ways failed to create change so often – putting data in front of people does not create change.’
A key question in CGIAR’s 2030 Research and Innovation Strategy1 is how to change from an organisation that sees impact as an add-on to research to one that designs research impact from the start. Innovation and impact run all the way through the new strategy, and a ToC is a key organising principle for deliberate transformation into a more impact-oriented organisation:
A ToC helps researchers think of themselves as part of an ecosystem of change, rather than centring them, and encourages strategic consideration of how to work as a partnership player. It also reminds researchers to humbly consider the role of science: generated scientific outputs are only useful within the context of what everyone else is already thinking about.
A ToC is a tool that can be used to share a common vision with partners and a common strategy to achieve it. It can provide a constant resource to check progress on the journey.
For CGIAR, a ToC is a dynamic, learning-enabling tool that can help determine what is being done well, what is being learnt, whether investments or resources are being allocated in the right place, or whether alternative thinking is required to achieve longer-term goals.
Sonja J. Vermeulen, personal communication, 2021
15.6 Way Forward
The AR4D domain is becoming increasingly complex as it grapples with the need for food-system transformation. A ToC can provide critical guidance in planning and implementing projects and programs with respect to engagement, partnerships, and research. To make ToC-based approaches as effective as possible, two gaps in particular must be filled. First, quantitative impact assessment methods can be blended with qualitative methods so that evaluation becomes about both the process and numbers. At the same time, new methods need to be developed for blended evaluation. Second, the evidence base concerning the efficiency, efficacy, and failings of ToC-based AR4D urgently requires further development and synthesis, and the lessons must be applied broadly.
Open access