2.1. Economic and social evaluation and its application to data investments and data governance

The role of the evaluation is to identify the benefits and costs of a project or intervention; assess whether the intervention is worthwhile and has delivered its aims and objectives; to communicate these results; and, ultimately, to incorporate into a broader project cycle so that lessons can be learned (Alves et al., Reference Alves, Tava, Whittard, Green, Beata Kreft and Ritchie2021). The principle of economic and social evaluation (ESE) is well established; for example, the “Magenta Book” (HMT, 2020) is a long-established guide to evaluators in the United Kingdom.

Although the literature talks of “ESE,” in practice evaluation is almost always just the economics. As Alves et al. (Reference Alves, Tava, Whittard, Green, Beata Kreft and Ritchie2021) note, EE is hard enough without the additional uncertainties brought about by considering less easily measured social costs and benefits.

Even for EE, this can be difficult. The Green Book (HMT, 2022) suggests that EE can take the form of a cost-effectiveness analysis, multi-criteria analysis, economic impact assessment, or cost benefit analysis; and ideally, the choice is between two alternatives considered ex ante. Baker (Reference Baker2000) suggests the minimum is a simple counterfactual: “doing X” versus “not doing X.”

Whittard et al. (Reference Whittard, Ritchie and Nyengani2021) summarize the guidelines on EE as follows:

• • The full costs should include direct and indirect costs and attributable overheads.

• • All benefits, both direct and indirect, should be valued unless it is clearly not practical to do—however, it remains important to consider valuing the differences between the options.

• • All costs and benefits should be valued at market price (opportunity costs).

• • Where possible, estimates for wider social and environmental costs and benefits, for which there is no market price, should be included.

• • The value should cover the useful lifetime of the asset encompassed.

• • Cost and benefits should be expressed in “real terms.”

• • Cost and benefits should be discounted due to private/social time preference.

• • Cost and benefits of different options should be valued and the net cost and benefit calculated.

This is an ideal world; these guidelines reflect the fact that methods for EE were designed for large investment projects with easily identified inputs and outputs, and a clear alternative. However, as Alves et al. (Reference Alves, Tava, Whittard, Green, Beata Kreft and Ritchie2021) discuss, these rapidly run into problems when consider more intangible investments such as those relating to data.

Multiple authors have highlighted the fundamental measurement problem in valuing the benefits of data investments and use. Key confounders include the complementarity between data, infrastructure and processes; the ability for data to be reused infinitely and simultaneously without reducing the “stock”; and the possibility that data appreciates in value over time, rather than depreciating as is normal with assets. Wdowin and Diepeveen (Reference Wdowin and Diepeveen2020) summarize both the issues and the wide range of resulting estimates arising from different assumptions, but even these are just for the valuation of data investments. As noted in the introduction, data are not the same as data governance, and there appear to be no evaluations of investments in the latter prior to Whittard et al. (Reference Whittard, Ritchie and Nyengani2021).

The issue of assumptions is perhaps the thorniest issue in relation to evaluations. For post hoc evaluations (i.e., the investment has been made, and its economic worth is now being assessed), the vested interest of both the funder and the investment recipient is about producing a “large number” (Alves et al., Reference Alves, Tava, Whittard, Green, Beata Kreft and Ritchie2021). Given the uncertainty embedded throughout the quantitative evaluation process, it does not take much in terms of loosening the parameters to turn a potentially “failing” project into one that records a strong positive return. Alves et al. (Reference Alves, Tava, Whittard, Green, Beata Kreft and Ritchie2021), who interviewed both delivery teams and stakeholders, found that almost all interviewees “saw relatively little real value in the number” (Alves et al., Reference Alves, Tava, Whittard, Green, Beata Kreft and Ritchie2021, p. 24).

When considering data governance, assumptions become even more important. Much of data governance is about reducing risk; that is, lowering the probability of something negative which might happen but ideally (if the project is well designed) will not. In other words, if data governance is well done, the outcome of it is unobservable by design.

Whittard et al. (Reference Whittard, Ritchie and Nyengani2021) throw an additional problem into the mix: evaluation in LMICs. While there have been a number of evaluation on projects in LMICs (including agriculture), evaluation methods designed for HICs may not be appropriate in places where discount rates, financing methods, outside options, and institutions differ significantly from the models envisaged. Whittard et al. (Reference Whittard, Ritchie and Nyengani2021) argue that there is a strong need to tailor the evaluation to the particular circumstances of the project, and that qualitative evaluations may be just as informative as quantitative evaluations, if not more.

2.2. The Five Safes: current application and uses

Data governance and data access are a complex issue, involving question of law, ethics, IT, statistics, contracts, HR, accreditation, and other factors (Green and Ritchie, Reference Green and Ritchie2023). The Five Safes splits data governance and access questions into five separable but comprehensive and interdependent “dimensions of control” (Table 1).

Table 1.

The Five Safes (adapted from Green and Ritchie, Reference Green and Ritchie2023)

The Five Safes was first created in 2003 at the UK Office for National Statistics to describe data management systems, particularly access to confidential data for research. In the United Kingdom, New Zealand, and Australia, it is used to describe the governance arrangements for most research data release arrangements in the health and social sciences, and for wider government data-sharing arrangements. It is an increasingly common framework across the public sector and academia in North America, Europe, and Japan. From a relatively quiet start, growth has been exponential in the last five years; the biggest stimulus to adoption was the need for innovative approaches to data governance in the pandemic.

The Five Safes is used in formal legislation, such as the UK Digital Economy Act, and in regulation. In the United Kingdom, for example, the Office for Statistics Regulation bases its guidance on the Five Safes (OSR, 2018), while key academic funders (Administrative Data Research UK, Health Data Research UK, The Innovation Hub), all require bidders to “address” the Five Safes in their data management plans. In Australia, the Five Safes has transmuted into five “Data Sharing Principles” as the basis for the new Data Access and Transparency Act 2022.

As the Five Safes has become better known in the last 10 years, it has become more common to use it actively for designing data strategies in the public sector (OECD, 2014; Green and Ritchie, Reference Green and Ritchie2016: ICON, 2016; OSR, 2018; Cranswick et al., Reference Cranswick, Tumpane and Stobert2019) and private sector (Security Brief, 2019; Arbuckle and El Emam, Reference Arbuckle and El Emam2020). Green and Ritchie (Reference Green and Ritchie2023) note that the scope of Five Safes has moved away from research data access and has now been applied to areas as diverse as managing HR systems or compliance modelling.

In summary, the Five Safes is a familiar data governance tool in HICs. While there have been some debates in recent years about whether the framework needs to be adapted or extended (Green and Ritchie, Reference Green and Ritchie2023, review these debates), in general its use as a framework is uncontroversial. As Green and Ritchie (Reference Green and Ritchie2023) note, the Five Safes does not itself provide specific guidance: it is a framework for organizing questions, rather than a checklist for answers. However, with the emergence of comprehensive implementation guides such as that being developed in Australia (McEachern, Reference McEachern2021), this is likely to change.

One area that is lacking is evaluation, both process (does the system perform as well as it could?) and economic (does the system generate a positive return on investment?). Ritchie (Reference Ritchie2017) noted that, in theory, the Five Safes provides a handy structure for evaluations; in practice, ONS (2011) appears to be the only example to date that has used the Five Safes in a formal evaluation, and that was a process evaluation for a system that mirrored ONS’ system. There appear to be no examples of using the Five Safes for EE.

A second gap is the application of the Five Safes in LMICs. It has been adopted by some public sector organizations in Mexico, Nepal, and South Africa, and in 2022, the UK National Institute for Health Research commissioned a 4-year program of virtual “summer schools” for LMIC researchers. However, these particular initiatives have not yet led to a wider adoption of the framework.

3.1. Project genesis

In 2019, the UWE team was commissioned by CAB International (CABI) to carry out an evaluation of data governance on BMGF projects. The initial aim was simply to provide an EE and to demonstrate how this could be done in practice. This was a retrospective EE, carried out on a completed (or nearly completed) project. An alternative to EE would have been a process evaluation (looking at processes from an operational efficiency perspective), but this was not explored as the aim was to see how EE could be generally employed in BMGF projects.

After initial discussions, it became clear that existing EE models were inappropriate due to the issues raised above: unmeasurability of inputs, outputs and outcomes; conceptual problems with what was being measured; and the need to situate the evaluation in the LMIC context. It was agreed that a new approach would be needed, and the project aims changed to include the development of an effective valuation method, as well as the application of that method to a BMGF project as proof-of-concept.

The evaluation was carried out on Supporting Soil Health Interventions in Ethiopia (SSHIiE). This is a $1.5 million project led by Gesellschaft für Internationale Zusammenarbeit (GIZ). The overall objective of the SSHIiE project was to provide proof of concept for the impact of leveraged geo-spatial soil, agronomic, and soil health data on transformative agricultural development. Specific aims of SSHIiE were to

• • increase the quality, efficiency, and reach of government and private sector services;

• • improve national and regional resource allocation and policies; and

• • increase the quality, availability, and utility of data and evidence.

The aim of the evaluation was not to provide a view on the value of the data acquired, but on the governance arrangements; that is, the value of how the data collection was designed, and use regulated. Accordingly, the evaluation focused on the data management arrangements for the SSHIiE project and not on the data collected.

The agreed approach was to split the evaluation of the project’s data governance into a qualitative component that would then direct and frame data collection for a quantitative component. Whittard et al. (Reference Whittard, Ritchie and Nyengani2021) describe the evaluation in detail. Whittard et al. (Reference Whittard, Ritchie, Musker and Rose2022) summarize and comment on the measurement issues associated with the evaluation. The evaluation contained several innovations. One of those concerns us in this article is the decision to use the Five Safes as a framework to structure the qualitative data collection.

3.2. Method

The first task was to identify the activities in SSHIiE that delivered “data governance.” This was not identified as a separate activity in the SSHIiE project planning, which concentrated on the practical delivery of the data collection mechanisms. The initial discussions between the review team and BMGF/GiZ sought to understand what data governance covered, and to find a common language. This is when we began to consider using the Five Safes as the organizing framework. Three of the five work packages were identified as primarily or mostly about improving the governance of data on the project.

The second stage was to identify what elements of data governance were being addressed in that work package. The Five Safes was broken down into additional subdomains, with the idea that each separate project work package with elements of governance would be tested against the subdomains, where relevant.

The third element was the link between activities and outcomes. Every evaluation requires a “logic model,” a statement of how inputs, outputs, impacts and outcomes interact. Table 3 shows a typical logic model suggested by the UK government.

Table 2.

Detailed governance questions for work packages (subset; from Whittard et al., Reference Whittard, Ritchie and Nyengani2021)

Table 3.

Typical logic model (HMT, 2020)

Combining these three elements (work packages, Five Safes, logic model) formed the basis for the team to structure its qualitative investigation.

Table 4 shows a partial view on the table generated for each of the work packages. Thus it can be seen that, for example, in work package 1, one of the activities was the Open Data Sensitization Workshop. Inputs and measurable outputs were identified. The only relevant governance domain is “people,” and this was used to identify the target outcomes.

Table 4.

Project mapping (partial view; from Whittard et al., Reference Whittard, Ritchie and Nyengani2021); asterisks indicate which ‘safe’ is relevant for this output

To undertake the evaluation, a three-staged approach was completed (see Whittard et al., Reference Whittard, Ritchie and Nyengani2021 for fuller details). In summary, the evaluation was based on 3 reports provided by BMGF, 5 interviews/focus groups covering 7 key stakeholders, and 12 targeted questionnaires, of which 8 were completed. Due to budgetary and time limitations, it was not feasible to interview individuals from all stakeholder groups or to identify a representative sample. Instead, a convenience/snowballing approach was taken. The individuals interviewed included representatives from the Ethiopian Government, agriculture sector groups, academics, and the voluntary sector. The questionnaires were sent to experts identified by the CABI team as being able to provide detailed insights into the project.

Measurable inputs and output associated with the activity and the outcomes were identified. This was used to generate specific questions for that activity. A simplified version of the full framework was then sent to the interviewees prior to interview, to indicate the topics that would be discussed, but the role of the Five Safes in structuring the questions was not emphasized to interviewees. The interviews were transcribed and analyzed.

As a post hoc evaluation with limited access to participants, Tables 2–4 represent the starting points of the analysis. Some of the questions were not appropriate or irrelevant, some were felt to be unanswerable by interviewees, and some did not seem likely to lead to useful answers. Nevertheless, the act of defining a general framework before moving to specific topics helped the team rapidly narrow the scope of what was feasible.

3.3. Analyzing findings

The interview findings had two functions: as direct evaluation data and to help structure the quantitative study for the more formal EE.

The direct analysis of the interviews was structured around the Five Safes. For each dimension, the team reported interview findings, and identified lessons learned for that governance dimension. These are tabulated below, in summary, with a subset of findings shown for illustration. For the report, the Five Safes was replaced with more meaningful descriptions of the dimensions (left-hand column of Table 5).

Table 5.

Lessons learned (matrix; extract based on Whittard and Ritchie, Reference Ritchie2021)

Having used the interviews to determine what factors seemed to help or hinder the successful exploitation of data, the qualitative data collection was designed to put values, as far as possible, on these factors. The interview findings were then reused to qualify and provide context for the quantitative findings.

For example, the quantitative study showed that, when restricting the evaluation to direct, reasonably measurable components, the investments in data governance had a negative benefit (in other words, BMGF would have been better off choosing to invest the money in something other than data governance). As Alves et al. (Reference Alves, Tava, Whittard, Green, Beata Kreft and Ritchie2021) note, the ability to generate positive returns through highly subjective assumptions makes “the number” all but meaningless when dealing with highly intangible outcomes. We calculated mostly cash and time costs, generating a negative return; we then present the funders with the non-monetary benefits (identified during the qualitative stage) associated with that notional loss, for them to consider whether the overall outcome was valuable. The evaluation then becomes “given these measurable financial outcomes, do the non-financial outcomes seem an appropriate return on the investment?”

However, the information collected on the qualitative section showed that a large number of intangible, persistent benefits had been generated in data governance, even if most of them were thought to be unmeasurable. For example, a key innovation in the project was the Coalition of the Willing (CoW)—an initial meeting and then subsequent activities designed explicitly to build relationships that could make the project work. In the evaluation, this was accounted for on largely cost-of-time measures, but the qualitative review showed that the effects of the CoW were pervasive across the Five Safes: getting stakeholder buy-in, identifying potential champions and blockers, demonstrating the importance of trust in policy-making and delivery. Moreover, interviewees recognized the long-term value of the CoW for changing attitudes, for introducing new ways of thinking about data governance, and for providing a positive example of collective planning. This highlighted that the timing of the evaluation was crucial to the perspective on whether it brought a net gain or not. Some very limited assumptions were sufficient to show that the project overall was of benefit to the community; for example, assuming that the research papers produced during the project were valued using the time-cost of production. This enables consistency with a more traditional measure-everything analysis, if the funders had wanted it.

4.1. The Five Safes as a structural framework

The review team decided to (a) split the review into quantitative and qualitative elements, and (b) use the Five Safes framework to provide structure for the qualitative component. Decision (a) was undoubtedly the more important conceptual choice, as it provided a practical way of identifying the scope of the quantitative study and a context for interpreting quantitative results. However, decision (b) simplified the qualitative element considerably.

The Five Safes does not directly address the measurement or timing of costs and benefits, or many of the other practical problems of evaluation noted in the literature review; as Green and Ritchie (Reference Green and Ritchie2023) discuss, the Five Safes is a mechanism for framing questions, but provides no direct answers per se. Nevertheless, using a ready-made framework, familiar to the reviewers and with an intellectual hinterland to support it, made the task of breaking one large problem into many smaller ones much easier.

It is worth noting that the qualitative/quantitative split and the use of the Five Safes was not the first solution considered for evaluating the CABI project. Several alternative conceptual frameworks were explored over some 6 months, often based on the more traditional quantitative or case study evaluation models. All proved unsatisfactory, mostly because they did not address the slipperiness of valuing the concept of data governance. Realizing that the Five Safes could provide the scope definition led to the qualitative/quantitative split, and the development of a practical method.

Would other approaches to the qualitative study also have worked? “Yes, probably” is the fairest answer. However, part of the reason for choosing to use the Five Safes is that it was explicitly designed to deal with the multifaceted nature of data governance, and the language of the Five Safes can be directly related to data governance questions. This helped to address the issues of what, exactly, on the SSHIiE project counted as “data governance” activities and outcomes. A particular strength of the Five Safes is that it is broad in its coverage, so it ensures that all important elements area captured, while it is subjective in its application. This empowers evaluators to work within a broad framework, while allowing them the flexibility to focus in on issues they feel are of particular importance within the specific context of the project. In addition, given its familiarity to the data science community, using the Five Safes to structure an evaluation can help to reduce some of the uncertainty in an area, which, by its intangible nature, is plagued by ambiguity.

4.2. Mapping outcomes to the Five Safes

It was notable that much of the project value came in the “safe projects” dimension: that is, the element of governance concerned with objective-setting, outcomes, approvals, engagement, and planning. This is unsurprising: given the innovative nature of the project and the data governance activities (such as the CoW), planning and clear project oversight were disproportionately important in determining the success of the data governance plans. Ritchie and Tava (Reference Ritchie and Tava2020) make the same point, but on a theoretical rather than empirical basis. It would perhaps be useful, in future, to see whether follow-on projects place the same emphasis on planning, or whether more technical activities such as staff training become the drivers of success or failure.

This does raise the issue of whether those doing the project planning have the necessary skills to do this effectively. Data governance does tend to have strong learning effects; that is, improvements made in governance persist. A framework such as the Five Safes can help to understand where these gains are made on both current and future projects. For example, training and practical experience (the “people” element) in this project should feed directly through into the “project planning” phase of the next investment.

If, as suggested in the qualitative studies, this project has led to permanent changes in attitude toward data sharing and data governance in Ethiopia, then one would expect future projects to reflect this. For example, “a shared understanding of the importance of trust” becomes an input, rather than an output. This is an important outcome for the donor community and would not have been picked up, had the evaluation stuck to a traditional quantitative approach. It is important for the SSHIiE project (as well as us) to reflect and recognize that this program evaluation using the Five Safes might therefore be an easy way to identify accumulative gains from investment (of course, the same could be said for any other structure which is used consistently across multiple evaluations).

One factor that came out strongly in the interviews was the importance of timing. In some of the “safes” benefits were quickly identified as structural shifts that could have long-term impacts (providing a successful example of stakeholder management; showing how attitudes changed); in others, the benefits were more directly related to the projects (such as the relationship between system design and user training). As well as reinforcing the sensitivity of results to the timing of the evaluation, such findings also show the usefulness of a structure that can highlight the different temporal impacts of different elements of governance.

4.3. The value of a consistent framework

The Five Safes is more commonly used to design data governance activities rather than evaluate them, but the implication is there is the potential to exploit this framework consistently throughout all stages of grantmaking (portfolio strategy, grant design, anticipated risk of achieving measurable impact). It was noted that data governance was not considered as an explicit planning process when setting up SSHIiE. This is not uncommon; data governance is often seen as a set of separate tasks, such as ethical review or data management, which is why the Five Safes was created. Using the Five Safes, or some other framework, to develop an explicit data governance strategy would have simplified evaluation, even a retrospective one as in this case.

The Five Safes also showed itself as adaptable to the LMIC situation with relatively little change; again, this is because the Five Safes is a framework, not a checklist, and so the details can be tailored to the local situation while keeping the “big picture” constant. This has been the authors’ (limited) experience of applying the Five Safes in other LMIC contexts where, for example, training has seen local adaptation while still retaining the same overall concepts. This flexibility does offer a basis for developing data governance planning and evaluation models across very different contexts.

The Five Safes began as a tool for designing access to confidential data in research facilities. Its growth in this area has followed the typical S-shaped pattern of a network tool: slow initial growth, quick adoption at the point at which it becomes “the” standard, then slowing growth as it reaches saturation. Outside of this specific use, it is still at the slow-growth stage, but the range of applications is increasing. As noted in the literature review, the use of the Five Safes in evaluation is rare, but then the evaluation of data governance itself is very rare. Providing an evaluation tool that is more widely consistent with the language used in data governance is likely to produce some synergy in both evaluation and planning.