1. Introduction

The extent to which the high costs of rural electrification are justified by its impacts on societies and economies has been a matter of debate for decades (see, for example, Rose, Reference Rose1940; Devine, Reference Devine1983; Barnes and Binswanger, Reference Barnes and Binswanger1986; Barnes, Reference Barnes2010). In recent years, academic contributions to this discussion have been influenced considerably by the so-called credibility revolution in economics (see Angrist and Pischke, Reference Angrist and Pischke2010). The claim is that ‘design-based research’ (Card, Reference Card2022) like randomized controlled trials (RCTs) and instrumental variables (IVs) leads to more credible and verifiable identification of causal effects. This ‘experimentalist paradigm’ (Biddle and Hamermesh, Reference Biddle and Hamermesh2017) is closely linked to the vision of evidence-based policy: well-identified causal effects, so the narrative goes, will eventually tell us which interventions work and hence should be scaled to shape future policies (Young et al., Reference Young, Ashby, Boaz and Grayson2002; Duflo, Reference Duflo2004, Reference Duflo2020; Panhans and Singleton, Reference Panhans and Singleton2017).

In this paper, we examine the case of rural electrification in the Global South, documenting that design-based research is much less effective in improving policy than it is often claimed. This is not a new verdict, and we build on previous critical reflection on the credibility revolution paradigm (Rodrik, Reference Rodrik2008; Ravallion, Reference Ravallion2009, Reference Ravallion, Bédécarrats, Guérin and Roubaud2020; Heckman and Urzua, Reference Heckman and Urzua2010; Basu, Reference Basu2014; Deaton and Cartwright, Reference Deaton and Cartwright2018; Deaton, Reference Deaton2020; Drèze, Reference Drèze2020; Muller, Reference Muller2023).Footnote ¹ We extend this line of discussion by a specific application to rural electrification, an important area of development policy that absorbs large amounts of public funding (World Bank, 2018; Blimpo and Cosgrove-Davies, Reference Blimpo and Cosgrove-Davies2019). While national governments often justify investments into rural electrification from a social justice and hence a rights-based perspective, donor agencies and international development banks are under pressure to prove that the investment is worthwhile, following an explicit or implicit cost-benefit analysis logic. There is also an interesting within-sector cost-effectiveness debate because expensive grid extension competes with infrastructure leapfrogging via lower cost decentralized solutions like stand-alone solar or mini-grids (Levin and Thomas, Reference Levin and Thomas2016).

To inform this debate, many empirical studies have been published in recent years that examine the impacts of rural electrification, increasingly also using design-based methods from the credibility revolution toolkit. Several systematic reviews and meta-analyses have summarized this growing literature. In a nutshell, these reviews show that the literature is divided, with some studies finding very large effects, and others very modest or no effects. This divide is consequential for policy, especially considering that, for the extension of the power grid, large effects are required to justify the high costs. This holds true under a cost-benefit principle as it is applied by many donors, but also under a rights-based principle because then grid extension competes with off-grid technologies for cost-effectiveness.

Such meta-analyses and systematic reviews are important because, while design-based research is good at generating well-identified causal effects, the external validity gap still needs to be bridged. For this, an accumulation of evidence is needed – something that Duflo (Reference Duflo2020) refers to as the ‘pointillist painting,’ with each causal study being one dot on the painting.Footnote ² We use the case of rural electrification to show that even in a rich literature the pointillist painting is hard to compile and the dots on the canvas leave a lot of room for interpretation. We further argue that in highly contested policy areas, even well-meaning policymakers will use this wiggle room to pursue their interests. Next, we argue that the practice of design-based research, despite its intellectual beauty in identifying causality, is not immune to other biases stemming from questionable research practices, underpowered designs, overgeneralization, and publication bias. This further complicates the use of evidence in the policy landscape.

To conclude, we argue that this observation is not particular to electrification. We therefore call for a debate on what this implies for the science-policy interface. More research is needed on how evidence is generated and synthesized as well as how it is used for policy.

2. The credibility revolution in the electrification literature

Prior to the credibility revolution, empirical research on rural electrification had been conducted for many decades and had recurrently featured insightful studies based on various methods. Nonetheless, it is a showcase example of what the credibility revolution rightly criticized in the 2000s: many studies made some sort of causal inference based on a naive comparison of people or regions with and without access to electricity, without accounting for endogenous selection processes (see Peters, Reference Peters2009).Footnote ³ That has changed over the past 15 years or so, with an increasing number of published studies revealing more sensitivity for the problems of selection bias. The methodological portfolio first covered quasi-experimental matching and difference-in-difference designs, but increasingly also IVs and sometimes RCTs.

In fact, IVs have been used in many papers on grid-based electrification. Dinkelman (Reference Dinkelman2011) and Lipscomb et al. (Reference Lipscomb, Mobarak and Barham2013) are the earliest examples and they have been influential and foundational for the literature. The decentralization of electricity access also facilitated randomization, so that the first RCTs appeared in the mid-2010s (Furukawa, Reference Furukawa2014; Aklin et al., Reference Aklin, Bayer, Harish and Urpelainen2017; Grimm et al., Reference Grimm, Munyehirwe, Peters and Sievert2017). RCTs for power infrastructure in most settings proved to be infeasible for political or budgetary reasons; Lee et al. (Reference Lee, Miguel and Wolfram2020a) is a notable exception. Yet, for on-grid electrification, quasi-experimental methods and especially IVs continue to be the dominant identification strategies, while for off-grid solar several RCTs exist.

This wave of intense design-based research was followed by a battery of overview papers and systematic reviews (henceforth ‘reviews’) (Bernard, Reference Bernard2012; Peters and Sievert, Reference Peters and Sievert2016; Bonan et al., Reference Bonan, Pareglio and Tavoni2017; Jimenez, Reference Jimenez2017; Bos et al., Reference Bos, Chaplin and Mamun2018; Morrissey, Reference Morrissey2018; Blimpo and Cosgrove-Davies, Reference Blimpo and Cosgrove-Davies2019; Hamburger et al., Reference Hamburger, Jaeger, Bayer, Kennedy, Yang and Urpelainen2019; Bayer et al., Reference Bayer, Kennedy, Yang and Urpelainen2020; Perdana et al., Reference Perdana, Glandon, Moore and Snilsveit2020; Lee et al., Reference Lee, Miguel and Wolfram2020b; Jeuland et al., Reference Jeuland, Fetter, Li, Pattanayak, Usmani, Bluffstone, Chávez, Girardeau, Hassen, Jagger, Jaime, Karumba, Köhlin, Lenz, Naranjo, Peters, Qin, Ruhinduka and Toman2021). The research community has hence not only generated the dots on Duflo's pointillist painting but also invested in compiling what the painting shows. All these reviews diagnose a divide in the literature, that is, one set of studies comes to very positive conclusions about the development effects of electrification while another set of studies rather observes small or no effects.

To understand the policy implication of this, the size of the effect must be assessed in relation to the costs. Here, it is important to distinguish between on-grid and off-grid electrification. Given the high cost of grid-based rural electrification, large positive effects are required to make the intervention cost-effective and even modest positive effects would advocate against the investment. Based on their finding of muted effects, Lee et al. (Reference Lee, Miguel and Wolfram2020a) conclude that the investment into grid extension entails a ‘social surplus loss’. In contrast, for off-grid electrification such as small-scale solar, even modest effects can render a cost-benefit analysis positive and suggest that promoting this technology is cost-effective – because of the considerably lower investment cost (Grimm et al., Reference Grimm, Lenz, Peters and Sievert2020).Footnote ⁴

The reviews refer to several potential explanations of the divide, but in our reading, two narratives stand out: a regional divide and a methodological divide.Footnote ⁵ Jeuland et al. (Reference Jeuland, Fetter, Li, Pattanayak, Usmani, Bluffstone, Chávez, Girardeau, Hassen, Jagger, Jaime, Karumba, Köhlin, Lenz, Naranjo, Peters, Qin, Ruhinduka and Toman2021) is an insightful starting point. It does not delve into a narrative for the divide in the literature. Its main purpose, rather, is to comprehensively take stock of the literature. Jeuland et al. (Reference Jeuland, Fetter, Li, Pattanayak, Usmani, Bluffstone, Chávez, Girardeau, Hassen, Jagger, Jaime, Karumba, Köhlin, Lenz, Naranjo, Peters, Qin, Ruhinduka and Toman2021) thereby illustrates how vast the evidence base is when a review is very inclusive. By covering a generous list of journals as well as the grey literature, it shows that the electrification literature comprises some 2,000 studies. As an extreme case, one can draw from this large pool to compile the pointillist painting, even if there is certainly a broad consensus that many of these 2,000 dots should be dismissed, for example because a study does not apply design-based methods. All other reviews employ much more exclusive selections of the literature and most include design-based studies only.

The regional narrative for the divide in the literature points to the different development potentials in different regions and target populations (see, for example, Peters and Sievert, Reference Peters and Sievert2016; Hamburger et al., Reference Hamburger, Jaeger, Bayer, Kennedy, Yang and Urpelainen2019; Lee et al., Reference Lee, Miguel and Wolfram2020b). Hamburger et al. (Reference Hamburger, Jaeger, Bayer, Kennedy, Yang and Urpelainen2019) reveal that large parts of the design-based electrification literature are concentrated in just a few countries. Especially Sub-Saharan Africa is largely ignored. Related to this, Peters and Sievert (Reference Peters and Sievert2016) argue that the large effects observed in some Latin American and Asian countries cannot be generalized to Sub-Saharan Africa because of different economic conditions at baseline. They also provide evidence for small effects from several Sub-Saharan African countries, which contrast with the much larger effects in the pre-existing literature. In a similar vein, Lee et al. (Reference Lee, Miguel and Wolfram2020b) emphasize that, historically, electrification in most industrialized countries happened while the economies were on a growth trajectory. Evidence from such contexts is hence not transferable to places today where remote areas are connected that are barely integrated in economic development processes.

The methodological narrative is raised mainly in Bayer et al. (Reference Bayer, Kennedy, Yang and Urpelainen2020) and Lee et al. (Reference Lee, Miguel and Wolfram2020b). Bayer et al. (Reference Bayer, Kennedy, Yang and Urpelainen2020) establish that studies using randomized designs typically deliver smaller effects than those using quasi-experimental designs. They explain this by the selection bias inherent to non-randomized methods that inflates impact estimates. The pattern in their data is indeed striking, but an important caveat is that with one exception all RCTs were done on off-grid electrification technologies, not the grid. Grid extension programs are mostly evaluated using IVs, and sometimes regression discontinuity and difference-in-difference designs. Lee et al. (Reference Lee, Miguel and Wolfram2020b: 131), focusing on grid electrification, point to the large number of IVs in that literature and suggest that ‘it is hard to rule out the possibility that the correlation between the instrument and the dependent variable runs through additional channels beyond electrification’.

In fact, the heavy reliance on observational data and especially IVs is conspicuous in the electrification literature, and it might import risks of bias. Above all, the geographic IVs that are often used in electrification evaluations such as the land gradient or water flow are suspected of violating exclusion restrictions because they affect the causal network through many pathways, not just through electrification, the instrumented variable.Footnote ⁶ Another reason to be concerned is that these geographical IVs are often weak IVs, which is not a problem per se if appropriate remedies are used. But these remedies are less effective if weakness concurs with violated exclusion restrictions (Bensch et al., Reference Bensch, Gotz and Ankel-Peters2020) and if scholars screen specifications based on first-stage strength (Ankel-Peters et al., Reference Ankel-Peters, Bensch and Vance2023). Related to the screening aspect, IVs are suspected of being more prone to publication bias and p-hacking (Brodeur et al., Reference Brodeur, Cook and Heyes2020: 3,636), because ‘when using a non-experimental method like IV there are many points at which a researcher exercises discretion in ways that could affect statistical significance’.Footnote ⁷ Relatedly, we are not aware of an IV-based study in the electrification literature with a null result (Bayer et al., Reference Bayer, Kennedy, Yang and Urpelainen2020).

It is furthermore conspicuous that those more recent studies that find smaller effects use self-collected primary data to evaluate specific electrification interventions, irrespective of whether they are RCTs or based on a difference-in-differences. This covers studies like Lee et al. (Reference Lee, Miguel and Wolfram2020a), an RCT, but also Bensch et al. (Reference Bensch, Cornelissen, Peters, Wagner, Reichert and Stepanikova2019), Chaplin et al. (Reference Chaplin, Mamun, Protik, Schurrer, Vohra, Bos, Burak, Meyer, Dumitrescu, Ksoll and Cook2017), Masselus et al. (Reference Masselus, Ankel-Peters, Sutil, Modi, Mugyenyi, Munyehirwe, Williams and Sievert2024a) and Lenz et al. (Reference Lenz, Munyehirwe, Peters and Sievert2017) as well as Peters et al. (Reference Peters, Vance and Harsdorff2011). We therefore raise the question of whether this evaluative setting – primary data and specific interventions under evaluation – could possibly lead to fewer incentives to publish large effects. One reason could be that the specific interventions under evaluation are often large and well-known investments, making a null effect more interesting. Self-collected data also allows for tracking potential effects much more meticulously along a theoretical results chain (e.g., by eliciting appliance adoption, productive appliance adoption, jobs in electricity-using firms, etc.). This is not to say that such evaluations are without problems. Regional scope is limited and cooptation by funding development agencies is possible. Primary data also often covers shorter time periods (Nag and Stern, Reference Nag and Stern2023; see Masselus et al. (Reference Masselus, Ankel-Peters, Sutil, Modi, Mugyenyi, Munyehirwe, Williams and Sievert2024a) for an exception).

In any case, the electrification literature should be evaluated in light of recent trends in the economics profession towards more transparency (Christensen and Miguel, Reference Christensen and Miguel2018). This requires sensitivity for pre-specification and robustness replicability as well as quantitative meta-analyses that account for potential publication bias (Andrews and Kasy, Reference Andrews and Kasy2019; Carter et al., Reference Carter, Schönbrodt, Gervais and Hilgard2019; Irsova et al., Reference Irsova, Doucouliagos, Havranek and Stanley2024) – something that has hitherto not been done.

3. Bayesian policymakers and reasoned intuition

The target audience of applied empirical research according to the evidence-based policy paradigm are policymakers.Footnote ⁸ Economists have started to examine the conditions under which policymakers indeed make use of available evidence (Banuri et al., Reference Banuri, Dercon and Gauri2019; Hjort et al., Reference Hjort, Moreira, Rao and Santini2021; Vivalt and Coville, Reference Vivalt and Coville2023). The underlying assumption often is that the evidence provides a scientifically clear picture. In practice, though, the evidence is often murky and contradictory, and subject to debates about methodological issues. The electrification literature is a showcase example of this. It is therefore important to ask how policymakers form their beliefs.

Ideally, policymakers and we, their academic advisors, are Bayesians: We have a prior which we update as new evidence comes in. The prior's responsiveness is a function of the evidence's methodological quality. That is, the prior is firmer and less responsive to new evidence the better the already existing evidence is. Likewise, it responds more to methodologically sound new evidence. This type of thinking, though, requires repeated appraisals of the incoming evidence. For this appraisal, there exist no standards. At best, these appraisals are based on experience and expertise. In other words, we must use what Basu (Reference Basu2014: 466) calls reasoned intuition: ‘intuition and gut feeling […] need to be held under the scanner of reason before we use them to translate experience and evidence into rules and behaviour and policy.’ Most policymakers have experience and expertise, so it is possible that reasoned intuition can work when policymakers come across new evidence.

Yet, so far, we have assumed benevolent policymakers while in practice they might have some sort of vested interests. This is, in many cases, not condemnable. For example, policymakers are typically civil servants and hence subscribe to a certain political agenda of the administration they represent. It is natural that policymakers extract from the evidence what serves their interest. A divided literature like the one on rural electrification provides the basis for confirmation bias as it is empirically diagnosed by Banuri et al. (Reference Banuri, Dercon and Gauri2019). In a similar vein, Vivalt and Coville (Reference Vivalt and Coville2023) provide empirical evidence for what they call ‘asymmetric optimism’: policymakers update more on good news than on bad news.

Policymakers managing electrification portfolios can have agendas. For example, major development banks have a long history of investing in large infrastructure through grants and lending, and it is understandable that they – or some of their staff members – prefer on-grid electrification over off-grid electrification. Confirmation bias and asymmetric optimism might tempt them to seize that part of the literature that suggests substantial development effects of grid extension programs. Staff of solar advocacy organizations or private sector representatives seeking subsidies for their off-grid solar programs might, by contrast, prefer evidence suggesting only modest impacts of on-grid electrification. This would strengthen the cost-effectiveness of off-grid technologies. The hawker's tray of the electrification literature has much evidence to offer for both camps.

An informed debate between these two camps based on reasoned intuition is hence problematic. An additional important layer of complexity is that applying reasoned intuition is harder the more prevalent methodological concerns are that are not well understood within academia.Footnote ⁹ For example, academic debates do not converge when it comes to publication bias and how to account for it when making inferences. Likewise, controversies about robustness in replications and reproductions are hard to settle among replicators and original authors (Ozier, Reference Ozier2021; Ankel-Peters et al., Reference Ankel-Peters, Fiala and Neubauer2024). And while external validity is an accepted barrier in economics between rigorous evidence and its policy relevance, the literature on how to account for it in the generalization of scientific results is nascent but so far inconclusive (Muller, Reference Muller2015, Reference Muller, Kincaid and Ross2021; Pritchett and Sandefur, Reference Pritchett and Sandefur2015; Peters et al., Reference Peters, Langbein and Roberts2018; Vivalt, Reference Vivalt2020; Dehejia et al., Reference Dehejia, Pop-Eleches and Samii2021; Gechter, Reference Gechter2024). Concerns about construct validity are less widely discussed and virtually absent in the economics literature, although they are of utmost importance for generalization across supposedly similar interventions (Pritchett et al., Reference Pritchett, Samji and Hammer2013; Esterling et al., Reference Esterling, Brady and Schwitzgebel2023; Masselus et al., Reference Masselus, Ankel-Peters and Petrik2024b). Such debates including their ambiguous outcomes are not a failure but rather a natural part of the scientific enterprise. Nevertheless, they do pose major hurdles for the evidence-policy interface.