Policy Significance Statement

This article engages with an emerging policy question: how legitimate, large-scale data benchmarking might be sustained under shrinking resources and growing demands for evidence in AI governance. It examines the potential of a hybrid human-AI architecture as one possible pathway for balancing efficiency with transparency, inclusivity, and accountability in global assessments. The pilot study suggests that while AI can support structured tasks and broaden evidence discovery, expert interpretation continues to play an important role in handling contextual nuance and maintaining democratic legitimacy. For governments, international organizations, and donors, these findings point toward the possible value of investing in hybrid and participatory evidence infrastructures that could leverage AI capabilities while preserving independent, Global South-led perspectives in digital governance.

1. Background

The global data landscape has become increasingly central to governance, development, and policy evaluation. In parallel, a wide array of indexes and assessment tools has emerged to benchmark countries’ progress, serving as a key component of the infrastructure for evidence-based policy. Yet these tools often grapple with fundamental limitations. As highlighted in the Data for Development (D4D) Stocktaking Report (Open Data Watch, 2021) and further discussed by Zuiderwijk et al. (Reference Zuiderwijk, Pirannejad and Susha2021), traditional assessments embed normative assumptions, methodological trade-offs, and institutional perspectives. Their reliance on composite scores and expert judgment has helped establish comparative benchmarks, while also revealing opportunities to better capture contextual nuance, enhance transparency, and strengthen reproducibility.

Research across governance domains shows that indicators do not merely describe policy environments but shape them. By ordering actors and defining performance categories, benchmarks influence regulatory behavior, reform priorities, and resource allocation (Davis, Reference Davis2012; Doshi et al., Reference Doshi, Kelley and Simmons2019). Similar dynamics appear in development and aid transparency initiatives, where assessments increasingly evaluate not only the existence of policies but their operational characteristics (Ghosh and Kharas, Reference Ghosh and Kharas2011; Palagashvili and Williamson, Reference Palagashvili and Williamson2021; Reinsberg and Swedlund, Reference Reinsberg and Swedlund2023). Benchmarking, therefore, operates as an accountability mechanism whose legitimacy depends on transparent evidence, practices, and sustained expert judgment.

At the same time, the political economy supporting such benchmarking infrastructures has become increasingly fragile. Many global indicators rely heavily on aid-funded expert research, and the withdrawal of key donors (most notably USAID at the end of 2024) has exposed the vulnerability of these models (Powell et al., Reference Powell, Lagomarsino and Melamed2025). Meanwhile, policy attention has shifted toward artificial intelligence, often overlooking that reliable data governance remains a prerequisite for trustworthy AI (Verhulst and Schüür, Reference Verhulst and Schüür2023). Before AI tools can be responsibly integrated, benchmarking initiatives must therefore grapple with the political and epistemic conditions under which data are produced, validated, and used.

The emergence of large language models (LLMs) and AI research agents introduces a potential response to this tension. Automation promises scalability and lower costs, yet also introduces opacity into processes whose legitimacy depends on interpretability and contestability. The question is therefore not simply whether AI can replicate expert work, but how its integration alters the conditions under which benchmarking remains accountable and credible.

This article approaches AI-assisted benchmarking as a governance problem rather than a purely technical one. We evaluate AI integration through three observable dimensions of democratic legitimacy—transparency of the assessment process, inclusivity of the evidence base, and accountability of outcomes. Using selected indicators from the Global Data Barometer (2025), we explore how automation reshapes the production of comparative policy knowledge and what institutional arrangements are required for it to remain trustworthy.

2. Methodological lessons from the Global Data Barometer

Large-scale benchmarking initiatives in the data domain have historically relied on human experts to interpret complex and context-dependent realities. The Global Data Barometer (GDB) offers one of the most recent and ambitious examples of this model. Conceived as a successor to earlier efforts such as the Open Data Barometer, the GDB sought not only to map availability of government data but also to capture broader conditions shaping how data contributes to the public good. This ambition required methodological innovation: rather than simply measuring datasets, the GDB embedded a theory of change that linked governance frameworks, capabilities, and evidence of availability and use (Global Data Barometer, 2025). As such, the GDB is a paradigmatic case for understanding the opportunities and limits of human-driven benchmarking, and a crucial reference point for exploring the potential of hybrid AI-assisted approaches.

The GDB was designed around a multi-level indicator framework that captured not only the existence of datasets or regulations but also the enabling conditions that allow their use. Indicators also included conditional logic guiding researchers through context-specific pathways. For instance, if the existence of a dataset was confirmed, subsequent questions then assessed its quality, scope, and accessibility. This approach produced results that moved beyond binary measures and generated comparative insights across 43 countries in the second edition (and 109 in the first one). Yet the very sophistication of this layered methodology made it resource-intensive, raising persistent questions about how such depth could be sustained at scale.

To meet this challenge, the GDB relied on a distributed global research network. More than 100 national researchers in the first edition and over 40 in the second, supported by regional hubs and thematic partners, contributed to data collection and peer review. This model safeguarded the diversity of local perspectives and ensured that evidence was validated by those with contextual expertise. At the same time, it exposed a fundamental trade-off: inclusivity and contextual richness came at the cost of efficiency, as quality control required multiple review layers and often slowed the research cycle. The Barometer, therefore, illustrates a central dilemma of benchmarking: how to reconcile legitimacy with sustainability, a tension that reappears with particular intensity in debates on AI-assisted methodologies.

The AI turn sharpens these trade-offs by challenging one of the Barometer’s most deliberate design choices: transparency. Every response in the GDB was backed by documentary evidence and is available for access and reuse on its website, ensuring traceability and accountability. Machine learning systems, by contrast, often obscure their training data and internal logic, leaving end-users unable to interrogate results. If hybrid benchmarking is to leverage AI, it must replicate the evidentiary rigor of human-led research. Automated outputs need to be made auditable, so that efficiency gains do not come at the expense of transparency and trust.

Bias presents a second challenge where the GDB’s experience is instructive. Just as the Barometer mitigated regional or linguistic bias through its distributed network of local researchers, AI systems risk reproducing and amplifying biases embedded in their training data or prompt design. Here, the lesson is clear: legitimacy in benchmarking does not emerge from the elimination of bias, which is impossible, but from procedural mechanisms that surface, contextualize, and counteract it. In hybrid models, this means building in human oversight not as a safeguard of last resort, but as a structural feature of the methodology.

Taken together, these design choices illustrate the dual challenge at the heart of global benchmarking: legitimacy requires methodological transparency, inclusivity, and openness, but these same commitments increase the costs and fragility of sustaining expert-driven models at scale. It is precisely these tensions that motivate our exploration of whether—and how—AI tools might complement rather than displace the human infrastructure on which the GDB rests. To address these tensions, we turned to exploratory experimentation.

3. Toward a hybrid benchmarking framework: a pilot experiment

These methodological tensions motivated us to move beyond conceptual reflection and test, in practice, whether AI could complement rather than replace the GDB model. To that end, we conducted an exploratory pilot to pursue a dual objective: first, to assess the feasibility of integrating AI into the GDB data collection process; and second, to evaluate the quality of AI-generated outputs, particularly in comparison to GDB data (second edition) contributed by human researchers, in order to identify potential risks and barriers of the AI approach.

Unlike a hypothesis-driven experiment that proceeds with predetermined assumptions, our pilot did not set fixed research hypotheses or predefined outcomes at the outset; instead, the progression of the pilot was guided by the results observed at each stage, and we determined next steps based on these results and the actual tasks of the GDB research process. This adaptive, result-oriented approach ensured that our exploration remained closely aligned with the practical requirements of the GDB research process, rather than pursuing abstract technical goals.

To anchor our investigation in practical realities, we grounded our experimental design in the GDB framework. In the second edition of the GDB, indicators are organized under three pillars: Governance and Capability, which primarily assess governance frameworks (such as laws, policies, and strategies); and Availability, which evaluates the presence of thematic data resources (e.g., public procurement data), regardless of whether they are published directly by the government. Each of these pillars comprises a distinct group of indicators. The data collection process requires country researchers to complete indicator surveys, where each survey follows a dual structure: an “Existence Question” to determine if the object of assessment (e.g., a regulatory framework or a dataset) exists, followed by a series of “Element Questions” to evaluate its specific features.

Based on this structure, we selected a range of indicators representing varying degrees of evidence complexity—defined primarily by the nature and structure of the required evidence—to serve as the testing ground for our pilot (see Table 1). To illustrate, while both the “Data Protection” and “Data Sharing” indicators assess existing governance frameworks, they differ significantly in evidence structure. “Data Protection” indicator is typically anchored in a single, consolidated piece of legislation (e.g., a Data Protection Act). In contrast, the “Data Sharing” indicator comprises a patchwork of frameworks governing diverse practices, such as government-to-government (G2G) and business-to-business (B2B) data sharing. This fragmented composition requires researchers to navigate and synthesize evidence from multiple sources to answer the element questions.

Table 1.

Selection of indicators for experimental testing

Similarly, the “Public Procurement” and “Land Tenure” indicators both assess the existence of thematic data resources. Unlike governance frameworks, which are text-based, assessing data resources requires researchers to locate and evaluate actual technical implementations, such as dataset download pages, API endpoints, or interactive visualization systems. As reported in the second edition of the GDB, public procurement data are generally more widely available and often centralized within a one-stop system. Conversely, land tenure data are frequently fragmented across multiple management systems even when available. Consequently, the search and assessment process for land tenure data entails a significantly higher level of complexity.

Guided by this framework, our exploration naturally unfolded into three distinct phases, with each phase’s research question building upon the previous one:

• • Phase 1: Feasibility test—automated analysis with predefined evidence. Our inquiry began with a foundational test: could an AI analyze a preselected piece of evidence to answer corresponding expert survey questions?

• • Phase 2: Capability expansion—automated text-based evidence discovery. Phase 1’s focus on predefined evidence raised a critical follow-up question regarding the preceding step in the research process, leading to our next logical inquiry: could AI automate text-based evidence discovery?

• • Phase 3: Complexity challenge—end-to-end task execution. While previous phases decoupled discovery from evaluation, Phase 3 aims to integrate them into a unified pipeline. This shift is essential for Availability indicators. Unlike text-based evidence (e.g., laws in PDF format) that can be physically stored and transferred between steps, evidence of data availability—often involving live web interactions or system functionality—cannot be easily “physicalized” for downstream processing. Thus, evaluating these indicators requires an end-to-end approach, where the AI agent assesses the evidence in situ immediately upon discovery.

Our tool selection was guided by two core principles: prioritizing publicly available, off-the-shelf solutions to ensure relevance to the technological landscape of June–July 2025, and adapting these tools to the evolving complexity of the assessment tasks (Table 2 details the specific technical setup).

Table 2.

Tools and configuration by research phase

For Phase 1, we integrated the Gemini 2.5 Flash API via Dify to automate the assessment of the “Data Protection” indicator across 34 countries, enabling a statistical evaluation of the results. This model was selected as a cost-effective, representative state-of-the-art LLM to test general feasibility. Crucially, its knowledge cutoff (January 2025) predates the release of the GDB second edition (June 2025); this ensures the model possesses no specific memory of the new dataset, although it may retain prior context from the GDB first edition.

For Phases 2 and 3, we engaged the three dominant deep research agents (ChatGPT, Gemini, and Perplexity) to handle complex evidence discovery. Given the nascent state of autonomous research, we adopted a multi-agent cross-validation strategy to mitigate individual model hallucinations. However, the absence of public APIs limited us to conversational interfaces, precluding the large-scale automation seen in Phase 1. Consequently, we shifted to a single-country qualitative case study, focusing on detailed comparisons across agents.

We adopted a straightforward prompting strategy to assess baseline feasibility, deferring detailed optimization and comparison to future research. Table 3 lists the core prompts used in Phase 3, which served as the foundation for the simplified variations used in Phases 1 and 2.

Table 3.

Example prompts used during Phase 3

4. Findings from a multistage exploration

This section presents the findings from our phased exploration. We begin with a high-level summary that systematically evaluates AI’s feasibility and quality against human responses across the different experimental phases (Table 4). This overview serves as a roadmap to the detailed breakdown of findings from each phase that follows.

Table 4.

Summary of findings on AI feasibility, quality, and implications for democratic legitimacy

4.1. Phase 1: feasibility of automated analysis

Focusing on the Data Protection Legislation (DPL) indicator, we built a workflow to assess 34 countries. The AI was tasked with analyzing the provided legal text and answering the corresponding “Elements” questions. Our analysis revealed an average match rate of 61.76 between AI-generated answers and the original GDB expert data. However, this overall figure masks a significant performance disparity based on question type, as shown in Table 5.

Table 5.

AI versus human answer alignment for DPL indicator by question type

As observed in the results, questions concerning foundational provisions relate to the basic characteristics of a Data Protection Law (DPL), such as the inclusion of data portability. These questions typically involve direct citation of specific clauses, leading to a high degree of alignment between human and AI answers.

In contrast, questions about advanced provisions pertain to cutting-edge issues, such as clauses addressing group privacy or automated decision-making. These topics may not be covered by explicit articles and must instead be inferred from general provisions or addressed by cross-referencing other governance frameworks. Consequently, more significant discrepancies are observed between human and AI responses in these areas.

Crucially, a mismatch did not always indicate an AI error. In several cases, AI systems identified relevant legal provisions across lengthy documents that had been partially overlooked in expert assessments, illustrating the potential value of AI for consistency and quality assurance within hybrid workflows. A detailed review revealed cases where the AI’s interpretation was more accurate than the human assessment, as well as cases where it faltered. Table 6 provides two illustrative vignettes.

Table 6.

Case studies of AI performance in DPL analysis

4.1.1. Key insight

While AI shows high feasibility as an evidence analyzer for structured tasks, its reliability diminishes significantly with increasing conceptual nuance, highlighting the critical need for human oversight in validating complex interpretations.

5. Discussion: rethinking sustainability and legitimacy

It is important to emphasize that the findings presented here are derived from an exploratory pilot rather than a comprehensive validation exercise. The purpose of the analysis is not to generalize about the performance of AI systems across all benchmarking contexts, but to generate empirically grounded hypotheses about the relationship between evidential complexity, normative judgment, and the limits of automation in global assessments. In particular, the observed variation in AI performance across indicators suggests testable propositions regarding when and where AI-assisted approaches may be appropriate, and where human oversight remains indispensable. These propositions invite further systematic investigation, rather than serving as definitive conclusions.

At the same time, the provisional nature of these findings does not diminish their significance. Even in this limited and exploratory setting, the results from our multistage exploration consistently demonstrate that AI is not a neutral tool; its integration into assessment workflows carries significant implications for democratic values (Mittelstadt et al., Reference Mittelstadt, Allo, Taddeo, Wachter and Floridi2016). Its performance is not uniform; it excels at structured, factual tasks but falters when faced with conceptual nuance. Its evidence discovery process can introduce biases, and its analytical methods can lead to factually incorrect conclusions without human validation. As such, the integration of AI into assessment workflows carries significant implications for the democratic legitimacy of the process—impacting its transparency, inclusivity, and accountability. Building on these findings, this discussion delves into the broader implications for human-AI collaboration and proposes a hybrid architecture designed to navigate these complexities.