Sharing, harmonizing, and analyzing participant-level data is of central importance in the rapid research response to emerging pathogens. Individual participant data meta-analyses (IPD-MAs), which synthesize participant-level data from related primary studies, have several advantages over pooling study-level effect estimates in a traditional meta-analysis. IPD-MAs enable researchers to more effectively separate spurious heterogeneity related to differences in measurement from clinically relevant heterogeneity from differences in underlying risk or distribution of factors that modify disease progression. This tutorial describes the steps needed to conduct an IPD-MA of an emerging pathogen and how IPD-MAs of emerging pathogens differ from those of well-studied exposures and outcomes. We discuss key statistical issues, including participant- and study-level missingness and complex measurement error, and present recommendations. We review how IPD-MAs conducted during the COVID-19 response addressed these statistical challenges when harmonizing and analyzing participant-level data related to an emerging pathogen. The guidance presented here is based on lessons learned in our conduct of IPD-MAs in the research response to emerging pathogens, including Zika virus and COVID-19.

The I-squared index was proposed in 2002 as a measure to help understand the consistency of study results in a meta-analysis. It was developed to overcome some of the limitations of existing measures, principally the chi-squared test for heterogeneity and the between-study variance as estimated in a random-effects meta-analysis. I-squared measures approximately the proportion of total variability in results that is due to true heterogeneity rather than random error; it is also conveniently interpreted as a measure of inconsistency in the results of the studies. The index has become extremely widely used, although it is often misinterpreted as an absolute measure of the amount of heterogeneity, which it is not. Here, we discuss the I-squared index and the different ways it can be defined, computed, and interpreted. We introduce a new interpretation of I-squared as a weighted sum of squares, which we propose may be helpful when setting up simulation studies. We discuss some of the extensions and repurposes that have been proposed for I-squared and offer some recommendations on the appropriate use of the index in practice.

Identifying studies for inclusion in realist syntheses using title and abstract screening is challenging given the need to unearth explanatory insights to build context–mechanism–outcome configurations. Such insights may only be uncovered through full-text paper reading. To address this issue, a novel approach for identifying studies has been developed called Reverse Chronology Quota Record Screening (RCQRS). Results of database searches are screened in reverse chronology, and in stages, to fill quotas matching the requirements of the review. RCQRS may be beneficial in any of the following circumstances: (a) the timeframe to complete the synthesis is short; (b) the scope of inquiry is not fully defined, (c) the availability of suitable literature is beyond the screening capacity of the reviewers; or (d) the availability of literature is sparse and reviewers seek to extrapolate insights from other areas. In contrast to RCQRS, exhaustive title and abstract screening (i.e., screening entire database results) may hamper study selection due to being overwhelming in volume and time-intensive, resulting in a causally thin cohort of papers for analysis. RCQRS used in stages, and in conjunction with other search strategies (e.g., hand searching, backward citation tracking, and expert solicitation) can support creative, robust analytical insights with causally rich extrapolations. Using the Horizon-EU funded SERENITY study on deprescribing in palliative care as a case example, the benefits and limitations of RCQRS are explored. Finally, a checklist template is offered for teams who wish to reflect on, and transparently report, the use of RCQRS in their realist synthesis.

Exponential growth in scientific literature has heightened the demand for efficient evidence-based synthesis, driving the rise of the field of automated meta-analysis (AMA) powered by natural language processing and machine learning. This PRISMA systematic review introduces a structured framework for assessing the current state of AMA, based on screening 13,216 papers (2006–2024) and analyzing 61 studies across diverse domains. Findings reveal a predominant focus on automating data processing (52.5%), such as extraction and statistical modeling, while only 16.4% address advanced synthesis stages. Just one study (approximately 2%) explored preliminary full-process automation, highlighting a critical gap that limits AMA’s capacity for comprehensive synthesis. Despite recent breakthroughs in large language models and advanced AI, their integration into statistical modeling and higher-order synthesis, such as heterogeneity assessment and bias evaluation, remains underdeveloped. This has constrained AMA’s potential for fully autonomous meta-analysis (MA). From our dataset spanning medical (67.2%) and non-medical (32.8%) applications, we found that AMA has exhibited distinct implementation patterns and varying degrees of effectiveness in actually improving efficiency, scalability, and reproducibility. While automation has enhanced specific meta-analytic tasks, achieving seamless, end-to-end automation remains an open challenge. As AI systems advance in reasoning and contextual understanding, addressing these gaps is now imperative. Future efforts must focus on bridging automation across all MA stages, refining interpretability, and ensuring methodological robustness to fully realize AMA’s potential for scalable, domain-agnostic synthesis.

Effective public health decision-making relies on rigorous evidence synthesis and transparent processes to facilitate its use. However, existing methods guidance has primarily been developed within clinical medicine and may not sufficiently address the complexities of public health, such as population-level considerations, multiple evidence streams, and time-sensitive decision-making. This work contributes to the European Centre for Disease Prevention and Control initiative on methods guidance development for evidence synthesis and evidence-based public health advice by systematically identifying and mapping guidance from health and health-related disciplines.

Structured searches were conducted across multiple scientific databases and websites of key institutions, followed by screening and data coding. Of the 17,386 records identified, 247 documents were classified as ‘guidance products’ providing a set of principles or recommendations on the overall process of developing evidence synthesis and evidence-based advice. While many were classified as ‘generic’ in scope, a majority originated from clinical medicine and focused on systematic reviews of intervention effects. Only 41 documents explicitly addressed public health. Key gaps included approaches for rapid evidence synthesis and decision-making and methods for synthesising evidence from laboratory research, disease burden, and prevalence studies.

The findings highlight a need for methodological development that aligns with the realities of public health practice, particularly in emergency contexts. This review provides a key repository for methodologists, researchers, and decision-makers in public health, as well as clinical medicine and health care in Europe and worldwide, supporting the evolution of more inclusive and adaptable approaches to public health evidence synthesis and decision-making.

Automating data extraction from full-text randomized controlled trials for meta-analysis remains a significant challenge. This study evaluates the practical performance of three large language models (LLMs) (Gemini-2.0-flash, Grok-3, and GPT-4o-mini) across tasks involving statistical results, risk-of-bias assessments, and study-level characteristics in three medical domains: hypertension, diabetes, and orthopaedics. We tested four distinct prompting strategies (basic prompting, self-reflective prompting, model ensemble, and customized prompts) to determine how to improve extraction quality. All models demonstrate high precision but consistently suffer from poor recall by omitting key information. We found that customized prompts were the most effective, boosting recall by up to 15%. Based on this analysis, we propose a three-tiered set of guidelines for using LLMs in data extraction, matching data types to appropriate levels of automation based on task complexity and risk. Our study offers practical advice for automating data extraction in real-world meta-analyses, balancing LLM efficiency with expert oversight through targeted, task-specific automation.

Critical interpretive synthesis was introduced in 2006 to address various shortcomings of systematic reviews such as their limitations in synthesizing heterogeneous data, integrating diverse study types, and generating theoretical insights. This review sought to outline the methodological process of conducting critical interpretive syntheses by identifying the methods currently in use, mapping the processes that have been used to date, and highlighting directions for further research. To achieve this, a scoping review of critical interpretive syntheses published between 2006 and 2023 was conducted. Initial searches identified 1628 publications and after removal of duplicates and exclusions, 212 reviews were included in the study. Most reviews focused on health-related subjects. Authors chose to utilize the method due to its iterative, inductive, and recursive nature. Both question-based and topic-based reviews were conducted. Literature searches relied on electronic databases and reference chaining. Mapping to the original six-phase model showed most variability in use of sampling and quality assessment phases, which were each done in 50.7% of reviews. Data extraction utilized a data extraction table. Synthesis involved constant comparison, critique, and consolidation of themes into constructs, and a synthesizing argument. Refining critical interpretive synthesis methodology and its best practices are important for optimizing the utility and impact and ensuring findings are relevant and actionable for informing policy, practice, and future research.

The ROBINS-I tool is a commonly used tool to assess risk of bias in non-randomised studies of interventions (NRSI) included in systematic reviews. The reporting of ROBINS-I results is important for decision-makers using systematic reviews to understand the weaknesses of the evidence. In particular, systematic review authors should apply the tool according to the guidance provided. This study aims to describe how ROBINS-I guidance is currently applied by review authors. In January 2023, we undertook a citation search and screened titles and abstracts of records published in the previous 6 months. We included systematic reviews of non-randomised studies of intervention where ROBINS-I had been used for risk-of-bias assessment. Based on 10 criteria, we summarised the diverse ways in which reviews deviated from or reported the use of ROBINS-I. In total, 492 reviews met our inclusion criteria. Only one review met all the expectations of the ROBINS-I guidance. A small proportion of reviews deviated from the seven standard domains (3%), judgements (13%), or in other ways (1%). Of the 476 (97%) reviews that reported some ROBINS-I results, only 57 (12%) reviews reported ROBINS-I results at the outcome level compared with 203 reviews that reported ROBINS-I results at the study level alone. Most systematic reviews of NRSIs do not fully apply the ROBINS-I guidance. This raises concerns around the validity of the ROBINS-I results reported and the use of the evidence from these reviews in decision-making.

With the rapid growth of scholarly literature, efficient artificial intelligence (AI)–aided abstract screening tools are becoming increasingly important. This study evaluated 10 different machine learning (ML) algorithms used in AI-aided screening tools for ordering abstracts according to their estimated relevance. We focused on assessing their performance in terms of the number of abstracts required to screen to achieve a sufficient detection rate of relevant articles. Our evaluation included articles screened with diverse inclusion and exclusion criteria. Crucially, we examined how characteristics of the screening data—such as the proportion of relevant articles, the overall frequency of abstracts, and the amount of training data—impacted algorithm effectiveness. Our findings provide valuable insights for researchers across disciplines, highlighting key factors to consider when selecting an ML algorithm and determining a stopping point for AI-aided screening. Specifically, we observed that the algorithm combining the logistic regression (LR) classifier with the sentence-bidirectional encoder representations from transformers (SBERT) feature extractor outperformed other algorithms, demonstrating both the highest efficiency and the lowest variability in performance. Nonetheless, the algorithm’s performance varied across experimental conditions. Building on these findings, we discuss the results and provide practical recommendations to assist users in the AI-aided screening process.

A meta-analysis of diagnostic test accuracy (DTA) studies typically synthesizes study-specific test sensitivity ($Se$) and specificity ($Sp$) to quantify the accuracy of an index test of interest. The bivariate linear mixed effects model with logit transformation of $Se$ and $Sp$ (BLMM-Logit) is commonly used to make statistical inferences, but may lead to misleading results due to the need for Haldane–Anscombe correction and an approximate estimation of variance within the study. Alternative models based on the arcsine square root and Freeman–Tukey double arcsine transformation have been proposed to address these issues; however, they still rely on approximate variance estimation, which is suitable only for large sample sizes. The bivariate generalized linear mixed effects model (BGLMM) is another option, but it faces convergence issues with small meta-analyses or sparse primary studies. To address these limitations, we proposed an exact within-study variance calculation method that does not require Haldane–Anscombe correction and is applicable regardless of the transformation used or the number of studies and participants. We evaluated this method against existing approaches using real-life and simulated DTA meta-analyses. The methods were comparable for large meta-analyses. However, BLMM-Logit demonstrated substantial negative bias in estimating variances between studies and consistently underestimated summary $Se$ and $Sp$ in all simulation scenarios. In contrast, the proposed exact methods (Exact-Logit, Exact-ASR, and Exact-FTDA) and BGLMM had minimal bias and better performance metrics, particularly for meta-analyses with sparse primary studies. Thus, the proposed exact methods should be preferred for DTA meta-analyses with small or sparse studies.

Large language models have shown promise for automating data extraction (DE) in systematic reviews (SRs), but most existing approaches require manual interaction. We developed an open-source system using GPT-4o to automatically extract data with no human intervention during the extraction process. We developed the system on a dataset of 290 randomized controlled trials (RCTs) from a published SR about cognitive behavioral therapy for insomnia. We evaluated the system on two other datasets: 5 RCTs from an updated search for the same review and 10 RCTs used in a separate published study that had also evaluated automated DE. We developed the best approach across all variables in the development dataset using GPT-4o. The performance in the updated-search dataset using o3 was 74.9% sensitivity, 76.7% specificity, 75.7 precision, 93.5% variable detection comprehensiveness, and 75.3% accuracy. In both datasets, accuracy was higher for string variables (e.g., country, study design, drug names, and outcome definitions) compared with numeric variables. In the third external validation dataset, GPT-4o showed a lower performance with a mean accuracy of 84.4% compared with the previous study. However, by adjusting our DE method, while maintaining the same prompting technique, we achieved a mean accuracy of 96.3%, which was comparable to the previous manual extraction study. Our system shows potential for assisting the DE of string variables alongside a human reviewer. However, it cannot yet replace humans for numeric DE. Further evaluation across diverse review contexts is needed to establish broader applicability.

AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews, version 2) and ROBIS are tools used to assess the methodological quality and the risk of bias in a systematic review (SR). We applied AMSTAR-2 and ROBIS to a sample of 200 published SRs. We investigated the overlap in their methodological constructs, responses by item, and overall, percentage agreement, direction of effect, and timing of assessments. AMSTAR-2 contains 16 items and ROBIS 24 items. Three items in AMSTAR-2 and nine in ROBIS did not overlap in construct. Of the 200 SRs, 73% were low or critically low quality using AMSTAR-2, and 81% had a high risk of bias using ROBIS. The median time to complete AMSTAR-2 and ROBIS was 51 and 64 minutes, respectively. When assessment times were calibrated to the number of items in each tool, each item took an average of 3.2 minutes per item for AMSTAR-2 compared to 2.7 minutes for ROBIS. Nine percent of SRs had opposing ratings (i.e., AMSTAR-2 was high quality while ROBIS was high risk). In both tools, three-quarters of items showed more than 70% agreement between raters after extensive training and piloting. AMSTAR-2 and ROBIS provide complementary rather than interchangeable assessments of systematic reviews. AMSTAR-2 may be preferable when efficiency is prioritized and methodological rigour is the focus, whereas ROBIS offers a deeper examination of potential biases and external validity. Given the widespread reliance on systematic reviews for policy and practice, selecting the appropriate appraisal tool remains crucial. Future research should explore strategies to integrate the strengths of both instruments while minimizing the burden on assessors.

The evaluation of the role of face masks in preventing respiratory infections is a paradigm case in synthesising complex evidence (i.e. extensive, diverse, technically specialised, and with multilevel chains of causality). Primary studies have assessed different mask types, diseases, populations, and settings using different research designs. Numerous review teams have attempted to synthesise this literature, in which observational (case–control, cohort, cross-sectional) and ecological studies predominate. Their findings and conclusions vary widely.

This article critically examines how 66 systematic reviews dealt with mask efficacy studies. Risk-of-bias tools produced unreliable assessments when—as was often the case—review teams lacked methodological expertise or topic-specific understanding. This was especially true when datasets were large and heterogeneous, with multiple biases playing out in different ways and requiring nuanced adjustments. In such circumstances, tools were sometimes used crudely and reductively rather than to support close reading of primary studies and guide expert judgments. Various moves by reviewers—excluding observational evidence altogether, assessing risk but not direction of biases, omitting distinguishing details of primary studies, and producing meta-analyses that combined studies of different designs or included studies at critical risk of bias—served to obscure important aspects of heterogeneity, resulting in bland and unhelpful summary statements.

We draw on philosophy to question the formulaic use of generic risk-of-bias tools, especially when the primary evidence demands expert understanding and tailoring of study quality questions to the topic. We call for more rigorous training and oversight of reviewers of complex evidence and for new review methods designed specifically for such evidence.

Critical appraisal is a core component of JBI qualitative evidence synthesis, offering insights into the quality of included studies and their potential influence on synthesized findings. However, limited guidance exists on whether, when, and how to exclude studies based on appraisal results. This study examined the methods used in JBI qualitative systematic reviews and the implications for synthesized findings. In this study, a systematic analysis of qualitative reviews published between 2018 and 2022 in JBI Evidence Synthesis was conducted. Data on decisions and their justifications were extracted from reviews and protocols. Descriptive and content analysis explored variations in the reported methods. Forty-five reviews were included. Approaches reported varied widely: 24% of reviews included all studies regardless of quality, while others applied exclusion criteria (36%), cutoff scores (11%), or multiple methods (9%). Limited justifications were provided for the approaches. Few reviews cited methodological references to support their decisions. Review authors reported their approach in various sections of the review, with inconsistencies identified in 18% of the sample. In addition, unclear or ambiguous descriptions were also identified in 18% of the included reviews. No clear differences were observed in ConQual scores between reviews that excluded studies and those that did not. Overall, the variability raises concerns about the credibility, transparency, and reproducibility of JBI qualitative systematic reviews. Decisions regarding the inclusion or exclusion of studies based on critical appraisal need to be clearly justified and consistently reported. Further methodological research is needed to support rigorous decision-making and to improve the reliability of synthesized findings.

Indirect treatment comparison (ITC) is widely used to estimate the comparative effectiveness of treatments when head-to-head trials are unavailable. For the typical scenario of anchored ITC where one trial compares drug A to drug C (AC trial) and another compares drug B to drug C (BC trial), the comparative effectiveness of drugs A versus B is calculated by subtracting (or dividing) the relative treatment effect of A versus C in the AC trial by that of B versus C in the BC trial, assuming the covariate distributions in both trials are balanced. This operation is valid only if the chosen effect measure is transitive, that is, in a three-arm randomized trial of drugs A, B, and C, the direct treatment effect of A versus B equals the indirect treatment effect of A versus B through their comparisons to C. For survival outcomes, many ITCs use the hazard ratio (HR) as the effect measure. In this article, we demonstrate that HR is generally not transitive and should be used with caution. As more reliable alternatives, we recommend effect measures with better transitivity properties: the restricted mean survival time (RMST) difference, the landmark survival probability difference (or ratio) at a prespecified time point, and the average hazard with survival weights (AH-SW) difference.

Bayesian hierarchical models offer a principled framework for adjusting for study-level bias in meta-analysis, but their complexity and sensitivity to prior specifications necessitate a systematic framework for robust application. This study demonstrates the application of a Bayesian workflow to this challenge, comparing a standard random-effects model to a bias-adjustment model across a real-world dataset and a targeted simulation study. The workflow revealed a high sensitivity of results to the prior on bias probability, showing that while the simpler random-effects model had superior predictive accuracy as measured by the widely applicable information criterion, the bias-adjustment model successfully propagated uncertainty by producing wider, more conservative credible intervals. The simulation confirmed the model’s ability to recover true parameters when priors were well-specified. These results establish the Bayesian workflow as a principled framework for diagnosing model sensitivities and ensuring the transparent application of complex bias-adjustment models in evidence synthesis.

A growing number of oncology treatments, such as bevacizumab, are used across multiple indications. However, in health technology assessment (HTA), their clinical and cost-effectiveness are typically appraised within a single target indication. This approach excludes a broader evidence base across other indications. To address this, we explored multi-indication meta-analysis methods that share evidence across indications.

We conducted a simulation study to evaluate alternative multi-indication synthesis models. This included univariate (mixture and non-mixture) methods synthesizing overall survival (OS) data and bivariate surrogacy models jointly modeling treatment effects on progression-free survival (PFS) and OS, pooling surrogacy parameters across indications. Simulated datasets were generated using a multistate disease progression model under various scenarios, including different levels of heterogeneity within and between indications, outlier indications, and varying data on OS for the target indication. We evaluated the performance of the synthesis models applied to the simulated datasets in terms of their ability to predict OS in a target indication.

The results showed univariate multi-indication methods could reduce uncertainty without increasing bias, particularly when OS data were available in the target indication. Compared with univariate methods, mixture models did not significantly improve performance and are not recommended for HTA. In scenarios where OS data in the target indication is absent and there are also outlier indications, bivariate surrogacy models showed promise in correcting bias relative to univariate models, though further research under realistic conditions is needed.

Multi-indication methods are more complex than traditional approaches but can potentially reduce uncertainty in HTA decisions.

1. Meta-analyses are a reliable method for a quantitative research synthesis. They are, however, prone to specific biases that can be introduced in the process. Such a bias could exist if primary literature produces similar results if coming from the same authors. Authorship network bias is the non-independence of effect sizes introduced by the overlap of authors of primary studies. If not accounted for, it can severely impact the quality of meta-analysis and the conclusions drawn from it.

2. To account for such non-independence, multilevel models with author clusters as an additional hierarchy level were recently suggested. We propose a new method for the detection of non-independent effect sizes based on authorship networks and for their correction.

3. An analysis of simulated data demonstrates the effectiveness of the here-suggested new method. We further applied our new method to nine exemplary meta-analyses.

4. Our new method for detection and effective correction can be easily integrated in existing meta-analysis workflows, using the functionality already offered by R’s metafor package.

5. Our goal is to enhance the reliability of meta-analyses by highlighting potential authorship network bias and offering a method to address this often-overlooked bias.

Meta-analytic confirmatory factor analysis (CFA) is a type of meta-analytic structural equation modeling (MASEM) that is useful for evaluating the factor structure of measurement scales based on data from multiple studies. Modeling the factor structure is just one example of the many potentially interesting research questions. Analyzing covariance matrices allows for the evaluation of measurement properties across studies, such as whether indicators are functioning the same across studies. For example, are some indicators more indicative of the common factor in certain types of studies than in others? The additional analysis of means of the observed variables opens up many other research questions to consider such as: “Are there mean differences in mental health between clinical and non-clinical samples?” To answer such questions, it is necessary to analyze both the covariance and the mean structure of the indicators. In this paper, we present, illustrate, and evaluate a method to incorporate the means of variables in the MASEM analyses of such datasets. We focus on meta-analytic CFA, with the aim of testing differences in latent means across studies. We provide illustrations of the comparison of latent means across groups of studies using two empirical datasets, for which data and analysis scripts are provided online. The performance of the new model was tested in a small-scale simulation study. The results showed adequate performance under the tested conditions. Finally, we discuss how the proposed method relates to other analysis options such as multigroup or multilevel structural equation modeling.