Introduction
Vaccines are a cornerstone of global public health, playing a crucial role in reducing the burden of infectious diseases worldwide [1]. While randomized controlled trials (RCTs) generate evidence of vaccine efficacy under ideal conditions, post-licensure observational studies provide essential real-world evidence on vaccine effectiveness (VE) and public health impact across diverse populations and settings [Reference Lopalco and DeStefano2]. Systematic reviews and meta-analyses (SR/MAs) synthesize findings from these studies to inform clinical practice, public health policies, and vaccination strategies. However, the reliability of SR/MA findings is contingent upon the quality of the included studies and their risk-of-bias (RoB) [Reference Metelli and Chaimani3].
Generic RoB assessment tools, designed to evaluate a broad spectrum of studies, may be insufficient in the context of VE research. Specifically, observational VE studies are susceptible to some important and often unique challenges compared to observational studies in other clinical areas, such as healthy vaccinee bias, healthcare-seeking behavior bias, temporal bias, and biases related to prior infection or immunization schedules [Reference Leis4]. Furthermore, across observational study designs, the types of biases vary. For instance, cohort studies may be affected by differential loss to follow-up and difficulties in accounting for changes in vaccination status, while case-control studies face risks of recall bias, misclassification of vaccination history, and challenges in selecting appropriate control groups [Reference Metelli and Chaimani3]. The increasingly popular case-control study using a test-negative design (TND) mitigates certain biases (e.g., healthcare-seeking behavior bias) by comparing symptomatic individuals who seek care and test positive for the target pathogen (cases) with those who test negative (controls) [Reference Stuurman5]; however, the TND design, when poorly designed or implemented, introduces its own challenges, including heterogeneity in testing procedures, case definitions, and diagnostic tools [Reference Leis4, Reference Amin6].
As in other areas of research, SR/MAs of observational VE studies also face challenges due to more pronounced clinical heterogeneity compared to SR/MAs of RCTs, stemming from variations in participant characteristics, interventions, and outcomes, reflecting real-world practices and less stringent inclusion criteria [Reference Metelli and Chaimani3]. Heterogeneity in exposure and outcome definitions and measurements across VE studies underscore the need for standardized reporting and rigorous RoB assessment to ensure the validity of pooled evidence from VE research.
To gain a more comprehensive understanding of current practices and explore the landscape of RoB assessment in VE research, we conducted a scoping review with the objective of identifying tools currently used to assess RoB in VE studies in published systematic reviews with particular attention to the use of current RoB standards, modifications to current standards, and the application of new original strategies employed by the authors.
Methods
We prospectively registered our scoping review protocol on the Open Science Framework (https://osf.io/euz9b/) [Reference Davoodi7]. We followed the Joanna Briggs Institute (JBI) methodological framework for scoping reviews, initially developed by Arksey and O’Malley [Reference Arksey and O’Malley8] and subsequently refined by Levac et al. [Reference Levac, Colquhoun and O’Brien9]. Additionally, this review is reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist [Reference Tricco10] and PRISMA 2020 statement [Reference Page11]. A completed PRISMA-ScR checklist is provided in Supplementary Appendix A. While our protocol initially specified an English-only inclusion criterion, we ultimately did not impose language restrictions, allowing the inclusion of studies published in any language.
Search strategy
A professional information specialist (BS), in consultation with the research team, developed the search strategy. The MEDLINE strategy was peer-reviewed by another senior information specialist using the Peer-Review of Electronic Search Strategies (PRESS) checklist [Reference McGowan12] (Supplementary Appendix B). Using the multifile option and deduplication tool on the Ovid platform, we searched Ovid MEDLINE® ALL and Embase Classic+Embase. We also searched the core databases of Web of Science. The search combined controlled vocabulary (e.g. ‘Vaccine Efficacy’, ‘Vaccines’, ‘Systematic Review’) with relevant free text (e.g. ‘vaccine effectiveness’, ‘immunization’, ‘evidence-based review’). We executed all searches on 17 May 2023. There were no language restrictions. To ensure feasibility and budgetary parameters, we limited results to the publication years 2013–2023. We downloaded the records to EndNote 9.3.3 (Clarivate Analytics) and deduplicated them prior to uploading them to Covidence (Veritas Health Innovation Ltd). Details of the full search strategy are provided in Supplementary Appendix C. To identify the relevant grey literature and unpublished studies, we utilized the Grey Matters checklist developed by the Canadian Drug Agency (CAD-AMC) [13] to include materials published between 1 January 2018 and 15 August 2023 (Supplementary Appendix D). Our searches were not updated after these dates given the methodological focus of this scoping review and the achievement of saturation.
Selection criteria
Eligibility criteria were defined using the Population-Concept-Context (PCC) framework [Reference Davoodi7]. We included systematic reviews of observational VE studies, with or without meta-analysis, that performed any form of RoB or quality assessment, whether using an established tool or the review authors’ own methodology. To maximize the capture of relevant data, we imposed no restrictions on population characteristics, vaccine types, or vaccine-preventable diseases. For the purpose of this review, we defined VE as any measure of the vaccine’s protective capacity against various outcomes under real-world conditions, evaluated through post-licensure (phase IV) studies [Reference Sullivan and Cowling14]. Systematic reviews that solely included RCTs evaluating vaccine efficacy or studies focused solely on immunogenicity outcomes (e.g. antibody titres, markers of cell-mediated responses) were excluded.
Review process
We imported the search results into Covidence (Veritas Health Innovation Ltd) for final deduplication and screening. The study selection process consisted of two stages. First, two independent reviewers screened all unique records by title and abstract, retaining any record selected by at least one reviewer. In the second stage, two reviewers independently conducted full-text screenings of all records selected in the previous stage, resolving any disagreements through team discussions until consensus was reached. To ensure consistency at each stage, all reviewers initially screened the same 10 publications. The results from this pilot phase informed refinements to the screening process. For non-English publications, we utilized our team’s language capabilities and employed ChatGPT (OpenAI) for translation when needed.
Data extraction and synthesis
We developed and piloted a standardized data extraction form in Covidence to systematically collect relevant information from the included systematic reviews (Supplementary Appendix E). A single reviewer extracted data from the included full-text articles using the form, which was verified by another reviewer. The following elements were extracted: study design(s) included in the systematic review, number of observational studies included, vaccine(s) assessed, definition(s) of VE, outcomes measured, and details about whether and how the review authors conducted RoB assessment(s) of included studies. For the purposes of our review, we included GRADE [Reference Guyatt15] and CHERG [Reference Walker16] as RoB tools when no other designated RoB tool was used. Although technically GRADE is an evidence assessment framework, one of the key domains is study limitations/RoB, which is represented by five key criteria relevant to observational studies: ‘failure to develop and apply appropriate eligibility criteria, flawed measurement of both exposure and outcome, failure to adequately control confounding, and incomplete or inadequately short follow-up’ [Reference Schünemann17]. Similarly, CHERG was derived from the GRADE framework and adapts its principles for assessing the quality of evidence on intervention effects relevant to child health [Reference Walker16]. Separately, we also documented instances where GRADE or CHERG were used alongside a designated RoB tool. Discrepancies during data extraction were resolved through discussion to maintain consistency and accuracy. Descriptive statistics were used to summarize the data.
Results
Search results
Our searches identified 2,509 unique records from three databases and 54 reports from the grey literature sources, all of which underwent title and abstract screening (Figure 1). From the databases, 367 records were eligible for full-text assessment, including 8 (2.2%) published in languages other than English (Chinese, French, Portuguese, Korean, and Farsi). Of these, 203 systematic reviews (202 in English and one in Portuguese) met our inclusion criteria and were included in the data synthesis (Supplementary Table S1, Supplementary Appendix F). A list of excluded studies with reasons for exclusion is provided in Supplementary Appendix G. Notably, 38 systematic reviews were excluded because no RoB or quality assessment was reported, thus making them ineligible (Figure 1).
The PRISMA flow diagram of study selection [Reference Haddaway54].
Figure 1 Long description
The flowchart is organized into four vertical phases labeled on the left.
1. Identification phase. Two top boxes show records identified through database searching n = 3779 and other sources n = 54. Databases include MEDLINE n = 1503, Web of Science n = 1367, and Embase n = 909. A horizontal dashed arrow points to a box for Duplicates removed n = 1324.
2. Screening phase. A central box for Records screened n = 2509 leads to a horizontal dashed arrow pointing to Records excluded n = 2115.
3. Eligibility phase. A central box for Full-text articles assessed for eligibility n = 394 includes 367 from databases and 27 from other sources. A horizontal dashed arrow points to a box for Full-text articles excluded, with reasons n = 182. Reasons include: Wrong outcome(s) n = 46, No RoB assessment n = 38, Conference abstract n = 36, Not a systematic review n = 23, Not peer-reviewed n = 19, Wrong study design n = 11, No observational studies n = 8, and Wrong intervention n = 1.
4. Included phase. The final central box shows Total studies included n = 212, consisting of Systematic reviews n = 203 and Grey literature reports n = 9.
From the 54 records identified through the grey literature sources, 27 remained after duplicate removal and were screened, yielding 9 eligible reports. Although these were not systematic reviews and thus excluded from the descriptive analysis, we reviewed their RoB assessment approaches. We have summarized these reports in Supplementary Appendix F and Supplementary Table S5, highlighting their valuable methodologies for quality assessment, despite not being discussed further in the main text.
Characteristics of the systematic reviews
Of the 203 systematic reviews included in our analysis, 112 (55.2%) included both RCTs and observational studies (Table 1). The systematic reviews included various observational study designs: cohort studies (175/203, 86.2%), case-control studies (131/203, 64.5%), and TND studies (45/203, 22.2%). On average, systematic reviews included 20 observational studies (range one to 290); 37.9% of reviews incorporated fewer than 10 primary studies and 19.2% more than 30.
Characteristics of systematic review articles included in the scoping review (N = 203)
Table 1 Long description
The table is organized into seven main categories with corresponding counts (N) and percentages (%).
* RCTs included in the systematic review: Yes (112, 55.2%), No (91, 44.8%).
* Number of observational studies included: Mean (S D) is 19.8 (plus or minus 25.7); Minimum to maximum range includes a range from 1 to 1290. Distribution: less than 10 (77, 37.9%), 10 to 20 (61, 30.0%), 21 to 30 (26, 12.8%), and greater than 30 (39, 19.2%).
* Observational study designs: Cohort (175, 86.2%), Case-control (131, 64.5%), Test-negative (45, 22.2%), Cross-sectional (31, 15.3%), Screening method (13, 6.4%), Other (31, 15.3%), and Not reported (6, 3.0%).
* Vaccine-preventable disease under investigation: COVID19 (66, 32.5%), Seasonal influenza (62, 30.5%), Pneumococcal disease (19, 9.4%), HPV (15, 7.4%), Varicella or Herpes zoster (11, 5.4%), MMR (10, 4.9%), Pandemic influenza (10, 4.9%), Pertussis (8, 3.9%), Rotavirus (7, 3.4%), and Other (18, 8.9%).
* Definitions of VE: Descriptive (51, 25.1%) and Meta-analyzed (152, 74.9%). Meta-analysis was based on Odds Ratio (89, 58.6%), Risk Ratio (73, 48.0%), Hazard Ratio (19, 12.5%), Incidence Rate Ratio (15, 9.9%), Diagnostic Odds Ratio (4, 2.6%), Alternative calculations (13, 8.6%), and Not specified (10, 6.6%).
* Outcomes assessed: Laboratory-confirmed infection (137, 67.5%), Symptomatic disease (130, 64.0%), Hospitalization (88, 43.3%), Death (70, 34.5%), Contact with HCP (15, 7.4%), Disease severity (4, 2.0%), and Other (9, 4.4%).
* Risk-of-bias assessment performed: Yes (190, 93.6%), No (13, 6.4%).
a Categories are not mutually exclusive, e.g., reviews could report more than one effect measure or outcome.
b VE = [1 − {(PCV × (1 − PPV)) / ((1 − PCV) × PPV)}], (PCV: proportion of cases vaccinated, PPV: proportion of population vaccinated); VE = [1 − {(c1 / N1) / ((c0 + 1) / (N0 + 1))}], where VE = [(1 − RR) × 100], RR = (c1 / N1) (risk in the vaccinated group) and the risk in the unvaccinated group is adjusted with a continuity correction (c0 + 1) / (N0 + 1); VE as a dichotomous outcome (e.g., incidence: yes or no), RR = {OR / [(1 − P₀) + (P₀ × OR)]}, (P₀: event incidence in control group, OR: odds ratio); VE(t) = A × e(−w × t), (A: initial effectiveness, w: waning rate, t: time); VE = [1 − (ARV/ARU) × 100], (ARV, ARU: attack rates in vaccinated and unvaccinated); VE = [(pooled HR − 1) / HR] or VE = [(pooled OR − 1) / OR]; Broome method; Pooled relative VE (rVE); Risk reduction.
c Examples include emergency department, urgent care or clinic/office visits.
HCP, healthcare professional; HPV, human papillomavirus; SD, standard deviation; RCT, randomized controlled trial; RoB, risk of bias; VE, vaccine effectiveness.
COVID-19 and seasonal influenza vaccines were the most frequently studied across included systematic reviews (66/203 (32.5%) and 62/203 (30.5%), respectively). Others commonly assessed included vaccines against pneumococcal disease (19/203, 9.4%) and human papillomavirus (HPV; 15/203, 7.4%). Outcomes evaluated included laboratory-confirmed infections (137/203, 67.5%), symptomatic disease (130/203, 64.0%), and hospitalization (88/203, 43.3%). The details are provided in Supplementary Table S1 (Supplementary Appendix F).
The included systematic reviews revealed diverse approaches to defining and calculating VE. Notably, 152/203 (74.9%) reviews reported meta-analysed VE estimates based on various adjusted effect measures, while 51/203 (25.1%) reviews reported VE descriptively as presented in the primary studies, without pooling. Among these 152 reviews, the most used effect measure was the odds ratio (OR), applied in 58.6% (89/152) of the reviews, using the formula VE = (1 − OR) × 100. The risk ratio (RR) was the second-most used measure, employed in 48.0% (73/152) of reviews, utilizing the formula VE = (1 − RR) × 100. Less commonly, hazard ratios were used in 19/152 reviews (12.5%), incidence rate ratios in 15/152 reviews (9.9%), and diagnostic odds ratios in 4/152 reviews (2.6%). Further details are provided in Table 1.
RoB assessment methods
Most of the systematic reviews (190/203, 93.6%) reported using one (176/203, 86.7%) or multiple (14/203, 7.2%) existing RoB and/or quality-of-evidence assessment tools or checklists. As shown in Table 2, the Newcastle-Ottawa Scale (NOS) [Reference Wells18] was the most employed (85/190, 44.7%), followed by Risk of Bias in Non-randomized Studies of Interventions (ROBINS-I) (46/190, 24.2%) [Reference Sterne19], and JBI tools (11/190, 5.8%) [20]. Ad-hoc author-developed tools were used in 13/203 (6.4%) reviews. Two quality assessment frameworks, GRADE and CHERG, were used to assess RoB or quality in 6 (3.2%) and 3 (1.6%) reviews, respectively. GRADE and CHERG were used alongside designated RoB tools in 37 and 1 review, respectively. Along with occasional instances of these certainty assessments (e.g. GRADE and CHERG) being used to assess RoB/quality, we identified other checklists and tools being used to assess RoB, such as reporting guidelines (e.g. STROBE) [Reference Von Elm21], tools designed for assessment of randomized trials (that can be used to assess prospective cohort studies) being applied to other study designs, such as case-control [Reference Higgins and Green22], and tools designed for animal studies (e.g. SYRCLE’s RoB tool) [Reference Hooijmans23].
Risk-of-bias assessment tools/checklists utilized in included systematic reviews (n = 190)
Table 2 Long description
The table consists of three columns: RoB tool/checklist, N, and percentage. The data is as follows:
* Newcastle-Ottawa Scale (N OS): 85, 44.7 percent.
* Risk of Bias in Non-randomized Studies of Interventions (R OBINS-I): 46, 24.2 percent.
* Joanna Briggs Institute (J BI): 11, 5.8 percent.
* Downs and Black Checklist: 7, 3.7 percent.
* Grading of Recommendations, Assessment, Development, and Evaluations (GRADE): 6, 3.2 percent.
* National Heart, Lung and Blood Institute (N HLBI) Study Quality Assessment Tool: 6, 3.2 percent.
* National Institutes of Health (N IH) quality assessment tool: 6, 3.2 percent.
* Cochrane RoB tool: 5, 2.6 percent.
* Checklist recommended by Agency for Healthcare Research and Quality (A HRQ) for cross-sectional/prevalence study quality: 4, 2.1 percent.
* Child Health Epidemiology Reference Group (C HERG): 3, 1.6 percent.
* Scottish Intercollegiate Guidelines Network (S IG N): 3, 1.6 percent.
* Critical Appraisal Skills Programme (C ASP): 3, 1.6 percent.
* Other: 20, 10.5 percent.
a Categories are not mutually exclusive as more than one tool was utilized in some systematic reviews.
b GRADE was reported to be used to assess quality and/or risk-of-bias. While technically GRADE is a quality assessment framework, one of the key domains is risk-of-bias. 37 of the systematic reviews used GRADE alongside a designated RoB tool.
c CHERG was reported to be used to assess quality and/or risk-of-bias. While technically CHERG is a quality assessment framework, one of the key domains is study quality. One of the of systematic reviews used CHERG alongside a designated RoB tool [Reference Fulton55].
Among the 190 systematic reviews that utilized a published RoB assessment tool, 34 (17.9%) reported adapting the tool(s) to better align with the reviews’ goals. These adaptations were made to align with characteristics of included primary studies, such as specific study designs, vaccine types, or other relevant features. Detailed descriptions of these modifications and their implications are presented in Supplementary Tables S2 and S3 (Supplementary Appendix F). Common adjustments included modifications to published scoring systems, refinements to specific assessment criteria, or inclusion/exclusion checklist items. For instance, the authors of some reviews adapted the NOS by converting its star ratings into categorical risk levels (e.g. three or four categories from low to high RoB based on whether a certain number of items were missing or inadequate) [Reference Sun24–Reference Meggiolaro27], while others applied scoring cut-offs (e.g. categorizing a study into low/moderate/high quality based on the number of points/stars assigned to each item) to interpret the results [Reference Zhu, Liu and Zhang28–Reference Berild31]. Two systematic reviews customized the NOS by integrating items from other checklists [Reference Gupta30, Reference Berild31]. For example, Gupta et al. (2022) [Reference Gupta30] modified the NOS by incorporating the three domains of selection, comparability, and outcome/exposure from the Agency for Healthcare Research and Quality (AHRQ) [Reference Viswanathan32] framework and converting NOS scores to align with the AHRQ quality assessment thresholds. Other reviews made what could be perceived as more substantial modifications, such as removing non-relevant criteria or adapting the assessment criteria of the selected RoB tool to better align with the specific vaccines, populations, or outcomes under evaluation [Reference Hughes33–Reference Markowitz37]. For instance, Hughes et al. [Reference Hughes33] added context-specific assessment criteria to their RoB assessment, for example, studies were classified as being at a high risk of bias if they reported the following issues: cold chain issues, age groups spanning 10 years, and unclear or missing age at initial dose. Similarly, Markowitz et al. [Reference Markowitz37] adapted questions to the specific context, in this case, making them specific to reduced dose VE studies, e.g., assessing whether there was sufficient interval between the first and second vaccine dose and whether there was a sufficient buffer time between vaccination and case counting to avoid incorrectly counting prevalent infections.
Thirteen systematic reviews (6.4%) did not assess RoB through a commonly used tool (Supplementary Table S4 and Supplementary Appendix F). One review by Kandeil et al. (2020) [Reference Kandeil38] reported considering the GRADE framework to assess quality but ultimately did not use it, citing the diverse study outcomes and extensive data recalculations required. Instead, the quality of studies was assessed using a rubric designed by the review authors. Similarly, the remaining 12 reviews employed custom methods to assess biases and study validity. These custom methods often involved creating a checklist, scale, or resource that captured RoB and/or quality concepts the authors found relevant. In cases where no applicable tool was available for a given study design, the authors provided a narrative description of their quality assessment, commenting on quality characteristics often discussed in the literature [Reference Jackson39–Reference Markowitz49]. For example, Darvishian et al (2014) [Reference Darvishian41] created a checklist and elicitation scale to evaluate internal and external biases focusing on their potential impact on VE estimates. Similarly, Andani et al. (2022) [Reference Andani46] developed an evaluation strategy targeting three key quality features related to case detection, data collection methods, and quantitative methods. This tool classified studies as weak, moderate, strong, or having a fatal error in each domain. Studies receiving a fatal error rating in any domain were excluded from further analysis. Similarly, in the absence of an RoB tool tailored to TND studies, review authors often rely on methodological features that are applicable across designs. For example, Okoli et al. (2021) [Reference Okoli47] examined studies on three quality features that could introduce bias: participant enrollment methods, methods for verifying vaccination status, and statistical adjustments for age and comorbidities.
Discussion
This scoping review identified 203 systematic reviews of observational VE studies, most of which assessed RoB using a validated risk-of-bias assessment tool. The systematic application of standardized tools and reporting guidelines is essential to promote research reproducibility and enhance the credibility of scientific findings used to inform clinical and policy decisions.
Importantly, our objective was not to evaluate the performance of individual RoB tools in detecting specific biases or in discriminating between low- and high-quality VE studies, as such an assessment would require reappraisal of primary studies. Instead, we aimed to document how systematic review authors apply RoB tools in practice and how they respond to perceived limitations. Because the included reviews did not usually report explicit rationales for modifying RoB tools, our findings should be interpreted as describing patterns of use and adaptation, rather than authors’ motivations.
Notwithstanding the widespread usage of RoB tools, and their frequent modification, such as the addition of criteria or alternative scoring approaches, our scoping review suggests significant heterogeneity of practices concerning the quality and risk-of-bias assessment of VE studies, reflecting limitations in how existing tools are operationalized within VE systematic reviews. It is noteworthy that in our initial screening 38 systematic reviews were excluded from our review due to the absence of any risk-of-bias or quality assessment.
Our findings also show that systematic reviewers often address the lack of tailored assessment tools for VE studies by modifying or adapting an existing tool, or in 13 cases, even developing their own methods. For example, Okoli et al. (2021) [Reference Okoli47] created their own approach to overcome the lack of a validated assessment tool for TND studies, despite TND being a common design in VE research. TND studies are increasingly employed over traditional case-control studies because they help mitigate confounding from healthcare-seeking behavior by selecting cases and controls among those who seek care at the same facilities and present with similar symptoms and offer greater efficiency than cohort studies [Reference Teerawattananon50, Reference Mésidor51].
The authors employed a diverse array of RoB or quality assessment tools, with NOS [Reference Wells18], ROBINS-I [Reference Sterne19], and JBI [20] being the most frequently used. Notably, adaptations of these tools, particularly of NOS [Reference Wells18] and ROBINS-I [Reference Sterne19], along with the Downs and Black checklist [Reference Downs and Black52], were implemented in 34 (16.7%) included reviews and 14 (7.2%) resorted to using multiple RoB tools to evaluate the methodological quality of included studies, reflecting the limitations of individual tools in capturing the variability inherent in observational studies specifically aimed at VE estimation. Each design presents distinct challenges that need to be properly accounted for in risk-of-bias and quality assessment. While some of these methodological challenges may also arise in other areas of observational research, their recurrence within VE systematic reviews underscores persistent difficulty in applying existing RoB tools to assess biases central to VE estimation. Cross-sectional studies cannot establish temporal relationships and may overestimate VE when conducted during periods of peak effectiveness. TND studies, while reducing confounding from healthcare-seeking behavior, remain sensitive to differences in testing practices [Reference Mésidor51]. Screening method studies rely heavily on accurate estimates of population vaccination coverage, but they are prone to overestimating VE due to respondent bias and limited ability to control for confounding [Reference Teerawattananon50]. Cohort studies face issues such as differential loss to follow-up, immortal time bias [Reference Remschmidt, Wichmann and Harder42] whereas case-control studies are vulnerable to recall bias and require careful selection of controls to ensure valid estimation of VE [Reference Lu53]. These biases are highlighted in the context of VE research in part because of the frequent reliance on observational designs and their importance for interpreting VE estimates. The complexity of VE estimation and the resulting difficulty of critically appraising VE studies in a standardized manner highlights the urgent need for a comprehensive, reliable, and user-friendly RoB tool tailored specifically to VE research, capable of addressing the unique challenges and biases associated with diverse observational study designs.
While certainty assessment frameworks such as GRADE [Reference Guyatt15] and CHERG [Reference Walker16] contain domains or criteria that include items related to RoB and/or quality, they are not technically RoB tools themselves. In addition, use of checklists, such as reporting guidelines, and application of tools intended for other designs or populations suggest a need for clarity. The occasional use of frameworks, checklists or other tools being used to assess RoB, the complete absence of RoB assessment, the frequent adaptations to tools, the use of multiple tools, and the use of ad hoc approaches together indicate a need for clearer guidance or a tailored framework for assessing RoB in VE systematic reviews.
Of the 203 systematic reviews, 51 (25.1%) reported VE descriptively, while 10 (4.9%) pooled data without specifying the effect estimates used to calculate VE, likely aiming to maximize the inclusion of highly diverse primary studies. From a methodological standpoint, failing to specify the type of effect estimate used for VE calculation in systematic reviews can complicate interpretation and synthesis. Different effect measures (e.g. odds ratios, risk ratios, hazard ratios) are not interchangeable, and pooling them without definition may lead to misleading or non-comparable results. This lack of clarity can hinder readers’ ability to assess the validity and applicability of pooled estimates and underscores the importance of transparent reporting of VE definitions and calculation methods in evidence syntheses.
In line with the descriptive nature of our review, we do not make recommendations about use of a single optimal RoB tool for VE systematic reviews. Instead, the observed patterns of tool adaptation, such as converting star ratings into categorical judgments or adding criteria to better reflect VE-specific concerns, ad hoc approaches, and use of multiple tools suggest that existing instruments may not fully meet the needs of VE evidence synthesis, highlighting a potential gap and the need for either the development of a VE-specific RoB tool or clearer, design-specific guidance on how existing tools should be applied and adapted across different observational VE study designs.
Our review has several strengths, including adherence to rigorous scoping review methods, independent duplicate screening and data extraction processes, and a comprehensive literature search that achieved saturation. However, heterogenous and suboptimal reporting practices across studies pose challenges in fully ascertaining risk of bias assessment methodologies, making it difficult to disentangle limitations of the RoB tools themselves from limitations in how they were applied, and therefore our findings reflect observed practices rather than the intrinsic adequacy of the tools. The quality of our review is inherently tied to the reporting quality of the included systematic reviews, underscoring once again the need for standardized and transparent reporting practices.
Summary
In summary, the identification and evaluation of important sources of bias in observational VE studies is highly dependent on the context and specific characteristics of each study, including its design. Future research should focus on systematically evaluating the main types of bias to be considered, mitigated, and accounted for in VE estimation through various types of VE studies. This would lay the foundation for developing and validating a new RoB tool, or modifying existing ones, to address the features commonly encountered in VE research (e.g. diverse observational designs, time-varying exposure definitions, and the frequent need to adapt RoB tools to vaccination-specific contexts). Standardized reporting practices are equally critical to ensure that findings are robust, comparable, and capable of informing evidence-based clinical and policy decisions.
Abbreviations
- VE
-
Vaccine effectiveness
- RoB
-
Risk-of-bias
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/S0950268826101794.
Data availability statement
The authors confirm that the data supporting the findings of this study are available within the article and its supplementary materials.
Acknowledgements
We thank Kaitryn Campbell, MLIS, MSc, for the peer review of the MEDLINE search strategy. We thank Emilie Chan for her help with title and abstract screening.
Author contribution
Conceptualization: M.B., G.S. Formal analysis: Z.D., C.L. Investigation: Z.D., C.L., K.T., N.S. Methodology: Z.D., C.L., K.T., B.S., N.S., M.S., G.S., M.B. Project administration: Z.D., K.T. Supervision: D.B.F., G.S., M.B. Validation: B.S. Visualization: Z.D. Writing – original draft: Z.D. Writing – review and editing: C.L., K.T., B.S., M.S., D.B.F., G.S., M.B.
Competing interests
D.B.F. has academic appointments at the University of Ottawa and the Children’s Hospital of Eastern Ontario Research Institute; she is employed by Pfizer and works on an unrelated topic. The other authors have no conflicts of interest to disclose.
Funding statement
This research was undertaken, in part, thanks to funding from (a) the Canada Research Chairs (CRC) Program [PI: Giorgia Sulis, Tier 2 CRC in Communicable Diseases Epidemiology]; and (b) funding received from CoVAR-NET (Knowledge Implementation and Translational Team [KITT]) via Canada Institutes of Health Research [PI: Melissa Brouwers, Professor and Director of School of Epidemiology and Public Health]. The funders had no role in the conceptualization, data collection, analysis, interpretation of findings, decision to publish, or preparation of the manuscript.
Open access
