Systematic reviews

Authors often submit articles that include the term ‘systematic’ in the title without realising that this term requires strict adherence to specific criteria. A systematic review follows explicit methodology to answer a well-defined research question; it involves a comprehensive literature search, rigorous evaluation of the quantity and quality of research evidence, and analysis of the evidence to synthesise an answer to the research question. The evidence gathered in systematic reviews can be qualitative or quantitative. However, if adequate and comparable quantitative data are available, then a meta-analysis can be performed to assess the weighted and summarised effect size of the studies included. Depending on the research question and the data collected, systematic reviews may or may not include quantitative meta-analyses; however, meta-analyses should be performed in the setting of a systematic review to ensure that all of the appropriate data are accessed. The components of a systematic review can be found in an important article by Moher et al., published in 2009, which defined requirements for systematic reviews and meta-analyses.^{Reference Moher, Liberati, Tetzlaff and Altman2}

In order to optimise reporting of meta-analyses, an international group developed the Quality of Reporting of Meta-Analyses (‘QUOROM’) statement at a meeting in 1996, which led to the publication of the Quality of Reporting of Meta-Analyses statement in 1999.^{Reference Moher, Cook, Eastwood, Olkin, Rennie and Stroup3} Moher et al. revised that document and re-named the guidelines the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (‘PRISMA’). The Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement included both meta-analyses and systematic reviews, and the authors incorporated definitions established by the Cochrane Collaboration.^{Reference Green and Higgins4}

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement established the current standard for systematic reviews. In order to qualify as a systematic review, the methods section should acknowledge use of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, and all Preferred Reporting Items for Systematic Reviews and Meta-Analyses components should be incorporated strictly in all facets of the paper, from the research question to the discussion.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement includes a checklist of 27 items that must be included when reporting a systematic review or meta-analysis.^{Reference Moher, Liberati, Tetzlaff and Altman2} A downloadable version of this checklist can be used by authors, reviewers and journal editorial staff to ensure compliance with recommended components.⁵ All 27 components will not be listed in this brief editorial (although authors and reviewers are encouraged to consult the article by Moher et al. and familiarise themselves with all items), but a few will be highlighted.

The research question, as reflected in the title, should be a hypothesis-based specific research inquiry. The introduction must describe the rationale for the review, and provide a specific goal or set of goals to be addressed. The type of systematic review, according to the Cochrane Collaboration, is based on the research question being asked, and may assess diagnostic test accuracy, review prognostic studies evidence, evaluate intervention effects, scrutinise research methodology or summarise qualitative evidence.⁶

In the methods section, the participants, interventions, comparisons, outcomes and study design (‘PICOS’) must be put forward. In addition to mentioning compliance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses, the methods section should state whether a review protocol exists and, if so, where it can be accessed (including a registration number). Systematic reviews are eligible for registration in the International Prospective Register of Systematic Reviews (‘PROSPERO’), as established at the University of York, UK. When the International Prospective Register of Systematic Reviews is used (it is available but not required for systematic reviews), registration should occur at the initial protocol stage of the review, and the final paper should direct to the information in the register.

The methods section also must include specific study characteristics, including databases used, years considered, languages of articles included, and specific inclusion and exclusion criteria for studies; the rationale for each criterion must be included. Which individuals specifically performed searches should be noted. Details regarding the following aspects also should be reported: electronic search strategy (with a full description of at least one electronic search strategy sufficient to allow replication of the search), process for article selection, data variables sought, assumptions and simplifications, methods for assessing bias risk of each individual study (such as selective reporting in individual studies) and utilisation of this information in data synthesis, principal summary measures (e.g. risk ratio, hazard ratio, difference in means), methods of data management and combining study results, outcome level assessment, and any other information.

The results section should mention the number of studies identified, screened and evaluated for eligibility (including rationale for exclusion), and the number of studies in the final synthesis. A Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram should be included to convey this information succinctly.⁷ The results section also should report the study characteristics, study results, risk of bias within and across studies, and a qualitative or quantitative synthesis of the results of the included studies. This level of rigor in acquiring and evaluating the evidence of each individual study is one of the criteria that distinguishes systematic reviews from other categories.

If the systematic review involves studies with paired samples and quantitative data, a summary of data should be provided for each intervention group, along with effect estimates and confidence intervals for all outcomes of each study. If a meta-analysis is performed, then synthesised effect size should be reported, with confidence intervals and measures of consistency (i.e. data heterogeneity such as I²), for each meta-analysis, and an assessment of bias risk across studies. A forest plot, which provides a graphical presentation of the meta-analysis results, should be included.

The discussion section should summarise: the main findings, commenting on the strength of evidence for each outcome, as well as relevance to healthcare providers, policymakers and other key stake-holders; limitations and outcomes of the study; and conclusions, highlighting the interpretation of results in the context of other research, and implications for future research.

Non-adherence to any of these criteria, and the others listed in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement and checklist, means the review does not qualify as being ‘systematic’.

Meta-analyses

Meta-analyses, when feasible based on available and comparable quantitative data, supplement a systematic review evaluation by adding a secondary statistical analysis of the pooled weighted outcomes of similar studies. This adds a level of objectivity in the synthesis of the review's findings.

Meta-analyses are appropriate when at least two individual studies contain paired samples (experimental group and control group) and provide quantitative outcome data and sample size. Studies that lack a control group may over-estimate the effect size of the experimental intervention or condition being studied, and are not ideal for meta-analyses.^{Reference Goodacre8} It also should be remembered that the conclusions of a meta-analysis are only as valid as the data on which the analysis is based. If the articles included are flawed, then the conclusions of the meta-analysis also may be flawed. Systematic reviews and meta-analyses are the most rigorous categories of review.

Mixed-methods reviews

Systematic reviews typically contain a single type of data, either qualitative or quantitative; however, mixed-methods reviews bring together a combination of data types or study types. This approach may be utilised when quantitative data, in the setting of an intervention study, only provide a narrow perspective of the efficacy or effectiveness of the intervention. The addition of qualitative data or qualitative studies may provide a more complete picture of the knowledge, attitudes, and behaviours of clinicians, patients or researchers regarding that intervention. This type of review could involve collecting either the quantitative or the qualitative data using systematic review methodology, but often the qualitative data are gathered using convenience sampling. Many qualitative studies provide useful insights into clinical management and/or implementation of research interventions; incorporating them into a mixed-methods review may provide a valuable perspective on a wide range of literature. Mixed-methods reviews are not necessarily systematic in nature; however, authors conducting mixed-methods reviews should follow systematic review methodology, when possible.

Literature and narrative reviews

Literature reviews include peer-reviewed original research, systematic reviews and meta-analyses, but also may include conference abstracts, books, graduate degree theses and other non-peer-reviewed publications. The methods used to identify and evaluate studies should be specified, but they are less rigorous and comprehensive than those required for systematic reviews. Literature reviews can evaluate a broad topic but do not specifically articulate a specific question, nor do they synthesise the results of included studies rigorously. Like mixed-methods reviews, they provide an overview of published information on the topic, although they may be less comprehensive than integrative reviews. In addition, unlike systematic reviews, they do not need to support evidence-based clinical or research practices, or highlight high-quality evidence for the reader.

Narrative reviews are similar to literature reviews and evaluate the same scope of literature. The terms sometimes are used interchangeably, and author bias in article selection and data interpretation is a potential concern in literature and narrative reviews.

Umbrella reviews

An umbrella review integrates previously published, high-quality reviews such as systematic reviews and meta-analyses. Its purpose is to synthesise information in previously published systematic reviews and meta-analyses into one convenient paper.

Rapid reviews

A rapid review uses systematic review methodology to evaluate existing research. It provides a quick synthesis of evidence and is used most commonly to assist in emergent decision-making, such as that required to determine whether coronavirus disease 2019 (Covid-19) vaccines should receive emergent approval.

Scoping, mapping and systematised reviews

If literature has not been reviewed comprehensively in a specific subject that is varied and complex, a mapping review (also called scoping review) may be useful to organise initial understanding of the topic and its available literature. While mapping reviews may be helpful in crystallising research findings and may be published, they are particularly useful in helping to determine whether a topic is amenable to systematic review, and to help organise and direct the approach of the systematic review or other reviews of the subject.

Systematised reviews are used most commonly by students. The systematised review provides initial assessment of a topic that is potentially appropriate for a systematic review, but a systematised review does not meet the rigorous criteria of a systematic review and has substantially more limited value.

Additional types of reviews exist, including critical review, state-of-the-art review and others.