Hostname: page-component-5d84bcc8dc-tnhbq Total loading time: 0 Render date: 2026-09-14T10:07:10.613Z Has data issue: false hasContentIssue false
  • English
  • Français

Systematic review search methods evaluated using the Preferred Reporting of Items for Systematic Reviews and Meta-Analyses and the Risk Of Bias In Systematic reviews tool

Published online by Cambridge University Press:  07 December 2020

Shelley de Kock Open the ORCID record for Shelley de Kock [Opens in a new window] ,
Lisa Stirk ,
Janine Ross ,
Steven Duffy ,
Caro Noake  and
Kate Misso
Shelley de Kock*
Affiliation:
Kleijnen Systematic Reviews Ltd, York, UK
Lisa Stirk
Affiliation:
Kleijnen Systematic Reviews Ltd, York, UK
Janine Ross
Affiliation:
Kleijnen Systematic Reviews Ltd, York, UK
Steven Duffy
Affiliation:
Kleijnen Systematic Reviews Ltd, York, UK
Caro Noake
Affiliation:
Kleijnen Systematic Reviews Ltd, York, UK
Kate Misso
Affiliation:
Kleijnen Systematic Reviews Ltd, York, UK
*
Author for correspondence: Shelley de Kock, E-mail: shelley@systematic-reviews.com
Get access Rights & Permissions [Opens in a new window]

Abstract

Objectives

To evaluate the methodological and reporting characteristics of search methods of systematic reviews (SRs) using the Preferred Reporting of Systematic Reviews and Meta-Analyses (PRISMA) checklist and the Risk Of Bias In Systematic reviews (ROBIS) tool.

Methods

A sample of 505 SRs published in 2016 was taken from KSR Evidence, a database of SRs, and analyzed to assess compliance with Information sources and Search of the PRISMA checklist. Domain 2 (D2) (Identification and Selection of Studies) of the ROBIS tool was used to judge the risk of bias in search methods.

Results

Regarding Information sources and Search of PRISMA, twenty percent of SRs which claimed to be PRISMA-compliant in their methods, were compliant; twenty-four percent of SRs published in journals that require PRISMA reporting were compliant; nineteen percent in total were found to be compliant. Twenty-eight percent of SRs were judged to be at a low risk of bias in D2 and so searched widely with an effective strategy and, finally, ten percent were both compliant with the reporting of Information sources and with Search of PRISMA and were judged to be at a low risk of bias in D2.

Conclusions

Ninety percent of SRs are failing to report search methods adequately and to conduct comprehensive searches using a wide range of resources. Editors of journals and peer reviewers need to ensure that they understand the requirements of PRISMA and that compliance is adhered to. Additionally, the comprehensiveness of search methods for SRs needs to be given more critical consideration.

Keywords

Systematic reviews as a topicReproducibility of resultsMethods

Information

Type
Assessment
Information
International Journal of Technology Assessment in Health Care , Volume 37 , Issue 1 , 2021 , e18
Copyright
Copyright © The Author(s), 2020. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Akobeng, AK. Understanding systematic reviews and meta-analysis. Arch Dis Child. 2005;90:845–8.CrossRefGoogle ScholarPubMed
Mulrow, CD. Systematic reviews: Rationale for systematic reviews. BMJ. 1994;309:597–9.CrossRefGoogle ScholarPubMed
Centre for Evidence-Based Medicine [Internet]. Oxford Centre for Evidence-Based Medicine—Levels of Evidence (March 2009). Oxford: University of Oxford; 2020 [accessed 2020 Feb 12]; Available from: https://www.cebm.net/2009/06/oxford-centre-evidence-based-medicine-levels-evidence-march-2009/.Google Scholar
Moher, D, Liberati, A, Tetzlaff, J, Altman, DG, The, PG. Preferred reporting items for systematic reviews and meta-analyses: He PRISMA Statement. PLoS Med. 2009;6:e1000097.CrossRefGoogle Scholar
Whiting, P, Davies, P, Savovic, J, Caldwell, D, Reeves, BCShea, B et al. ROBIS: tool to assess risk of bias in systematic reviews [Internet]. Bristol: University of Bristol; 2013 [accessed 2020 Feb 14]; Available from: http://www.bristol.ac.uk/media-library/sites/social-community-medicine/robis/ROBIS%201.2%20Clean.pdf.Google Scholar
de Kock, S, Stirk, L, Noake, C, Duffy, S, Ross, J, Misso, K et al. Systematic reviews assessed as high risk of bias due to avoidable failures in searching: analysis of a data set of critically appraised systematic reviews. Presented at Global Evidence Summit 2017; 2017 Sep 13–16; Cape Town, South Africa.Google Scholar
Whiting, P, Savović, J, Higgins, JPT, Caldwell, DM, Reeves, BC, Shea, B et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.CrossRefGoogle ScholarPubMed
Kleijnen Systematic Reviews. KSR Evidence [Internet]. York: KSR; 2020 [accessed 2019 Jan 30]; Available from: www.ksrevidence.com.Google Scholar
Liberati, A, Altman, DG, Tetzlaff, J, Mulrow, C, Gøtzsche, PC, Ioannidis, JPA et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Ann Intern Med. 2009;151:W-65-94.CrossRefGoogle ScholarPubMed
Tam, WWS, Lo, KKH, Khalechelvam, P. Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: A cross-sectional study. BMJ Open. 2017;7:e013905.CrossRefGoogle ScholarPubMed
Riado Minguez, D, Kowalski, M, Vallve Odena, M, Longin Pontzen, D, Jelicic Kadic, A, Jeric, M et al. Methodological and reporting quality of systematic reviews published in the highest ranking journals in the field of pain. Anesth Analges. 2017;125:1348–1354.CrossRefGoogle ScholarPubMed
Panic, N, Leoncini, E, de Belvis, G, Ricciardi, W, Boccia, S. Evaluation of the endorsement of the preferred reporting items for systematic reviews and meta-analysis (PRISMA) statement on the quality of published systematic review and meta-analyses. PLoS One. 2013;8:e83138.CrossRefGoogle ScholarPubMed
Leclercq, V, Beaudart, C, Ajamieh, S, Rabenda, V, Tirelli, E, Bruyere, O. Meta-analyses indexed in PsycINFO had a better completeness of reporting when they mention PRISMA. J Clin Epidemiol. 2019;115:46–54.CrossRefGoogle ScholarPubMed
Koffel, JB, Rethlefsen, ML. Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: A cross-sectional study. PLoS One. 2016;11:e0163309.CrossRefGoogle ScholarPubMed
Page, MJ, Shamseer, L, Altman, DG, Tetzlaff, J, Sampson, M, Tricco, AC et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: A cross-sectional study. PLoS Med. 2016;13:e1002028.CrossRefGoogle ScholarPubMed
Page, MJ, Moher, D. Evaluations of the uptake and impact of the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement and extensions: a scoping review. Syst Rev. 2017;6:263.CrossRefGoogle ScholarPubMed
Nawijn, F, Ham, WHW, Houwert, RM, Groenwold, RHH, Hietbrink, F, Smeeing, DPJ. Quality of reporting of systematic reviews and meta-analyses in emergency medicine based on the PRISMA statement. BMC Emer Med. 2019;19:19.CrossRefGoogle Scholar
Rethlefsen, M, Ayala, AP, Kirtley, S, Koffel, J, Waffenschmidt, S. Preferred Reporting Items for Systematic Reviews and Meta-Analysis Search Extension (PRISMA-S) 2019: Explanation and elaboration [Internet]. OSF Preprints; 2019 [accessed 2019 Oct 4]; Available from: https://osf.io/sfc38/.Google Scholar