Hostname: page-component-89b8bd64d-46n74 Total loading time: 0 Render date: 2026-05-12T08:56:56.772Z Has data issue: false hasContentIssue false
  • English
  • Français

Barriers to searching for grey literature in systematic reviews: A survey of authors and editors

Published online by Cambridge University Press:  12 May 2026

Jane O’Sullivan ,
Sarah Dawson  and
Julian Piers Thomas Higgins Open the ORCID record for Julian Piers Thomas Higgins [Opens in a new window]
Jane O’Sullivan*
Affiliation:
Anaesthesia and Critical Care, University Hospital Limerick, Ireland Department of Population Health Sciences, University of Bristol Population Health Sciences, UK
Sarah Dawson
Affiliation:
Population Health Sciences, University of Bristol Medical School, UK
Julian Piers Thomas Higgins
Affiliation:
Population Health Sciences, University of Bristol, UK
*
Corresponding author: Jane O’Sullivan; Email: osullij5@tcd.ie
Rights & Permissions [Opens in a new window]

Abstract

To explore the perceptions of, and barriers to, grey literature searching among medical researchers and journal editors. A cross-sectional survey of authors of systematic reviews and the editors of the journals in which the reviews were published. Systematic reviews indexed in MEDLINE, spanning a 4-week period in 2019. We excluded protocols. We asked whether the reviewers were performing a grey literature search. If they were, we asked about their approach to the grey literature search and relevant guidance. If they were not, we asked about their rationale for this. We elucidated understandings of grey literature from all reviewers. The survey to journal editors asked about their perceptions towards grey literature. A consecutive sample of 1,229 systematic reviews was included. A total of 155 authors responded, and a total of 46 journal editors responded. The majority (57%) of reviewers reported performing a grey literature search. However, there was no consensus on types of grey literature items or sources among the reviewers. The most frequent barrier to grey literature searches was concern about the quality/detail of the grey literature items. Editors expressed negative perceptions towards grey literature, rooted in suspicion of its quality. While many reviewers reported performing a grey literature search, there was a diverse understanding of the term grey literature. Concerns about the quality of grey literature exist among reviewers and editors alike. There is a marked discrepancy between the best-practice guidelines and the gatekeepers of journals.

Keywords

grey literaturereview methodologysearch strategy

Information

Type
Research Article
Information
Research Synthesis Methods , First View , pp. 1 - 13
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (https://creativecommons.org/licenses/by/4.0), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.
Copyright
© The Author(s), 2026. Published by Cambridge University Press on behalf of The Society for Research Synthesis Methodology

Highlights

What is already known on this topic?

  • • Systematic reviewers are generally encouraged to perform a grey literature search.

  • • There is considerable confusion about how a grey literature search ought to be performed for a systematic review.

What is new?

  • • Among 155 review authors and 46 journal editors surveyed, definitions and practices relating to grey literature varied substantially.

  • • Although 57% of review authors reported performing a grey literature search, many used sources unlikely to retrieve grey literature comprehensively.

  • • The most frequently reported barrier was concern about the quality or level of detail in grey literature items, and many journal editors expressed reluctance to accept non-peer-reviewed or unpublished material.

Potential impact for RSM readers

  • • Systematic review authors exhibit inconsistent definitions and practices for grey literature, leading to variability in search strategies.

  • • Key barriers include concerns over quality, resource intensiveness, and misaligned expectations from journal editors regarding grey literature.

  • • There is a clear need for standardised, structured guidance to harmonise grey literature searching and improve its integration in systematic reviews.

1 Introduction

Grey literature, sometimes referred to as ‘fugitive literature’, has traditionally been defined as information produced outside conventional publishing and distribution channels, often lacking formal peer review. This definition was formalised in the Luxembourg Definition (1997) and was later expanded in New York in 2004.Reference Lefebvre, Glanville, Briscoe, Higgins, Thomas, Chandler, Cumpston, Li, Page and Welch1 While this foundational definition remains influential, recent literature has expanded on its practical implications and relevance to evidence synthesis. For example, Adams et al.Reference Adams, Hillier-Brown and Moore2 provide critical reflections on grey literature use in public health reviews, and the Campbell Collaboration has issued detailed guidance on identifying, searching for, and documenting grey literature in systematic reviews.Reference MacDonald, Comer and Foster3 An alternative definition of grey literature has been as ‘manifold document types produced on all levels of government, academics, business and industry in print and electronic formats that are protected by intellectual property rights, of sufficient quality to be collected and preserved by library holdings of institutional repositories, but not controlled by commercial publishers’.Reference Schöpfel4 Further definitions exist.Reference Adams, Smart and Huff5

Whatever the definition, grey literature is recognised as a diverse body of material. Examples of grey literature include government reports, working papers, press releases, theses, and conference proceedings. It can represent anything from a dissertation to a government document to a blog.6 Given the diversity of formats and sources encompassed by grey literature, variation in how systematic reviews search for it and report it is expected. Thus, understanding grey literature can be extremely challenging.Reference Adams, Hillier-Brown and Moore2

A review of existing guidance highlights the importance of comprehensive literature searching in systematic reviews.Reference Cooper, Booth, Varley-Campbell, Britten and Garside7 However, there is no consensus on how to perform a grey literature search consistently across reviews. The Cochrane Handbook for Systematic Reviews of Interventions recommends searching for sources of grey literature. It describes grey literature as reports published outside the traditional commercial publishers.Reference Lefebvre, Glanville, Briscoe, Higgins, Thomas, Chandler, Cumpston, Li, Page and Welch1 Search strategies for comprehensive grey literature searches have been proposed.Reference Egger, Juni, Bartlett, Holenstein and Sterne8 – Reference Rothstein, Hopewell, Cooper and Valentine11 Existing guidance has attempted to address these issues, most notably in the Cochrane Handbook’s Technical Supplement to Chapter 4Reference Lefebvre, Glanville, Briscoe, Higgins, Thomas, Chandler, Cumpston, Li, Page and Welch1 , Reference Lefebvre, Glanville, Briscoe, Higgins, Thomas, Chandler, Cumpston, Li and Page12 and the Campbell Collaboration’s recent recommendations for grey literature search strategies and reporting.Reference MacDonald, Comer and Foster3 However, there is no consensus on how to perform a grey literature search consistently across reviews, reflecting the heterogeneity of sources and approaches involved. Consensus may be more feasible at the level of specific types of grey literature rather than the category as a whole.

Despite the guidelines, a previous literature survey found that only a very small proportion of systematic reviews included unpublished literature.Reference Hartling, Featherstone, Nuspl, Shave, Dryden and Vandermeer13 Another older study found that participants from grey literature studies contributed 11% of the study participants included in a sample of 135 meta-analyses.Reference McAuley, Pham, Tugwell and Moher14 These low proportions may be explained by difficulties that researchers face when performing a grey literature search. While there is anecdotal evidence of such difficulties,Reference Godin, Stapleton, Kirkpatrik, Hanning and Leatherdale9 there is no current evidence based on surveying reviewers about the collective difficulties that they face while performing a grey literature search. Moreover, a study more than 30 years ago determined that 30% of journal editors would not publish a meta-analysis with unpublished studies.Reference Cook, Guyatt and Ryan15 However, there is a paucity of recent work to determine if this attitude still persists.

2 Objective

To determine what are the perceptions of, and barriers to, grey literature searching among medical researchers and journal editors.

3 Methods

We undertook a survey of authors of systematic reviews and the editors of the journals in which the reviews were published. We identified the systematic reviews in the context of an empirical study reported in detail elsewhere.Reference O’Sullivan, Dawson, Cooper and Higgins16 In brief, we searched MEDLINE to capture all reports of systematic reviews indexed during a 4-week period (spanning October/November 2019) to capture a representative sample. Survey invitations were distributed in May 2020, and the survey remained open for 6 weeks. We included the reviews in our analysis if they satisfied the following criteria: (i) the report provided a clear statement of eligibility criteria for included studies; (ii) the review was ‘systematic’ in the sense of appearing to have searched comprehensively for studies meeting the eligibility criteria (either because this is stated directly or it can be inferred indirectly from the authors’ use of two or more databases); (iii) the report documented the search strategy, providing at a minimum the information sources for the review; and (iv) the report was published in English. The empirical study included 1,262 systematic reviews, of which 66 were Cochrane Reviews and 1196 were non-Cochrane reviews. For the present study, we excluded reviews reported only in protocols and a small batch of other records where the authors had only searched one database (Epistemonikos).

For each of the included systematic reviews, we recorded the corresponding author and journal of publication. We contacted each corresponding author via a personalised email, which cited the relevant systematic review. The email included a link to a survey via Survey Monkey, asking a series of questions about their understanding of searching for grey literature. If we received an automated error message, specifying that the email address of the corresponding author (as listed in the review) was no longer valid, we sought the most up-to-date email address elsewhere. The Cochrane and non-Cochrane authors received separate links to an identical survey. This enabled us to distinguish easily between the survey respondents as either Cochrane or non-Cochrane authors. The survey asked 14 questions which covered: the search process for the systematic review, understanding the meaning of ‘grey literature’, use of guidance on searching for grey literature, and what barriers might impact the search for grey literature (see Appendix A in the Supplementary Material).

We collated a list of all journals that had published the reviews in the sample and identified the (senior) journal editor. Where multiple journal editors were listed, we selected the content editor. We contacted each selected journal editor via email or via the online queries portal of the journal’s website, again providing a link to a survey via Survey Monkey. We sent a single survey to any editor who covered multiple different journals in our list. The survey asked five questions about the editor’s perceptions and acceptance of grey literature (see Appendix B in the Supplementary Material).

We produced simple descriptive summaries of the survey responses and present these mainly as bar charts. For the survey of authors, we present overall results as well as separate results for Cochrane and non-Cochrane reviews.

4 Results

From the 1,262 reviews included in the empirical study, we excluded 33 protocols and rapid reviews (Figure 1), leaving 1,229 eligible reviews. We attempted to contact 1,229 authors and 661 editors (Figure 1). Of the eligible reviews, 155 authors and 46 editors completed the survey, corresponding to response rates of 12.6% and 7%, respectively. A breakdown of response rates by survey question and review type can be found in Appendix C in the Supplementary Material.

Figure 1

Flow diagram illustrating numbers of authors and editors contacted and completing the survey.

4.1 Review authors

With respect to the search, 53% (82/155) of all reviewers who responded to our survey reported that they received assistance from an information specialist or librarian with 47% non-Cochrane and 95% Cochrane authors (Figure 2) (N.B. the proportion of Cochrane authors in our sample was small). The majority (43%) of non-Cochrane authors reported spending ‘weeks’ on the search strategy (rather than ‘hours’, ‘days’, ‘months’, or ‘years’), while the majority (50%) of Cochrane authors reported spending ‘days’ (Figure 2). The median number of non-Cochrane authors and Cochrane authors involved in the planning and execution of the search strategy was 3 and 2 authors, respectively.

Figure 2

Planning and execution of the literature search: (a) the use of an information specialist for the planning and execution of the search; (b) the full-time equivalents of time taken to plan and undertake the search.

The items most commonly considered by non-Cochrane authors to be grey literature were conference proceedings and dissertations (69% each). The majority of non-Cochrane authors also considered non-peer reviewed journal articles (56%), unpublished trials (66%), government reports (65%), and regulatory documents or clinical study reports (63%) to be grey literature items. Apart from books (35%), editorials (30%), and social media posts (15%), the majority of Cochrane authors thought all of the listed items were grey literature (Figure 3a).

Figure 3

Understanding the meaning of grey literature by reviewers: (a) the items considered to be grey literature; (b) the information resources considered to be sources of grey literature.

The literature resource most commonly considered by non-Cochrane authors to be a source of a grey literature was Google Scholar (62%). The majority of non-Cochrane authors also considered study registries (50%) and Open Grey (60%) to be sources of grey literature. Apart from PubMed (5%) and Web of Science (15%), the majority of Cochrane authors considered all the listed resources to be sources of grey literature (Figure 3b).

A clear majority of Cochrane authors (80%) reported performing a grey literature search, compared with about half (49%) of non-Cochrane authors (Figure 4a). For those who reported performing a grey literature search, the majority of non-Cochrane authors sought the following grey literature items: conference proceedings (66%), dissertations/theses (54%), and unpublished trials (54%). Of the Cochrane authors who reported performing a grey literature search, the majority sought conference proceedings (87%), dissertations and theses (67%), and unpublished trials (67%) (Figure 4b).

Figure 4

Conduct of a grey literature search: (a) performance of a grey literature search; (b) grey literature items sought within the reviews; (c) sources of grey literature searched by the reviewers.

For the non-Cochrane authors who said they performed a grey literature search, the majority of authors sought grey literature in reference lists (73%), PubMed (67%), and Google Scholar (60%). Of the Cochrane authors who said they performed a grey literature search, the majority sought grey literature in trial registries (64%) and reference lists (57%) (Figure 4c).

We asked the authors who reported performing a grey literature search about whether they used written guidance. A minority of non-Cochrane (31%) and Cochrane (7%) authors sought written guidance (Figure 5a). Of those who sought guidance, the majority of respondents sought guidance in the Cochrane Handbook (see Table 1). The majority (72%) of non-Cochrane authors found the guidance useful. Only a minority (20%) of Cochrane authors found the guidance useful (Figure 5b). The majority of non-Cochrane (64%) and Cochrane (71%) authors reported that structured guidance would either be extremely useful, very useful, or somewhat useful (Figure 5c).

Figure 5

Use of guidance for grey literature search: percentage of authors who (a) sought guidance on their grey literature search; (b) found the written guidance on grey literature searching to be useful, among those who reported using any form of guidance; (c) would find structured guidance for performing a grey literature to be useful.

Table 1

Responses from authors to the question ‘What source of guidance was sought?’

The majority of non-Cochrane authors reported concerns regarding the quality/detail of reporting in grey literature (66%) and the potential for bias in studies reported in grey literature items (52%). The majority of Cochrane authors (58%) reported concerns regarding the number of possible sources of grey literature (Figure 6). Appendix D in the Supplementary Material outlines the freeform responses regarding grey literature that were submitted by the survey participants.

Figure 6

What were the barriers to performing a grey literature search.

4.2 Journal editors

Of the small number of journal editors who responded to our survey, 83% reported that the search strategy is scrutinised as part of their peer-review process. In response to a follow-up question, 50% of editors reported that grey literature searches were being performed in the systematic reviews submitted to their journals. However, 65% of journal editors reported that a grey literature search was not expected for the systematic reviews submitted to their journals. Moreover, 50% of journal editors reported that they would not accept a systematic review that contained non-peer reviewed items. Nearly half (43%) of journal editors reported that they would not accept a systematic review that contained unpublished items.

5 Discussion

In our survey of a consecutive sample of 1,229 authors of systematic reviews, 135 non-Cochrane and 20 Cochrane authors responded. In a second survey to elicit attitudes of journal editors to grey literature, 46 journal editors responded.

We observed discrepancies and a lack of clarity about what constitute grey literature items and resources, among both non-Cochrane and Cochrane authors. Even an item we would clearly regard as ‘grey’, such as a dissertation/thesis, was not considered to be a grey literature item by all review authors. Similarly, many authors failed to describe OpenGrey as a grey literature resource. It should be noted that OpenGrey is no longer an actively maintained resource, which may impact its future utility. The discrepant responses suggest that there is a general lack of consensus on how to classify grey literature items and resources. This may be explained by the lack of authors using guidelines to perform the grey literature search. The majority of authors expressed a desire for clear structured guidance on how to perform a grey literature search.

When probed, the majority of authors reported that they performed a grey literature search. Among those authors who did not perform a grey literature search, the most commonly reported reason was the concern about the quality of and/or level of detail reported in a grey literature item. While concerns about the credibility of grey literature persist, peer review is not a universal proxy for quality. Grey literature undergoes varying levels of scrutiny, from rigorous academic evaluation—such as doctoral theses reviewed by examination committees—to internal institutional reports or policy documents that may involve only limited oversight. As such, the internal validity and methodological quality of grey literature should be appraised within the systematic review process itself, rather than being assumed based solely on publication type or venue.

In 2013, a large survey of 2,184 Cochrane authors was performed, focussing on their experience of searching for grey literature.Reference Schroll, Bero and Gotzsche17 Their response rate of 36.9% was similar to our response rate from Cochrane review authors. Of the review authors, 75.3% of respondents reported performing a grey literature search. This corresponds with our own report of Cochrane authors who performed a grey literature search (80%). Our study additionally included non-Cochrane authors, who were excluded from this earlier study, and in fact comprised the large majority of our respondents.

The results of our survey show that there is misunderstanding in the nature of grey literature and the conduct of a grey literature search. This confirms the observations made in previous studies on grey literature.Reference Godin, Stapleton, Kirkpatrik, Hanning and Leatherdale9 – Reference Rothstein, Hopewell, Cooper and Valentine11 Although the most commonly accepted definition of grey literature is as print or electronic information not controlled by commercial or academic publishers,Reference Farace and Schopfel18 there appears to be considerable variation in awareness of what constitutes grey literature among reviewers. This can possibly be explained by fluidity in the meaning of grey literature among information specialists.Reference Farace and Schopfel18

The information specialist appears to be an underused resource in the conduct of non-Cochrane reviews. One of the main roles of information specialists, especially in the context of healthcare research, is to support systematic reviews.Reference Crum and Cooper19 Of the authors in our survey who reported performing a grey literature search, conference proceedings were the items most commonly sought. In a previous study, conference abstracts or unpublished data (trial registers, file drawer data, data from trialists) were the most common grey literature items included in the systematic reviews.Reference Hopewell, McDonald, Clarke and Egger20 Many review authors, including non-Cochrane and Cochrane authors, used PubMed as part of the grey literature search. While some non-journal documents, such as US government agency reports (e.g. Agency for Healthcare Research and Quality [AHRQ]), may be included in PubMed, it is unlikely that authors will encounter many grey literature items within this resource.

In the case of both non-Cochrane and Cochrane reviews, our respondents reported that a considerable amount of time was dedicated to the search strategy. In a previous study, 81 respondents who detailed their search strategy reported a median time of 471 minutes spent searching for studies for their systematic review.Reference Saleh, Ratajeski and Bertolet21 Moreover, a scoping review found that the median time for the search process to identify eligible studies for systematic reviews was between 7.85 and 22 hours. Additionally, the median time for the additional development of a search strategy for systematic reviews was between 1 and 13.5 hours.Reference Nussbaumer-Streit, Ellen and Klerings22 While our survey respondents reported spending much more time on their search strategy, this difference may be explained by the fact that their survey only asked about the amount of time spent searching and ours asked about both the planning and execution of the search strategy. Equally, the scoping review mentions reported outcomes in systematic reviews, whereas this study directly surveyed the authors.

There are several barriers to conducting searches for grey literature. The most, commonly cited barrier in this study was apprehension about the internal validity of the research identified. This corresponds with the previous findings by Saleh et al, which identified the lack of peer review and the quality of the individual studies as the greatest concern of review authors and journal editors.Reference Saleh, Ratajeski and Bertolet21 The rationale for excluding studies that are not peer reviewed is because peer-review is seen as a proxy for research quality.Reference Moradzadeh, Sedghi and Panahi23 Our view is that the authors of a systematic review should be able to conduct an assessment of internal validity at least as effectively as a peer reviewer, such that restriction to peer-reviewed literature should not be necessary. However, we suspect that there are many systematic review teams without such skills. A relevant point is that some items of grey literature are insufficiently detailed to enable full assessment of the study quality. To date, we are unaware of any study that empirically examines the internal validity of studies reported in the grey literature versus non-grey literature.

Half of journal editors in our survey reported that review authors are performing a grey literature search as part of their systematic review. This is similar to the proportion of authors who report having performed a grey literature search. The majority of journal editors did not expect a grey literature to be performed as part of a systematic review. This conflicts with the current recommendation in the Cochrane Handbook, which advocates a grey literature search be performed where eligible and appropriate.Reference Lefebvre, Glanville, Briscoe, Higgins, Thomas, Chandler, Cumpston, Li, Page and Welch1 In a 2006 survey, a total of 93 journal editors responded, and 70% of these felt that grey literature probably or definitely should be considered as part of a systematic review.Reference Tetzlaff, Moher, Pham and Altman24 However, a much older study in 1993 reported that only 47% of journal editors expected grey literature to be included in the systematic review.Reference Cook, Guyatt and Ryan15 The discrepancy may be due to the fact that in 1993, Cochrane was in its infancy and the number of systematic reviews published before 1993 would have been very small in comparison with today.Reference Hoffmann, Allers and Rombey25 A lot of methodological initiatives, by Cochrane and other organisations, have been created to ensure that reviewers minimise reporting bias when conducting a search. It remains unclear whether there is an ongoing discrepancy between current guidance and the expectations of the journal editors in this present study.

Editors are the gatekeepers of their journals.Reference Resnik and Elmore26 Our findings suggest that resistance to the inclusion of non-peer reviewed and unpublished materials may have increased rather than diminished over time. Whereas a study conducted more than 30 years ago reported that approximately 30% of journal editors would not accept a meta-analysis including unpublished studies, our findings indicate that nearly half of editors surveyed would not accept a systematic review containing non-peer reviewed items, and 43% would not accept one containing unpublished studies. This highlights the persistence, and potential strengthening, of editorial barriers to comprehensive evidence synthesis.

A key strength of our study is its cross-sectional nature, with the survey being sent to authors of all systematic reviews published over a 4-week period in October/November 2019. While the overall response rate was low, a considerable number of authors still completed our survey. It highlights some of the misunderstanding of what is grey literature, among review authors. It further examines the apprehensions certain authors possess about including grey literature. The authors seem to express a desire for more structured guidance on the topic. Finally, it highlights the prejudices that editors have against grey literature.

There are limitations to this study. First, the response rate was low, as might be expected for a survey of this nature. It is unclear whether the authors received the emails but did not read them (no delivery notifications or read receipts were requested in the emails sent) or whether the emails were captured as spam. We suspect that authors who failed to respond to our survey were less likely to have performed a grey literature search. Thus, this current survey may not be representative of all authors in the included sample of systematic reviews. Second, while the Luxembourg and Prague definition are widely used to define grey literature,Reference Schöpfel4 , Reference Farace and Schopfel18 there is no universally accepted, working definition of grey literature or moreover a grey literature search.Reference Bonato27 Hence, the survey respondents might not be expected to have the same shared understanding of grey literature. Finally, the survey was conducted in 2020 and, given the rapidly evolving nature of information retrieval methods, results may not reflect current practice.

This study suggests some future areas for research. There is a need for better understanding of the internal validity of studies reported in grey literature. There is also a need to learn more about the ability to assess internal validity based on grey literature items compared with non-grey literature such as journal articles. A structured tool or guidance tailored to each of the various forms of grey literature would also be of value to systematic review authors. In addition to developing new tools, increased awareness of existing resources—such as the Cochrane Handbook’s Technical Supplement—would likely enhance authors’ ability to identify and use grey literature effectively.

In conclusion, the search for (and inclusion of) grey literature is deemed highly desirable by the Cochrane Handbook. However, there appears to be a misunderstanding about the nature of grey literature, both in terms of what constitutes grey literature and where it can be found. While a slight majority of authors in our survey reported seeking grey literature for their review, they often sought it in resources unlikely to include grey literature items. There appears to be a need for clear, structured guidance on grey literature in the context of systematic reviews (what, where, and how to find it), which the respondents to our survey appear to endorse. Existing resources such as the Cochrane Handbook’s Technical Supplement to Chapter 4 provide detailed recommendations on grey literature searching and reporting, and wider awareness and uptake of such resources may help address this gap. We recommend that future systematic reviews explicitly state which types of grey literature were sought (e.g. theses, conference proceedings) and potentially adopt a checklist approach to improve transparency.

Author contributions

J.O’S.: conceptualisation, data curation, formal analysis, writing—original draft; S.D.: conceptualisation, supervision, writing—review and editing; J.P.T.H.: conceptualisation, formal analysis, supervision, writing—review and editing.

Competing interest statement

The authors declare that no competing interests exist.

Data availability statement

Data availability is not applicable to this article as no new data were created or analysed in this study.

Funding statement

J.P.T.H. is a National Institute for Health and Care Research (NIHR) Senior Investigator (NIHR203807). S.D. and J.P.T.H. are supported by the NIHR Evidence Synthesis Groups (NIHR153861, NIHR168894) and by the NIHR Applied Research Collaboration West (ARC West) at University Hospitals Bristol and Weston NHS Foundation Trust (NIHR200181). The views expressed in this article are those of the authors and do not necessarily represent those of the NHS, the NIHR, or the Department of Health and Social Care.

Supplementary material

To view supplementary material for this article, please visit http://doi.org/10.1017/rsm.2026.10087.

References

Lefebvre, C, Glanville, J, Briscoe, S et al. Searching for and selecting studies. In: Higgins, JP, Thomas, J, Chandler, J, Cumpston, M, Li, T, Page, MJ, Welch, VA, eds. Cochrane Handbook for Systematic Reviews of Interventions Version 6.5. Cochrane; 2024.Google Scholar
Adams, J, Hillier-Brown, FC, Moore, HJ et al. Searching and synthesising ’grey literature’ and ’grey information’ in public health: critical reflections on three case studies. Syst Rev. 2016;5(1):164.10.1186/s13643-016-0337-yCrossRefGoogle ScholarPubMed
MacDonald, H, Comer, C, Foster, M et al. Searching for studies: a guide to information retrieval for Campbell systematic reviews. Campbell Syst Rev. 2024;20(3):e1433.10.1002/cl2.1433CrossRefGoogle ScholarPubMed
Schöpfel, J. Towards a Prague definition of grey literature. Paper presented at: Twelfth international conference on grey literature: transparency in grey literature. Grey tech approaches to high tech issues; 6–7 December 2010; Prague, 2010.Google Scholar
Adams, RJ, Smart, P, Huff, AS. Shades of grey: guidelines for working with the grey literature in systematic reviews for management and organizational studies. Int J Manage Rev. 2017;19(4):432–454.10.1111/ijmr.12102CrossRefGoogle Scholar
GreyNet. Document types in grey literature. 2025. Accessed January 28, 2025. https://www.greynet.org/greysourceindex/documenttypes.html.Google Scholar
Cooper, C, Booth, A, Varley-Campbell, J, Britten, N, Garside, R. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol. 2018;18(1):85.10.1186/s12874-018-0545-3CrossRefGoogle ScholarPubMed
Egger, M, Juni, P, Bartlett, C, Holenstein, F, Sterne, J., How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study . Health Technol Assess. 2003;7(1):1–76.10.3310/hta7010CrossRefGoogle ScholarPubMed
Godin, K, Stapleton, J, Kirkpatrik, SI, Hanning, RM, Leatherdale, ST. Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Syst Rev. 2015;4:138.10.1186/s13643-015-0125-0CrossRefGoogle Scholar
Mahood, Q, Van Eerd, D, Irvin, E. Searching for grey literature for systematic reviews: challenges and benefits. Res Synth Methods. 2014;5(3):221–234.10.1002/jrsm.1106CrossRefGoogle ScholarPubMed
Rothstein, HR, Hopewell, S. Grey literature. In: Cooper, LVH, Valentine, JC, eds. The Handbook of Research Synthesis and Meta-Analysis. 2nd ed. Russell Sage Foundation; 2009:103–125.Google Scholar
Lefebvre, C, Glanville, J, Briscoe, S et al. Technical supplement to chapter 4: searching for and selecting studies. In: Higgins, JP, Thomas, J, Chandler, J, Cumpston, M, Li, T, Page, MJ, et al, eds. Cochrane Handbook for Systematic Reviews of Interventions Version 6.5. Cochrane; 2024.Google Scholar
Hartling, L, Featherstone, R, Nuspl, M, Shave, K, Dryden, DM, Vandermeer, B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol. 2017;17(1):64.10.1186/s12874-017-0347-zCrossRefGoogle Scholar
McAuley, L, Pham, B, Tugwell, P, Moher, D. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet. 2000;356(9237):1228–1231.10.1016/S0140-6736(00)02786-0CrossRefGoogle ScholarPubMed
Cook, DJ, Guyatt, GH, Ryan, G et al. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA. 1993;269(21):2749–2753.10.1001/jama.1993.03500210049030CrossRefGoogle ScholarPubMed
O’Sullivan, J, Dawson, S, Cooper, C, Higgins, J. How comprehensive is comprehensive? Prevalence of searches beyond peer-reviewed journal articles and bibliographic sources: a cross-sectional study of systematic reviews. [Manuscript submitted for publication]. 2025.10.1017/rsm.2026.10086CrossRefGoogle Scholar
Schroll, JB, Bero, L, Gotzsche, PC. Searching for unpublished data for Cochrane reviews: cross sectional study. BMJ. 2013;346:f2231.10.1136/bmj.f2231CrossRefGoogle ScholarPubMed
Farace, D, Schopfel, J. Grey Literature in Library and Information Studies. De Gruyter; 2010.10.1515/9783598441493CrossRefGoogle Scholar
Crum, JA, Cooper, ID. Emerging roles for biomedical librarians: a survey of current practice, challenges, and changes. J Med Libr Assoc. 2013;101(4):278–286.10.3163/1536-5050.101.4.009CrossRefGoogle ScholarPubMed
Hopewell, S, McDonald, S, Clarke, M, Egger, M. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev. 2007;(2):MR000010.Google ScholarPubMed
Saleh, AA, Ratajeski, MA, Bertolet, M. Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized. Evid Based Libr Inf Pract. 2014;9(3):28–50.10.18438/B8DW3KCrossRefGoogle ScholarPubMed
Nussbaumer-Streit, B, Ellen, M, Klerings, I et al. Resource use during systematic review production varies widely: a scoping review. J Clin Epidemiol. 2021;139:287–296.10.1016/j.jclinepi.2021.05.019CrossRefGoogle ScholarPubMed
Moradzadeh, M, Sedghi, S, Panahi, S. Towards a new paradigm for ‘journal quality’criteria: a scoping review. Scientometrics. 2023;128(1):279–321.10.1007/s11192-022-04520-5CrossRefGoogle Scholar
Tetzlaff, J, Moher, D, Pham, B, Altman, D. Survey of views on including grey literature in systematic reviews. Paper presented at: 16th Cochrane Colloquium; 2006, Dublin, Ireland.Google Scholar
Hoffmann, F, Allers, K, Rombey, T et al. Nearly 80 systematic reviews were published each day: observational study on trends in epidemiology and reporting over the years 2000-2019. J Clin Epidemiol. 2021;138:1–11.10.1016/j.jclinepi.2021.05.022CrossRefGoogle ScholarPubMed
Resnik, DB, Elmore, SA. Ensuring the quality, fairness, and integrity of journal peer review: a possible role of editors. Sci Eng Ethics. 2016;22(1):169–188.10.1007/s11948-015-9625-5CrossRefGoogle ScholarPubMed
Bonato, S. Searching the Grey Literature: A Handbook for Searching Reports, Working Papers, and Other Unpublished Research. Rowman & Littlefield; 2018.10.5040/9798216436355CrossRefGoogle Scholar
Figure 0

Figure 1 Flow diagram illustrating numbers of authors and editors contacted and completing the survey.

Figure 1

Figure 2 Planning and execution of the literature search: (a) the use of an information specialist for the planning and execution of the search; (b) the full-time equivalents of time taken to plan and undertake the search.

Figure 2

Figure 3 Understanding the meaning of grey literature by reviewers: (a) the items considered to be grey literature; (b) the information resources considered to be sources of grey literature.

Figure 3

Figure 4 Conduct of a grey literature search: (a) performance of a grey literature search; (b) grey literature items sought within the reviews; (c) sources of grey literature searched by the reviewers.

Figure 4

Figure 5 Use of guidance for grey literature search: percentage of authors who (a) sought guidance on their grey literature search; (b) found the written guidance on grey literature searching to be useful, among those who reported using any form of guidance; (c) would find structured guidance for performing a grey literature to be useful.

Figure 5

Table 1 Responses from authors to the question ‘What source of guidance was sought?’

Figure 6

Figure 6 What were the barriers to performing a grey literature search.

Supplementary material: File

O’Sullivan et al. supplementary material

O’Sullivan et al. supplementary material
Download O’Sullivan et al. supplementary material(File)
File 19.3 KB