Skip to main content Accessibility help


  • Mick Arber (a1), Julie Glanville (a2), Jaana Isojarvi (a2), Erin Baragula (a2), Mary Edwards (a2), Alison Shaw (a2) and Hannah Wood (a2)...

This study investigated which databases and which combinations of databases should be used to identify economic evaluations (EEs) to inform systematic reviews. It also investigated the characteristics of studies not identified in database searches and evaluated the success of MEDLINE search strategies used within typical reviews in retrieving EEs in MEDLINE.


A quasi-gold standard (QGS) set of EEs was collected from reviews of EEs. The number of QGS records found in nine databases was calculated and the most efficient combination of databases was determined. The number and characteristics of QGS records not retrieved from the databases were collected. Reproducible MEDLINE strategies from the reviews were rerun to calculate the sensitivity and precision for each strategy in finding QGS records.


The QGS comprised 351 records. Across all databases, 337/351 (96 percent) QGS records were identified. Embase yielded the most records (314; 89 percent). Four databases were needed to retrieve all 337 references: Embase + Health Technology Assessment database + (MEDLINE or PubMed) + Scopus. Four percent (14/351) of records could not be found in any database. Twenty-nine of forty-one (71 percent) reviews reported a reproducible MEDLINE strategy. Ten of twenty-nine (34.5 percent) of the strategies missed at least one QGS record in MEDLINE. Across all twenty-nine MEDLINE searches, 25/143 records were missed (17.5 percent). Mean sensitivity was 89 percent and mean precision was 1.6 percent.


Searching beyond key databases for published EEs may be inefficient, providing the search strategies in those key databases are adequately sensitive. Additional search approaches should be used to identify unpublished evidence (grey literature).

Hide All

This research received no specific grant from any funding agency or other body. YHEC is a consultancy company conducting systematic reviews of the effectiveness and cost-effectiveness of interventions and provides search services and training in search conduct.

Hide All
1.York York Health Economics Consortium. Economic evaluation [online]. 2016. (accessed September 30, 2018).
2.Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies: Canada. 4th ed. 2017. (accessed September 30, 2018).
3.National Institute for Health and Care Excellence. Methods for the development of NICE public health guidance. Process and methods [PMG4]. 3rd ed. 2012. (accessed September 30, 2018).
4.National Institute for Health and Care Excellence. Developing NICE guidelines: The manual. Process and methods [PMG20]. 2014. (accessed September 30, 2018).
5.National Institute for Health and Care Excellence. Guide to the processes of technology appraisal. 2018. (accessed September 30, 2018).
6.Scottish Medicines Consortium. How we make our decisions. 2018. (accessed September 30, 2018).
7.Borah, R, Brown, AW, Capers, PL, Kaiser, KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7:e012545.
8.Alton, V, Eckerlund, I, Norlund, A. Health economic evaluations: How to find them. Int J Technol Assess Health Care. 2006;22:512-517.
9.Pitt, C, Goodman, C, Hanson, K. Economic evaluation in global perspective: A bibliometric analysis of the recent literature. Health Econ. 2016;25(Suppl 1):9-28.
10.Royle, P, Waugh, N. Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technol Assess. 2003;7(34):iii, ix-x, 1-51.
11.Sassi, F, Archard, L, McDaid, D. Searching literature databases for health care economic evaluations: How systematic can we afford to be? Med Care. 2002;40:387-394.
12.Centre for Reviews and Dissemination. Undertaking the review. In: Centre for Reviews and Dissemination, editor. Systematic reviews: CRD's guidance for undertaking reviews in health care. York. University of York. 2009.!SSL!/WebHelp/1_3_UNDERTAKING_THE_REVIEW.htm (accessed September 30, 2018).
13.Shemilt, I, Mugford, M, Byford, S, et al. Incorporating economics evidence. In: Higgins, JPT, Green, S, eds. Cochrane handbook for systematic reviews of interventions version 5.1.0. The Cochrane Collaboration. 2011. 449-80 (accessed September 30, 2018).
14.Wood, H, Arber, M, Glanville, JM. Systematic reviews of economic evaluations: How extensive are their searches? Int J Technol Assess Health Care. 2017;33:25-31.
15.Briscoe, S, Cooper, C, Glanville, J, Lefebvre, C. The loss of the NHS EED and DARE databases and the effect on evidence synthesis and evaluation. Res Synth Methods. 2017;8:256-257.
16.Thielen, FW, Van Mastrigt, G, Burgers, LT, et al. How to prepare a systematic review of economic evaluations for clinical practice guidelines: Database selection and search strategy development (part 2/3). Expert Rev Pharmacoecon Outcomes Res. 2016;16:705-721.
17.Kaunelis, D, Glanville, J. Costs and economic evaluation. Summarized research in information retrieval for HTA. 2018. (accessed September 30, 2018).
18.Kaunelis, D, Farrah, K. Appraisal of: Pitt, C, Goodman, C, Hanson, K. Economic evaluation in global perspective: A bibliometric analysis of the recent literature. Health Econ. 2016;25(Suppl 1):9-28.
19.Sampson, M, Zhang, L, Morrison, A, et al. An alternative to the hand searching gold standard: Validating methodological search filters using relative recall. BMC Med Res Methodol. 2006;6:33.
20.Aagaard, T, Lund, H, Juhl, C. Optimizing literature search in systematic reviews - Are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders? BMC Med Res Methodol. 2016;16:161.
21.Ayiku, L, Levay, P, Hudson, T, et al. The medline UK filter: Development and validation of a geographic search filter to retrieve research about the UK from OVID medline. Health Info Libr J. 2017. doi:10.1111/hir.12187.
22.Selva, A, Sola, I, Zhang, Y, et al. Development and use of a content search strategy for retrieving studies on patients’ views and preferences. Health Qual Life Outcomes. 2017;15:126.
23.National Institute for Health and Care Excellence. Guide to the methods of technology appraisal [PMG9]. London: National Institute for Health and Care Excellence; 2013. (accessed September 30, 2018).
24.London School of Hygiene & Tropical Medicine, Public Health England, University College London. Evidence review & economic analysis of excess winter deaths for the National Institute for Health and Care Excellence (NICE). Review 2: Interventions and economic studies. Undated. (accessed September 30, 2018).
25.Lam, MT, De Longhi, C, Turnbull, J, Lam, HR, Besa, R. Has Embase replaced MEDLINE since coverage expansion? J Med Libr Assoc. 2018;106:227-234.
26.Lefebvre, C, Manheimer, E, Glanville, JM. Chapter 6: Searching for studies. In: Higgins, JPT, Green, S, eds. Cochrane handbook for systematic reviews of interventions [Version 5.1.0] [updated March 2011]. The Cochrane Collaboration. 2011.
27.Duffy, S, de Kock, S, Misso, K, et al. Supplementary searches of PubMed to improve currency of MEDLINE and MEDLINE In-Process searches via Ovid. J Med Libr Assoc. 2016;104:309-312.
28.University of York, Centre for Reviews and Dissemination. Search strategies: NHS EED. (accessed September 30, 2018).
29.McGowan, J, Sampson, M, Salzwedel, DM, et al. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40-6.
30.Meert, D, Torabi, N, Costella, J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. J Med Libr Assoc. 2016;104:267-277.
31.Koffel, JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: A cross-sectional survey of recent authors. PLoS One. 2015;10:e0125931.
32.Rethlefsen, ML, Farrell, AM, Osterhaus Trzasko, LC, Brigham, TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol. 2015;68:617-626.
33.Effective Practice and Organisation of Care (EPOC) Cochrane Group. A collection of databases, web sites and journals relevant to Low-and Middle-Income Countries (LMICs). 2013. (accessed September 30, 2018).
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

International Journal of Technology Assessment in Health Care
  • ISSN: 0266-4623
  • EISSN: 1471-6348
  • URL: /core/journals/international-journal-of-technology-assessment-in-health-care
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed