Skip to main content Accessibility help

How much searching is enough? Comprehensive versus optimal retrieval for technology assessments

  • Andrew Booth (a1)


Objectives: The aim of this study is to review briefly different methods for determining the optimal retrieval of studies for inclusion in a health technology assessment (HTA) report.

Methods: This study reviews the methodology literature related to specific methods for evaluating yield from literature searching strategies and for deciding whether to continue or desist in the searching process.

Results: Eight different methods were identified. These include using the Capture–recapture technique; obtaining Feedback from the commissioner of the HTA report; seeking the Disconfirming case; undertaking comparison against a known Gold standard; evaluating retrieval of Known items; recognizing the Law of diminishing returns, specifying a priori Stopping rules, and identifying a point of Theoretical saturation.

Conclusions: While this study identified a variety of possible methods, there has been very little formal evaluation of the specific strengths and weaknesses of the different techniques. The author proposes an evaluation agenda drawing on an examination of existing data together with exploration of the specific impact of missing relevant studies.



Hide All
1. Allen, IE, Olkin, I. Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA. 1999;282:634635.
2. Bayliss, SE, Dretzke, J. Health technology assessment in social care: A case study of randomized controlled trial retrieval. Int J Technol Assess Health Care. 2006;22:3946.
3. Bennett, DA, Latham, NK, Stretton, C, et al. Capture-recapture is a potentially useful method for assessing publication bias. J Clin Epidemiol. 2004;57:349357.
4. Booth, A. The number needed to retrieve: A practically useful measure of information retrieval? Health Info Libr J. 2006;23:229232.
5. Chilcott, J, Brennan, A, Booth, A, et al. The role of modelling in prioritising and planning clinical trials. Health Technol Assess. 2003;7:iii, 1125.
6. Dickersin, K, Scherer, R, Lefebvre, C. Identifying relevant studies for systematic reviews. BMJ. 1994;309:12861291.
7. Egger, M, Juni, P, Bartlett, C, et al. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;7:176.
8. Grant, MJ, Booth, A. A typology of reviews: An analysis of 14 review types and associated methodologies. Health Info Libr J. 2009;26:91108.
9. Greenhalgh, T, Peacock, R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: Audit of primary sources. BMJ. 2005;331:10641065.
10. Ioannidis, J. Why most published research findings are false. PLoS Med. 2005;2:e124.
11. Kastner, M, Straus, S, Goldsmith, CH. Estimating the Horizon of articles to decide when to stop searching in systematic reviews: An example using a systematic review of RCTs evaluating osteoporosis clinical decision support tools. AMIA Annu Symp Proc. 2007;389–393.
12. Kastner, M, Straus, S, McKibbon, K, et al. The capture-mark-recapture technique can be used as a stopping rule when searching in systematic reviews. J Clin Epidemiol. 2008;62:149157.
13. Khan, KS, Coomarasamy, A. A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Med Educ. 2006;6:59.
14. Löhönen, J, Isohanni, M, Nieminen, P, et al. Coverage of the bibliographic databases in mental health research. Nord J Psychiatry. 2010;64:181188.
15. Noyes, J, Popay, J, Pearson, A, et al. Qualitative research and Cochrane reviews. In: Higgins, JPT, S, Green, eds. Cochrane handbook for systematic reviews of interventions. Version 5.0.1 [updated September 2008]. The Cochrane Collaboration, 2008.
16. Ogilvie, D, Hamilton, V, Egan, M, et al. Systematic reviews of health effects of social interventions: 1. Finding the evidence: How far should you go? J Epidemiol Community Health. 2005;59:804808.
17. Pirolli, P, Card, S. Information foraging in information access environments. In: Proceedings of the Human Factors in Computing Systems, CHI '95. Association for Computing Machinery, 1995.
18. Rosenthal, R. The file drawer problem and tolerance for null results. Psychol Bull. 1979;86:638641.
19. Rotstein, D, Laupacis, A. Differences between systematic reviews and health technology assessments: A trade-off between the ideals of scientific rigor and the realities of policy making. Int J Technol Assess Health Care. 2004;20:177183.
20. Royle, P, Milne, R. Literature searching for randomized controlled trials used in Cochrane reviews: Rapid versus exhaustive searches. Int J Technol Assess Health Care. 2003;19:591603.
21. Sampson, M, Shojania, KG, Garritty, C, et al. Systematic reviews can be produced and published faster. J Clin Epidemiol. 2008;61:531536.
22. Shojania, KG, Sampson, M, Ansari, MT, et al. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007;147:224233.
23. Spoor, P, Airey, M, Bennett, C, et al. Use of the capture-recapture technique to evaluate the completeness of systematic literature searches. BMJ. 1996;313:342343.
24. Stevinson, C, Lawlor, D. Searching multiple databases for systematic reviews: Added value or diminishing returns? Complement Ther Med. 2004;12:228232.
25. Watt, A, Cameron, A, Sturm, L, et al. Rapid reviews versus full systematic reviews: An inventory of current methods and practice in health technology assessment. Int J Technol Assess Health Care. 2008;24:133139.
26. Watt, A, Cameron, A, Sturm, L, et al. Rapid versus full systematic reviews: Validity in clinical practice? ANZ J Surg. 2008;78:10371040.
27. Weale, AR, Lear, PA. Randomised controlled trials and quality of journals. Lancet. 2003;361:17491750.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

International Journal of Technology Assessment in Health Care
  • ISSN: 0266-4623
  • EISSN: 1471-6348
  • URL: /core/journals/international-journal-of-technology-assessment-in-health-care
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed