Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Home
Hostname: page-component-846f6c7c4f-tkmnj Total loading time: 0.301 Render date: 2022-07-07T02:35:30.314Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "useRatesEcommerce": false, "useNewApi": true } hasContentIssue true
  • English
International Journal of Technology Assessment in Health Care International Journal of Technology Assessment in Health Care

Article contents

How much searching is enough? Comprehensive versus optimal retrieval for technology assessments

Published online by Cambridge University Press:  06 October 2010

Andrew Booth
Andrew Booth*
Affiliation:
University of Sheffield
Get access Rights & Permissions[Opens in a new window]

Abstract

Objectives: The aim of this study is to review briefly different methods for determining the optimal retrieval of studies for inclusion in a health technology assessment (HTA) report.

Methods: This study reviews the methodology literature related to specific methods for evaluating yield from literature searching strategies and for deciding whether to continue or desist in the searching process.

Results: Eight different methods were identified. These include using the Capture–recapture technique; obtaining Feedback from the commissioner of the HTA report; seeking the Disconfirming case; undertaking comparison against a known Gold standard; evaluating retrieval of Known items; recognizing the Law of diminishing returns, specifying a priori Stopping rules, and identifying a point of Theoretical saturation.

Conclusions: While this study identified a variety of possible methods, there has been very little formal evaluation of the specific strengths and weaknesses of the different techniques. The author proposes an evaluation agenda drawing on an examination of existing data together with exploration of the specific impact of missing relevant studies.

Keywords

DatabasesBibliographicTechnology assessmentBiomedicalInformation retrievalSelection biasSensitivity and specificity
Type
THEME SECTION: INFORMATION RETRIEVAL FOR HTA
Information
International Journal of Technology Assessment in Health Care , Volume 26 , Issue 4 , October 2010 , pp. 431 - 435
Copyright
Copyright © Cambridge University Press 2010

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

1. Allen, IE, Olkin, I. Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA. 1999;282:634–635.CrossRefGoogle ScholarPubMed
2. Bayliss, SE, Dretzke, J. Health technology assessment in social care: A case study of randomized controlled trial retrieval. Int J Technol Assess Health Care. 2006;22:39–46.CrossRefGoogle Scholar
3. Bennett, DA, Latham, NK, Stretton, C, et al. Capture-recapture is a potentially useful method for assessing publication bias. J Clin Epidemiol. 2004;57:349–357.CrossRefGoogle Scholar
4. Booth, A. The number needed to retrieve: A practically useful measure of information retrieval? Health Info Libr J. 2006;23:229–232.CrossRefGoogle Scholar
5. Chilcott, J, Brennan, A, Booth, A, et al. The role of modelling in prioritising and planning clinical trials. Health Technol Assess. 2003;7:iii, 1–125.CrossRefGoogle ScholarPubMed
6. Dickersin, K, Scherer, R, Lefebvre, C. Identifying relevant studies for systematic reviews. BMJ. 1994;309:1286–1291.CrossRefGoogle ScholarPubMed
7. Egger, M, Juni, P, Bartlett, C, et al. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;7:1–76.Google ScholarPubMed
8. Grant, MJ, Booth, A. A typology of reviews: An analysis of 14 review types and associated methodologies. Health Info Libr J. 2009;26:91–108.CrossRefGoogle ScholarPubMed
9. Greenhalgh, T, Peacock, R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: Audit of primary sources. BMJ. 2005;331:1064–1065.CrossRefGoogle Scholar
10. Ioannidis, J. Why most published research findings are false. PLoS Med. 2005;2:e124.CrossRefGoogle ScholarPubMed
11. Kastner, M, Straus, S, Goldsmith, CH. Estimating the Horizon of articles to decide when to stop searching in systematic reviews: An example using a systematic review of RCTs evaluating osteoporosis clinical decision support tools. AMIA Annu Symp Proc. 2007;389–393.Google Scholar
12. Kastner, M, Straus, S, McKibbon, K, et al. The capture-mark-recapture technique can be used as a stopping rule when searching in systematic reviews. J Clin Epidemiol. 2008;62:149–157.CrossRefGoogle ScholarPubMed
13. Khan, KS, Coomarasamy, A. A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Med Educ. 2006;6:59.CrossRefGoogle ScholarPubMed
14. Löhönen, J, Isohanni, M, Nieminen, P, et al. Coverage of the bibliographic databases in mental health research. Nord J Psychiatry. 2010;64:181–188.CrossRefGoogle ScholarPubMed
15. Noyes, J, Popay, J, Pearson, A, et al. Qualitative research and Cochrane reviews. In: Higgins, JPT, S, Green, eds. Cochrane handbook for systematic reviews of interventions. Version 5.0.1 [updated September 2008]. The Cochrane Collaboration, 2008. www.cochrane-handbook.org.Google Scholar
16. Ogilvie, D, Hamilton, V, Egan, M, et al. Systematic reviews of health effects of social interventions: 1. Finding the evidence: How far should you go? J Epidemiol Community Health. 2005;59:804–808.CrossRefGoogle Scholar
17. Pirolli, P, Card, S. Information foraging in information access environments. In: Proceedings of the Human Factors in Computing Systems, CHI '95. Association for Computing Machinery, 1995.Google Scholar
18. Rosenthal, R. The file drawer problem and tolerance for null results. Psychol Bull. 1979;86:638–641.CrossRefGoogle Scholar
19. Rotstein, D, Laupacis, A. Differences between systematic reviews and health technology assessments: A trade-off between the ideals of scientific rigor and the realities of policy making. Int J Technol Assess Health Care. 2004;20:177–183.CrossRefGoogle ScholarPubMed
20. Royle, P, Milne, R. Literature searching for randomized controlled trials used in Cochrane reviews: Rapid versus exhaustive searches. Int J Technol Assess Health Care. 2003;19:591–603.CrossRefGoogle ScholarPubMed
21. Sampson, M, Shojania, KG, Garritty, C, et al. Systematic reviews can be produced and published faster. J Clin Epidemiol. 2008;61:531–536.CrossRefGoogle Scholar
22. Shojania, KG, Sampson, M, Ansari, MT, et al. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007;147:224–233.CrossRefGoogle Scholar
23. Spoor, P, Airey, M, Bennett, C, et al. Use of the capture-recapture technique to evaluate the completeness of systematic literature searches. BMJ. 1996;313:342–343.CrossRefGoogle ScholarPubMed
24. Stevinson, C, Lawlor, D. Searching multiple databases for systematic reviews: Added value or diminishing returns? Complement Ther Med. 2004;12:228–232.CrossRefGoogle ScholarPubMed
25. Watt, A, Cameron, A, Sturm, L, et al. Rapid reviews versus full systematic reviews: An inventory of current methods and practice in health technology assessment. Int J Technol Assess Health Care. 2008;24:133–139.CrossRefGoogle ScholarPubMed
26. Watt, A, Cameron, A, Sturm, L, et al. Rapid versus full systematic reviews: Validity in clinical practice? ANZ J Surg. 2008;78:1037–1040.CrossRefGoogle ScholarPubMed
27. Weale, AR, Lear, PA. Randomised controlled trials and quality of journals. Lancet. 2003;361:1749–1750.CrossRefGoogle ScholarPubMed
53
Cited by

Save article to Kindle

To save this article to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

How much searching is enough? Comprehensive versus optimal retrieval for technology assessments
Available formats
×

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

How much searching is enough? Comprehensive versus optimal retrieval for technology assessments
Available formats
×

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

How much searching is enough? Comprehensive versus optimal retrieval for technology assessments
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *