Skip to main content Accessibility help

Evaluation criteria to assess the value of identification sources for horizon scanning

  • Joanna Smith (a1), Alison Cook (a1) and Claire Packer (a1)


Objectives: The English National Horizon Scanning Centre routinely scans thirty-five sources to identify new and emerging health technologies. The aim of the study was to develop and apply evaluation criteria and scores to assess the value of sources, and to identify a cutoff score below which sources would be recommended for removal from routine horizon scanning. Criteria to evaluate each source scanned could result in a more efficient approach in the selection process.

Methods: Evaluation criteria were developed following a review of the literature and discussions with horizon analysts. Proposed criteria were piloted on a random selection of six sources, and then applied to all thirty-five sources. The criteria were assessed using the Analytic Hierarchy Process.

Results: Eight criteria were identified as being most relevant for assessing the value of scanning sources. The three most important (primary) criteria were coverage (approximate percentage of relevant information), quality (reliable, accurate, objective), and efficiency (estimated time to identify one potentially significant health technology or other relevant information). Seven sources fell beneath the cutoff score and were recommended for removal from routine scanning.

Conclusions: The criteria were considered useful in the assessment of current sources, and have the potential to be used to assess new ones. These criteria may be useful for other horizon scanning centers to pilot and validate.



Hide All
1. Douw, K, Vondeling, H. Selecting new technologies for evaluation: Can clinical experts predict which new anticancer drugs will impact Danish health care. Soc Sci Med. 2007;64:283286.
2. Douw, K, Vondeling, H, Eskildsen, D, et al. Use of the Internet in scanning the horizon for new and emerging health technologies: A survey of agencies involved in horizon scanning. J Med Internet Res. 2003;5:e6.
3. Formulary. Advanstar Communications. (accessed May 6, 2009).
4. Packer, C. The National Horizon Scanning Centre (NHSC): Early warning for new and emerging health technologies in England. Evid Based Healthc Public Health. 2005;9:410413.
5. Perras, C, Topfer, LA. A review of sources of new drug information used for horizon scanning. The Canadian Agency for Drugs and Technologies in Health (CADTH). Book of Abstracts, IV Annual Meeting Health Technology Assessment International (HTAI), Barcelona; 2007:92, S22–01.
7. Robert, G, Gabbay, J, Stevens, A. Which are the best information sources for identifying emerging health care technologies? An international Delphi survey. Int J Technol Assess Health Care. 1998;14:636643.
8. Saaty, TL. The analytic hierarchy process. New York: McGraw Hill; 1980.
9. Stevens, A, Milne, R, Lilford, R, et al. Keeping pace with new technologies: Systems needed to identify and evaluate them. Br Med J. 1999;319:1291.
10. Thygesen, M, Vondeling, H, Douw, K, et al. Development of an Internet search strategy to identify new medical technologies in prostate cancer. Ugeskr Laeger. 2007;169:19121916.
11. Wild, C, Langer, T. Emerging health technologies: Informing and supporting health policy early. Health Policy. 2008;87:160171.



Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed