Hostname: page-component-77c89778f8-n9wrp Total loading time: 0 Render date: 2024-07-21T05:13:40.491Z Has data issue: false hasContentIssue false


Published online by Cambridge University Press:  02 December 2013

Gail MacKean
Health Technology Assessment Unit
Tom Noseworthy
Health Technology Assessment Unit
Adam G. Elshaug
School of Population Health, Department of Health Care Policy
Laura Leggett
Health Technology Assessment Unit
Peter Littlejohns
Division of Health and Social Care Research
Joan Berezanski
Clinical Advisory and Research Branch
Fiona Clement
Health Technology Assessment Unit
Rights & Permissions [Opens in a new window]


Background: Health technology reassessment (HTR) is “a structured, evidence-based assessment of the clinical, social, ethical, and economic effects of a technology currently used in the healthcare system, to inform optimal use of that technology in comparison to its alternatives.” The purpose of this study is to describe the key themes in the context of current HTR activities and propose a way forward for this newly emerging field.

Methods: Data were gathered from a workshop held as part of the 2012 Canadian Agency for Drugs and Technology in Health (CADTH) symposium. The workshop consisted of two panel presentations followed by discussion; data gathered, including presentations and rich audience discussion transcripts, were analyzed for key themes emerging in the field of HTR using constant comparative analysis.

Results: The language chosen to describe HTR will set the tone for engagement. The identification of champions at multiple levels and political will are essential. Key lessons from international experience are: disinvestment is difficult, focus on clinical areas not specific technologies, identify clear goals of the HTR agenda. Six key themes were identified to move the HTR agenda forward: emphasize integration over segregation, focus on development of HTR methods and processes, processes are context-specific but lessons must be shared, build capacity in synergistic interdisciplinary fields, develop meaningful stakeholder engagement, strengthen postimplementation monitoring and evaluation.

Conclusions: To move this field forward, we must continue to build on international experiences with a focus on developing novel methodological approaches to generating, incorporating, and implementing evidence into policy and practice.

Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NCCreative Common License - SA
The online version of this article is published within an Open Access environment subject to the conditions of the Creative Commons Attribution-NonCommercial-ShareAlike licence . The written permission of Cambridge University Press must be obtained for commercial re-use.
Copyright © Cambridge University Press 2013

Technologies, broadly defined as drugs, diagnostic tests, including indicators and reagents, devices, equipment and supplies, medical and surgical procedures, support systems, and organizational and managerial systems used across the spectrum of health care, are a major source of expenditure within the healthcare system (Reference Joshi, Stahnisch and Noseworthy1). For decades, the healthcare system has focused on managing the entry of technologies, including the requirements for licensure mandated by Health Canada for drugs and devices. For specific technologies, including new drugs which must be assessed in Canada by the Common Drug Review, a demonstration of clinical value and cost-effectiveness is also required (Reference Tierney and Manns2). However, once a technology enters the system, there is no standardized process for monitoring its use, nor managing its exit if it is superseded by advances in knowledge (Reference Joshi, Stahnisch and Noseworthy1;Reference Elshaug, Wat, Moss and Hiller3).

The proposed working definition for health technology reassessment (HTR) is: “a structured, evidence-based assessment of the clinical, social, ethical, and economic effects of a technology currently used in the healthcare system, to inform optimal use of that technology in comparison to its alternatives.” (Reference Noseworthy and Clement4) (Box 1). The language chosen to describe the assessment of technologies currently in use in the healthcare system can be influential in setting the tone for engagement. For example, language such as “disinvestment” and “reinvestment” focus attention on the cost component of technology, and the potential inference of a foregone conclusion to the process, rather than the actual goal of HTR; optimizing the use of technologies in health care and achieving the greatest clinical benefit with our social dollars. The term HTR is proposed as it is value neutral and does not pre-suppose the outcome of the process. Although HTR has perhaps traditionally been thought of as managing the exit of particular technologies from the system, a complete removal of a technology is unlikely to be warranted in most circumstances; experience to date suggests that a change in scope of use is far more common (Reference Docherty, Choudury and Littlejohns5;Reference Elshaug, Moss, Hiller and Maddern6).

While intimately linked to both health technology assessment (HTA) and disinvestment, HTR is a distinct concept. While HTA application has been mainly limited to managing the entrance of “new things” into the healthcare system, HTR is a process to support on-going evidence informed policy regarding the optimal use of technology throughout the lifecycle of technology. Technologies undergoing HTR may or may not have undergone HTA as part of their adoption decision. However, all technologies selected for HTR are actively being used within the healthcare system with a requirement to manage their scope of use. Disinvestment is one potential outcome of HTR; other outcomes include no change in use, narrowing the scope of use (including the potential to increase use of a comparator technology in place), increasing scope of use and in rare circumstances, stopping use all together. In the past, HTR has been an implicit part of HTA as often usage of a competing technology will be affected by the adoption of a new technology. However, there is now a desire to see HTR become standard practice with a goal of optimizing the use of technologies in health care and achieving the greatest clinical benefit with our social dollars (Reference Joshi, Stahnisch and Noseworthy1;Reference Elshaug, Wat, Moss and Hiller3;79).

A recent systematic review of the literature on HTR identified eight countries with some evidence of past or current work related to HTR (Reference Leggett, Noseworthy and Zarrabi10). Many organizations realize the potential benefits of HTR and are interested in integrating it into their healthcare system. However, a lack of top-down support, push back from clinicians, financial and human resource limitations, and a lack of expertise in health technology reassessment are inhibiting program development (Reference Elshaug, Hiller, Tunis and Moss8;Reference Elshaug, Hiller and Moss11Reference Watt, Hiller and Braunack-Mayer14). Several critical factors for success were also identified, including the importance of early and ongoing stakeholder engagement, the identification of champions at multiple levels, embedding HTR into existing structures if and where appropriate, and developing HTR models in context that are adaptable (Reference Elshaug, Hiller, Tunis and Moss8;Reference Garner and Littlejohns12;Reference Watt, Hiller and Braunack-Mayer14;Reference Elshaug, Moss and Littlejohns15).

Given the mounting interest in continued assessment beyond the adoption decision, the 2012 Canadian Agency for Drugs and Technology in Health (CADTH) symposium held in Ottawa, Canada in April 2012 featured a preconference workshop entitled Health Technology Reassessment (HTR): Promoting Value and Evidence-Based Practice. The purpose of this study is to describe the key themes in the context of current HTR activities and propose a way forward for this newly emerging field.


The workshop was hosted by the Health Technology Assessment Unit at the University Calgary, who are actively engaged with the provincial Ministry of Health to develop a HTR model for Alberta, Canada. The workshop was open to all attendees of the CADTH symposium. Attendees came from a variety of contexts including representation from provincial, national, and international HTA agencies, ministries of health, not-for-profit health agencies, academia, and industry. The workshop began with a brief presentation to establish a common understanding of HTR and its development as a field. Two panel presentations followed. The first panel focused on methodological challenges and consisted of a health economist, an expert in implementation science and an international expert in HTR. The second panel focused on current practices with representation from three jurisdictions (Alberta, Australia, and United Kingdom) each at different stages with their HTR programs. During the panel sessions, each panelist was given 5 minutes to provide an introduction followed by audience discussion. The workshop was recorded and transcribed. All data gathered, including presentations and rich discussion transcripts, were analyzed for key themes emerging in the field of HTR using constant comparative analysis.

Table 1. Definitions

Table 2. Summary of Themes


Themes from International Experiences

The United Kingdom and Australia report the most experience in HTR. At the national (Medicare) level in Australia, HTR has been integrated under the responsibility of the Department of Health and Ageing, including the Medical Services Advisory Committee (MSAC), which makes recommendations to the Health Minister about what medical services offer sufficient safety and (cost)effectiveness to warrant public subsidy. Since 2009, a Quality Framework for Australia's Medicare Benefits Schedule was developed under the auspices of the MSAC (Reference Elshaug, Watt, Mundy and Willis16). As part of this initiative, the Health Minister commissioned a review of HTA in Australia, with one outcome being the development of a postmarket surveillance system that included a parallel approach to HTR. This parallel approach involved a review of individual items (Reference Merlin, Street and Holton17), which dovetailed with existing HTA processes, as well as a whole-of-specialty review (Reference Merlin, Street and Holton17). Both proved to be novel, effective approaches to engaging clinicians from the beginning of the HTR process.

For the review of individual items, a proposed scanning process was adapted to identify appropriate candidates for HTR (Reference Elshaug, Moss and Littlejohns15). Using this process, 156 possible candidates were identified a nd provided to government for consideration, from which they identified fifteen for initial rapid review and potential full HTR (Reference Elshaug, Watt, Mundy and Willis16). Outcomes from review include: an amendment to the item description such that it better captures the patient group/s most likely to benefit from any procedure, for example; an increase, decrease or maintenance of the fee; or a complete stop to public funding of the item. One example of an individual technology that underwent a complete review was vertebroplasty for osteoporotic vertebral fractures. The recommendation from MSAC was that Government stop public funding of this procedure because of a lack of evidence of effectiveness. Some clinicians noted their dissent to MSAC's decision yet concomitantly, multiple reports from around the world have been developed supporting MSAC's position (Reference Elshaug and Garber18).

In ophthalmology, a whole-of-specialty review was conducted. The initial impetus for the review was a government decision to cut the fee for cataract surgery, which resulted in dissent from ophthalmologists. The government's observation was that advances in cataract surgery technology delivered on its promise: procedures could be done far more quickly and safely then when it was first introduced (and priced), so a decrease in fee seemed the appropriate market response. The ensuing whole-of-specialty (ophthalmology) review of the Medical Benefits Schedule was approached collaboratively with ophthalmology stakeholders, asking the field what in the schedule they thought required some modification. Of the sixty-one item descriptors on the schedule only twenty remained unchanged through the process. Items were either clarified, modified (e.g., refinement of indications and/or eligible patient groups), split, merged, or entirely removed (Reference Merlin, Street and Holton17). In all respects, this was considered a success with good outcomes for all including patients. The whole-of-specialty approach is now rolling out more broadly.

In the United Kingdom, the National Health Service (NHS), through the National Institute for Health and Clinical Excellence (NICE), has launched a HTR program. NICE has developed two streams; the “do-not-do” recommendations and the Cochrane Quality and Productivity Topics. Both are passive HTR activities resulting in guidance that clinicians may choose to follow. The “do-not-do” recommendations is a searchable database of recommendations made since 2007. The recommendations are identified during the process of guidance development where NICE's independent advisory bodies often identify NHS clinical practices that they recommend should be discontinued completely or should not be used routinely. Each record contains the “do not do” recommendation and includes additional information including the intervention, health topic, the guidance it comes from (with a link to the relevant paragraph in the guidance), and the other “do not do” recommendations from the same guidance. Each recommendation also includes the healthcare setting that describes the main clinical environments in which the intervention or investigation may be initiated.

In addition to the “do-not-do” recommendations, NICE has also developed the Cochrane Quality and Productivity Topics. These are based on Cochrane reviews with the goal of helping the NHS identify practices that could be reduced or stopped freeing up cash and resources that could reinvested in more effective practices. NICE's experience over the past 12 years, however, is that although reassessment and potential disinvestment has the potential to increase efficiency and quality, actual cost savings, and therefore the potential for reinvestment has been less than anticipated. In the year 2011–2012, half of the efficiency savings came from the freezing of health professionals’ salaries, not disinvestment in technologies (Reference O'Dowd19).

An analysis of the shared international experiences resulted in three common themes emerging:

Disinvestment is difficult

Even in the presence of very strong evidence that a particular technology is harmful and/or (cost)ineffective, withdrawing the technology from the healthcare system is difficult. Players, both within and outside the healthcare system, must be prepared for political and professional fall-out. Strong, unwavering political will is critical.

Focus on clinical areas in addition to specific technologies

It is difficult to tease out the appropriate scope of use of a particular technology from the broader pathway of care. Very little done in health care is done in isolation, but rather procedures and devices are used in combination with other treatments within a pathway of care. A change in scope of use of a technology is likely to have ripple effects, resulting in unintended consequences, such as an increase in use of an alternative treatment. A clear understanding of the care pathway within which the technology is embedded, along with social and ethical implications, is critical. The unintended consequences may be mitigated by focusing on clinical areas where multiple comparators can be assessed side-by-side in any care treatment algorithm. In addition, this approach provides a natural way for engaging clinicians as one could begin by approaching a particular department or specialty. Clinical specialty support of any disinvestment list was described as critical to moving forward.

Clear goals of the HTR agenda are required to prioritize candidate technologies

If the focus is actual cost savings, then it is likely that much could be achieved by focusing on redesigning service delivery mechanisms, and optimizing the human resources within the healthcare system. However, if the goal is to optimize the use of technology, which may include cost-savings, focus must move to technologies which are misused in the system.

Moving Forward with Health Technology Reassessment: A Call to Action

Analyzing the audience discussion in the context of current literature and expert presentations, several key themes emerged to move the field of HTR forward.

Emphasize integration over segregation

Integrating HTR with on-going work such as the development of clinical practice guidelines and care pathways, as well as quality improvement initiatives will likely result in the greatest uptake of HTR. Advocating for the integration of economic considerations into guideline and care pathway development will support value for money as one criterion to inform decisions about what should and should not be done.

Focus on development of HTR methods and processes

The need for methodology to measure costs, benefits, and values is dire. These methods are particularly necessary for non-drug technologies. Methodologies must maximize usefulness for decision makers, and incorporate outcomes of importance to both clinicians and patients. In addition, they must represent feasible tools to use within a policy/decision-making context; they must be relevant, timely, and interpretable.

Processes are often context-specific

Although there is much to be learned from processes developed in various jurisdictions, processes ought to be context-specific. Learnings from other jurisdictions must be tempered by contextual and system realities, and processes are likely to require local adaptation. This might include retro-fitting new HTR processes to existing policy structures. Key stakeholders must be engaged in this work. This will ensure that processes developed are supported by, and meet the needs of, the system using the outcomes to inform their decisions.

Build capacity in synergistic interdisciplinary fields

Highly skilled people are required in a variety of disciplines including: economics, implementation science, KT, health services research, policy analysis, change management, evaluation, and stakeholder engagement. It will take innovative teams that are interdisciplinary and cross institutional in nature to take on the HTR challenge.

Meaningful stakeholder engagement

Meaningful stakeholder engagement is crucial. Essential elements to engage stakeholders are: transparent processes for identifying and prioritizing potential HTR candidates, involvement throughout the process of evidence generation and review, and authentic meaningful engagement (Reference Elshaug, Hiller and Moss11;Reference Hodgetts, Elshaug and Hiller13;Reference Watt, Hiller and Braunack-Mayer14;Reference Watt, Willis, Hodgetts, Elshuag and Hiller20). Stakeholders should be defined broadly to include any group impacted by the resulting decision. As clinicians are likely to be an important stakeholder group in any HTR, an effective strategy should “grab both the hearts and the minds of clinicians.” Thoughtful consideration of patient and public engagement should be considered (Reference Street, Hennessy, Watt, Hiller and Elshaug21).

Strengthen postimplementation monitoring and evaluation

HTR must include a developed postimplementation monitoring and evaluation process with an emphasis on active monitoring for unintended consequences.

“There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don't know. But there are also unknown unknowns. There are things we don't know we don't know. (22)”

Active monitoring for these “unknown unknowns” is critical to ensure that health outcomes, access, and equity are not adversely impacted. For example, within a surgical program, the operation suite will never be un-used. Thus, if one procedure is stopped, the operational suite time could be filled with another procedure perhaps equally as, or more, harmful and wasteful. Active monitoring for these situations will ensure that the vacuum created by removing one procedure is back-filled by a more effective procedure resulting in better value for money. This point emphasizes the need for buy-in by clinicians to mitigate unintended consequences. Monitoring for these consequences will require a variety of evaluation techniques drawing on both quantitative and qualitative research methodologies.


HTR is not merely an academic exercise. If HTR is to be successful it must generate useful knowledge to inform real world decisions around the optimal use of technologies. The term “inform” is deliberately used here, as it is recognized that research evidence is only one of many kinds of knowledge that decision makers use when making these decisions, and that these decisions are always informed by values. In addition, decision making is neither rational nor linear, and as Jonathon Lomas has noted “is not an event; rather it is a generally diffuse, haphazard & somewhat volatile process that is ethereal in nature” (Reference Lomas and Brown23).

As one panelist noted, evidence-based guidance can be viewed as a practical manifestation of social contracts in deliberative democracies to ensure the most efficient and ethical allocation of finite healthcare resources. To achieve this goal, social values as well as technical issues need to be considered and guidance developed from HTR should reflect the social/political milieu.

Healthcare budgets will not continue to increase and we need to shift our thinking about how we can use these dollars differently with the goal of optimizing “value for money.” Managing technologies throughout their lifespan is required to ensure that technologies continue to achieve optimal value for money. HTR is required to support evidence-informed continued management. To move this field forward, we must continue to build on international experiences with a focus on developing novel methodological approaches to generating, incorporating, and implementing evidence into policy and practice.


Gail MacKean, PhD, Research Consultant, Health Technology Assessment Unit, Department of Community Health Sciences, Institute of Public Health, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada

Tom Noseworthy, MD, MPH, Professor, Health Technology Assessment Unit, Department of Community Health Sciences, Institute of Public Health, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada

Adam G. Elshaug, PhD, Associate Professor, Menzies Centre for Health Policy, School of Public Health, The University of Sydney, NSW, Australia

Laura Leggett, BSc., Research Assistant, Health Technology Assessment Unit, Department of Community Health Sciences, Institute of Public Health, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada

Peter Littlejohns, PhD, Professor, Division of Health and Social Care Research, School of Medicine, King's College London, Capital House, London

Joan Berezanski, Assistant deputy minister, Clinical Advisory and Research Branch, Alberta Health, Edmonton, Alberta, Canada

Fiona Clement, (), PhD, Assistant Professor, Health Technology Assessment Unit, Department of Community Health Sciences, Institute of Public Health, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada


All authors report they have no potential conflicts of interest.



1.Joshi, NP, Stahnisch, FW, Noseworthy, T. Reassessment of health technologies: Obsolescence and waste. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2009.Google Scholar
2.Tierney, M, Manns, BJ, For the members of the Canadian Expert Drug Advisory Committee. Optimizing the use of prescription drugs in Canada through the Common Drug Review. CMAJ. 2008;178:432435.CrossRefGoogle ScholarPubMed
3.Elshaug, A, Wat, A, Moss, J, Hiller, J. Policy perspectives on the obsolescence of health technologies in Canada. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2009.Google Scholar
4.Noseworthy, T, Clement, FM. Health technology reassessment: Scope, methodology and language. Int J Technol Assess Health Care. 2012;28:201202.CrossRefGoogle Scholar
5.Docherty, MJ, Choudury, M, Littlejohns, P. Infective endocarditis. NICE calls for a register. BMJ. 2011;342:d633.Google Scholar
6.Elshaug, AG, Moss, JR, Hiller, JE, Maddern, GJ. Upper airway surgery should not be first line treatment for obstructive sleep apnoea in adults. BMJ. 2008;336:4445.CrossRefGoogle Scholar
7.Basque Office for Health Technology Assessment (Osteba). Report on the development of the GUNFT guideline: Quality plan for the NHS MHSP. Health Technology Assessment Reports No 2007/11. Vitoria-Gasteiz Spain: Osteba; 2009.Google Scholar
8.Elshaug, A, Hiller, J, Tunis, S, Moss, J. Challenges in Australian policy processes for disinvestment from existing, ineffective health care practices. Aust New Zealand Health Policy. 2007;4:23.CrossRefGoogle ScholarPubMed
9.National Institute for Health and Clinical Excellence. Draft Project PLan: Introduction of new disinvestment programmes. 2006. (accessed October 31, 2012).Google Scholar
10.Leggett, L, Noseworthy, T, Zarrabi, M, et al.Health technology reassessment of non-drug technologies: Current practices. Int J Technol Assess Health Care. 2012;28:220227.CrossRefGoogle ScholarPubMed
11.Elshaug, AG, Hiller, JE, Moss, JR. Exploring policy-makers’ perspectives on disinvestment from ineffective healthcare practices. Int J Technol Assess Health Care. 2008;24:19.CrossRefGoogle ScholarPubMed
12.Garner, S, Littlejohns, P. Disinvestment from low value clinical interventions: NICEly done? BMJ. 2011;343:d4519.CrossRefGoogle Scholar
13.Hodgetts, K, Elshaug, AG, Hiller, JE. What counts and how to count it: Physicians’ constructions of evidence in a disinvestment context. Soc Sci Med. 2012;75:21912199.CrossRefGoogle Scholar
14.Watt, A, Hiller, J, Braunack-Mayer, A, et al.The ASTUTE Health study protocol: Deliberative stakeholder engagements to inform implementation approaches to healthcare disinvestment. Implement Sci. 2012;7:101.CrossRefGoogle ScholarPubMed
15.Elshaug, AG, Moss, JR, Littlejohns, P, et al.Identifying existing health care services that do not provide value for money. Med J Aust. 2009;190:269273.CrossRefGoogle Scholar
16.Elshaug, AG, Watt, AM, Mundy, L, Willis, CD. Over 150 potentially low-value health care practices: An Australian study. Med J Aust. 2012;197:556560.CrossRefGoogle ScholarPubMed
17.Merlin, T, Street, J, Holton, C, et al. Review of medical benefit scheme items for specific ophthalmology services under the medical benefit scheme quality framework. Australian Department of Health and Ageing. Canberra: Commonwealth of Australia; 2011.Google Scholar
18.Elshaug, AG, Garber, AM. How CER could pay for itself - Insights from vertebral fracture treatments. N Engl J Med. 2011;364:13901393.CrossRefGoogle ScholarPubMed
19.O'Dowd, A.Half of latest NHS efficiency savings came from staff pay freeze. BMJ. 2012;345:e7174.CrossRefGoogle ScholarPubMed
20.Watt, AM, Willis, CD, Hodgetts, K, Elshuag, AG, Hiller, JE. Engaging clinicians in evidence-based disinvestment: Role and perceptions of evidence. Int J Technol Assess Health Care. 2012;28:211219.CrossRefGoogle ScholarPubMed
21.Street, JM, Hennessy, SE, Watt, AM, Hiller, JE, Elshaug, AG. News and social media: Windows into community perspectives on disinvestment. Int J Technol Assess Health Care. 2011;27:376383.CrossRefGoogle ScholarPubMed
22.Secretary Donald Rumsfeld. United States Department of Defence News Briefing. 2002. (accessed October 31, 2012).Google Scholar
23.Lomas, J, Brown, AD. Research and advice giving: A functional view of evidence-informed policy advice in a Canadian Ministry of Health. Milbank Q. 2009;87:903926.CrossRefGoogle Scholar
24.International Network of Agencies for Health Technology Assessment. Compiled by Dr. Facey, Edited by Topfer and Chan. Health technology assessment glossary. Stockholm, Sweden: International Network of Agencies for Health Technology Assessment; 2006.Google Scholar
Figure 0

Table 1. Definitions

Figure 1

Table 2. Summary of Themes