Health technology assessment (HTA) aims to provide health services with evidence-based conclusions and guidance on the value of health interventions. Applying HTA to medical devices and diagnostics can be challenging for several reasons relating to the nature and quality of the evidence and the many purposes and ways in which devices can be used.
First, the amount and quality of published evidence for devices and diagnostics is generally much poorer than for pharmaceuticals. This is the result of less regulatory demand for evidence; less commercial reward to compensate for research and development costs (because the period of profitability for an innovative device can be relatively brief before imitators become available); and an industry that includes a large proportion of small and medium sized manufacturers, many of whom are inexperienced in the types of clinical research needed for HTA and shortage of funds to support research.
Second, the available studies on devices and diagnostics often focus on endpoints which are not the most useful to support decisions on adoption: they commonly fail to address relevant clinical utility issues and practical value propositions. One reason for this may be pressure by funders on the inventors of technologies to offer a quick return on their investment.
Third, devices often undergo fairly rapid sequential development, with repeated modifications that can confound evidence if included in studies; and similar devices may be marketed by several different manufacturers.
Finally, and in contrast to medicines, there are many factors that influence the benefits which devices and diagnostic technologies achieve, including the setting in which they are used, the staff who use them and (in the case of diagnostics) whether and how much subsequent patients’ outcomes are changed.
In the United Kingdom (UK), as elsewhere in the world, the agenda for rational innovation in healthcare has gathered pace as a result of an increased focus on outcomes in the context of escalating costs (Reference Kent, Fendrick and Langa1;2). Central to this agenda is a demand to identify clinically beneficial and cost-effective new technologies; to foster their early adoption; and ideally to replace other less effective or more costly technologies and practices (2). In response to this demand, the National Institute for Health and Care Excellence (NICE) created a new program (in December 2009) to assess medical devices and diagnostics and to produce guidance for the National Health Service in England (NHS) on their adoption.
The NICE Medical Technologies program and its processes were developed through a complex process of deliberation and engagement with stakeholders (see the Results section). Its methods of assessment and the nature of the guidance it produces on adoption of technologies drew upon the principles of existing NICE work but required significant innovation, such as the ability to consider all levels of evidence and the use of cost consequence models. In addition, the program was designed with a capacity to foster collaborative research between companies (manufacturers or commercial distributors) and the health service to answer relevant questions about the benefits and clinical utility of technologies: that will not be described in detail here. In this study, we set out the principles on which the new system of evaluation of devices and diagnostics is based, the process and methods of assessment, and our first 5 years’ experience of technologies notified to NICE.
METHODS
For the purposes of this study, all technologies notified during the first 5 years were reviewed, documenting their eligibility, the decisions made about them and the type of guidance produced on them by NICE, for each successive 1-year period. Reasons for not selecting technologies were derived from letters sent to companies which include brief details of the main considerations for the decision not to select.
For the purposes of the study, technologies were classified according to the Global Medical Device Nomenclature (GMDN) and the Food and Drug Administration (FDA) systems, using the methodology described by Keltie et al. (Reference Keltie, Bousfield, Cole and Sims3). Associations were sought between the classes in each system and whether or not technologies were selected by the committee for assessment, using Fisher's Exact test (as done for a previous smaller sample of procedures described by Keltie et al. (Reference Keltie, Bousfield, Cole and Sims3).
RESULTS
The main recommendation in Medical Technologies Guidance is based on whether or not “the evidence supports the case for adoption” of the technology. This recommendation is accompanied by a brief statement of the benefit/s and cost consequences of adoption. The recommendation may be qualified by statements about patient subgroups or the settings to which the case for adoption applies. If the case for adoption is not supported by the evidence then the reasons for that are described, and are reviewed here.
The Medical Technologies Evaluation Programme (MTEP) was developed during 2009 through a wide ranging series of discussions within NICE, informed by the deliberations of working groups which included representatives from the medical devices and diagnostics industries, clinicians, patient organizations, health service managers, and government agencies. As a result of these discussions, the following scheme was devised: it is described graphically, and in detail, in the MTEP Process Guide and Methods Guides (4).
-
(A) The Programme deals with technologies that fit accepted definitions of medical devices and diagnostic technologies: the latter include genetic tests.
-
(B) Technologies are normally notified to the program by the company (manufacturer or a UK distributor), but exceptionally by a clinical sponsor invited by NICE if a technology seems to have special promise for the health service.
-
(C) Notification requires responses to a series of factual questions about the product, and its use. The notification form is available online (5). In addition, the company is asked to describe the claimed advantages of the technology over current management: these claims are divided into benefits for patients (clinical outcomes, acceptability, access to care, quality of life), benefits for the health service (staff or resource use, capital and maintenance costs), and sustainability (less waste or energy use).
-
(D) To be eligible for assessment, technologies must have a CE (Conformité Européene) mark as a medical device (i.e., regulatory approval under the European Union Medical Devices Directive) and be ready for marketing in the United Kingdom (or plans for CE marking within 1 year). They should not be the subject of other ongoing national evaluation. Some consideration is given as to whether there are any users in the United Kingdom (even in the context of research) so that it is practical to get at least some expert advice from UK clinicians.
-
(E) The MTEP team (analysts with expertise in health technology assessment) prepares a Briefing Note for each technology, based on the information from the notification, input from expert clinical and scientific advisors (nominated by professional organizations, by companies and by NICE), and, when obtainable, input from patient organizations. Additional information about the condition being treated and aspects such as equality considerations (issues relating to race, sex, religion) is included. The company has an opportunity to check the Briefing Note for accuracy.
-
(F) The Briefing Note is considered at a meeting of the Medical Technologies Advisory Committee (hereafter “the committee”). Membership of this independent advisory committee is available online (6). Expert clinical or scientific advisors attend the meeting to answer the committee's questions about the product and its use in clinical care. The committee decides whether the technology shows sufficient promise of benefits to patients and the health system to be selected for assessment, and, therefore, whether guidance from NICE would be desirable. Important elements of that decision include what “current management” the product should be compared against; the quantity, quality, and relevance of proffered evidence to the normal UK care pathway; and a clear understanding of the setting in which the technology is to be used (e.g., community or hospital care).
-
(G) If a technology is selected the committee then decides which of NICE's assessment programs would be most suitable. Options selected to date are:
-
(a) The committee may decide to retain the technology for its own assessment, to produce Medical Technologies Guidance. Only single technologies are assessed and cost-consequences modeling is used, so technologies that are likely to be cost-incurring are not suitable.
-
(b) Diagnostic technologies that are likely to require complex assessment (for example, because there are several other alternative or supplementary tests, significant changes to care pathways, or a need for cost-per-quality-adjusted life-year modeling) or that are likely to be more costly than current management are referred to the NICE Diagnostics Assessment Programme (7).
-
(c) Devices that are likely to have a major financial impact on the health service may be referred to the NICE Technology Appraisals Programme, which assesses clinical and cost-effectiveness (8).
-
-
(H) Technologies selected for Medical Technologies Guidance are then subject to a standardized NICE assessment process, consisting of:
-
(a) Production of a scope that defines the most important questions about clinical and resource impacts, based on the company's claims. It sets the boundaries for assessing the evidence and for the committee's decision making and is prepared in the standard HTA PICO format, addressing the population, intervention, comparator, and outcomes for the assessment.
-
(b) Production of a detailed submission by the company, which is in two parts: first the clinical evidence, with a clear description of the decision problem in the PICO table in the scope and the relationship of the evidence to it; and second, the economic evidence with an executable economic model (based on a template supplied by NICE). The economic model is designed to calculate the cost consequences of adopting the technology instead of current management and is based on explicit assumptions about both.
-
(c) A critical analysis of the company's submission by one of four independent academic centers commissioned by NICE. These academic centers were chosen for their expertise in various aspects of HTA of medical devices and diagnostics. They produce an assessment report which identifies errors, uncertainties, and issues in the company's submission that might invite argument (9). If the academic center considers that the submission does not adequately address issues in the scope, it may suggest to NICE additional analyses, which could include a new cost model. When the academic center's assessment report is complete, with any new analyses, the company has an opportunity to review it and to proffer written comments.
-
(d) Production of an “overview” by the MTEP team, highlighting the significant findings of the assessment report as they are to be considered by the committee in developing guidance. This overview may include additional information that the team considers relevant.
-
(e) Where needed, further advice from clinical or scientific experts and patient and carer organizations. In addition, for some technologies, individual patients are identified and asked for their input.
-
-
(I) The committee meets and considers all the information listed above. Expert advisers (usually two) attend that meeting, and for some technologies, a patient also attends. Representatives of the company are present. The data are presented and committee members have the opportunity to pose questions to the experts, manufacturer, and patient. The committee then drafts recommendations on the technology in private session, in view of their commercial sensitivity. Aspects of the committee's discussion are noted for inclusion in the guidance.
-
(J) Draft guidance is then subject to 1 month of public consultation on NICE's Web site. At a subsequent meeting, the committee considers all the comments received at public consultation and may revise the guidance in the light of these. Before publication, there is a final step whereby consultees who believe that there are factual errors or that NICE's processes were breached during preparation of the guidance can request resolution of these issues. When these processes are complete, guidance is published.
From the start of the system in December 2009 to November 2014, there were 186 notifications of technologies. The primary purpose of the technologies was diagnostic in 100 (54 percent) and therapeutic in 86 (46 percent). Perusal by the NICE team found 39 (21 percent) to be ineligible (because, for example, of no CE mark), so the number seen by the committee was 147: this is the denominator for the analyses below.
Following consideration of Briefing Notes by the Committee, sixty-seven (47 percent) technologies were selected for assessment and production of NICE guidance. The majority (35–52 percent) were considered suitable for Medical Technologies Guidance, while twenty-nine (43 percent) were routed to the NICE Diagnostics Assessment Programme (7), because of their complexity or cost. Just three were routed to other NICE programs: two to Technology Appraisals and one for a Clinical Guideline (8;10). Table 1 shows the notifications and the numbers eligible and selected, year by year.
Number of Technologies Notified to NICE from December 2009 – November 2014
The main reasons that technologies were not selected are shown in Table 2. These reasons were derived from the brief letters sent to companies. By far the commonest reason was lack of evidence of patient or system benefit. Specific uncertainties were documented more often about benefits to the health service (in 54 percent cases) than about benefits to patients (39 percent) or about cost (24 percent).
Reasons Given in Letters to Companies for not Selecting Technologies for Development of NICE Guidance
Table 3 shows the numbers (percent) of technologies selected, based on the FDA and GMDN classifications. Analysis by Fisher's exact test showed no association between any classes of procedure, in either classification, and decisions to select or not select for assessment.
Classification of Technologies Notified to NICE.
Supplementary Table 1 shows the twenty-seven technologies for which Medical Technologies guidance has been published. It shows the types of recommendations made. Overall, the main recommendation encouraged adoption by the health service for eighteen (75 percent) technologies; it recommended research to address clinical utility for five (21 percent), and for just one (4 percent) it stated that the evidence did not support the case for adoption.
DISCUSSION
The system we have described is a novel approach to a longstanding yet topical problem, namely how to spot which of the very many new medical devices (including diagnostics) will provide benefits to patients and health services, and how to encourage their adoption. This is a challenge to health systems worldwide and it may be that elements of NICE's process, or recommendations resulting from it, could be useful to others in addressing uptake of innovative technologies in different healthcare environments.
Our experience has been that some technologies are highly innovative and supported by good evidence, allowing a high level of certainty in recommending their adoption. An example was the EXOGEN ultrasound system for long bone fractures with nonunion or delayed healing, for which the evidence supported a recommendation for its use in nonunion fractures (11). The recommendation to the NHS to adopt EXOGEN was specific about the advantages to patients and about the likely saving of 1,164 GBP per patient compared with current management. This recommendation was circulated throughout the NHS and is also used by the company on its promotional literature. Other examples include the MAGEC system for spinal lengthening in children with scoliosis, which avoids repeated surgical procedures and may save around 12,000 GBP per child in the first 6 years; and the Mega Soft Patient Return Electrode, a reusable monopolar diathermy pad that replaces single use adhesive pads and that offers advantages for some patients and increased convenience for staff without increased costs (12;13).
By contrast, many of the technologies notified had less clear and plausible claims and many were supported by little or no good evidence of clinical advantage. The fact that one-fifth (21 percent) were ineligible and that over half (54 percent) of the remainder were not selected for full assessment does call into question the way in which technologies are being notified to our system. Could there be other more promising devices and diagnostics, with better evidence of benefit, that are not being notified, and if so, how could they be identified?
Like the other elements of our process, the notification method was agreed with industry, but experience and discourse with companies has suggested that some are reluctant to engage with the system. There seem to be three main reasons. First, some have concerns about the risk of a “negative” recommendation: that has occurred for just one product on the grounds of claims that were not judged likely to be realized in real-world practice (14). Typically, such technologies would not be selected for development of guidance, but sometimes it is unclear at the time of selection whether the company's claims will be plausible.
The NICE team frequently encounters a contrast between optimistic marketing claims and the reality of published clinical evidence. The solution to this, for companies, is simply not to notify products which have very little or poor evidence; and to avoid making claims which are implausible. The NICE team spends time advising companies on issues like these, but some decide to notify despite advice to the contrary.
A second reason that some companies seem reluctant to notify technologies is the work involved in preparing a full submission to NICE. For large companies, that ought to pose little problem, but for small companies that develop new technologies, the task can seem daunting, both in terms of time and lack of experience. That is understandable, but the process is a very clear one and even the cost modeling is relatively straightforward.
A third reason suggested for reluctance to notify is that some companies are uncertain about the commercial impact of NICE Medical Technologies Guidance. Although any positive recommendation from NICE has a strong influence on the healthcare system, the recommendations cannot mandate use of a particular device; they can only state explicitly that the evidence supports adoption, in place of defined current management, and provide figures from modeling of the cost savings to be made from adoption. Those costs have been given on a per patient basis, but there are currently plans to include in the recommendations a global figure for cost savings which would result if the technology were to be adopted for all eligible patients. This may be more helpful, especially for technologies that offer a small financial saving with each use, but for a large number of patients. It could encourage policy makers and payers to adopt technologies and should provide a powerful marketing tool for companies.
The strength of recommendations and the influence of NICE notwithstanding, there are still uncertainties about the effects of Medical Technologies Guidance on uptake of technologies, and NICE continues to look for new ways of promoting the guidance and of trying to measure its impact. There is a team within NICE (the Adoption and Impact team) that works at a local level to promote adoption of recommended technologies, to identify barriers to adoption, and to promulgate useful ideas for changing practice. Guidance is aimed at clinicians, as well as payers and institutional providers of healthcare: all these groups should be motivated to adopt technologies that have evidence of advantages to patients and/or to the healthcare system.
The reasons for technologies not being selected were derived from the brief letters sent to companies, which are not meant to be comprehensive, but to give an insight into the most relevant issues discussed by the committee. The content of these letters has evolved during the 5 years under study, but the reasons given in them were amenable to being categorized as shown in Table 2. Lack of evidence was by far the most common: this is not one of the selection criteria specified by NICE, but it meant that the “likelihood” of claimed benefits would be so uncertain that they could not be properly assessed (15).
One of the ulterior aims of the program is to encourage the generation of better evidence for medical devices, which is widely recognized to be an important issue (Reference Ergina, Cook and Blazeby16). With this in mind, NICE has initiated mechanisms to facilitate research for technologies that appear promising but for which more evidence is needed, when recommendations on a technology specify a particular need for research. These activities are described elsewhere (Reference Pomfrett, Campbell, Pugh, Campbell and Marlow17). The selection process is confidential between the company and NICE; decisions are only disseminated outside NICE if technologies are selected for full evaluation. The NHS is not informed about those technologies that are not selected for evaluation (NICE is not part of the NHS), and so nonselection does not affect the adoption of technologies by the NHS.
Classification of technologies according to the FDA and GMD systems was done only for the purposes of this report, and it offers an interesting reflection of the amount of innovation occurring in different areas. There was no association toward selecting or not selecting technologies in any particular class.
During our first 5 years of experience with this system for HTA of new devices and diagnostics, the complex process of assessment and production of recommendations by the committee has developed and evolved. The committee's decision-making process seems to work well and some particular aspects have been described elsewhere (Reference Campbell18). The finding that many of the notified technologies were not selected for assessment suggests that the process for identifying the best technologies to introduce into the system needs reconsideration. Such reconsideration could usefully involve the medical technologies industry and clinicians in identifying unmet need. Whatever evolution may occur, the system we have described provides the basis for addressing an important area of innovation in health care, aimed at getting the best new devices and diagnostics into widespread use by the clinical community.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: https://doi.org/10.1017/S0266462317000253
CONFLICTS OF INTEREST
The authors declare that they have nothing to disclose.
Open access
