Introduction
Health technology assessment (HTA) has emerged as a discipline that aims to use the best available evidence to inform healthcare systems’ decisions about how to allocate scarce resources to achieve the maximum possible value. The traditional HTA processes of thorough evidence review and careful deliberation were not well suited to informing the rapid healthcare decision making by governments to curb the impact of COVID-19 during 2020 and 2021. Other scientific disciplines provided crucial, timely contributions, including health-related (like epidemiology, public health, immunology, and virology) and others (like behavioral science, logistical modeling, and constrained-optimization modeling). The limited role for HTA may be explained by some as an rational application of the “rule of rescue” (Reference McKie and Richardson1), while in some countries, there are inherent sensitivities around using economic evaluation to guide health-related resource allocation (Reference Neumann2). Additionally, HTA professionals have never faced a situation like COVID-19, and some may have been uncertain about how applicable their standard assessment methods and processes would be during a pandemic (Reference Padula, Reid and Tierce3). Many of these reasons are understandable. However, the absence of value assessments may have in part led to the observed inefficient allocation of resources on some COVID-19 interventions that were later proven to be of limited, if any, value to population health (Reference Dawoud, Chalkidou, Sullivan, Ruiz and Adler4;Reference Campbell, Whittington, Rind and Pearson5).
We argue that the core rationale for using HTA to guide decisions relating to the adoption of health technologies is immediately relevant, and even necessary, in the ongoing pandemic context (Reference Kaló, Németh and Zemplényi6), particularly as healthcare systems and budgets return to normal. However, we hypothesized that COVID-19 would expose areas of existing HTA practice that could be improved, and posit that traditional processes may need some modifications to improve agencies’ flexibility and responsiveness. Modifications are proposed as part of best-practice guidance recommendations.
The guidance is intended to support HTA bodies to undertake appraisals of therapeutics for COVID-19 and diagnostics for SARS-CoV-2. Medicines to treat COVID-19 are particularly likely to fall under the remit of most HTA agencies (7), and diagnostic tests featured prominently in decision making during the first years of the pandemic. Other types of intervention, such as vaccines and public health measures, are less likely to be assessed by HTA agencies going forward, and may have important nuances that would distract from the recommendations that are most relevant to most agencies. We propose that HTA agencies give due consideration to this guidance in their assessments, and maintain a robust approach while allowing appropriate pragmatism to adapt to the quickly changing decision context (Reference Elvidge, Summerfield and Knies8). We hope it also helps to enable HTA bodies to prepare a pandemic strategy and operational plan that adopts a living, life-cycle approach, to respond to a future pandemic, epidemic, or other healthcare-related emergency.
The guidance has been developed as part of Next Generation Health Technology Assessment (HTx). HTx is a Horizon 2020 project supported by the European Union lasting for 5 years from January 2019. Its main aim is to create a framework for the next generation of HTA to support patient-centered, societally oriented, real-time decision making on access to and reimbursement for health technologies throughout Europe.
Background
COVID-19 poses challenges for existing HTA processes
To understand the HTA challenges posed by COVID-19 and assess the need for modified or novel HTA processes, we used a multifaceted approach. First, to identify the key challenges, we sent a survey to forty-seven HTA agencies, composed of European agencies identified from the European Network for Health Technology Assessment (7) and a similar recent HTx survey (Reference Hogervorst, Vreman, Quik, Mantel-Teeuwisse and Goettsch9), and non-European agencies known to have been engaged in COVID-19 assessments. It was completed by twenty-one HTA agencies (sixteen from Europe, three from Australasia, and two from North America), of whom eleven respondents agreed to join a follow-up roundtable workshop. The survey and its findings have been fully reported elsewhere (Reference Elvidge and Dawoud10). In short, thorough HTA – especially cost-effectiveness analysis – was not a priority for policy makers in the early stage of the pandemic, due to: the need for governments and health systems to act quickly; uncertainty about traditional concepts of “value for money” during a health emergency, and in the context of a rapid expansion of central funds; and the risk of some treatments being subject to a shortage of global supply. HTA representatives perceived key challenges to assessing tests and treatments for COVID-19 as being the lack of high quality, comparative, long-term clinical evidence; nontraditional reporting formats, such as press releases without peer review (11); and heterogeneity in study populations, settings, and outcomes.
Additionally, the COVID-19 context of rapidly evolving scientific understanding and clinical practice meant HTA agencies’ usual “static” decision-making processes could become out of date quickly without a commitment to regular updates. They also faced increased public focus relating to COVID-19 decision making, which created pressure to rapidly approve promising technologies (Reference O’Rourke, Orsini and Guerino12) despite uncertain evidence.
To identify particular barriers to assessing the value for money of therapeutics and diagnostics for COVID-19, we also convened a workshop of thirteen health economists and modelers. Participants were identified as economists known by the authors to be engaged in COVID-19 modeling, and were from academia (n = 9) or HTA (n = 4) in a range of countries (three from England, three from the Netherlands, two from Canada, two from US, one from Hungary, one from Scotland, and one from Ukraine). We also conducted a systematic review of published economic evaluations of COVID-19 therapeutics and diagnostics, to learn from early attempts to assess such technologies; its methods and results have been reported elsewhere (Reference Elvidge, Summerfield, Nicholls and Dawoud13). From these activities, it was clear that early COVID-19 modeling efforts had lacked the high-quality data needed to inform the model structure and parameter inputs. Scientific understanding and the clinical pathway were developing quickly, but relatively little research has focused on the resource use, quality of life and long-term outcomes required by cost-effectiveness models (Reference Lorgelly and Adler14). The outputs of economic evaluations using incomplete or poor-quality evidence are less informative to decision makers. Therefore, many modeling efforts have necessarily been exploratory or highly uncertain, which would result in uncertain HTA deliberations. Even when good data are available, rigorous economic evaluations are generally time consuming, which is not consistent with the need to act fast in response to a health emergency. Additionally, to fully understand the value of effective interventions for an infectious disease pandemic, it may be necessary to capture complex and nontraditional modeling elements, such as disease transmission and system capacity effects.
HTA agencies would value best-practice guidance for COVID-19
In many countries, the pandemic context has now shifted away from the urgent situation of 2020 and 2021. Vaccination programs have taken effect and populations have developed a level of immunity to the virus. Healthcare systems are no longer at breaking point, and many novel and repurposed interventions are in development for moderate and severe disease. Management of COVID-19 is returning to the responsibility of traditional, fixed healthcare budgets, as emergency government funding is removed and economies recover (see Figure 1). We conducted a targeted search for existing methods guidance about how to assess technologies for COVID-19 on the web sites of five regulatory bodies (from Australia, Canada, the EU, UK, and US) and forty-two HTA organizations (from twenty-five European countries (n = 35) and five non-European countries (n = 7)) on 16 March 2021. This identified little methods guidance specifically for COVID-19 technologies; just four documents, from two regulators and one HTA agency, mostly reiterating traditional best practices (15;16). This may be inappropriate, as HTA representatives at our roundtable considered that many of the COVID-19-related challenges are likely to remain for some time, meaning assessments of COVID-19 technologies are unlikely to be straightforward (Reference Lorgelly and Adler14;Reference O’Rourke, Oortwijn and Schuller17). Some challenges are not new, such as uncertainty about long-term outcomes, but will continue to exist in a context of wider COVID-related instability. Therefore, best-practice guidance to support HTA agencies would be useful, to: (i) make agencies aware of the myriad issues that they should consider; (ii) provide a framework for consistent assessments, balancing the rigor and pragmatism; and (iii) help prepare HTA processes for potential future pandemics.
Figure 1. Potential stages of an infectious disease pandemic for HTA agencies.
Methods
Following the survey of HTA agencies, roundtable discussion, workshop with health economists and reviews of economic evaluations and existing methods guidance – to identify gaps in HTA processes for the assessment of interventions for COVID-19 – we sought to develop novel guidance to help HTA agencies approach the challenges presented by the pandemic.
We used a “policy sandbox” approach (Reference Leckenby, Dawoud, Bouvy and Jónsson18) to co-develop the recommendations with multiple HTA stakeholders. Sandboxes provide a way of framing engagement where the objective is to develop new methods, services or processes, by creating a “safe space” to consider the problems and innovate solutions without affecting ongoing work (e.g., this process did not affect the concurrent NICE COVID-19 rapid evidence reviews (19)). For this, we convened a multi-stakeholder group composed of twenty-one participants representing HTA agencies (n = 7), clinicians (n = 3), academia (n = 3), industry (n = 3), patient experts (n = 2), the Professional Society for Pharmacoeconomics Research (n = 2), and a national payer (n = 1). Representatives were from countries with various income levels across Europe and North America (4), to make the key principles of the guidance applicable to different economic settings. Stakeholders were first presented with findings from the background work, such as the key challenges for HTA and economic modeling. They were then presented with proposed guidance to address those challenges, which had been drafted by the HTx project team as a “straw man” to encourage discussion. Stakeholders critiqued, revised, removed, and added to the recommendations, at a workshop that held on 5 August 2021 and through subsequent engagement by email, thereby co-developing the final best-practice guidance for HTA agencies.
Separately, we conducted a targeted review of known existing clinical guidelines and pathways (19–27), and engaged with clinical experts known to the authors, to identify the most important outcomes associated with COVID-19 and form a clear understanding of the disease and clinical pathway. The eight clinicians were from Australia, the Netherlands, and UK, and covered the following medical specialties: emergency and critical care, general practice, infectious disease, intensive care, and rehabilitation. The purpose of this was to include a schematic alongside the best-practice guidance recommendations, as a starting point for (i) HTA agencies to consider where interventions may be positioned in the clinical pathway, and what outcomes are most important there, and (ii) the development of future COVID-19 decision models.
Results
Key recommendations for HTA
The best-practice guidance represents the consensus view of the multi-stakeholder “COVID-19 HTA best-practice development group,” derived from the sandbox process as described above. Its intended primary audience is HTA agencies planning or undertaking assessments of tests for SARS-CoV-2 or treatments for COVID-19. The recommendations provide pragmatic approaches to consider when appraising technologies for COVID-19 as their value for money becomes a more important policy objective. Developers of COVID-19 technologies can also use the guidance to inform their evidence generation and submission dossiers, while it can help inform other stakeholders in their engagement with, and expectations of, HTA processes.
The key recommendations are summarized below, and available in full at: https://www.htx-h2020.eu/publications/ (“Best-practice guidance for the health technology assessment of diagnostics and treatments for COVID-19”) (Reference Elvidge, Summerfield and Knies8).
Assessing clinical effectiveness
Randomized controlled trials (RCTs) should remain the gold standard for clinical evidence for technologies for COVID-19. However, during a pandemic, especially in the early stages, high-quality long-term RCT data are likely to be lacking, while evidence from traditionally less robust sources may be generate more quickly (such as real-world evidence (RWE), studies from other settings, studies published as pre-prints). In these circumstances, HTA agencies should be open-minded about incorporating such evidence into their assessments technologies for COVID-19; weighing the benefit to the healthcare system of timely HTA guidance at a time of great need, against the risk that the available evidence is too weak for accurate decision making. To increase their confidence in doing so, agencies should utilize and engage with the many European research initiatives seeking to enhance the availability of RWE and developing methods to analyze it appropriately (such as HTx, European Health Data and Evidence Network, GetReal Institute, and RWE4Decisions). Pragmatically, agencies should also consider using existing “living” systematic reviews to inform their clinical effectiveness assessments (Reference Boutron, Chaimani and Meerpohl28–Reference Siemieniuk, Bartoszko and Ge30), to save time and resources and avoid duplication of efforts. Similarly, published core outcome sets for COVID-19 could be used to identify the most appropriate clinical outcomes (Reference Boutron, Chaimani and Meerpohl28;31).
Assessing value for money
HTA agencies that use cost–utility analysis should continue to do so when assessing technologies for COVID-19. Other types of economic evaluation, such as cost–consequences analysis, can also be considered, where they are likely to be useful. Both the healthcare (payer) perspective and the societal perspective should be considered relevant to decision making about COVID-19 technologies. A societal perspective is likely to be more important when the pandemic situation is urgent (Reference Schnitzler, Janssen and Evers32) and when funding is provided from a central government, rather than a specific healthcare budget. HTA agencies should continue to use their standard cost-effectiveness thresholds in COVID-19 assessments for now, but should engage in research to identify whether there are different societal preferences during and following a pandemic.
Economic modeling
To inform assessments of cost effectiveness across a wide range of COVID-19 technologies, or sequences of technologies, the use of a whole-disease pathway model including both diagnosis and treatment is recommended. The model should ideally be an individual-level simulation to capture the heterogeneous patient population, the complex pathway, and the potential impact of some technologies on transmission or system capacity.
It should accommodate a wide range of options, including allowing it to run an individual-level or cohort-level model, capture disease transmission (e.g., by linking to an epidemiological or infectious disease model), and incorporate system capacity effects. Capacity effects will be more important to consider during surges of infection that lead to increased pressure on healthcare resources beyond their capacity constraints. The model should also include the flexibility to disable any of these options, to facilitate a less complex modeling approach where appropriate. For example, a simple cohort-level model may be acceptable when assessing a straightforward, narrowly defined decision problem, or multiple technologies at the same position in the treatment pathway. Any economic model should capture the long-term COVID-19 outcomes (33;34) and treatment effects.
The model should be frequently updated to reflect the most up-to-date understanding of the disease and changes in the standard of care. A “living,” adaptable whole-disease pathway model could be collaboratively developed by HTA agencies, with input from multiple stakeholders, and made freely available for use and adaptation to support a globally responsive HTA approach to COVID-19 technologies. As a starting point for such efforts, we have developed a summary COVID-19 disease and clinical pathway, derived from clinical guidance and expertise (see Figure 2).
Figure 2. COVID-19 clinical and disease pathway derived from clinical guidelines and expertise.
Considering uncertainty
Most COVID-19 assessments will have a high level of uncertainty. Agencies should be transparent about the data gaps and assumptions made and should consider the results of extensive sensitivity and threshold analyses. Probabilistic results should still be used and could be accompanied by value of information analysis to inform future research needs. A transparent, pragmatic, “living” HTA approach should be implemented, with a commitment to responsively reviewing decisions as new and better evidence emerges, including potentially reversing previous decisions by reinvesting or disinvesting in technologies.
Affordability and procurement
HTA agencies should routinely consider the expected budget impact (Reference Sullivan, Mauskopf and Augustovski35) of COVID-19 technologies, including any required service redesign and system burden, to identify technologies that would be difficult for the healthcare system to implement. This could trigger commercial discussions between the technology developer and healthcare payer. Alternatively, it could trigger the HTA agency to explore subgroup analyses, to identify groups for whom a technology is most cost effective and inform the prioritization of its use. Commissioners and payers may explore novel payment models for COVID-19 technologies. Managed access agreements, including a period of data collection to resolve key uncertainties, may be particularly well supported by a living HTA approach.
Considering other elements of value
HTA agencies should consider whether there are relevant benefits that are not adequately captured in the clinical and cost-effectiveness assessments of COVID-19 technologies, such as reduced inequity, reduced fear of infection, and scientific innovation. If it is not possible to include these effects quantitatively (e.g., by capturing them in utility values), then they should be considered using a qualitative and deliberative approach.
Stakeholder engagement
The far-reaching effects of the pandemic mean HTA agencies should ensure a broad range of stakeholders can contribute to the assessment of COVID-19 technologies. Input from clinical and patient experts may be particularly informative where there are evidence gaps and uncertainties. A tiered approach to patient and public engagement is recommended, including circumstances to approach citizens’ groups and organizations that represent high-risk groups. HTA agencies could consider using innovative methods to facilitate broad engagement, such as digital and online communication tools.
Conclusions
The pandemic has undoubtedly created challenges for HTA agencies, and many of these will persist for some time. To address them, we encourage agencies to formally consider this best-practice guidance when assessing technologies for COVID-19, and to implement its recommendations where possible. It has been co-developed with a range of stakeholders, providing a framework to support timely, evidence-based decisions about the value of the many technologies for COVID-19 in the coming months and years, as healthcare systems move away from the emergency pandemic response and once again seek cost-effective, affordable options. In the event of a future infectious disease pandemic, the guidance may also help HTA agencies to contribute more quickly, by planning their assessments and modeling requirements earlier and preparing for more flexible, responsive decision making. We acknowledge that not all elements of the guidance will be applicable for every HTA agency, but individual agencies may still wish to adopt the recommendations that are most suited to their preferred processes and methods.
While this guidance will support the medium-term response of HTA agencies to the pandemic, traditional HTA approaches, like cost-effectiveness analysis, might not be optimal or practical during the initial, emergency response to healthcare crises. Further research, perhaps using a policy sandbox approach, to explore the best ways for HTA agencies to provide an immediate contribution at such times – including for other types of intervention, such as vaccines and public health strategies – would be valuable.
Acknowledgments
We would like to thank the following individuals co-developed this guidance as part of the COVID-19 HTA Best-Practice Development Group:
-
• Neil Bertelsen, Ailsa Brown, Jon D. Campbell, Arturo Cabra, Jaime Caro, Stijntje Dijk, Trudy Foster, Peter Graham, Eline Krijkamp, James Larcombe, Anne Lee, Ben Messer, Mark Miller, Dionysios Ntais, Brian O’Rourke, Oresta Piniazhko, Nina Pinwill, Juan Carlos Rejon-Parrilla, Kawaldip Sehmi, Matthew D. Stevenson, Sheela Upadhyaya, Thomas Walker.
We would also like to thank the following individuals for their input into, or support for, the development of this guidance:
-
• Roisín Adams, Tarry Ahuja, Lucy Beggs, Jacoline Bouvy, Adam Brooke, Jagpreet Chhatwal, Y-Ling Chi, Carla Deakin, Diana Delnoij, Jane Eddleston, Julian Elliott, Anna Melissa Guerrero, Alex Haines, Kevin Harris, Jasdeep Hayre, Daniel Horner, Li Jiu, Páll Jónsson, Justine Karpusheff, Jeanette Kusel, Karen Lee, Francois Maignen, Kamila Malinowska, Gergő Merész, Andrew Mitchell, Diane Playford, Johan Ponten, Maarten Postma, Rachid Rafia, Mihai Rotaru, Murali Shyamsundar, John Spoors, Tina Taube, Isabelle Tongio, Michael Toolan, Tari Turner, Goedele van Haasteren, Albert Vollaard, Junfeng Wang, Alan Wailoo, Melanie Whittington, Darshan Zala.
Funding statement
This work was supported by the European Union’s Horizon 2020 research and innovation program (Grant Agreement No. 825162). This dissemination reflects only the authors’ views and the European Commission is not responsible for any use that may be made of the information it contains. The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Competing interest
J.E., S.K., B.N., Z.K., W.G., and D.M.D. disclosed that this work was supported by the European Union’s Horizon 2020 research and innovation program (Grant Agreement No. 825162). A.S. reported no competing interest.
Open access
