Brief background

In the United States, AHRQ created and operated the national Healthcare Horizon Scanning System to inform its Effective Health Care Program from December 2010 to December 2015. Built on concepts and approaches developed for the AHRQ horizon scanning system, PCORI initiated its HCHSS in December 2018 and defined its project scope to focus on PCORI’s high-priority conditions and topics and interventions with high potential for disruption in health care in the United States.

As an extension to its horizon scanning program, to obtain timely evidence summaries on emerging interventions and to inform its CER investments on primary research, PCORI initiated the Emerging Technologies and Therapeutics Reports in 2019. Figure 1 illustrates the general methodologies in place for these reports.

Figure 1.

PCORI’s process for developing Emerging Technologies and Therapeutics Reports. This figure describes the following processes: topic scoping and development, topic refinement, evaluation, report development, stakeholder engagement in report development, and dissemination. Abbreviation: PCORI, Patient-Centered Outcomes Research Institute.

Stakeholder engagement

Stakeholders nominate the topics for Emerging Technologies and Therapeutics Reports and are engaged throughout the development of the report. A broad range of stakeholders including patients, caregivers, and other representatives offer subject matter expertise based on real-world or lived experience; explore, refine, or validate the scope of the project; and provide feedback throughout the continuum of the development of the report, raise concerns, or suggest opportunities for continued alignment of these reports with their priorities, with an intent to enhance the reports’ utility to inform healthcare decisions. These reports not only focus on the examination of clinical outcomes, but also include patient-centered outcomes deemed important, as informed by stakeholders (Reference Forsythe, Carman and Szydlowski1).

Description of the general approach

PCORI engages stakeholders for the nomination of potential topics that have the highest potential for disruption in health care. Although there are many different definitions of disruptive innovation (Reference Christensen, Raynor and McDonald15), PCORI’s HCHSS defines healthcare disruption as “disruption to the current standard of care in terms of patient outcomes, health disparities, care delivery, infrastructure, access, and/or costs (Reference Hulshizer, Lynch and De Lurio2).” PCORI’s working definition of new and emerging technologies includes “new and emerging interventions or therapies that are currently cleared by the U.S. Food and Drug Administration (FDA) for phase II/III trials, or those that may be approved in the next 1–3 years, and interventions or therapies in clinical use without the need for the FDA approval (such as off-label or repurposed for use in a particular condition or diagnostics) but lack sufficient evidence of efficacy.” After deliberations, PCORI staff conduct preliminary literature searches and narrow down eligible topics that have the highest potential for disruption, develop an initial set of guiding research questions, further evaluate topics of interest through stakeholder engagement, and contract the project report development to an external vendor for further in-depth evaluation of the topic (Figure 1). As required by PCORI, the contracted vendor utilizes systematic literature search and selection to identify relevant studies from published and gray literature sources to answer guiding research questions. The systematic literature search and selection methodology is described in detail in the appendices of each report to ensure that the yield is replicable and reproducible. During the conduct of the report, selected representatives (about 8–10 members) from different stakeholder groups with either relevant subject matter expertise or lived experience serve as key informants (KIs). KIs help to refine guiding research questions, identify relevant outcomes, and highlight relevant contextual issues in these interventions. The report includes a succinct summary of therapeutics or other technologies, a section on contextual issues identified by KIs, and a descriptive review along with evidence maps (i.e., visual charts) to depict the breadth of the available evidence. The objective of the evidence gathering is to provide a better understanding of the body of evidence supporting currently approved technologies of interest, including evidence gaps, but also to inform about those that are in the pipeline to be approved soon by the FDA. The reports undergo a technical review by selected KIs to ensure that guiding questions have been fully addressed.

The final report is publicly posted on PCORI’s Web site and is followed by the development of an issue brief and other communication and dissemination initiatives including blog posts, social media initiatives, and PCORI’s email newsletters. The issue brief succinctly summarizes the final report’s key findings in three to four pages using a lay language format and simple infographics, with an emphasis on highlighting findings that are most relevant to a range of stakeholder groups, including clinicians, patients and caregivers, researchers, health system managers, payers, and policy makers.

To date, PCORI has completed five Emerging Technologies and Therapeutics Reports, and the description of these reports is available in Table 2. The five completed reports include Landscape Review and Evidence Map of Gene Therapy as two-part reports (Reference Richardson, Apaydin and Baxi16;Reference Richardson, Apaydin and Baxi17) with two accompanying articles (Reference Apaydin, Richardson and Baxi18;Reference Apaydin, Richardson and Baxi19); Proteomic Testing in Cancer and Cardiovascular Diseases (Reference Stapleton, Siddiqi and Apaydin20); Artificial Intelligence (AI) in Clinical Care (Reference Girosi, Mann and Kareddy21); Genomic Sequencing to Guide Cancer Management (Reference Oristaglio, Manso-Share and Madireddi22); AI in the COVID-19 Response (Reference Mann, Berdahl, Baker and Girosi23); and an accompanying article (Reference Mann, Berdahl, Baker and Girosi24).

Table 2.

Completed emerging technologies and therapeutics reports

Abbreviations: AAV, adeno-associated virus vector; AI, artificial intelligence; CAR-T, chimeric antigen receptor T cell; CRISPR, clustered regularly interspaced short palindromic repeats; NGS, next-generation genomic sequencing; RNAi, antisense, ribonucleic acid interference; ZFN, zinc finger nuclease

Reflections from evaluation of completed reports

These reports provide a means for conveying patient-centric information about an emerging innovation while supporting the identification of evidence gaps that can be used to generate new topics for future research (Figure 2). PCORI has the opportunity to develop a systematic process for nomination and working with stakeholders to establish a pipeline for future projects as well as establish various approaches to select the topics of interest. PCORI’s reports have examined multiple interventions to date. Future-focused rapid reports of single technological evaluations may yield a more expedient process for identifying evidence and addressing gaps pertaining to emerging interventions. Currently, PCORI’s Emerging Technologies and Therapeutics Reports program does not have a process in place to assess the uptake of its report findings among different stakeholders, and future efforts are needed to assess the impact of these reports on evidence generation to address gaps. Additionally, PCORI needs to develop criteria for when to update or “sunset” an existing report.

Figure 2.

PCORI’s process of the evaluation of technologies and therapeutics. This figure describes the short-, medium-, and long-term products from PCORI. Abbreviations: M, million; PCORI, Patient-Centered Outcomes Research Institute.

Comparative overview of processes across HTA organizations

We provide a comparative overview of selected national and international HTA organizations, to understand similarities and differences across programs as well as to consider incorporating relevant processes and methodologies into PCORI’s Emerging Technologies and Therapeutics Reports.

We selected five publicly funded international HTA organizations that are comparable in terms of healthcare practices and demographics to which our findings will be applied, including AHRQ from the United States (25), Canadian Agency for Drugs and Technologies in Health from Canada (26), the National Institute for Health and Care Excellence (NICE) from the United Kingdom (27), Institute for Quality and Efficiency in Health Care (IQWiG) from Germany (28), and Pharmaceutical Benefits Advisory Committee from Australia (29) (Table 3). We obtained information from the methodology reported on the Web sites of these agencies and from relevant publications (30–32). The programmatic process of conducting HTA varied considerably across the reviewed bodies in terms of funding, types of assessments, the role of manufacturers, stakeholder engagement in various stages of HTA, and timeline to completion of a draft report publication.

Table 3.

Comparison of the evidence assessment process of technologies and therapeutics by various publicly funded organizations

^a Fee exemptions or fee waivers for rare disease medications or those that are of public interest, respectively.

^b Applicable for the assessment of clinical effectiveness and the safety of pharmaceutical drugs that are commissioned through G-BA according to AMNOG.

^c For the assessment of all other topics (excluding pharmaceutical drugs), submission of proposals occurs via the IQWiG Web site and is conducted through the ThemenCheck Medizin (TopicCheck Medicine) program.

^d Referred to NICE using criteria reviewed by the Dept of Health.

Abbreviations: AHRQ, Agency for Healthcare Research and Quality; AMNOG, The Act on the Reform of the Market for Medical Products (Arzneimittelmarkt-Neuordnungsgesetz); CADTH, Canadian Agency for Drugs and Technologies in Health; G-BA, Gemeinsamer Bundesausschuss (Joint Federal Committee); HTA, health technology assessment; IQWiG, Institute for Quality and Efficiency in Health Care; NHS, National Health Services; NICE, National Institute of Health and Care Excellence; PBAC, Pharmaceutical Benefits Advisory Committee; SR, systematic review.

While the international HTA organizations in this review are publicly funded, a few charged a fee or an optional fee from industry or manufacturers for the evaluation of their products (26;28;29). Only two HTA organizations assessed clinical effectiveness alone (25;28), whereas the remaining three organizations assessed both clinical and cost-effectiveness. All organizations engaged manufacturers, although the level of their involvement varied across the evaluated HTAs and included patients or patient advocacy organizations as stakeholders during their evaluations. All but one HTA organization stated their topic selection criteria (29). All organizations utilized in-house staff expertise in varying capacities to conduct their evaluations or to compile their reports. All HTAs included a review period for review by one or more stakeholders, including manufacturers, payers, patient groups, and physicians and hospital representatives. AHRQ and NICE allow general public comments for their reports. IQWiG engages the public for its public HTA program (the ThemenCheck Medizin) or during deliberations of draft guidance through its supporting agency Federal Joint Committee (G-BA). The communication of uncertainty for informed decision making also varied across agencies.

Comparison of PCORI’s processes of evaluation of emerging technologies with publicly funded HTA organizations

PCORI’s Emerging Technologies and Therapeutics Reports initiative has many similarities to publicly funded organizations reviewed here. However, there are some key differences between the reviewed HTA organization programs and PCORI’s program that serve their respective missions. While most of the publicly funded HTA organization programs listed here generally focused on the effectiveness and/or value of technologies, in the role of informing coverage decisions, PCORI’s reports serve to understand the landscape of evidence to support the generation of topics for funding primary research, to inform stakeholders, and to understand the real-world issues during diffusion and aid in healthcare decision making. PCORI uses KI interviews to understand contextual issues such as regulatory factors that vary by technologies (therapeutics, diagnostics, devices, etc.), systems issues (e.g., need for specialty centers during the administration of CAR-T therapies), and patient-relevant issues (e.g., the durability of response with gene therapies and the need for re-administration).

Key considerations and challenges in evaluating emerging interventions

In summary, our review identified some strengths and limitations of PCORI’s Emerging Technologies and Therapeutics Reports and offered lessons, informed by HTA programs, to improve processes. The review identified potential ways PCORI’s reports and methods could contribute to HTA organizations. PCORI’s Emerging Technologies and Therapeutics Reports convey patient-centric information about an emerging innovation and identify evidence gaps that can be used to generate new topics for future primary research.

Even though policies and practices of engaging patients and their families during the conduct of HTAs are evolving (Reference Wale, Thomas, Hamerlijnck and Hollander33), PCORI conducts research guided by patients, caregivers, patient advocacy groups, and the broader healthcare community. PCORI’s Emerging Technologies and Therapeutics Reports do not constitute a full HTA, but they do incorporate standard HTA processes from topic nomination through a final draft review and their methodologies such as systematic searches and selection. PCORI’s reports can serve as models to conceptualize and incorporate patient centricity in many ways during HTA development that could spur additional primary evidence generation. PCORI currently funds the Science of Engagement awards to further the evidence base for effective stakeholder engagement in research.

To date, PCORI’s reports have examined broader topics of multiple interventions. Rapid reports on single technologies are conducted routinely by many of the publicly funded HTA organizations that we reviewed, and this could be an approach to strengthen PCORI’s program (30). Implementing rapid reports could be important for two reasons – first, it could lead to a reduction in the time frame for report development, thereby expediting the process of identifying evidence and addressing gaps. Second, a focused review of single technologies in an expedient time frame can provide patients, their families, clinicians, and other providers with timely information that facilitates informed healthcare decision making (e.g., clinical trial participation).

PCORI’s program could be strengthened by incorporating HTA processes of topic nomination, topic selection, and technology evaluation and adapting them to the needs of patients and other stakeholders in the United States. Working with relevant stakeholders, PCORI could develop a systematic process for the nomination and selection of topics for future projects. Currently, PCORI’s Emerging Technologies and Therapeutics Reports program does not have a process in place to assess the uptake of its report findings among different stakeholders, and future efforts are needed to evaluate the impact of these reports on evidence generation to address gaps. Additionally, PCORI needs to develop a framework for when to update or “sunset” an existing report. Currently, updating a report under the Emerging Technologies and Therapeutics Reports program, PCORI relies on internal deliberations of the current relevance of key questions evaluated in the completed reports and the available new evidence.

Healthcare horizon scanning is a systematic process to identify new and emerging healthcare interventions that address unmet medical needs and have the highest potential for impact on health care. From horizon scanning, selected candidate topics would need to be evaluated if and how these new interventions can be of use, or how they should be monitored. PCORI and HTA organizations utilize healthcare horizon scanning to identify candidate assessment topics and develop accompanying reports on emerging healthcare interventions to set priorities. These reports can increase the awareness of available innovative care delivery practices, increase uptake, and further promote innovation. Horizon scanning and its accompanying emerging technology reports can adequately prepare users for future changes and can assist in policy making and in prioritizing resources by identifying important needs or gaps. For example, from PCORI’s perspective, identified gaps can inform prioritizing how resources should be allocated for future patient-centered outcomes research to increase evidence for emerging interventions. PCORI’s Emerging Technologies and Therapeutics Reports fit into the larger milieu of its broader U.S. Congressional remit of primary comparative CER and dissemination and implementation of research findings.