Background
Following regulatory approval, a successful reimbursement decision by funding bodies is required for patient access to new treatments. In England, this body is the National Institute for Health and Care Excellence (NICE), who publish nationally mandated guidance regarding the acceptance of new health technologies by the National Health Service (NHS). Guidance produced by NICE on health technologies is also applied selectively in Northern Ireland, Scotland, and Wales. Evidence submissions from companies to NICE typically include analysis of clinical trial data, economic modeling, and the synthesis of other relevant information, such as clinical expert opinion and epidemiological data. Since the inception of NICE in 1999, a total of 544 technology appraisals (TAs, as of January 2019) have been published across a range of different disease areas, of which 240 were of cancer drugs (1).
In this area, when NICE are faced with uncertainty in regards to the clinical- and/or cost-effectiveness of cancer drugs that may be addressed with further data collection, interim funding may instead be obtained via the Cancer Drugs Fund (CDF) (2). The CDF was set up in 2011 as a mechanism to improve patient access to cancer drugs, and was reformed in 2016 to ensure further evidence collection as part of managed access agreements to minimize delays to patient access (3). Additional nontrial data may provide useful information to decision makers, as it is seldom the case that trials are able to provide all evidence necessary to inform decision making. This is particularly prevalent in oncology, where treatments for rare cancers may receive regulatory approval based on data from an uncontrolled (Phase II) trial, which may lack necessary evidence for health technology assessment (HTA).
Data reflecting the use of a given intervention within a nontrial (or “true-to-life”) clinical setting are often termed “real-world data” (RWD)— the analysis of which is termed “real-world evidence” (RWE). The NICE Decision Support Unit (DSU) currently defines RWD and RWE as:
RWD is a commonly used term to describe data generated from sources that relate to everyday clinical practice, generally outside the artificial constraints of randomized controlled trials. In its broad definition, RWD can include data generated as part of pragmatic controlled trials, however most RWD does not produce randomized evidence of treatment effect. In the context of Health Technology Assessment (HTA), RWD typically presents as observational data from registries, administrative databases and surveys. Bell et al. (Reference Bell, Wailoo, Hernandez, Grieve, Faria and Gibson4).
RWE has the potential to provide useful information regarding the use and/or outcomes of a given treatment within a setting most relevant to UK routine practice or to fill in data gaps not addressed by clinical trials (e.g. informing the underlying nature of a disease or long-term disease outcomes). However, a number of potential challenges in the collection of RWE data in NICE appraisals have been identified including difficulties in collecting baseline health-related quality of life (HRQoL) data and patient consent, securing adequate funding to maintain a database and support staff, information technology issues, and a lack of a clear control group. In spite of these challenges, over time RWE has gained increasing use to inform decision making for reimbursement authorities (Reference Bell, Wailoo, Hernandez, Grieve, Faria and Gibson4).
Study Objectives
In light of the increasing acceptance and profile of RWE to support decision making by NICE, this study aims to establish how RWE has been used to inform NICE appraisals of cancer drugs. We conducted a review of NICE cancer single technology appraisals (STAs) to establish where RWE played a pivotal role in the decision-making process. To explore RWE more thoroughly, we identify exemplar cancer STAs for which RWE was integrated appropriately, and conversely STAs for which RWE were provided and subject to negative criticism throughout the NICE process by the independent Evidence Review Group (ERG) and NICE Appraisal Committee (henceforth termed “the Committee’).
Methods
Identification of Appraisals
Searches were conducted in October 2018 to capture all relevant STAs published from April 2011 until the search date. April 2011, the data of introduction for the CDF, was the chosen cut-off for the review as the CDF actively encouraged the collection of RWE to support NICE decision making. The STAs were identified by viewing guidance by topic on the NICE Web site, as well as downloading the relevant material to check for eligibility.
Eligibility Criteria
Completed STAs published by NICE from April 2011 to October 2018 that evaluated pharmacological interventions in populations with cancer were included within the review. Appraisals for noncancer populations were excluded, as well as any terminated appraisals. Multiple technology appraisals (MTAs) were excluded as the MTA process does not require companies to submit evidence (though they may choose to do so). Between April 2011 and October 2018, NICE published 39 MTAs across all disease areas versus 288 STAs (correct as of October 2018). Therefore, while some relevant information may be reported within MTA documentation, these appraisals were excluded in the interest of comparing interventions assessed under the same NICE appraisal process. In addition, STAs focusing on interventions related to complications of cancer (as opposed to treatment of cancer specifically) were also excluded. Two reviewers assessed the eligibility of STAs and cross-checked results before deciding on an agreed list of relevant studies.
Data Extraction
A data extraction form was developed prior to the identification of relevant STAs. The data extraction form was designed to capture data regarding the use of RWE to inform cost-effectiveness modeling. The focus of this study was placed on RWE used in the company model, given that these inputs will feature in the calculation of the ICER, and therefore ultimately have an impact (however small this may be) on decision making. Categories of RWE use were extracted to establish which aspects of submitted cost-effectiveness analyses are most frequently supported using RWE—for example, use of RWE to support the quantification of costs, patient outcomes, HRQoL, and so on. Other data extracted included the key source(s) of clinical effectiveness, economic model type, and broad comments raised by relevant parties on the use of RWE. Missing documents (such as earlier iterations of appraisals that were superseded) were requested from NICE via email. A second reviewer checked 20 percent of extracted STAs.
For the purpose of this study, RWE was defined as follows: “analysis of data regarding the health state and/or delivery of health care collected in a non-interventional trial setting,” which while simplified is broadly aligned with published definitions (4;5). The focus of this study was placed upon the use of RWE to directly inform the company-submitted cost-effectiveness analysis; therefore, evidence used to support the interpretation and/or perception of the cost-effectiveness results (but not used to inform them) was not considered. Analysis of data collected from national databases that were not specific to a particular clinical indication, and are used as standard (e.g. national statistics, NHS reference costs, etc.), was not deemed as relevant RWE.
Synthesis
Across all identified STAs, a summary of the use of RWE was obtained via the response to the following questions: (1) did the company provide RWE to support the cost-effectiveness analysis submitted? (2) if so, was this provided upfront or in response to a criticism raised throughout the NICE process? and (3) was the use of RWE submitted by the company ultimately rejected by the Committee? Where an explicit rejection of the use of RWE was not provided, it was assumed that the RWE use was accepted.
STA case studies were identified through the data extraction and discussion with the research team. Case studies were selected where the use of RWE played a substantial role within the cost-effectiveness analysis, and was subject to discussion as part of the NICE process. The final collection of case studies was designed to cover a range of different treatments, disease areas, RWE use, and timeframe of assessment. Review findings for the identified case studies are presented as a narrative summary.
Results
Identified NICE Appraisals
From April 2011 to October 2018, NICE published a total of 288 TAs. Of these, a total of 114 were completed (i.e. nonterminated) STAs that evaluated pharmacological interventions in populations with cancer. Submission dossier material for n = 113 STAs was available from either the NICE Web site or following a request sent to NICE. Materials for the missing STA (TA457, carfilzomib for previously treated multiple myeloma) could not be provided by NICE at the time of this review due to a potential breach of confidence (under Section 41 of the Data Protection Act 2018) (6). The identification of relevant NICE STAs is presented as a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) diagram in Figure 1 (Reference Moher, Liberati, Tetzlaff and Altman7).
Figure 1. PRISMA diagram of included NICE STAs. NICE, National Institute for Health and Care Excellence; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; STA, single technology appraisal.
Generalized Summary of RWE Use
Nearly all assessments (n = 108 of 113, 96 percent) incorporated some degree of RWE to inform the cost-effectiveness analysis as part of the STA process (provided in the company's original submission dossier). We identified only ten STAs wherein RWE was found to be provided solely as a direct response to an ERG or Committee request or criticism; however, in practice, this number acts as a lower bound as it is possible that RWE may have been provided in response to documentation not available in the public domain (e.g. following the decision problem meeting held as part of the NICE STA process).
A summary of the type of RWE provided as part of the STA process is presented in Figure 2. The most common types of RWE presented include analysis relating to the HRQoL of patients (“Utilities,” n = 80, 71 percent), the costs used to populate the model (“Costs,” n = 52, 46 percent), and the quantification of medical resource utilization (“Resource use,” n = 45, 40 percent). Real-world utility data were primarily sourced from published studies (n = 72), either related to health state utility values, adverse event-related utility losses, or age-adjustments incorporated within the company's economic model. In a small number of STAs (n = 8), utilities were obtained from unpublished data (e.g. company-led data collection), predominantly to inform health state utility values. The most common cost item cited from RWE sources was end-of-life care. Resource use frequencies were typically obtained via chart reviews, registry databases, or published literature.
Figure 2. Use of real-world evidence by category. Admin, administration; AE, adverse event; RWE, real-world evidence; STA, single technology appraisal.
Unsurprisingly, there were several instances where specific source(s) and/or application(s) of RWE were subject to critique by either the ERG, Committee, or both. For example, the relevance of an identified RWE source to the UK population may be questionable, or the choice of statistical model applied to the RWD may be subject to criticism. The phrasing within the final recommendation for each STA precluded a formal analysis of RWE source or application acceptance. This is because it is relatively commonplace for the Committee to consider “a range of scenarios” to inform decision making. To illustrate where the use of RWE has been disputed in previous STAs conducted by NICE, case studies are discussed later in the article.
In only two STAs did the ERG outrightly reject the use of RWE to inform the estimation of an intervention's cost-effectiveness—in both instances, the Committee agreed that RWE presented was inappropriate to inform the cost-effectiveness analysis. The first of these was TA523 (midostaurin for untreated acute myeloid leukemia) in which the company used matching to a historical control (observational data) to inform overall survival estimates in the economic model, though the pivotal trial for midostaurin was a randomized controlled trial. The ERG commented that as the historical control study was nonrandomized, it was susceptible to bias, and as life expectancy had increased in recent years, data from historical controls were less relevant. The trial-based economic model was preferred by the ERG and the Committee.
The second instance of RWE rejection was in TA370 (bortezomib for previously untreated mantle cell lymphoma), where the company provided end-of-life costs based on a sample of forty patients. The ERG opted to remove this cost from the economic model entirely as it was considered to be taken from a small and heterogeneous population, and included costs for not-for-profit and nongovernmental healthcare organizations (beyond the scope of costs considered by NICE). The Committee agreed with the removal of these costs.
Case Studies
Four case studies were identified to highlight a range of RWE usage in company submissions to NICE. The case study STAs were chosen to reflect a number of different types of cancer, submitting companies, dates of assessment, nature of RWE use, and perception by the ERG and Committee. Table 1 summarizes the included case studies.
Table 1. Case Studies of RWE Use in NICE Appraisals
CDF, Cancer Drugs Fund; ERG, Evidence Review Group; FAD, Final Appraisal Determination; NICE, National Institute for Health and Care Excellence; RWE, real-world evidence; TA, technology appraisal.
Discussion
RWE has been used in a multitude of applications to inform healthcare decision making by NICE, particularly in the field of oncology where clinical trials often do not provide all evidence required to inform cost-effectiveness analysis (though these issues are by no means unique to studies conducted in cancer). The CDF was introduced to facilitate RWD collection to reduce uncertainty and inform decision making, thus the focus of our review was to consider how RWE has factored into NICE appraisals of cancer treatments. This study illustrates how RWE has been used by companies to support the submission of cost-effectiveness evidence to NICE and highlights typical uses via the presentation of case studies. Compared with clinical trial data, RWE may be considered a better representation of local patient demographics, more reflective of clinical practice, and potentially be closer aligned with contemporary treatment patterns.
The key criticisms around RWE sources are typically concerned with the comparability of real-world patient cohorts to the clinical trial patients, and how relevant RWE patient cohorts are to contemporary practice (4;18). This is in addition to the commonly recognized concerns around bias when interpreting evidence collected outside of a controlled (trial) setting. We only identified two STAs where RWE was rejected in its entirety (i.e. both the source and approach were considered inappropriate for decision making). Instead, it was much more commonplace for the ERG and/or Committee to dispute the specific source(s) or application(s) of RWE provided (but not both simultaneously). Nevertheless, RWE has been shown to have a clear role in decision making, by addressing data gaps in cost-effectiveness analyses submitted by companies, and ultimately being used to inform the decision made (through impacting the estimate of the ICER).
Previous studies regarding RWE use to support NICE decision making have either considered a holistic view of the STA process, or have adopted a relatively narrow perspective as to what constitutes RWE (4;18–21). One of these studies is the NICE DSU report (Reference Bell, Wailoo, Hernandez, Grieve, Faria and Gibson4) which presents a comprehensive overview of RWE use in submissions to NICE across a range of therapeutic areas; including available NICE method manuals, methodological techniques to analyze RWD, and opinions on the application of RWE to the STA process from NICE personnel (Reference Bell, Wailoo, Hernandez, Grieve, Faria and Gibson4). The DSU report provides a useful source of information concerning the general use of RWE, yet is not specific to the issues that typically afflict cancer trials (only one cancer case study is noted within the DSU report: TA401, bosutinib for previously treated chronic myeloid leukemia) (Reference Bell, Wailoo, Hernandez, Grieve, Faria and Gibson4). The DSU report calls for the development of guidelines on the application of appropriate statistical methods to RWE, and highlights that current guidance across NICE programs is somewhat misaligned in relation to the discussion of RWE.
Makady et al. (Reference Makady, van Veelen, Jonsson, Moseley, D'Andon and de Boer21) considered the use of RWE to inform a range of HTA reports across five European agencies within the context of treatments for patients with melanoma. The authors of this study found that RWE had been predominantly used to estimate epidemiological information (such as prevalence), long-term effectiveness (i.e. survival), and quantify drug-related costs, though these findings were based on a limited number of melanoma-based case studies.
In our study, NICE STAs of cancer treatments were systematically identified and carefully searched for the specific use(s) of RWE to inform the submitted cost-effectiveness analysis. A large number of relevant STAs were identified, and several of these were subject to multiple rounds of evidence submission, meetings, and (where relevant) appeals. The sheer volume of materials necessitated a degree of interpretation as to which opinions may be taken to be reflective of the Committee's final decision (e.g. where a preliminary recommendation discussed a topic which was not included within the final recommendation). The need to differentiate between opinions that may be considered “interim” or “final” is not specific to the use of RWE, but is notably relevant to our study as perceptions of RWE may have changed over time.
It should be noted that the definition of RWD (and consequently, RWE) differs across studies, and so the findings of our review should be interpreted with this in mind. Some studies consider data from single-arm clinical trials as RWD, though this does not align with the majority of published definitions (4;5). It is the view of the authors that RWE should not be collected within a controlled, clinical trial setting—if evidence collected in this way were considered RWE, the same could be said by looking at results from individual arms of controlled trials. Our review does not consider the interpretation of different evidence types (of which RWE is one specific type), and so some of the findings may indeed apply to other types of evidence. In addition, our study relies upon accurate reporting in NICE appraisal documentation, and so a degree of subjectivity with regards to determining RWE sources was required.
In our study, we focused on the use of RWE to inform the company's cost-effectiveness analysis only, and in doing so were able to illustrate how this evidence has informed NICE appraisals of cancer drugs. Through reviewing published NICE appraisals, we intended to illustrate how RWE has been used to inform contemporary HTA decision making. However, the use of RWE may feature more broadly to contextualize and/or lend support to the findings of the economic evaluation. It is often the case that companies provide or refer to RWE to support the claim that a given intervention meets NICE's end-of-life criteria (i.e. that survival expectancy with current care is <2 years). Use of RWE in this manner does not influence the model results per se, but may have a profound influence on the interpretation of analyses, and thus impact the final decision made by the Committee. Further research is needed to explore more broad usage of RWE in submissions to NICE outside of the submitted economic evaluation.
A further limitation of this study was the reliance upon published materials. Materials were requested from NICE, but it was only possible to review those that were obtainable by the authors. For instance, it was not possible to review any RWE obtained via the CDF that was not presented in the public domain or any data provided to the ERG that were unreported or heavily redacted; as well as information for the missing STA (TA457, which NICE was unable to share). Furthermore, the determination of RWE sources is subject to the description of data collection provided within the relevant studies—some of which were unpublished and therefore it could not be ascertained with absolute certainty whether the sources aligned with the definition of RWE used in our study. The NICE STA process is recognized as one of the most transparent in the field of HTA, yet the confidentiality and/or opacity of some information submitted as part of the STA process precludes a fully exhaustive review of all evidence presented to the Committee, and thus establishes with greater certainty the type(s) and/or quality of data considered acceptable.
Our review provides an overview of the key criticisms raised by NICE that should be addressed prior to submission—for example, providing a clear justification of the similarities between the pivotal trial population and patients considered within the RWE. By addressing RWE criticisms commonly raised throughout the STA process upfront, the ability for the Committee to make decisions in a timely manner may be improved, thus limiting the extent to which patient access may be delayed. While our study focuses on submissions for cancer drugs, the findings of our study may be useful more broadly wherever RWE is utilized. The development of HTA-focused guidance regarding the appropriate use of RWE may also improve the usefulness of the evidence submission by companies to the Committee. Through improved presentation of evidence, a clearer understanding of common themes regarding the acceptance and rejection of RWE may be ascertained. Such guidance may be improved through collaboration with regulatory bodies for whom guidelines have previously been developed (Reference Klonoff5).
In June 2019, NICE announced its “statement of intent” concerning the potential for an increasing use of health and social care data to inform guidance development (22). This document explains NICE's wish to expand its methods and processes to enable more extensive and effective use of broader sources of data, such as RWE, to inform its decision making. Outside of NICE, various other initiatives have been developed to incorporate the use of RWE into decision making. For example, in Australia, the melanoma drug ipilimumab (Yervoy®, Bristol Myers-Squibb) was conditionally recommended by the Pharmaceutical Benefits Advisory Committee (PBAC) provided 2-year overall survival outcomes observed in practice aligned with the pivotal clinical trial (Reference Kim, Comey, Hausler and Cook23). From 2018, the Scottish Medicines Consortium (SMC) are now able to make interim decisions for medicines that have been given a conditional marketing authorization by the European Medicines Agency (EMA), subject to ongoing evaluation and future reassessment (24). The future role of RWE in HTA decision making relating to cancer drugs, both in the UK and internationally, is expected to increase over time based on relatively recent changes to the mechanisms under which interventions are able to be assessed and recommended for practice. This research may serve as a useful source of information for those involved in establishing how RWE may be used in HTA assessments.
In conclusion, our review found the use of RWE in NICE submissions of cancer drugs to be extensive, and in general appears to have provided a valuable source of information to aid decision making. Common criticisms of RWE were related to specific sources or analytical methods, though the use of RWE in general was rarely rejected in its entirety. Where possible we recommend that submissions to NICE should aim to proactively acknowledge these common criticisms, through clear justification of the approaches taken to analyze RWE and the relevance of the RWD source, which may be aided by the development of best practice guidelines for reporting standards of RWD and its application as RWE.
Financial Support
Takeda Pharmaceutical Company (authors TP, MR, EB, and RS) provided research funds for Delta Hat (authors AB and AJH) and Azurite Research (author GES) to undertake some of the research described within the article.
Conflict of Interest
TP, MR, EB, and RS are employees of Takeda Pharmaceutical Company. Delta Hat (authors AB and AJH) have previously provided consultancy services to Takeda Pharmaceutical Company outside of the research described within this article. The views expressed within this article are those of the author(s) and not necessarily those of the organizations to which individuals are affiliated.
Author Contributions
TP, MR, EB, and RS conceived and designed the analysis presented in this article. AB and GES collected the data and performed the quantitative analysis presented. AB and AJH drafted the initial manuscript. All authors reviewed, edited, and approved each subsequent draft of the manuscript.
Open access
