Introduction
Surrogate endpoints have become an integral part of the development of novel health technologies and in particular in oncology. Before a new cancer therapy can reach patients, it must first be approved by regulatory agencies, such as the European Medicines Agency in the European Union or the Food and Drug Administration in the United States. Traditionally, regulatory approvals were based on the data from randomized controlled trials (RCTs) demonstrating the treatment effect of the new health technology on overall survival (OS), which was considered the most objective and clinically meaningful outcome of most importance to patients as well as policy makers. However, increasingly regulatory agencies have been carrying out accelerated approvals of health technologies based on the treatment effects observed on surrogate endpoints. Surrogate endpoints are intermediate outcomes, which can substitute for and predict final clinical outcomes of interest such as mortality (Reference Ciani, Buyse, Drummond, Rasi, Saad and Taylor1–Reference Robb, McInnes and Califf6). The use of surrogate endpoints as primary outcomes in clinical trials can result in smaller, cheaper, and shorter studies compared to trials, which are designed to evaluate treatment effectiveness on the final clinical outcome such as OS.
Although the use of surrogate endpoints accelerates the evaluation of therapies in clinical trials and regulatory approvals, in many countries patient access to the new treatment also depends on subsequent scrutiny by health technology assessment (HTA) agencies such as the National Institute for Health and Care Excellence (NICE) who provide reimbursement recommendations for England and Wales. While regulatory decisions are based on the safety and efficacy of the product, HTA agencies are primarily concerned with the long-term clinical and cost-effectiveness of the new health technology in order to ensure that scarce healthcare resources are used efficiently. Therefore, the increasing reliance on surrogate endpoints for regulatory approvals creates problems for HTA agencies as long-term estimates of clinical and cost-effectiveness are unavailable at the time of HTA submissions that closely follow regulatory approvals. As such, HTA agencies generally express a strong preference for final clinical outcomes over surrogate endpoints and where these are not available require that surrogate endpoints are appropriately validated.
Over recent years, there has been a wealth of literature published regarding the prolific use of surrogate endpoints in HTA and in particular in oncology indications (Reference Kemp and Prasad7–Reference Ciani, Grigore, Blommestein, de Groot, Möllenkamp, Rabbe, Daubner-Bendes and Taylor9). There has also been a significant amount of criticism regarding such use of surrogate endpoints and calling for more transparent frameworks and harmonization of guidelines across HTA agencies (Reference Ciani, Grigore, Blommestein, de Groot, Möllenkamp, Rabbe, Daubner-Bendes and Taylor9;Reference Dawoud, Naci, Ciani and Bujkiewicz8;Reference Grigore, Ciani, Dams, Federici, de Groot, Möllenkamp, Rabbe, Shatrov, Zemplenyi and Taylor10). A plethora of statistical methods have been developed to ensure robust evaluation of surrogate endpoints (Reference Ensor, Lee, Sudlow and Weir11), including NICE Decision Support Unit’s technical support document 20, which describes methods for multivariate meta-analysis to evaluate surrogate endpoints (Reference Bujkiewicz, Achana, Papanikos, Riley and Abrams12). NICE have also updated their methodology guide to ensure improved evaluation of surrogate endpoints in submissions to their technology appraisals (TAs). While available meta-analytic methods serve well the purpose of surrogate endpoint validation and predicting the relative treatment effect on the final clinical outcome (which can be used in decision modeling frameworks), they do not capture all aspects of the role a surrogate endpoint can play in the decision modeling. A methodological review, carried out by the NICE Decision Support Unit in preparation for the update of NICE methodology guide, recommended a review of past TAs to fully understand how surrogate endpoints have been used in decision making by NICE to inform further methodological considerations and identify most optimal modeling approaches utilizing surrogate endpoints (Reference Welton, Phillippo, Owen, Jones, Dias, Bujkiewicz, Ades and Abrams13). With the context of such recent developments in the field of surrogate endpoints in the HTA setting, the aim of this project was to review and assess how surrogate endpoints have recently been used in NICE oncology TAs. Specifically, we are interested in how often surrogate endpoints have been used, the justification for their use, the level of evidence provided to support the use of the surrogate endpoint and in what way they informed health economic modeling.
Methods
Documents relating to NICE TAs are published on the NICE website. We retrieved documents from all NICE TAs for cancer drugs evaluated between February 1, 2022 and May 3, 2023 from the NICE Guidelines website. February 1, 2022 was chosen to coincide with the NICE methods guide update that included updated guidelines on how surrogate endpoints should be used in NICE TAs (14). Terminated, withdrawn or replaced appraisals were removed, as documentation was unavailable. A quality appraisal of the extracted TAs was not conducted as every TA conducted by NICE is subject to review by an evidence review group (ERG) who assess the clinical and cost-effectiveness analyses submitted by the company. An overview of the workflow can be observed in Figure 1.
Flowchart for inclusion of technology appraisals.
In the first stage, we adapted computer script from Polak et al. (Reference Polak, Cucchi, van Rosmalen and Uyl-de Groot15) to automatically list and download all documentation (e.g., manufacturer submissions, ERG reports and final appraisal determinations) available through the NICE website. For all TAs with documents available, the following general information was extracted by LW from either the company submission, ERG report, or final appraisal decision documents:
-
• Appraisal ID
-
• Appraisal date
-
• Treatment
-
• Disease area
-
• Primary endpoint(s) of pivotal trial
-
• Secondary endpoint(s) of pivotal trial
-
• Recommendation
In the second stage, these documents were screened by LW to assess whether they contained either of the terms “surrogate” or “surrogacy.” If either of these terms were present, the context of the term was reviewed to make an initial assessment of whether surrogate endpoints were utilized in either the clinical or cost-effectiveness analysis.
Following this initial assessment, in the third and final stages, the clinical and cost-effectiveness sections of the committee papers were reviewed to further assess whether a surrogate endpoint was utilized. For studies that included discussion of surrogate endpoints, it was important to understand not only which surrogate endpoints were used, but also if the surrogate endpoint was validated. Validation of a surrogate endpoint requires evidence from the literature that achieving a treatment effect on the surrogate endpoint will reliably predict a clinically meaningful treatment effect on the final clinical outcome (Reference Fleming and Powers16). Ciani et al. (Reference Ciani, Buyse, Drummond, Rasi, Saad and Taylor2) suggest a three-stage approach for validating surrogate endpoints for use in healthcare decision making. The first stage is to establish the level of evidence available for the surrogate. The authors suggest that there are three levels of evidence where the lowest level is biological plausibility of the surrogate relationship, the second level is an association between the surrogate endpoint and final outcome, and the highest level is where the treatment effect on the surrogate endpoint is associated with the treatment effect on the final outcome. Once the level of evidence is established, the second step is to assess the strength of association between the surrogate endpoint and final outcome and the final step is to quantify the relationship between the treatment effects on the surrogate endpoint and the final outcome. Based on this framework, the following information was extracted from either the company submission or ERG report:
-
• what was the surrogate endpoint
-
• what was the final clinical outcome
-
• what was the level of evidence for surrogate relationship
-
• whether association between surrogate endpoint and final outcome was investigated and method used or measure of such association
-
• whether association between the treatment effects on the surrogate endpoint and on the final outcome was investigated and method or measure of such association used
-
• whether predictions of the treatment effect on the final outcome from the effect on the surrogate endpoint were considered, including methods used.
We also assessed whether and how information on the surrogate endpoint was utilized in the health economic evaluation.
Results
A total of 65 TAs in oncology were published on the NICE website between February 1, 2022 and May 3, 2023. Of these 65 TAs, 16 appraisals did not provide a submission and 2 appraisals did not provide committee papers for review. The remaining 47 appraisals had 663 documents, which were downloaded and screened.
Ninety-four percent of appraisals (44/47), included at least one mention of the terms “surrogate” or “surrogacy.” However, initial assessment of the context of the terms found that for 36 percent of appraisals (16/44), there was no meaningful discussion of surrogate endpoints. For example, in 15 TAs, the word “surrogate” was included as part of a question in the clinical expert statement or professional organization submission asking, “if surrogate outcome measures were used, do they adequately predict long-term clinical outcomes.” While the majority of these TAs did not provide a response to this question, several included surrogacy terms by making statements, such as “no surrogate outcome measures were used” (TA819) (17).
The exclusion of three TAs for failing to contain any terms relating to surrogacy and a further 16 TAs for failing to contain any meaningful discussion of surrogate endpoints resulted in 28 TAs for a more in depth review (17;48;48–49;53–58;61–63;65;66; 69;70;72). Upon reviewing the clinical and cost-effectiveness sections of the committee papers for these TAs, a further 10 appraisals were excluded for not including surrogate endpoints in the technology assessment (51;52;59;60;64;67;68;71;73;74). This left 18 appraisals that discussed the use of surrogate endpoints and were included for full data extraction (18–35). See supplementary material for further information on the data extracted.
Of the 49 appraisals where there was a submission, 14 (29 percent) appraisals were in indications for blood and bone marrow cancers (including large B-cell lymphoma, multiple myeloma, Waldenstrom’s macroglobulinemia (WM), acute myeloid leukemia, chronic myeloid leukemia, chronic lymphocytic leukemia (CLL), and Hodgkin’s lymphoma) and the remaining 35 (71 percent) of appraisals were in solid tumors (including lung cancer, breast cancer, renal cancer, skin cancer, metastases, and bladder cancer). For the 49 appraisals that were fully reviewed, the health technology being appraised was recommended for use in the NHS in 40/49 (82 percent) of appraisals and recommended for use in the cancer drugs fund (CDF) in 4/49 (8 percent) of appraisals. Table 1 summarizes the characteristics of those 18 TAs where use of the surrogate endpoints was discussed explicitly. Of those 18 appraisals utilizing surrogate endpoints the health technology being appraised was recommended for use in the NHS in 15/18 (83 percent) of appraisals and recommended for use in the CDF in 2/18 (11 percent) of appraisals.
Characteristics of NICE technology appraisals using surrogate endpoints
TA, technology appraisal; NSCLC, non-small cell lung cancer; PFS, progression-free survival; pCR, pathologic complete response; EFS, event-free survival; RFS, recurrence-free survival; DFS, disease-free survival; VGPR, very good partial response; CR, complete response; MMR, major molecular response; CyR, cytogenic response; CCyR, complete cytogenic response; ToT, time on treatment; IDFS, invasive disease free survival; DRFS, distant relapse free survival; MRD, minimal residual disease; sCR, stringent complete response; OS, overall survival; DMFS, distant metastasis free survival; TAR, Technology Appraisal Recommendation; CDF, Cancer Drugs Fund; STM, state-transition model; PSM, partitioned survival model; SACT, systemic anticancer therapy data set.
Rationale for and scope of use of surrogate endpoints and indications
Surrogate endpoints were used in either the clinical-effectiveness or cost-effectiveness analyses of 18 TAs (Table 1). In the majority of appraisals (83 percent), it was stated that use of surrogate endpoints was required due to “immature OS data,” indicating that surrogate endpoints are primarily used when trials are ongoing, resulting in limited evidence on the final clinical outcome.
There were three TAs where “immature OS data” were not given as a reason for the use of surrogate endpoints. In two appraisals (TA795 (31) and TA796 (30)), time on treatment (ToT) was used as a proxy, or surrogate endpoint, for progression-free survival (PFS), as data on PFS were unavailable from the primary source of evidence for the submission. In the CDF appraisal TA784 (32), mature data on OS were collected from a pivotal trial conducted between the original submission and updated CDF review. However, despite the availability of mature OS data, TA784 utilizes PFS as a surrogate endpoint for OS stating that mature OS data were difficult to interpret due to high levels of missing data and crossover between treatment arms.
TAs utilizing surrogate endpoints included many cancer types and the majority of indications (61 percent) were in solid tumors including breast cancer, lung cancer, skin cancer, cervical cancer, renal cancer, bladder cancer and ovarian cancer. The remaining 39 percent of indications were in blood and bone marrow cancers including WM, large B-cell lymphoma, chronic myeloid leukemia, Hodgkin’s lymphoma, multiple myeloma, and CLL.
Types of outcome measures used as surrogate endpoints
Ten of the eighteen (56 percent) TAs discussed the use of a single surrogate endpoint (e.g., PFS as a surrogate endpoint for OS). However, the remaining eight appraisals considered more than one surrogate endpoint in a single appraisal. For example TA876 (19), evaluating nivolumab with chemotherapy for the treatment of non-small cell lung cancer, discussed pathological complete response (pCR) as a surrogate endpoint to event-free survival (EFS) and OS in addition to EFS as a surrogate endpoint to OS. In total, 37 surrogate endpoints were considered in the 18 TAs.
The most frequently considered surrogate endpoint was PFS, which was used in the evaluation of 5 (28 percent) technologies (pembrolizumab in two indications, polatuzumab vedotin, trastuzumab deruxtecan, and niraparib). EFS and disease-free survival (DFS) were each used in the evaluation of 3 (17 percent) technologies (EFS: nivolumab, polatuzumab vedotin, and pembrolizumab; DFS: pembrolizumab, atezolizumab, and nivolumab). Recurrence-free survival (RFS) and pCR were each used to evaluate two (11 percent) technologies (RFS: pembrolizumab in two indications; pCR: nivolumab and pembrolizumab). ToT was used as a surrogate endpoint to PFS in two appraisals (venetoclax and ibrutinib). Other surrogate endpoints utilized in the TAs were very good partial response/complete response, major molecular response rate, cytogenic response (CyR), complete CyR, invasive DFS, distant RFS, minimal residual disease (MRD), and stringent complete response.
The level of evidence provided to validate surrogate endpoints
According to Ciani et al. (Reference Ciani, Buyse, Drummond, Rasi, Saad and Taylor2), the first stage in assessing the validity of a surrogate endpoint is to evaluate the level of evidence provided for the surrogate relationship, where the lowest level of evidence is a biologically plausible relationship, the second level of evidence is a consistent association between the surrogate endpoint and the final outcome observed in observational studies and the highest level of evidence is a relationship between the treatment effects on the surrogate endpoint and final outcome observed in randomized controlled trials.
There were 37 separate surrogate endpoints discussed in the 18 TAs using surrogate endpoints. For 11 (30 percent) (19;21;23;26;27;34;35) of these surrogate endpoints, evidence from RCTs was provided to validate the relationship. TAs typically cited a systematic review and meta-analysis of these trials or studies. For 7 (19 percent) (32;28;31;34) of putative surrogate endpoints, evidence from observational studies was cited (see Section 3.4) and a further 12 (32 percent) (18;22;23;24;28;29;32;33) stated that the use of surrogate endpoints was based on the clinical opinion. Seven (19 percent) (25;28;30) of the surrogate endpoints discussed did not provide any evidence supporting their use.
Association between the surrogate endpoint and the final clinical outcome
Following the recommendations by Ciani et al. (Reference Ciani, Buyse, Drummond, Rasi, Saad and Taylor2), the second stage in assessing the validity of a surrogate endpoint is to evaluate the strength of association between the surrogate endpoint and final outcome. The strength of association is often evaluated through measures, such as the individual-level correlation and
$ {R}^2 $
. When assessing this association, it would be preferable to obtain individual participant data from RCTs in order to estimate the association at the trial and individual levels. However, such data are not always available.
For the 37 putative surrogate endpoints, a measure of association was explicitly reported in the TAs for 6 surrogate endpoints. Three of these were obtained from TA876 (19) investigating pCR as a surrogate endpoint for EFS and OS and EFS as a surrogate for OS. For the surrogate relationship between pCR and EFS/OS, the TA states that “patients who have a pCR have improved EFS and OS outcomes versus patients who do not achieve a pCR.” The TA then goes on to state that patient-level EFS by pCR has an HR = 0.49 (95 percent CI: 0.41, 0.60) and that patient-level OS by pCR has an HR = 0.49 (95 percent CI: 0.42, 0.57). For the surrogate relationship between EFS and OS, the TA states that there is a weighted Pearson’s correlation coefficient of r=0.819 (95 percent CI: 0.73, 0.92) between median EFS and median OS and a positive linear correlation of 0.864 (95 percent CI: 0.81, 0.99) between logHRs on EFS and OS. Furthermore, the TA provided an estimate of trial-level association in the form of
$ {R}^2 $
value of
$ 0.777 $
obtained from a random-effects meta-regression. Additional measures of association were reported in this TA; however, some measures could not be extracted due to censoring of the committee papers.
However, not all TAs referenced specific association measures between surrogate endpoints and final clinical outcomes. For those surrogate endpoints where no measure of association was reported, this was often due to lack of evidence available to support the surrogate relationship. For example, some TAs were unable to provide a measure of association, as use of the surrogate endpoint was based on the clinical opinion alone with no evidence from observational studies or RCTs. However, several TAs made statements, such as “RFS appears to be a valid surrogate endpoint for OS” (TA766 (35)) and cited a paper describing this relationship. While some of the papers cited provided a measure of association between the surrogate endpoint and final outcome (e.g., the paper cited in TA766 reported a correlation of 0.89 between RFS and OS), others simply cited trial reports which provided no measure of association between the surrogate endpoint and final outcome (TA817 (27)).
The relationship between the treatment effects on the surrogate endpoint and final outcome
Continuing to follow the recommendations set out by Ciani et al. (Reference Ciani, Buyse, Drummond, Rasi, Saad and Taylor2), the final stage of assessing the validity of a surrogate endpoint is to evaluate the relationship between the surrogate endpoint and final outcome, and in particular between the treatment effects on the two endpoints. This typically involves estimating (predicting) the expected treatment effect on the final clinical outcome and can include estimating the surrogate threshold effect (STE), which is the magnitude of treatment effect on the surrogate endpoint which would predict a statistically significant/clinically meaningful treatment effect on the final outcome.
Of the 37 surrogate endpoints evaluated in 18 TAs, the relationship between the surrogate endpoint and final outcome was only quantified within the TAs in 2 cases. In the committee papers for TA862 (21), it is stated that “a difference in median PFS of 5, 10, 15, and 20 months between an intervention and a comparator would be expected to translate into approximately 8.7, 17.4, 26.2, and 35.0 months’ additional median OS for the intervention.” The company goes on to say that “the 17.9-month increase in median PFS … is expected to translate into a clinically significant OS advantage.” In the second instance, in TA553 (original appraisal for CDF appraisal TA766 (35)), an STE of 0.77 is reported, indicating that for studies with an HR
$ \le 0.77 $
for RFS, there is a meaningful treatment benefit predicted on OS. In TA553, the HR on RFS is 0.57, supporting the argument for a treatment effect on OS.
While only two putative surrogate endpoints had the prediction explicitly reported in the TA, several more make references to papers which conduct STE analysis. For example, TA823 (26) references a paper by Mauguen et al. (Reference Mauguen, Pignon, Burdett, Domerg, Fisher, Paulus, Mandrekar, Belani, Shepherd, Eisen and Pang36) which states an STE of 0.88 and suggests that “cross-validation results confirmed the accurate prediction of the treatment effect on OS based on the effects observed on DFS.”
Use of nonrandomized evidence
Traditionally, reimbursement decisions would be made based on the data from a pivotal phase three RCTs. Of the 47 TAs reviewed, 35 (74 percent) were supported by data from a pivotal phase 3 RCT. However, 10 TAs were supported by either a phase 1 or phase 2 study and 2 TAs were supported by real-world evidence (RWE).
The two TAs, which were supported by RWE, were CDF appraisals TA795 (31) of Ibrutinib monotherapy for treatment of WM and TA796 (30) of Venetoclax for treating CLL. These reviews both use the systemic anticancer therapy (SACT) data set to support their submission. SACT is a data set, which collects information on anticancer therapies used across all NHS England Trusts, including those approved for use in the CDF. SACT was chosen as the primary source of evidence for these TAs, as it was deemed more generalizable to practice in NHS England. The limitation of SACT is that it typically does not record information on PFS and thus cannot be used to inform clinical or cost-effectiveness estimates. However, ToT is collected within SACT and in both TA795 and TA796; ToT was used as a surrogate, or proxy, to PFS.
Of the 10 TAs that were supported by phase 1 or 2 trials, 8 of these were also single-arm trials, including 3 first-in-human phase 1 single-arm trials. The use of single-arm trials in HTA poses a problem for assessment of clinical and cost-effectiveness, as there is no comparator arm to use as a baseline to estimate the relative treatment effects. Instead, studies that utilize single-arm trials as the pivotal study supporting the submission must use external evidence, such as historical trials or RWE to estimate the relative treatment effects. While none of the TAs reviewed contained both use of surrogate endpoints and a single-arm trial, the increasing use of both means that it is likely that in the future, we will see more submissions to NICE, which are based on single-arm trials investigating the treatment effect on a surrogate endpoint.
Use of surrogate endpoints in cost-effectiveness analysis
An economic model was presented for each of the 18 TAs using surrogate endpoints. The most commonly used economic models were state-transition models (STMs), which were used in 10 (56 percent) of the appraisals and partitioned survival models (PSMs), which were used in 5 (27 percent) of the appraisals. The majority of appraisals justified the use of STMs by stating that such models are better able to model the disease pathway. However, some also stated that in scenarios where survival data are immature, STMs are preferable to PSMs as OS can be modeled as a function of all other health states in the model rather than requiring extrapolation from immature data. Appraisals utilizing the partitioned survival approach justified using this method as it (a) directly utilizes survival data from pivotal trials, (b) is intuitive, and (c) is frequently used in oncology appraisals. Other economic models utilized were a cumulative survival model (TA813 (28)), surrogate survival model (TA813), means-based model (TA784 (32)), and response-based model (TA763 (34)).
For the five TAs that utilized PSMs (TA874 (20), TA862 (21), TA833 (37), TA796 (30), and TA772 (33)), all appraisals employed a PSM with three states of “progression free,” “progressed disease” and “dead.” In a PSM with these states, the value of each health state is determined using time-to-event data represented by the survival curves for PFS and OS for the treatment of interest. In three appraisals using PSMs (TA874, TA862, and TA833), the surrogate endpoint was not used to support the analysis as long-term OS was obtained via extrapolation of immature OS data from the pivotal trial. The use of such immature data for extrapolation to inform the economic model was noted as a concern by the ERG in each appraisal. In TA772, OS data from the pivotal trial were too immature for extrapolation. Instead, the company estimated long-term OS by extrapolation of OS from an external study, assuming no treatment benefit on OS. Therefore, in the absence of mature OS data, TA772 assumes no treatment benefit on OS rather than utilizing information on the surrogate endpoint to estimate the treatment effect on OS. Finally, for TA796, long-term PFS was estimated via extrapolation of ToT from the SACT RWE data set. The use of ToT as a proxy for PFS was necessitated by the lack of reporting of PFS in SACT.
For the 10 TAs that utilized STMs (TA885 (18), TA876 (19), TA851 (22), TA837 (23), TA830 (24), TA823 (26), TA817 (27), TA810 (29), TA795 (31), TA766 (35)), the number and definition of health states varied across the appraisals. However, in an STM patients move between different health states representing stages of a disease. How quickly patients move between these health states is defined by transition probabilities, which are generally estimated from time-to-event data in pivotal trial. For six appraisals (TA885, TA876, TA851, TA810, TA766, and TA817), the majority of transition probabilities were estimated directly from their pivotal trials. Therefore, despite all six TAs providing statements from both the company and ERG indicating that data on OS is immature, mortality events from pivotal trials were still used to inform the economic model and the surrogate relationship was not utilized. However, in three appraisals (TA837, TA830, and TA823), the number of mortality events was so low that RWE and external data were used in place of the pivotal trial to extrapolate over the model horizon. Therefore, these three appraisals preferred to use external but more mature data to inform their economic model rather than estimate mortality based on surrogate endpoints measured in their pivotal trials. Finally, for TA795, transition probabilities from the progression-free state were informed by estimating an HR for time to treatment discontinuation (TTD) in SACT versus TTD in the Rory Morrison Register (RMR). This HR was applied to an exponential model fitted to the PFS data from the RMR to obtain a long-term estimate of PFS in SACT. Therefore, TTD was not used directly as a proxy for PFS in TA795; instead, it was assumed that the relationship that holds for TTD between SACT and RMR also holds for PFS. The use of this approach was necessitated by the lack of reporting of PFS in SACT.
Finally for TA813 (28), TA784 (32), and TA763 (34), alternative cost-effectiveness models to PSM and STM were used. In TA813, the company used a cumulative survival approach that uses TTD as a surrogate for survival outcomes. However, the ERG was concerned about the lack of evidence to link TTD with survival and preferred a surrogate survival approach which modeled PFS as a function of cytogenic and hematological response, as they felt these responses as surrogates were supported by the literature. In TA784, the company used a mean-based model that is comprised of three health states: “progression-free,” “progressed disease,” and “death.” Like a PSM, it estimates long-term PFS and OS but calculates area under the curve (to obtain the mean costs and utilities that are applied to the average time spent in each health state. For PFS in both arms and OS in the treatment arm, long-term survival is estimated via extrapolation of pivotal trial data. However, the company argue that OS for the control arm cannot be extrapolated due to confounding and instead use a 1:1 PFS:OS relationship to estimate OS for the control arm. The ERG criticized this approach citing a lack of consistent evidence for the PFS-OS surrogate relationship. In TA763, a response-based model used KM curves from the pivotal trial up to a landmark point and split those alive at this point by MRD status. The company followed a five-stage approach to estimate PFS and OS for control and treatment arms. First PFS and OS for MRD+ in control arm were extrapolated. Second, a meta-analysis estimated HRs reflecting the association between MRD status and PFS and OS. Third, the HRs were applied to extrapolated curves from the first stage. Fourth, HRs obtained from analysis of pivotal trial were applies to curves to obtain PFS and OS for treatment groups split by MRD status. Finally, survival curves were weighted by MRD status. Therefore, the surrogate relationship between MRD status and PFS and OS was utilized within this model.
Discussion
In this project, we investigated the use of surrogate endpoints in 18 NICE TAs in oncology out of 47 TAs available for review. We found that 37 separate surrogate endpoints were discussed in a range of oncology indications including hematological and solid tumors. The majority of TAs used surrogate endpoints as a result of immature OS data from pivotal trials.
Based on a previously proposed three-step approach for validating surrogate endpoints (Reference Ciani, Buyse, Drummond, Rasi, Saad and Taylor2), we found that 11 (30 percent), 7 (19 percent), and 12 (32 percent) of the surrogate endpoints were supported by evidence from RCTs, observational data and biological plausibility, respectively. For only six (16 percent) of the putative surrogate endpoints, did the TA present a measure of association between the surrogate endpoint and final outcome. For those TAs that did report a measure of association between the surrogate endpoint and final outcome, the correlations reported between treatment effects on the surrogate endpoint and final outcome were reasonably strong (0.40–0.95). However, while such associations might be acceptable to NICE, who do not provide thresholds for the recommended acceptability of a surrogate, they would likely be less acceptable to the HTA body in Germany, IQWiG, who recommend that the lower confidence interval for correlation,
$ R $
, is greater than 0.85 (38). Only two (5 percent) of the putative surrogate endpoints were further supported by a quantification of the expected treatment effect on the final outcome based on the observed treatment effect on the surrogate endpoint. It is possible that a lack of evidence from RCTs on specific treatments and disease areas contributes to the paucity of statistical evidence for surrogate relationships. However, methods are emerging to explore borrowing information across treatment classes, disease indications, and from different sources of evidence (such as registry data) to improve validation of surrogate relationships where data from RCTs on the relevant disease area and treatment are sparse (Reference Papanikos, Thompson, Abrams, Stadler, Ciani, Taylor and Bujkiewicz39–Reference Wheaton, Papanikos, Thomas and Bujkiewicz41).
For NICE oncology TAs using surrogate endpoints, STMs were the most commonly used in cost-effectiveness analysis, with several appraisals noting that where there is immature survival data, such models are preferable as OS does not need to be directly modeled. However, PSMs were also frequently used, sometimes utilizing highly uncertain extrapolations of immature OS curves from pivotal trials. The majority of appraisals preferred to utilize immature OS data from pivotal trials, external trials or RWE rather than incorporate information on the relationship between surrogate endpoints and final outcomes in the economic model.
There are several limitations of this project. First, the selection of TAs is based on documents citing the terms “surrogate” or “surrogacy.” While this approach expedites the review process, it is possible that this decision resulted in excluding TAs where there was non-explicit discussion of surrogate endpoints as there can be discrepancies in how patients, clinicians, regulators, and HTA experts define and describe a surrogate endpoint (Reference Manyara, Davies, Stewart, Weir, Young, Wells, Blazeby, Butcher, Bujkiewicz, Chan and Collins42;Reference Ciani, Manyara, Davies, Stewart, Weir, Young, Blazeby, Butcher, Bujkiewicz, Chan and Dawoud43). A future, more comprehensive review, could consider inclusion of terms such as “intermediate endpoint” and “biomarker” to more fully encompass all discussions around surrogate endpoints. Second, this review only covered oncology drugs evaluated between February 2022 and May 2023. In the period of time, since this review was conducted several more oncology products will have been reviewed by NICE and there may have been further development of methods or guidelines on the use of surrogate endpoints in HTA. Therefore, future reviews of more recent TAs may show a more rigorous approach to the use of data on surrogate endpoints. Finally, this review has potentially limited generalizability, as it was only conducted within NICE which conducts decision making for England and Wales. However, while this review evaluated the use of surrogate endpoints in oncology TAs conducted by NICE for England and Wales, we hope that the outcomes of this review are beneficial to HTA agencies internationally. We hope that by highlighting the lack of standardization of methods for evaluating surrogate endpoints in NICE TAs that this will prompt other HTA bodies to investigate how surrogate endpoints are used within their own appraisals. Furthermore, we hope that this will foster further international collaboration of HTA bodies to work toward a more standardized approach to the use of surrogate endpoints in TAs.
While the relatively newly updated NICE methods guide, recommending more explicit validation of surrogate endpoints, is still to gain more traction and future appraisals will indicate the uptake of methods and approaches the guide recommends, there is still need for further guidance and methods development to support the appropriate and effective use of surrogate endpoints in HTA decision making. For example, throughout this project, it was observed in several TAs that while there was no meaningful discussion of surrogate endpoints, the pivotal trial used to support the appraisal did not include OS as a primary endpoint. This potentially suggests that further guidance is required to support the appropriate reporting of surrogate endpoints in TAs. A number of recent initiatives highlighted such need. Recent extensions of the CONSORT and SPIRIT statements to include appropriate reporting of use of surrogate endpoints and their justification (Reference Manyara, Davies, Stewart, Weir, Young, Butcher, Bujkiewicz, Chan, Collins, Dawoud and Offringa44) will hopefully improve the transparency of the use of surrogate endpoints in clinical trials and in turn in HTA policy making. The International Society of Pharmacoeconomics and Outcomes Research set up a task force currently developing guidance on emerging good practices for surrogate endpoint evaluation and validation of the relationship between outcomes quantitatively informing HTA (45), while NICE is currently working with international organizations to develop guidelines for incorporating surrogate endpoints in cost-effectiveness analyses (Reference Bouvy46).
Conclusions
Over recent years, there has been significant discussion in the literature about the increasing use of surrogate endpoints in supporting regulatory and reimbursement decisions. This review supports the statement that surrogate endpoints are frequently used in oncology TAs in England and Wales. Despite the wide availability of statistical methods and guidance on how to appropriately validate surrogate endpoints, this review highlights that the use and validation of surrogate endpoints can vary dramatically between TAs, which leads to uncertainty in decision making. To reduce such uncertainty, it is important that TAs utilizing surrogate endpoints provide clear evidence supporting the (a) association between the surrogate endpoint and final outcome, (b) association between treatment effects on the surrogate endpoint and final outcome, and (c) prediction of the treatment effect on the final outcome based on the observed effect on the surrogate endpoint. Providing clear evidence to support the surrogate relationship will enable greater confidence in the clinical and cost-effectiveness estimates obtained by HTA bodies which in turn will allow more robust reimbursement decisions and enable patients’ to access the best available treatments.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/S0266462325000017.
Acknowledgments
This research was funded by the University of Leicester and Medical Research Council methodology research grant (MR/T025166/1). The authors received additional support from the Leicester NIHR Biomedical Research Centre (BRC). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.
Competing interest
LW has no competing interests to declare. SB is a member of the NICE Decision Support Unit (DSU) and the NICE Guidelines Technical Support Unit (TSU). She has served as a paid consultant, providing methodological advice, to NICE, Roche, RTI Health Solutions and IQVIA and has received payments for educational events from Roche and the University of Bristol.
Open access
