Background
Despite important breakthroughs in the treatment of cancer, there are concerns about the quality of evidence and the extent of the clinical benefit provided by new cancer medicines (Reference Schnog, Samson, Gans and Duits1). In the United States, less than a third (27 percent) of the US Food and Drug Administration (FDA) approvals in solid tumor oncology for the period 2017–2021 were supported by evidence of an overall survival (OS) benefit (Reference Cherny2). The use of single-arm studies and surrogate outcome measures such as progression-free survival (PFS) or overall response rates (ORRs) has increased over time (Reference Chen, Haslam and Prasad3). Evidence on OS takes a longer time to acquire in clinical trials than for surrogate outcomes. The demand to expedite patient access to new cancer medicines is the main reason why mature OS data are often missing from the initial approval. However, after several years of follow-up, the majority of these agents had either no benefit or an unknown benefit in OS (Reference Cherny2).
Several studies have examined the evaluation of cancer medicines by regulators such as the US FDA, the European Medicines Agency and Health Technology Assessment (HTA) institutions in several countries (Reference Farina, Moro and Fasslrinner4–Reference Michaeli and Michaeli6). In Australia, a few studies have examined the HTA processes and the quality of evidence supplied in submissions (Reference Gao, Laka and Merlin7;Reference Karikios, Chim and Martin8) but none has assessed the clinical benefit of cancer medicines.
The objective of this study was to assess the level of evidence and the extent of clinical benefit for all cancer medicines submitted to the Pharmaceutical Benefits Advisory Committee (PBAC) between 2010 and 2020. PBAC is an independent HTA body appointed by the Australian Government that provides recommendations for listing on the Pharmaceutical Benefits Scheme (PBS), the main source of subsidized public funding for medicines in Australia. PBAC reviews the evidence on clinical and cost-effectiveness in the submission prepared by the manufacturers. Applicants are required to provide evidence to support the effectiveness and safety of the proposed medicine with reference to a comparator, the current alternative therapy used in Australian clinical practice. PBAC strongly prefers direct evidence from randomized controlled trials (RCTs) comparing the medicine to the comparator, but also accepts indirect evidence provided by the comparison of RCTS with the proposed medicine and with the comparator involving a common reference (e.g., placebo or other active therapy).
In addition, we used the European Society for Medical Oncology Magnitude of Clinical Benefit Scale version 1.1 (ESMO-MCBS) (Reference Cherny, Dafni and Bogaerts9) to grade the extent of benefit provided by cancer medicines for solid tumors, and examined the associations between level of evidence, clinical benefit, and funding recommendations.
Methods
We reviewed all submissions for cancer medicines submitted to the PBAC between 2010 and 2020. We excluded submissions for supportive therapies (e.g., antiemetics, antiresorptive agents, colony-stimulating factors), medicines for prevention of cancer, biosimilar medicines, and submissions for new formulations and new strengths for indications already recommended. Given that PBAC submissions may include several requests that yield different outcomes, we chose a medicine/indication pairing approach (Reference Lybrand and Wonder10). PBAC guidelines for submissions require the inclusion of the best available evidence comparing the clinical performance of the proposed medicine with that of the main comparator (i.e., the therapy[ies] likely to be most replaced by prescribers in practice) (11). For medicines that met the inclusion criteria, the following variables were extracted from the Public Summary Documents (PSD) published online by the PBAC (12): cancer type; PBAC recommendation; number of submissions; type of cancer medicine; indication; treatment setting; characteristics of the clinical studies (design, comparator, number of patients); clinical outcomes (OS, PSF, ORR); quality of life (QoL); type of economic analysis; and modeling (direct/indirect comparison). In instances where there were several submissions for the same indication (e.g., following PBAC rejection), data were extracted from the submission(s) that included the most detailed and latest data. When several studies were cited as contributing to the body of evidence for a submission, we selected the most relevant clinical trial. When the evidence considered by PBAC came from an indirect comparison of trials involving a common reference (e.g., placebo or other active therapy) instead of direct randomized trials, data were extracted from the key trial involving the medicine under consideration. Two researchers (AV, CC) extracted all data from the submissions independently and any discrepancies were resolved by discussion. The type and date of marketing authorization (priority review, conditional approval, orphan status) were obtained from the Therapeutic Good Administration website (13). The ESMO-MCBS scores for each medicine/indication pairing were derived from existing ESMO-MCBS scorecards posted on the ESMO open access portal (14). The ESMO-MCBS is the most widely validated tool to evaluate the magnitude of clinical benefit provided by cancer medicines for solid organ tumors (Reference Cherny2). The ESMO-MCBS assigns a score of 1 (low clinical benefit) to 5 (high clinical benefit) for drugs used in the advanced or metastatic setting with scores of 4 or 5 indicating a substantial magnitude of clinical benefit. In the potentially curative or adjuvant setting, the scale assigns scores of C (low clinical benefit) to A (high clinical benefit) with A and B representing a substantial magnitude of clinical benefit (Reference Cherny, Dafni and Bogaerts9).
Descriptive statistics were used to summarize the characteristics of all submissions. We calculated the median gains in OS when there was a significant difference between the medicine and the comparator in all the submissions as well as in the subgroups that received positive and negative reimbursement recommendations. We compared the characteristics of the trials and the clinical benefits between recommended versus rejected recommendations using the Chi-squared test and one-way analysis of variance (15). Probability values of p < 0.05 were considered statistically significant.
Results
Between 2010 and 2020, PBAC considered the funding of 182 indications for 100 different cancer medicines, including 124 (68.1 percent) medicines for solid tumors and 58 (31.9 percent) for hematological cancers (Table 1). Of the medicines for solid tumors, the vast majority were in the noncurative setting (n = 115, 92.7 percent). Monoclonal antibodies (36.8 percent) and protein kinase inhibitors (34.6 percent) were the most common therapeutic classes. These were followed by other antineoplastics (12.1 percent), cytotoxic chemotherapy (11 percent), and hormone and hormone antagonists (5.5 percent). Fifty (27.5 percent) indications had an orphan designation. Overall, 137 (75.3 percent) of the indications were recommended for funding by PBAC, 40 (21.9 percent) were rejected, and 5 (2.7 percent) had the decision deferred (i.e., the decision may become recommended or rejected at a later date after the end of a study period).
Table 1. Characteristics of applications
a Cancer medicines were categorized with the anatomic therapeutic chemical classification (31).
b Number of submissions: when the first PBAC recommendation does not support listing, companies can resubmit an application. Consequently, several review cycles may take place until a positive recommendation.
c Priority review and provisional approval are pathways that fast track prescription medicines onto the market which started in July 2017.
d If studies comparing the medicine with the main comparator are not available, PBAC may consider indirect comparison with other studies involving a common reference.
e When several studies were cited as contributing to the body of evidence for a particular submission, the most relevant trial was chosen.
f Includes one retrospective review.
Direct evidence with an appropriate comparator was provided in 63.7 percent of the indications (Table 1). Randomized clinical trials (RCTs) were presented as the supporting evidence for 154 indications (84.6 percent) including in 114 indications (62.6 percent) involving direct evidence. RCTs were available in 113 (91.1 percent) and 41 (70.7 percent) of the solid tumors and hematological cancer indications, respectively (p < 0.05). Single-arm studies provided the main evidence in 28 (15.4 percent) indications.
Statistically significant differences in OS were reported in 80 (44 percent) indications, including 63 (50.8 percent) and 17 (29.3 percent) indications for solid tumors and hematological cancers, respectively (Table 2). Of the remaining trials without an OS benefit, statistically significant improvements in PFS were found in 26.4 percent of indications. QoL data were reported in 45 (24.7 percent) of indications and showed an improvement in 13 (7.1 percent).
Table 2. Overall survival, progression-free survival, and quality of life
a Improved OS if a positive difference was found between the intervention and comparator groups and if the confidence intervals of the hazard ratio did not cross 1.
b showing improved PFS if a positive difference was found between the intervention and comparator groups and if the confidence intervals of the Hazard Ratio did not cross 1.
For indications with a statistically significant mature improvement in OS, the median OS gain was 3.0 months (range 0.9–17.0) for solid tumors and 8.2 months (1–49.1) for hematological cancers (Table 3). The ESMO-MCBS scores were available for 99 solid tumor indications, of which 51 (51.5 percent) showed substantial clinical benefit according to ESMO-MCBS. Using this scale, there was substantial clinical benefit for medicines in 40 (54.1 percent) of PBAC-recommended indications and 9 (42.9 percent) of PBAC indications -rejected (Table 4).
Table 3. Overall survival gain in randomized clinical trials
Table 4. ESMO-MCBS v1.1 scores
a Substantial benefit for ESMO-MCBS v1.1 was defined as a grade A or B for (neo)adjuvant intent and a score of 4 or 5 for palliative intent.
Comparisons between the indications that were recommended and rejected by the PBAC (Table 5) showed there was no difference in the proportion of RCTs (p = 0.34), type of comparison (p = 0.26), type of economic analysis (p = 0.19), proportion of submissions with OS statistically significant (p = 0.48), proportion of submissions with ESMO clinically significant (p = 0.36), or OS survival (p = 0.33).
Table 5. Comparison of characteristics of applications recommended versus rejected by PBAC
a Calculated for indications with statistically significant differences.
Discussion
Three-quarters of medicines for cancer indications were recommended for funding by PBAC between 2010 and 2020. This is similar to the PBAC listing rate of cancer medicines between 2005 and 2014 (83 percent) (Reference Karikios, Chim and Martin8) and substantially higher than the positive recommendation rate of 48 percent for all medicines observed between 2010 and 2018 (Reference Lybrand and Wonder10). RCTs were the main source of evidence (84.6 percent) and available in 113 (91.1 percent) and 41 (70.7 percent) of the solid tumors and hematological cancer indications, respectively. These findings are comparable to studies in the European setting. Eighty percent of cancer medicines approvals by the European Medicine Agency (EMA) between 2010 and 2019 were supported by at least one RCT, including 85.2 percent for solid tumors and 67.2 percent for hematological cancers (Reference Farina, Moro and Fasslrinner4).
Data on QoL were presented in less than a quarter of indications with improvement in this measure seen in only 7.1 percent of all submissions. Improvements in QoL was observed in seven of 68 (10 percent) of the indications approved for cancer medicines by the European Medicines Agency (Reference Davis, Naci and Gurpinar16). QoL is an important patient-orientated outcome for those receiving treatment for cancer and cannot easily be predicted from the known toxicity of the drug or effect on radiographic progression. Interestingly, an association has been reported between QoL benefit and OS but not the surrogate of PFS (Reference Samuel, Booth and Eisenhauer17). Notwithstanding the difficulty in measuring QoL within the context of a clinical trial, this highlights the importance of considering QoL in drug submissions especially for agents where the activity is based on surrogate outcomes such as PFS or ORR.
We found that 44.0 percent of the indications were supported by statistically significant benefits in OS including 29.3 percent for hematological cancers and 50.8 percent for solid cancers. In the absence of OS benefit, statistically significant benefits in PFS were found in 26.4 percent of cases. These results were similar to those found in a European study (Reference Pinto, Naci, Neez and Mossialos18). Of 58 indications approved for cancer medicines in Europe between 2012 and 2016, 25 (43 percent) were categorized as exhibiting OS benefit and 14 (24 percent) were categorized as exhibiting PFS benefit in the absence of OS benefit (Reference Pinto, Naci, Neez and Mossialos18).
The type and level of evidence available for the evaluation of hematological cancers was lower than for solid tumors with lower proportions of RCTs, and statistically significant benefits in OS. This can be partly explained by the natural history of hematological cancers, which can range from fulminant to almost benign. Hence, some hematological cancers are characterized by very long PFS and OS, which are difficult to measure in RCTs and may have valid disease-specific surrogate endpoints, such as complete cytogenetic response for chronic myeloid leukemia (Reference Kiesewetter, Cherny and Boissel19). RCTs can also be challenging to run when there is no available treatment and a single-arm trial has demonstrated a durable response rate. However, response rates may not be a good predictor of long-term outcomes, and confirmatory trials may take several years before the results are available (Reference Agrawal, Arora and Amiri-Kordestani20;Reference Deshmukh, Kesselheim and Rome21).
In our study, the majority of cancer indications (54 percent) had to be assessed based on surrogate outcomes such as PFS although no surrogate measure in oncology has been found to have absolute surrogacy for true clinical benefit across diseases and treatments (Reference Gyawali, de Vries and Dafni22). There was no difference between recommendations and rejections with regard to the availability of RCTs or the availability of statistically significant OS or PFS data. In Canada, drugs that received a positive recommendation compared with those with a negative recommendation were more likely to have a RCT design (92.3 percent vs. 53.8 percent) (Reference Meyers, Jenei, Chisamore and Gyawali5). However, similar to our study, an evaluation of the British, German, and French HTA decisions on cancer medicines for solid tumors found that the availability of RCTs, mature OS data, and statistically significant OS data were not associated with reimbursement decisions (Reference Smith, Fu, Fisher, Meletiche and Pawar23).
In our study, the median OS gains for solid tumors were 4.1 months for PBAC-recommended and 2.2 months for PBAC-rejected indications. This is consistent with international studies from equivalent regulatory and funding bodies. Submission to the pan-Canadian Oncology Drug Review had median OS gains of 3.7 months and 1.9 months in recommended and rejected submissions, respectively (Reference Meyers, Jenei, Chisamore and Gyawali5). Similarly, in applications to the FDA, the overall median OS gains were 2.80 months and 4.35 months for solid tumors and hematological cancers, respectively (Reference Michaeli and Michaeli6).
For the indications with an ESMO score available, 54.1 percent of the indications recommended and 42.9 percent of the indications rejected by PBAC had demonstrated a substantial clinical benefit. This may indicate that factors not reflected in this score are influencing funding decisions. Interestingly this finding is different from a similar study in the Canadian setting where significantly more cancer drugs (61.5 percent) that received a positive recommendation had demonstrated substantial clinical benefit compared to those that received negative recommendations (19.2 percent) (Reference Meyers, Jenei, Chisamore and Gyawali5). This may reflect differences in the factors considered in HTA methods. In Europe, 33 percent of the 144 indications approved by the EMA for solid tumors in 2009–2020 met the substantial clinical benefit criteria (Reference Grossmann, Wolf, Rothschedl and Wild24), and in the United States, 29 percent of the 146 indications for advanced-stage solid tumors approved by the FDA in 2017–2021 met the substantial clinical benefit criteria (Reference Cherny2). The higher proportion of indications that met the meaningful benefit criteria in HTA assessments in Australia and in Canada compared to regulatory bodies, such as the EMA and FDA can be explained by differences in evidentiary requirements (Reference Vreman, Naci and Goettsch25). The FDA and the EMA focus on the drug’s efficacy and allow placebo to be a comparator. Conversely, PBAC considers the effectiveness of the medicines with regard to an appropriate comparator, either a medicine prescribed on the PBS to treat that target population or the standard medical management if there are no currently listed PBS medicines as well as cost-effectiveness and the financial impact (26).
There was no significant difference in PBAC-recommended and PBAC-rejected indications with regard to the availability of RCTs, the availability of statistically significant OS or PFS data or the presence of a substantial clinical benefit (p = 0.36). PBAC decisions are influenced by quantitative factors, such as the comparative health benefit, comparative cost-effectiveness, patient affordability, predicted use in practice, and financial implications for the PBS and the Australian Government health budget. Other factors are less easily measurable such as overall confidence in the evidence, equity concerns, clinical need, severity of the disease, ability to target the medicine to patients who are likely to benefit the most, and public health issues (26;Reference Turkstra, Bettington, Donohue and Mervin27). An analysis of the characteristics strongly associated with PBAC rejections of cancer medicines between 2005 and 2014 were problematic or uncertain clinical evidence, problematic or uncertain economic evidence, and inactive comparator used (Reference Karikios, Chim and Martin8). The clinical evidence supplied to PBAC for cancer medicines was often of poor quality and had been deteriorating over time (Reference Gao, Laka and Merlin7). Decision-making by funding bodies, including PBAC, is likely to become increasingly challenging in the years to come due to increasing uncertainty, less incremental gains and pursuit of therapeutics for rare diseases.
Limitations
This analysis has several potential limitations. We retrieved data available in the PSD published by PBAC. PBAC’s assessments are complex and other factors driving decisions, such as comparative medicine safety, unmet need, cost-effectiveness, and budget impact were not examined. Not all outcomes were reported in PSDs and the commercially sensitive information about the medicine was frequently redacted. We only used the ESMO scores published on their website and did not attempt to calculate ESMO scores when not available. An important drawback of the current ESMO-MCBS scale is that it may not take into account important methodological biases (Reference Gyawali, de Vries and Dafni22), which is important given that many trials for cancer drugs are at high risk of bias based on their design, conduct, or analysis (Reference Naci, Davis and Savovic28).
Conclusion
PBAC has to make recommendations on cancer indications with unclear scientific evidence often involving indirect comparisons, single-arm studies, and surrogate endpoints, and with uncertain clinical value of incremental benefits. Approximately half of medicines for solid tumor indications were not offering substantial benefit according to the ESMO-MCBS. Although the prevailing narrative that all new cancer treatments are potentially beneficial (Reference Gyawali and Booth29), our results confirm the importance of balancing early access to new cancer medicines against population-level exposure to therapies with no benefit over standard of care and potential toxicity. A national consultation is currently underway to better refine Australian HTA policy and methods to meet these ongoing challenges (30).
Funding statement
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Competing interest
The authors declare none.
Open access
