1. Introduction
While regulatory approval is a necessary condition for the placement of new therapies on the market, it is no longer sufficient in ensuring timely patient access to healthcare innovations; health technology assessment (HTA) is a critical step that new therapies must undergo before a healthcare system makes them available to patients at public expense. The role of HTA is critical in ensuring that valuable innovations, offering significant additional clinical benefit to patients, are included in the benefits health systems provide.
Over time, many HTA agencies have been criticised in that they lack transparency, clear evaluation criteria, and flexible approaches for the assessment of drugs with atypical evidence submissions (e.g. orphan or ultra-orphan drugs) (Angelis, Lange, and Kanavos Reference Angelis, Lange and Kanavos2018). Despite calls to streamline and standardise HTA principles and methods across agencies or countries (O’Mahony Reference O’Mahony2019; Kristensen Reference Kristensen2012), there remain significant differences in how agencies are structured (Martelli et al. Reference Martelli, Torre, Ghionno, Staniscia, Neroni, Cicchetti, Bremen and Ricciardi2007; Allen et al. Reference Allen, Pichler, Wang, Patel and Salek2013) and in the methods they employ to reach decisions (Mathes et al. Reference Mathes, Jacobs, Morfeld and Pieper2013; Vreman et al. Reference Vreman, Mantel-Teeuwisse, Hövels, Leufkens and Goettsch2020; Akehurst et al. Reference Akehurst, Abadie, Renaudin and Sarkozy2017). This heterogeneity in HTA principles, processes and methods has led to disparities in coverage decisions between agencies often within the borders of the same country (Nicod and Kanavos Reference Nicod and Kanavos2012; Fischer, Heisser, and Stargardt Reference Fischer, Heisser and Stargardt2016; Salas-Vega, Bertling, and Mossialos Reference Salas-Vega, Bertling and Mossialos2016). This is understandable and, in some way, expected, since healthcare systems vary, the access to services they provide varies, and priorities differ. Nevertheless, understanding whether commonalities exist among clinical, economic and contextual factors influencing coverage decisions can help approximate decision processes and promote the perceived rationality of HTA processes amongst citizens.
Evidence suggests that a significant proportion of new drug submissions to HTA agencies receive a positive recommendation (Kanavos, Visintin and Gentilini, Reference Kanavos, Visintin and Gentilini2023), but a fair number are rejected. In principle, rejections of new drugs by HTA agencies safeguard the public from spending scarce resources on novel drugs with little or no added clinical value, but considering that the same drug can be approved in one setting but be rejected in another, raises broader equity concerns.
There are multiple dimensions of ‘value’ considered either formally or informally across national HTA bodies. These dimensions can be roughly grouped into considerations of the drug’s added therapeutic benefit (focus of agencies that do not use a cost-effectiveness criterion like Germany’s IQWiG and France’s HAS), comparative clinical and economic value (focus of agencies requiring demonstration of cost-effectiveness like England’s NICE), budgetary impact and perceived societal importance (Beletsi et al. Reference Beletsi, Koutrafouri, Karampli and Pavi2018). When reviewing the clinical and economic evidence on a new drug, decision-makers make value judgements, guided (at least in principle) by their agency’s evidence evaluation framework. The relative importance placed on these different elements of value differ considerably across agencies.
The divergence in HTA recommendations across countries is well-documented in the literature. Most comparative empirical studies to date have employed descriptive statistical methods (Fischer, Heisser, and Stargardt Reference Fischer, Heisser and Stargardt2016; Grepstad and Kanavos Reference Grepstad and Kanavos2015; Allen et al. Reference Allen, Liberti, Walker and Salek2017; Vreman et al. Reference Vreman, Mantel-Teeuwisse, Hövels, Leufkens and Goettsch2020), with some attempting to further explain differences in HTA outcomes through case studies (Nicod and Kanavos Reference Nicod and Kanavos2012; Clement et al. Reference Clement, Harris, Jing Li, Yong, Lee and Manns2009) and mixed methods approaches (Nicod and Kanavos Reference Nicod and Kanavos2016; Nicod Reference Nicod2017; Nicod et al. Reference Nicod, Maynou, Visintin and Cairns2020). However, the difficulty of quantitatively analysing the topic of HTA coverage recommendations has led to a paucity of rigorously designed economic modelling studies, many of which are statistically underpowered and/or constrained to a single jurisdiction.
Some quantitative studies have emerged over time studying the factors that influence HTA coverage recommendations; for example, evidence uncertainty and disease burden were found to be important decision modifiers beyond incremental cost-effectiveness ratios (ICER) in the context of NICE (Devlin and Parkin Reference Devlin and Parkin2004). Subsequent studies supported the assumption that ICER thresholds are not deterministic and that multiple factors can affect the probability of rejection (Dakin, Devlin, and Odeyemi Reference Dakin, Devlin and Odeyemi2006; Dakin et al. Reference Dakin, Devlin, Feng, Rice, O’Neill and Parkin2015). Similar single-country studies utilising logistic regressions exist for Scotland (Charokopou et al. Reference Charokopou, Majer, Raad, Broekhuizen, Postma and Heeg2015), Australia (Harris et al. Reference Harris, Hill, Chin, Jing Li and Walkom2008; Segal, Dalziel, and Mortimer Reference Segal, Dalziel and Mortimer2010; Chim et al. Reference Chim, Kelly, Salkeld and Stockler2010) and France (Le Pen, Priol, and Lilliu Reference Le Pen, Priol and Lilliu2003).
The cross-country econometric literature is more limited. A study using survey data from 13 OECD countries to separately analyse the stages of evidence assessment and appraisal (i.e. decision-making), highlighted that evidence uncertainty has a significant influence on the likelihood of rejection, mitigated only by the participation of a greater number of stakeholders in the appraisal stage (Fischer et al. Reference Fischer, Rogowski, Leidl and Stollenwerk2013). A study focusing on oncology drugs in the EU showed that macro-level health system characteristics do not account for differences in reimbursement of cancer drugs, apart from a significant positive association between SHI systems and coverage probability (Maynou and Cairns Reference Maynou and Cairns2019). A further study has found that agencies adopt an extra-welfarist approach in decision-making by going beyond the clinical evidence and conventional metrics of societal welfare, taking into account dimensions of value like drug innovativeness and unmet need (Kanavos, Visintin, and Gentilini Reference Kanavos, Visintin and Gentilini2023).
To date, there have been no published studies carrying out a comprehensive cross-country comparative analysis of HTA outcomes from the perspective of negative recommendations to understand what factor or combination of factors contribute to drugs being rejected by HTA agencies.
Against this background, therefore, this paper empirically studies the factors associated with HTA rejections (i.e., drugs receiving negative recommendations by HTA agencies) in a sample of prescription drugs assessed between 2009 to 2020 across eight HTA agencies in seven OECD countries. In doing so, we adopt a comparative approach to identify these factors. Additionally, we study the divergence in negative recommendations between agencies and explore agency-specific differences in the assessment of different types of drugs. Overall, we seek to answer two questions: first, what are the most significant factors contributing to HTA rejections across our sample? And, second, how does the importance of those factors differ across countries and agencies?
2. Data & methods
2.1. Sample selection and data collection
We extracted and used data from publicly available HTA reports. Eight HTA agencies across seven OECD countries were selected into the sample and fulfilled all four of the following criteria: first, they had established HTA processes operated by dedicated assessment bodies; second, they had diverse and explicitly stated decision-making criteria; third, they made HTA reports and information around recommendations publicly available; and, fourth, the information provided in the reports covered all aspects of assessment and appraisal to enable comparative analysis, particularly in what concerns the rationale for decision-making and the handling of clinical and economic uncertainties. The countries (and associated HTA agencies) that qualified were: Australia (PBAC – Pharmaceutical Benefits Advisory Committee); Canada (Canadian Drug Agency – CDA, formerly known as CADTH – Canadian Agency for Drugs and Technologies in Health, until 2023; and INESSS – Institut national d’excellence en santé et en services sociaux in Quebec); England (NICE – National Institute for Health and Care Excellence); France (HAS – Haute Autorité de Santé); Germany (IQWiG – Institute for Quality and Efficiency in Health Care); Scotland (SMC – Scottish Medicines Consortium); and Sweden (TLV – The Dental and Pharmaceutical Benefits Agency). Supplementary Table 1 reviews the regulatory structure and assessment methods followed by the sample HTA agencies.
Data extraction commenced with drugs that have received regulatory approval. As a first step, therefore, we considered all drugs that received marketing authorisation (MA) across the regulatory agencies in the study countries (European Medicines Agency - EMA for the European countries in the sample; Health Canada, for Canada and Quebec; and Therapeutic Goods Administration – TGA for Australia) between January 1st, 2009 and June 30th, 2018. In order to capture HTA outcomes for drugs that received MA in 2017 and 2018, we extended the study period to December 31st, 2020.
The selection process ensured that a representative sample of drugs and their respective indications (‘drug-indication pairs’) approved by regulatory authorities (EMA in Europe, TGA in Australia, and Health Canada in Canada) over the study period was created, by not restricting the sample only to drug-indication pairs assessed by all HTA agencies and including in the sample drug-indication pairs which were assessed by at least two HTA agencies in order to ensure comparability and minimise the sample’s bias towards certain therapeutic areas. HTA reports that were published by the eight agencies between January 1st, 2009 to December 31st, 2020 were obtained by directly accessing the HTA agency websites. A spreadsheet dataset was constructed and populated with information found within the reports. We excluded from the sample generics and biosimilars, and, therefore, our sample consisted exclusively of originator prescription drugs.
The unit of analysis was defined as the ‘drug-indication’, which meant that HTA outcomes on different indications for the same drug were entered into the dataset separately. The dataset was structured so that each entry represented a single drug-indication-HTA agency triplet. This resulted in a single HTA decision per drug-indication and agency (or a missing observation if no HTA report was found by that agency for that drug-indication). This was achieved by applying the following rules when analysing HTA reports: first, the most up-to-date full submissions were selected for each drug-indication; second, for instances where a resubmission was made (e.g., following an earlier rejection), the outcome of the resubmission was recorded and not the outcome of any previous assessments; and, third, reports of minor amendments to previous recommendations, which did not impose any further important clinical or economic restrictions, were excluded from the analysis.
2.2. Dataset description
Our dataset is an unbalanced panel of 217 drug-indication pairs, representing 175 unique drugs (molecules), assessed by at least two out of the eight HTA agencies over the study period 2009-2020, resulting in a sample of N = 1,405 observations. No agency had appraised all 217 drug-indication pairs in our dataset. Overall, there were 49 drug-indication pairs (22.6%), which were assessed by all eight agencies over the study period, while a majority of 180 drug-indications (82.9%) were assessed by at least six out of the eight agencies (see Supplementary Table 2).
2.3. Variables of interest
Our outcome is a binary variable indicating a negative HTA outcome (=1 for “do not list” recommendation; =0 for “list or list with conditions” recommendation). The explanatory variables utilised in our analysis (Table 1) included the following: First, regulatory variables (defined to control for the pathway used in obtaining marketing authorisation and distinguishing between standard marketing authorisation, conditional/accelerated marketing authorisation and parallel review; and whether the drug-indication pair had a previous rejection by the HTA agency). It is hypothesised that a standard approval increases the odds of rejection compared with accelerated approval, since accelerated approval provides a signal that the drug in question holds significant promise of clinical benefit. Equally, it is hypothesised that a previous rejection by an HTA agency could increase the odds of a (final) rejection. Second, disease-related variables (classifying drugs by cancer, non-cancer, orphan and non-orphan indications), whereby the hypothesis to be tested was whether the use of a drug for a specific indication (cancer- vs non-cancer-related and orphan- vs non-orphan-related) increased the odds of a rejection by HTAs. Third, clinical evidence variables were included (controlling for the quality of evidence, using a composite index of quality drawing on similar research in earlier studies (Kanavos, Visintin, and Gentilini Reference Kanavos, Visintin and Gentilini2023; Sterne et al. Reference Sterne, Savović, Page, Elbers, Blencowe, Boutron and Cates2019), and which was derived from information on (i) trial design (whether the evidence submitted was based on phase III Randomised Control Trial(s) (RCT) vs other form of trial design), (ii) comparator choice (direct comparator vs. placebo) and (iii) study primary endpoints utilised in the pivotal trial(s) submitted to the HTA agency (whether clinical vs surrogate). The hypothesis to be tested was whether poor quality submissions (e.g., relying on surrogate rather than clinical endpoints, or on phase II RCTs or single arm studies, rather than Phase III RCTs), increased the odds of a rejection by HTAs. Finally, clinical and economic uncertainty variables were included that could not be addressed by sponsors and which controlled for the perceived evidence uncertainties raised by HTA agency proceedings. Key clinical uncertainties included uncertainties relating to clinical benefit, comparator, practice generalisability, population generalisability, study design, follow-up period and adverse events. Key economic uncertainties included uncertainties relating to cost-effectiveness, utility input, cost input, economic modelling and economic model comparator. Each of these variables summarised concerns raised by HTA agencies around the extent to which the submitted clinical or economic evidence fulfilled optimal or acceptable thresholds. The hypothesis to be tested here was that submissions to HTA agencies with significant uncertainties that could not be addressed by sponsors carried a higher risk of rejection by HTAs.
Dataset variable descriptions and hypothesised effect on probability of rejection
Notes: *Bin = binary (YES = 1, NO = 0) variable; Cat = categorical variable.
† An uncertainty was categorised as ‘unaddressed’ if none of the following criteria were fulfilled: A) The manufacturer presented more evidence; B) External stakeholders (e.g. professional medical bodies) interfered to address uncertainty; C) The HTA agency dismissed the uncertainty as unimportant.
The data and variables informing our analysis have been extracted manually by two of the authors (PK and EV) from the extracted HTA reports and were cross-validated by all authors. Specifically, for clinical and economic uncertainties, manual extraction involved the authors identifying the relevant concerns raised by HTA agencies in the appraisal part of the HTA report, which informed the final recommendation. Table 1 summarises the variables, their definitions, the hypotheses they are testing and the expected or likely direction of the effect.
2.4. Empirical methods
A descriptive analysis preceded our econometric analysis and guided our selection of independent variables and model specification. In all our models, we employ multivariate logistic regressions, with a binary outcome capturing the likelihood of a negative HTA outcome for a drug-indication by an agency. Our models controlled for a comprehensive set of independent variables, as defined in Table 1. First, we accounted for assessment-specific characteristics, which vary at both drug-indication and agency levels (regulatory pathway, quality of evidence, previous rejection, clinical and economic uncertainties). Second, we included agency-invariant controls, which change only across drug-indications, such as disease-related variables (cancer vs non-cancer drugs; orphan vs non-orphan drugs). Last, we also controlled for agency fixed-effects to account for time-invariant differences across agencies, and year fixed effects to account for year-specific variation(s). We further examined whether the association between drug-indication characteristics and HTA decisions varied across agencies, by including interaction terms in our regression models (e.g., cancer drugs with orphan indications). The strength of associations are quantified using odds ratios and marginal effects (MEs), which allow us to illustrate how the predicted probability of a negative HTA outcome changes with potential changes in the independent variables presented above (Norton et al, Reference Norton, Dowd and Maciejewski2019). Further detail about the model specification can be found in the Supplementary Material.
All analyses were carried out in STATA 16.1 (‘StataCorp. 2019. Stata Statistical Software: Release 16. College Station, TX: StataCorp LLC.’). Two sets of results are shown, one for the full sample of 8 HTA agencies and one focusing only on countries pursuing cost-effectiveness analysis, therefore, excluding France and Germany, which are pursuing HTA with comparative clinical benefit assessment. Estimates are presented in the form of odds ratios (OR). When examining inter-agency differences, we additionally compute average MEs to aid the interpretation of interactions in our models. Standard errors clustered at the agency level are reported throughout.
3. Results
3.1. Descriptive analysis
Out of a total of 1,405 HTA assessments, there were 181 ‘do not list’ (reject) HTA outcomes by the end of the study period (Table 2). Rejections represented 12.9% of all assessments, largely because only final decisions were recorded even if that drug-indication had been previously rejected by the agency. There were 281 HTA assessments where agencies had previously rejected a drug-indication but subsequently issued a positive coverage recommendation (“list” or “list with criteria”).
Descriptive statistics for variables employed in the econometric models, presented by HTA outcome (rejection vs. listing)
Notes: Chi-square and t-statistics reported compare sample of rejections with sample of listings.
* L = recommend drug for listing within the marketing authorisation (MA) indication; LWC = recommend drug conditional on indication restriction (population, dosage, cycles of therapy, etc.); DNL = do not recommend drug for reimbursement.
¶ CEA = cost-effectiveness assessment; includes all countries & agencies apart from France (HAS) and Germany (IQWiG).
† An uncertainty was categorised as ‘unaddressed’ if none of the following criteria were fulfilled: A) The manufacturer presented more evidence; B) External stakeholders (e.g. professional medical bodies) interfered to address uncertainties; C) The HTA agency dismissed the uncertainty as unimportant.
Of the 217 drug-indications assessed by at least two agencies, there were 110 drug-indications (50.7%) which at least one agency rejected while some or all others recommended it for reimbursement, suggesting a significant divergence in HTA outcomes. Out of the 110 drug-indications rejected, most were rejected by less than half of the agencies that assessed them (n = 93, 84.5%). Only a single drug-indication pair was rejected (Nivolumab for treating locally advanced unresectable or metastatic urothelial cancer after platinum-containing chemotherapy) by both agencies that assessed it, namely, NICE and SMC (Supplementary Table 3).
Examining the pathways that manufacturers used prior to HTA submission, we find that the ‘early approval’ pathway, known as conditional marketing authorisation or accelerated approval, seems to be sought in submissions with lower quality of evidence, reflecting the early stage of the trial data and/or the difficulty of conducting high-quality studies due to restricted patient population sizes, as is often the case in most orphan indications (Figure 1). The drug-indications that are eventually rejected seem to make greater use of the early approval pathway (18% of ‘rejected’ drug-indications apply for ‘early approval’ vs 15% of those that have achieved ‘listed’ or ‘listed with conditions’) and have lower quality evidence associated with these submissions compared to listed drugs (82% of rejected submissions following an ‘early approval’ have low quality evidence vs 47% of listed); the higher proportion of low-quality evidence in rejected drugs versus listed drugs in early approvals is statistically significant (t= −3.76, p < 0.001). By contrast, the parallel review pathway seems to be used when evidence of high or average quality is subsequently submitted to the HTA regulator (60% of ‘rejected’ and 66% of ‘listed’ submissions assessed via parallel review had evidence of high or average quality), possibly reflecting the manufacturer’s confidence in the submission being accepted first-time round.
Breakdown of HTA submissions by (a) marketing authorisation (MA) pathway, (b) quality of evidence (low, average, and high), and (c) HTA outcome (rejected vs listed drug-indications)1. Note: 1Each of the evidence quality bars represents the % of all submissions (n) that utilised a particular MA pathway and had one of the two HTA outcomes.
A comparative analysis of rejection rates across country agencies shows several trends (Figure 2). First, rejection frequency across agencies varies, with Quebec and Australia having rejected the most submissions at 36% and 20%, respectively (Figure 2a). We also note that Germany and France have particularly low rejection rates when compared to the other countries (because only a single drug-indication in our sample has been rejected in Germany – Olaparib for ovarian cancer – we exclude Germany when examining inter-agency differences). This is in line with their different HTA approach that focuses solely on comparative clinical benefit assessment and the particularities of their pricing and reimbursement system, which allow drugs to be subjected to internal reference pricing when their comparative clinical benefit is found to be non-important, instead of being outright rejected.
Rejection rates across countries (a) by drug indication and (b) over time. Notes: An indication is recognised as orphan if any agency in the sample designated that drug-indication as an orphan one. This methodology is adopted because definitions of a rare disease may vary and some countries do not formally recognise orphan status in HTA proceedings (Australia, Canada, Quebec). Germany is excluded from this analysis because it rejected only a single drug (Olaparib) in the study period.
Second, we observe that drugs with a cancer and/or orphan indication represent a significant proportion of rejections across all agencies, with England’s NICE rejections being skewed most heavily towards cancer drugs despite the agency rejecting a below-average number of the cancer drugs it assessed over the study period (14% of assessed cancer drugs were rejected by NICE vs. sample average of 17%). On that point, it is interesting to note that one-third of the 80 cancer drugs in the sample also have an orphan indication (n = 26, 33%), which reflects the ‘personalised medicine’ shift over the past decade that has brought highly-specific and targeted cancer therapies to market.
Third, examining rejection trends over time reveals that Quebec, Australia and Scotland have, in general, rejected a greater proportion of submitted drugs from 2016 onwards (Figure 2b).
Finally, both listed and rejected drug-indication pairs present clinical and economic uncertainties, but a significantly higher proportion of rejected drugs is linked to both clinical and economic uncertainties; for example, 57% of rejections are linked to clinical benefit uncertainties compared with 33% of listings, and the same holds for practice generalisability (23% vs 16%), population generalisability (15% vs 12%), adverse events (22% vs 15%), inadequate follow up (42% vs 38%) and comparator concerns (12% vs 10%). Similar differences exist in the context of economic uncertainties, for example, in 42% of all rejections cost-effectiveness concerns were raised compared with 26% in all listing recommendations, and similar was the case in utility inputs (22% vs 12%), cost inputs (25% vs 20%) and economic model comparator inputs (11% vs 8%).
3.2. Factors associated with HTA rejections
In the model including all study countries (full sample), we used two alternative covariate specifications, the first (Model [1]) controlling for regulatory, disease, submission and clinical evidence characteristics, while the second (Model [2]) additionally controlling for non-addressed clinical evidence uncertainties; models [3] and [4] replicate the analysis for a restricted sample of countries using cost-effectiveness analysis (‘CEA countries’) for resource allocation purposes, therefore excluding France and Germany (the countries using comparative clinical benefit as the main decision criterion) (Table 3). In the restricted sample, we additionally control for whether an economic model was submitted, and whether a set of unaddressed economic evidence uncertainties were reported. All regressions control for agency and assessment year fixed effects (FE). Sensitivity analyses were conducted by including and excluding independent variables in the regression models to ensure robustness and absence of multicollinearity. Finally, a detailed look into the impact of unaddressed clinical and economic uncertainties on odds of rejection can be found in Supplementary Table 4.
Logistic regressions of HTA rejection for the full sample and a restricted sample of countries using cost-effectiveness analyses in assessments which excludes France (HAS) and Germany (IQWiG)
Notes: *CEA = cost-effectiveness assessment; includes all countries & agencies apart from France (HAS) and Germany (IQWiG).
Cluster robust SE in parentheses; Significance levels: ***p < 0.01, **p < 0.05.
In the full sample models, of the three regulatory pathways, only a parallel review process seems to be consistently associated with a lower probability of rejection (OR ranging between 0.43 and 0.41, 95%, p < 0.01). Interestingly, this relationship is driven by only three agencies that allow manufacturers to submit a parallel review request (Canada, Australia and Quebec), which are also the agencies with the highest overall rejection rates (15%, 20% and 36%, respectively). We can link this finding to our original hypothesis that drugs with evidence of higher quality for which manufacturers are confident about their approval by HTA agencies utilise the parallel review pathway to achieve faster market access. This relationship holds even when controlling for the quality of evidence and reported unaddressed evidence uncertainties, signalling a favourable treatment of the drugs that utilise this pathway over and above what the quality of evidence alone would predict. For the early approval pathway, we find a similar reduction in the odds of rejection as with parallel reviews but the strength and statistical significance of the association are less compelling.
In non-cancer drugs, an orphan indication is associated with a strong increase in the odds of rejection (OR ranges between 2.30 and 2.37, p < 0.01). Similarly, a cancer indication is associated with higher odds of rejection (model [1], OR 2.17, p < 0.05, model [1]), although this diminishes in significance when controlling for the likely poorer clinical evidence and higher uncertainties in their submissions (model [2], OR 1.76, where the effect is not significant at the 5% level). However, when considering the interaction term, it appears that the combination of an orphan and cancer indication considerably reduces the likelihood of rejection (OR ranges between 0.28 - 0.26, p < 0.01). Overall, an orphan indication seems to be a more powerful predictor of a negative outcome for both cancer and non-cancer drugs.
The impact of quality of evidence on rejection probability is significant when considering the composite index made up of the type of trial (at least one Phase III), the type of comparator (at least one active comparator/head-to-head trial) and the type of endpoints (at least one clinical endpoint). Low-quality evidence expectedly increases the odds of rejection (OR ranging between 2.07, in model [1], p < 0.01; and 2.20, in model [2], p < 0.01). When considering the impact of unaddressed clinical uncertainties on the odds of rejection, only concerns around clinical benefit increase the odds of rejection (model [2], OR 2.59, p < 0.01). Finally, a previous HTA rejection does not seem to change the odds of rejection vs listing and the effect is not statistically significant.
The above results are very similar to those obtained in the restricted sample comprising cost-effectiveness countries only (‘CEA countries). For the CEA countries only, the use of economic modelling in submissions is protective against a negative recommendation, although virtually all submissions to CEA countries make use of an economic model (96%). Uncertainties around cost-effectiveness has an important effect on increasing the probability of rejection (model [4], OR 2.07, p < 0.01) and the same holds for utility evidence used in economic models (model [4], OR 2.38, p < 0.01).
3.3. Inter-agency differences
The observed divergence in decision-making is assessed by, first, exploring differences in overall rejection rates and, second, examining the differences in the effect of selected regressors by agency. Canada, Quebec and Australia have considerably higher overall rejection rates compared to England, while France expectedly has lower rejection rates. Surprisingly, the closely related agencies of Scotland and Sweden also seem to exhibit higher rejection rates compared to England’s NICE, although these apparent differences become insignificant when accounting for the effect of unaddressed economic evidence uncertainty (particularly cost-effectiveness concerns) in the restricted sample of CEA countries (Supplementary Table 5).
Examining the differences in the effect of selected predictors by agency in Table 4, reveals a mixed picture. The findings suggest that England, Sweden and Quebec are more likely to reject cancer drugs (9.3%, 8.2% and 9.3%, higher probability of rejection compared to non-cancer drugs, respectively), while Australia and France are more likely to approve cancer drugs (9.4% and 2.3% reduction in the probability of rejection, respectively). Sweden is more likely to reject orphan drugs as well (10.1% higher probability of rejection compared to non-orphan drugs), whereas England seems more likely to approve these (4.1% lower probability of rejection). Scotland and Canada are also more likely to reject drugs with orphan indications (16.6% and 5.4% higher probability of rejection, respectively). However, for cancer drugs with an orphan indication, we find a strong universal reduction in the probability of rejection across all agencies (2.7% [Sweden] to 21.5% [Quebec] reduction in the probability of rejection across the study countries compared to non-cancer, non-orphan drugs). All agencies have a higher propensity to reject drugs when there are unaddressed clinical benefit uncertainties, the effect being most prominent in Quebec (17% increase in probability of rejection compared to submissions with no clinical benefit uncertainties). The same applies for unaddressed cost-effectiveness uncertainties, although a significant effect is found only in England (11% increase in rejection probability), Scotland (12% increase) and Quebec (20% increase).
Average marginal effects (ME) of interactions of selected variables with agency dummies1
Notes: 1 Each column represents a logistic regression of HTA rejection after interacting the respective regressor with agency dummies, controlling for covariates specified in model [2];
† Marginal effects for France (HAS) are reported from the full sample model [2] which excludes unaddressed economic uncertainty dummies. For all other countries, the CEA country restricted model [2] is used, which includes unaddressed economic uncertainty dummies.
Cluster robust SE in parentheses; Significance levels: ***p < 0.01, **p < 0.05.
4. Discussion
Our empirical approach presents a novelty which extends beyond simple descriptive analyses of HTA rejections. This is because controlling for several important confounders simultaneously, allows us to make inferences about the systematic differences in drug assessments across and between HTA agencies in our sample. There are several important points which should be taken away from our results.
First, while our research largely confirms earlier findings on the important drivers of HTA decision-making, it goes beyond to highlight that higher quality evidence in the form of head-to-head Phase III trials, which utilise direct clinical endpoints, have fewer clinical benefit uncertainties and higher cost-effectiveness all act protectively against a ‘do not list’ recommendation. Our analysis of cross-country differences in rejection rates confirms that important differences exist in the assessment of the same evidence by different agencies, even after accounting for the regulatory individualities of each. Such differences may be due to differences in the interpretation of evidence but also the existence of differences in social value judgements, such as rarity, severity, unmet medical need, or impact on family, among others, which may be explicitly incorporated into decision-making, or implicitly recognised on a per case basis at appraisal level.
Second, we find that, ceteris paribus, an orphan indication significantly reduces the risk of rejection in cancer drugs compared with non-cancer drugs. There are several challenges associated with the assessment of cancer medicines, which complicate decision-making, including high degrees of uncertainty, particularly in the size of clinical benefit, the type of endpoint used, and generalisability of the evidence and study populations, as there are often multiple indications per product, with some being combination therapies. However, an orphan indication in oncology medicines seems to sufficiently overcome the evidence shortfalls that may otherwise exist, reducing the likelihood of rejection. Presumably, the smaller expected budgetary impact of orphan oncology drugs (where the indication is narrower) and the higher level of unmet need makes them less prone to rejection compared to non-orphan cancer drugs.
Third, the consistency of our findings with respect to the association between orphan indications, cancer drugs and rejection probability may be misleading if we do not consider the ‘early access’ funding arrangements that are often in place for such treatments. For example, the UK’s Cancer Drugs Fund, which operated separately from NICE until 2016, has provided funding for many promising oncology therapies which had not demonstrated sufficient cost-effectiveness at the time of their initial assessment (Wood and Hughes Reference Wood and Hughes2020). Similar earmarked funds and compassionate use programmes for highly-specialised medicines have operated throughout other OECD countries (e.g. the French ATU, provincial cancer agencies in Canada) (Degrassat-Théas et al. Reference Degrassat-Théas, Paubel, Curzon, Le Pen and Sinègre2013; Srikanthan et al. Reference Srikanthan, Penner, Chan, Sabharwal and Grill2018). The existence of such ‘safety nets’ for access to important therapies which are rejected by HTA agencies may be creating a propensity for rejecting drugs which can be made available to patients through alternative funding avenues, particularly when considering the non-orphan oncology drugs. We should not also discount the uptake and use of financial risk-sharing agreements as an integral part of coverage negotiations across OECD countries (OECD 2020).
Fourth, we illustrate that certain process-related differences across agencies are non-trivial. Significant inter-dependencies in HTA practices seem to exist for closely linked health systems such as those in the UK and Sweden, as well as in France and Germany. HTA rejection rates are generally low for the aforementioned countries but are quite high in Canada and Australia, statistically powering the samples of the latter countries to reveal some important trends. The lack of an explicit orphan drug assessment framework in Canada (McMillan and Campbell Reference McMillan and Campbell2017) during our study period has likely led to our finding of a considerably increased rejection rate for orphan drugs by CDA. Presumably, if some of these drugs were formally designated as orphan, the impact of the orphan indication on rejection probability would have been lower. On the contrary, the establishment of the pan-Canadian Oncology Drug Review (pCODR) in the Canadian context in 2010 does seem to have led to non-biased reviews of cancer drugs, as the lack of a similar such review pathway by the Canadian province of Quebec seems to be driving a strong positive association between oncology indications and rejections in that province. In Australia, there is a similar lack of an orphan drug assessment framework as in Canada, but a separate funding arrangement for orphan drugs exists in the form of the Australian Life Saving Drugs Program (LSDP). The LSDP seems to have had a positive impact on the assessments of orphan drugs, reducing the likelihood of their rejection in contrast to the situation in Canada. The LSDP provides patients with access to drugs treating life-threatening rare diseases but strict listing criteria has meant that few drugs have been included in the LSDP list since its inception in 1995 (Maguire and Tilden Reference Maguire and Tilden2013; Australian Government Department of Health 2020).
4.1. Policy implications
A number of policy-relevant conclusions can be drawn from our analysis. First, the finding that certain agencies do not have processes tailored to the assessment of highly specialised or personalised treatments points to a need for reforming review pathways to consider drugs with atypical submissions separately. This is particularly important as an increasing number of highly individualised treatments are coming to market, rendering the conventional requirements of HTA agencies for randomised-controlled trial (RCT) evidence and robust cost-effectiveness analyses unrealistic. Indirect treatment comparisons and real-world evidence generation are obvious ways of overcoming evidentiary inadequacies, and are often combined with risk mitigation strategies, but their acceptability is often variable across settings. While randomised controlled trials have an important place in generating evidence, they are sometimes impossible as, for example, in the case of very rare diseases; and the lack of equipoise may render them unethical. Alternative trial designs, may also need to be implemented more broadly, particularly in the context of ever-increasing reliance on personalised medicine. These newer, or less familiar, study designs include basket trials, umbrella trials, Mendelian randomisation trials, step-wedge trials, ring trials, Baskerville trials and micro-randomised trials. HTA agencies will have to learn to use these newer designs if they are to remain relevant to evaluating evidence and to the practice of medicine and public health (Rawlins, Reference Rawlins2016). Although this may be an issue for the future, current pharmaceutical innovation may expedite this process.
Second, the widespread use of ‘early patient access’ schemes may be inadvertently affecting decision-making at the HTA level, relaxing requirements for cost-effectiveness and making it easier for manufacturers to enter markets prematurely. On that note, we argue that risk-mitigation strategies will be vital to ensure that only innovation offering significant additional clinical benefits to patients is rewarded and perverse incentives for skipping regulatory hurdles are not created.
Third, the persisting divergence in coverage recommendations across agencies strengthens the argument for increasing cooperation in HTA decision-making and reinforcing the role of international collaborative initiatives in sharing HTA ‘best practices’ (e.g. INAHTA, EUnetHTA), at least in the context of having some common denominators vis-à-vis the assessment of new drugs and dealing with uncertainty. The EU regulation on HTA cooperation is one step in that direction, at least in the European context, provided a common ground is reached on the complexity of requirements to suit different Member States’ needs (Kanavos et al., Reference Kanavos, Angelis and Drummond2019; Main et al, Reference Main, Schäfer and Kanavos2025).
Finally, the association of parallel review processes with a lower probability of rejection needs to be linked to the fact that drugs with high-quality evidence stand better chances of being approved by HTA agencies. Hence, parallel review pathways should be more widely available in order to expedite access to drugs that come to market with high-quality evidence and uncertainties that can be addressed by sponsors in a reasonable timeframe. Outstanding clinical evidence uncertainties can be addressed in a variety of ways, including evidence development in real-world settings and pay-for-performance schemes.
4.2. Study limitations
Our study has a number of limitations. The data collection methods which relied on analysts coding information found in published sources inevitably entail a certain degree of bias, partially mitigated by having multiple analysts review the same HTA reports and resolve conflicts through discussion. The absence of quantitative data on cost-effectiveness ratios and budget impact are also severely limiting our ability to draw meaningful conclusions about the results of our models. However, the frequent absence of these variables due to price confidentiality are a challenge for all such HTA studies. The modelling approach we employ allows us to raise some important points about the agencies in our sample, but the external validity of our conclusions remains unavoidably limited. Despite the considerable number of rejections which were analysed in our study compared to previous published papers, the rarity of the rejection event for some of the agencies leads to imprecisely calculated estimates and restricts statistical inference. Furthermore, a mixed methods approach using qualitative data from surveys or interviews with decision-makers may have been more appropriate to uncover the underlying mechanisms behind some of our findings. Future analyses with a larger sample would also enable a hierarchical model to be applied. Finally, the study findings pertain to specific HTA agencies within a defined time period, during which particular drug-indication pairs were assessed. Therefore, they should be interpreted with caution and should not be generalised to other contexts.
5. Conclusion
Having studied a sample of 1,405 drug-indication pairs, we have found that only 12.9% of the drugs are rejected by the 8 HTA agencies in our sample. Failure to satisfy HTA agencies is due to several factors relating to a product’s indication (cancer or orphan indication not seen in combination), the quality of evidence (low quality evidence in the form of a placebo-controlled, early phase trials with surrogate endpoints) and the existence of clinical benefit and cost-effectiveness uncertainties raised during the assessment process and which remain unaddressed by sponsors. We have demonstrated that persisting differences exist in the stringency of assessments carried out by different HTA agencies for different subsets of drugs, highlighting the importance of considering contextual factors when understanding the reasons behind reimbursement decisions. Future research will aim to examine drug assessments longitudinally, to determine how drug reimbursement decisions can change over time and analyse the implications of delays caused by multiple assessments on timely patient access to life-saving treatments.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S1744133125100212
Acknowledgements
The authors are grateful to three anonymous referees for helpful comments and suggestions in earlier drafts of the paper. All outstanding errors are the authors’ own.
Availability of data and material
The data informing this analysis are available upon request to the corresponding author.
Financial support
This work was funded by the European Union’s ‘Horizon 2020’ Research and Innovation programme (Grant Agreement No. 779312, ‘IMPACT HTA’). The views expressed in this paper are those of the authors and do not represent the views of the European Commission.
Competing interests
There are no conflicts of interest for any of the authors.
Open access
