Over the past few decades, health technology assessment (HTA) agencies in many countries (1–5) have been established to inform decisions about public reimbursement of new drugs after an assessment of the drugs’ economic profiles as well as efficacy and safety. These agencies review manufacturers’ submissions of selected new drugs to the HTA agency that include estimates of the cost-effectiveness of the new drug and the expected financial impact of the drug on short- and long-term healthcare budgets when the new drug is reimbursed at the requested price. Their submission guidelines include detailed guidance on the estimation of the cost-effectiveness of the new drug. The guidance for financial impact analysis provided by the HTA agencies is generally less detailed than guidance for cost-effectiveness analysis. Limited guidance on methods for financial impact analysis is given in the Institute for Quality and Efficiency in Health Care (IQWiG) and WellPoint guidelines (2;5). However, PBAC provides very detailed guidelines about how the financial impact should be estimated (4).
Within the Australian system, new drugs submitted for approval for reimbursement on the national Pharmaceutical Benefits Scheme (PBS) are assessed for relative safety, effectiveness, cost-effectiveness, and financial impact by PBAC. The PBAC operates two subcommittees that review and comment on the economic evaluation, if included; and the proposed use and financial impact. The outputs of an external evaluation by an academic center and the two PBAC subcommittees are collated for the appraisal discussion at the full PBAC, and a recommendation is made to the Minister of Health to list, with or without restrictions; not to list; or defer. If a rejection is recommended, the manufacturer may resubmit the application at the next available submission date. The final decision on whether to list the drug includes consideration of additional factors such as extent of unmet need and ethical implications.
A series of recent articles has focused on the possibility that financial (or budget impact) analysis has, at least implicitly, an impact on the probability that a new drug will be reimbursed (Reference Cohen, Stolk and Niezen6–Reference Niezen, Busschbach, Cohen and Stolk8) in addition to the well-proven impact on the likelihood of reimbursement of therapeutic value, cost-effectiveness, and burden of disease. These articles have pointed out that financial impact analysis is needed to fully understand the opportunity costs associated with the new intervention and its affordability (Reference Cohen, Stolk and Niezen7;Reference Mauskopf, Earnshaw and Mullins9–Reference Mauskopf11). However, despite the requirement to include financial impact estimates in submissions to HTA agencies, Niezen and colleagues (Reference Niezen, Busschbach, Cohen and Stolk8) concluded that “policy makers do not easily admit that they consider budget impact and are even reluctant to explicitly use budget impact as a formal criterion.”
Three recent quantitative studies (Reference Chim, Kelly, Salkeld and Stockler12–Reference Harris, Hill, Chin, Li and Walkom14) have estimated the impact of different explanatory variables on PBAC recommendations about reimbursement for new drugs. The primary focus of these studies was to estimate the impact of estimates of drug cost-effectiveness on the probability of being approved. All of these studies found that the probability of a drug being recommended for reimbursement in Australia by PBAC was correlated with its estimated cost-effectiveness ratio: the higher the ratio, the less likely approval. These studies used different analytical methods, including descriptive analysis (Reference Clement, Harris and Li13), probit multivariable analysis (Reference Harris, Hill, Chin, Li and Walkom14), and logistic regression analysis (Reference Chim, Kelly, Salkeld and Stockler12). All of these studies also looked at the effects of other economic and clinical variables on the reimbursement recommendation, including a measure of annual financial impact on the Australian drug budget. In all these studies, the financial impact on the Australian drug budget also was found to be a significant predictor of the reimbursement recommendation.
In this study, we extended these published analyses of PBAC data to include more recent data and to focus more specifically on the financial impact to the Australian drug budget. To provide a more complete understanding of the relationship between the financial impact on the Australian drug budget and the probability of PBAC recommendation for reimbursement, we subdivided financial impact into several categories and performed descriptive analysis as well as logistic regression, survival analysis using logistic regression, and recursive partitioning decision analysis. Survival analysis was performed to estimate the impact of multiple submissions on the reimbursement recommendation. Recursive partitioning decision analysis was performed to provide an indication of the relative importance of the different explanatory variables for the reimbursement decisions. All the multivariable analyses controlled for the cost-effectiveness ratio values and other variables that have previously been shown to be associated with the probability of a PBAC recommendation for reimbursement in Australia (Reference Chim, Kelly, Salkeld and Stockler12–Reference Harris, Hill, Chin, Li and Walkom14).
METHODS
The data file of recommendations by PBAC was created by abstracting data from the PBAC Web site (http://www.health.gov.au/internet/main/publishing.nsf/Content/public-summary-documents-by-product). Data were taken from recommendations made from July 2005 through November 2009. In the data file, a unique identification number was created for each product and its indication. If a product had more than one submission for the same indication, more than one record was created under the same unique identification number. However, if a product had multiple submissions that included a different indication, a new unique identification number was created for that product and indication. The data file included PBAC's recommendation (to recommend a listing with or without restrictions, not to recommend a listing, or to defer), the incremental cost-effectiveness ratio (cost per quality-adjusted life-year [QALY]), and the highest value of the financial impact presented in the summary document for each product. In addition, other variables were abstracted that have been shown to be associated with PBAC's reimbursement recommendations in previous studies (Reference Chim, Kelly, Salkeld and Stockler12;Reference Clement, Harris and Li13) (Supplementary Table 1, which can be viewed online at www.journals.cambridge.org/thc2013079).
The PBAC recommendation was the outcome variable in this study. The four categories of outcomes abstracted were combined to create a binary variable with categories recommended (yes, restricted yes), and not recommended (no, deferred). The predictors of the PBAC recommendation that were tested in the analyses were either abstracted or derived variables that were classified as either economic variables or clinical variables as seen in previous studies (Reference Chim, Kelly, Salkeld and Stockler12–Reference Harris, Hill, Chin, Li and Walkom14).
Economic variables included the following: (i) Population size, created from reported patient population size treated per year: low (<10,000), medium (10,000–50,000), and high (>50,000). (ii) Estimated financial impact per annum, using the upper bound value presented and estimated for the impact on only the drug budget: ≤A$0, >A$0 to <A$10 million, ≥A$10 million to ≤A$30 million, and >A$30 million. A categorical variable was used for the analysis because the PBAC summaries presented financial impact as an upper bound or range rather than a continuous variable for most of the submissions. (iii) Cost per QALY, using the upper bound base case analyses: >A$0 to ≤A$45,000, >A$45,000 to ≤A$75,000, and >A$75,000 as used in the Chim and colleagues study (Reference Chim, Kelly, Salkeld and Stockler12) of the impact of cost-effectiveness on reimbursement decisions. A fourth category, no cost-effectiveness analysis presented, was assigned to those submissions that used a cost-minimization approach for the economic evaluation.
Clinical variables included the following: (i) Active comparator (yes or no) that indicated whether an active comparator was used as the comparison group in at least one of the pivotal studies; (ii) Manufacturer claim for the clinical benefits of the new product: noninferior or equivalent or superior; (iii) Comparative clinical evidence available from randomized clinical trials only (RCT) or from RCT data plus a meta-analysis or indirect comparison analysis (RCT plus meta-analysis or indirect comparison analysis); (iv) Disease category (oncology or other), as a proxy measure of likelihood of reduced life expectancy and the “dread” factor associated with the disease (Reference Chim, Kelly, Salkeld and Stockler12); and (v) Surrogate end point (yes OR no), derived from a review of the end points in the submission.
The unit of analysis for all analyses was the unique drug and indication submission after July 2005. Only the first observed submissions of the unique drug and indication combination within our database were included in the univariate and initial multivariable logistic analyses because subsequent resubmissions were correlated with the first observed submission. All analyses were performed in SAS 9.3 or JMP 8. A test result was declared statistically significant if p value was < .05 and marginally statistically significant if p value was > .05 but ≤ .1.
First, a univariate analysis was performed to explore the association between the PBAC recommendation and the variables described previously. The association was tested by Pearson's chi-squared test. Next, a multivariable logistic regression was performed to evaluate the relationship between the probability of a positive recommendation and the categorical financial impact, while adjusting for other factors. The variables included in the logistic model were those that had an association with the recommendation with a p value ≤.30 in the univariate analysis (Reference Hosmer and Lemeshow15).
A discrete time-to-event analysis was performed, including all extracted data: both the first observed submission data and all resubmission data to determine the relationship between multiple submissions and PBAC recommendations while accounting for the correlations between repeated submissions and to determine the impact of the omission of the resubmissions on our estimates for financial impact. We performed the analysis using the logistic model as described in Allison (Reference Allison16). Total submission count was determined by counting the number of times the same drug plus indication was submitted. Only nine records had a total count of four or more, and these records were omitted from the analysis. The variable time since previous submission was included because the resubmissions happened at irregular intervals. A submission or resubmission for a drug plus indication that happened once or more than once but that was not recommended for reimbursement was considered to be a right-censored observation. Also, submissions could be left censored if data for the first observed submission indicated that previous submissions had occurred before July 2005.
Finally, a decision tree analysis was performed using the recursive partitioning algorithm in JMP analysis software (SAS, Cary, NC). Recursive partitioning is a nonparametric classification technique that splits into subsets, called nodes, observations with similar response values for predictor variables (17). In each node, the predictor variable with the strongest association to the outcome variable is chosen for splitting the node. For a categorical outcome, the recursive partitioning algorithm in JMP analysis software uses log10 (p value), also called logworth, where the p value is an adjusted p value given by a likelihood ratio test. To avoid overfitting, only partitions that had a logworth value ≥1.12, which corresponded to log10 (0.05), were performed. Because this is a nonparametric method, it has an advantage over logistic regression by not assuming any functional form for the association between predictors and outcome. Furthermore, recursive partitioning has the advantage of detecting possible complex interactions between predictors that may not have been detected by the logistic regression, and because it is easy to visualize and interpret, it is suitable for a decision-making process (18). Moreover, the relative importance of the predictors can be inferred by the order in which they partition the data (i.e., the earlier the predictor is used by the partition algorithm, the more important it is).
The logistic regression and decision tree analyses were also performed using only those submissions with a reported cost per QALY to assess the importance of financial impact in this subset of the total submissions.
RESULTS
A total of 260 submissions, representing 214 unique drug plus indication combinations and 46 resubmissions during the data abstraction time period, were extracted from the PBAC Web site. Most, or 170 of the 214 (79.5 percent) unique drug plus indication combinations, were not submitted before July 2005; 27 (12.6 percent) were submitted once before July 2005; and 17 (7.9 percent) were submitted multiple times before July 2005. Of the 260 submissions, 106 (40.8 percent) were recommended for reimbursement, 47 (18.1 percent) were partially recommended for reimbursement (i.e., recommended with restrictions), 100 (38.4 percent) were not recommended for reimbursement, and 7 (2.7 percent) were deferred. Therefore, the binary variable PBAC recommendation for reimbursement had 153 submissions (58.9 percent) that were recommended and 107 submissions (41.1 percent) that were not recommended.
Table 1 presents the univariate association between recommendation category and potential predictors. Five variables—financial impact, cost per QALY, manufacturer's claim, active comparator, and disease category—had a statistically significant association with the PBAC recommendation. With the exception of the financial impact category >A$30 million, the percentage of submissions that were recommended decreased as the financial impact and the cost per QALY increased. In addition, 74.3 percent of the submissions that did not report a cost per QALY were recommended as compared with 40.0 percent for those submissions with a reported cost per QALY. The percentage of recommendations was higher for entries that claimed noninferiority or equivalence, that had used an active control as the comparator in at least one pivotal clinical trial, and that were not in the oncology category. The other included variables (population size, comparative clinical evidence, and surrogate end points) did not have a statistically significant association with the PBAC recommendation. Based on the univariate analyses, six possible predictors of PBAC recommendations with a univariate p value ≤.30 were included in the logistic models.
Univariate Association Between PBAC Recommendations and Predictors
a Percentages were calculated out of the available data for the respective variable category.
bP value was calculated using Pearson's chi-square test for difference between the variable categories.
PBAC, Pharmaceutical Benefits Advisory Committee.
Figure 1 and Supplementary Table 2, which can be viewed online at www.journals.cambridge.org/thc2013079, present the results of the multivariable logistic analyses. Only the effect of financial impact (p = .0242), cost per QALY (p = .0235), and active comparator (p = .0365) were statistically significant. After adjusting for the other factors in the model, the only statistically significant odds ratios for financial impact were for comparing either category ≥A$10 million to ≤A$30 million (0.12; 95 percent confidence interval [CI]: 0.03–0.51) or >A$0 to <A$10 million (0.16; 95 percent CI: 0.04–0.60) with the category ≤A$0. The odds ratio for the financial impact category >A$30 million compared with the category ≤A$0 was not significant (0.25; 95 percent CI: 0.05–1.34). For cost per QALY, the only statistically significant odds ratios were for comparing the category >A$75,000 with either the category >A$0 to A$45,000 (0.11; 95 percent CI: 0.02–0.55) or to the category none (0.06; 95 percent CI: 0.01–0.40). The odds of recommending a drug submission that used an active comparator were 2.49 (95 percent CI: 1.06–5.85) times the odds of recommending a drug submission that used placebo as the primary comparator. Figure 1 presents a plot of selected odds ratios obtained from the first logistic model. Financial impact was not statistically significant (p = .1801) when the same model was run using only submissions that reported a cost per QALY (n = 103). Cost per QALY was the only statistically significant factor (p = .0158) in this model.
Multivariable logistic regression results (n = 204): odds ratios with 95% CI plots. CI, confidence interval; QALY, quality-adjusted life-year; RCT, randomized controlled trial.
Figure 2 and Supplementary Table 2 present the results of the analysis of discrete-time survival data, using multivariable logistic analysis. The statistically significant effects were total submission count (p = .0029), financial impact (p = .0021), cost per QALY (p = .0135), and an active comparator (p = .0229). The odds ratio estimates for financial impact, cost per QALY, and an active comparator were very similar to the corresponding odds ratios obtained in the first model. After adjusting for other factors in the model, the odds of recommending a drug that was submitted a third or second time were 9.62 (95 percent CI: 2.47–37.42) or 3.66 (95 percent CI: 1.35–9.96) times the odds of recommending a drug submitted the first time, respectively. The odds of recommendation were 0.83 lower for a 1-trimester increase in time since the previous submission. Figure 2 presents the odds ratio plots for the second logistic model. When the same model was run only using submissions (n = 126) with a reported cost per QALY, financial impact was still statistically significant (p = .0223).
Multivariable logistic regression results for discrete time-to-event data (n = 238): odds ratios with 95% CI plots. CI, confidence interval; QALY, quality-adjusted life-year; RCT, randomized controlled trial.
Figure 3 presents the recursive partition decision tree. In addition to the variables used in the logistic analyses, population size and surrogate end point were also included. Categorical financial impact was the most important factor to make the first partition (logworth = 7.18) by grouping the three financial impact categories, >A$0 to <A$10 million, ≥A$10 million to ≤A$30 million, and >A$30 million, into a single category and comparing it with ≤A$0. Cost per QALY, active comparator, and disease category were the next predictors selected for recursive partitioning. For cost per QALY, the categories none and >A$0 to A$45,000 were combined by the model program and compared with >A$45,000 to ≤A$75,000 and >A$75,000. The results of the recursive partitioning model indicated that the chance of being recommended for reimbursement for drug submissions with a financial impact ≤A$0 was 91.4 percent, and the chance for drug submissions with a financial impact >A$0 and a cost per QALY either not estimated or A$0 to ≤A$45,000 was 57.5 percent as compared with 24.4 percent for submissions with a financial impact >A$0 and cost per QALY >A$45,000. Lower chances for reimbursement recommendation were estimated for drug submissions with a financial impact >A$0, a cost per QALY either not calculated or >A$0 to ≤A$45,000, and without an active comparator (29.6 percent). When a new tree was constructed considering only submissions with a reported cost per QALY, the cost per QALY was the only important factor in the partitioning analysis.
Recursive partition decision tree (n = 204).
DISCUSSION
The results of the analyses presented in this study indicate that the estimated financial impact of a drug on the Australian drug budget is a predictor of the PBAC reimbursement recommendation, even when controlling for the cost-effectiveness ratio and other confounding variables. In the descriptive analysis, there was a gradient in probability of reimbursement, with the highest probability for drugs that were estimated to be cost-saving and the lowest probability for drugs that were estimated to increase annual costs between A$10 million and A$30 million. However, probability of recommendation was higher for those submissions with a financial impact of >A$30 million compared with those with a financial impact of ≥A$10 to ≤A$30 million. The logistic analyses demonstrated that this pattern was similar even when controlling for the cost-effectiveness ratio and other confounding variables and the number of submissions and even when only including those submissions that presented a cost per QALY estimate. A review of the submissions with an estimated financial impact >A$30 million (22 submissions) found that products in this category that were recommended for reimbursement either had cost per QALY estimates in the lower end of the A$0 to A$45,000 range, or very favorable clinical benefits, or indications where there were no alternative treatments that might explain this seemingly anomalous result.
The impact of multiple submissions on the probability of recommendation for reimbursement was significant with an odds ratio of 9.62 for a third and 3.66 for a second observed submission compared with the first observed submission. Supplementary Table 3, which can be viewed online at www.journals.cambridge.org/thc2013079, presents a summary of the changes in the categorical values among the 29 multiple submissions. A review of these changes indicated that, although reductions in price were likely key factors in obtaining a positive reimbursement recommendation in many cases, changes in the clinical data submitted were also influential in obtaining a positive recommendation.
Finally, the recursive partitioning decision analysis supported the importance of a positive financial impact for the reimbursement decision with the full database because the variable with the greatest discriminative power for reimbursement recommendations was shown to be a positive financial impact of any magnitude, followed by the cost per QALY. However, the results for the subset of submissions that report a cost per QALY, all of which had a positive financial impact, indicated that the cost per QALY variable had the greatest discriminatory power.
A threshold value of A$10 million was used for the financial impact analysis because full approval by the cabinet of the federal government was needed for drugs when their annual financial impact was expected to exceed A$10 million in any 12-month period within the first 4 full years of product listing. Having mandated that a multiyear financial impact analysis be included in submissions has allowed the PBS to put in place price-volume contracts for those drugs that exceed the A$10 million per year limit (Reference Duckett19), although the specific agreements are confidential.
The results of our analyses are supported by and extend those from three previous analyses of PBAC data (Reference Chim, Kelly, Salkeld and Stockler12–Reference Harris, Hill, Chin, Li and Walkom14). All three studies included financial impact on the healthcare sector as an explanatory variable. In the supplemental appendix of Clement et al. (13), in a descriptive analysis of submissions not needing a cost-per-QALY estimate, the probability of reimbursement in Australia was much higher (81 percent) than for those needing a cost-per-QALY estimate (44 percent) (Reference Clement, Harris and Li13). In the Harris et al. (Reference Harris, Hill, Chin, Li and Walkom14) study, for a sample of the decisions that also included an estimate of the cost-per-QALY gained, Harris and colleagues (Reference Harris, Hill, Chin, Li and Walkom14) estimated that for each increase of A$5 million in financial impact above the mean value, the probability of a PBAC recommendation for reimbursement would decrease by 0.03. However, by excluding any submissions that estimated financial cost-savings and many that estimated a budget impact <A$10 million, they may have underestimated the importance of the financial impact for reimbursement decisions, especially for drugs with multiple submissions. Finally, Chim and colleagues (12) compared the impact on reimbursement of two financial impact categories: <A$10 million and ≥A$10 million. In a logistic analysis, they estimated a statistically significant odds ratio of reimbursement of 0.46 for submissions with a financial impact of ≥A$10 million compared with those with a financial impact of < A$10 million.
The strengths of our analysis included the use of multiple analytic techniques that all provided similar results. The use of the survival analysis allowed us to include data from multiple submissions as well as right- and left-censored submissions, appropriately accounting for the correlation among repeat submissions of the same drug/indication pair. The use of the recursive partition decision analysis estimation technique is important because the results provided a simple stepped process for predicting whether a submission will be recommended for reimbursement in Australia.
The analysis also has some limitations. The major limitation is that, in many cases, the financial impact on the drug budget or the cost per QALY was provided in the public summary document only as an inequality or range. Thus, it was not possible to enter financial impact into the model as a continuous variable. Instead, we created four financial impact categories. Also, the sample size was relatively small (N = 260), limiting our ability to include a high number of explanatory variables or more than four categories of financial impact. There was also the possibility of multicollinearity among the independent variables used in the analysis. To test for this, we estimated “tolerance” scores for each independent variable and found that all variable scores were above 0.40, values not considered to be of concern for multicollinearity (Reference Allison16). Supplementary Table S4, which can be viewed online at www.journals.cambridge.org/thc2013079, presents the regression results. The choice of oncology as a proxy for disease severity was also a source of uncertainty because other diseases may also be associated with very limited life expectancy and “dread.”
The implications of the findings in this study are that, in Australia, the financial impact on the drug budget is an important determinant of whether a new drug is recommended for reimbursement with the strongest positive impact for products that have a zero or negative financial impact. Our findings are consistent with the conclusions of the recent published analyses on the possibility that financial impact plays a role in the HTA agency reimbursement recommendations, although not specifically listed as a criterion (Reference Cohen, Stolk and Niezen6–Reference Niezen, Busschbach, Cohen and Stolk8). In addition, the explicit instructions provided in the PBAC submission guidelines for performing financial impact analysis indicate the potential importance of these results for the PBAC review. Although financial impact is a specific part of the decision making for reimbursement in Australia, this is not necessarily the case in other countries. Whether financial impact influences reimbursement decisions in these countries is an empirical question that can be resolved only by performing similar analyses in these jurisdictions.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: www.journals.cambridge.org/thc2013079
Supplementary Table 2: www.journals.cambridge.org/thc2013079
Supplementary Table 3: www.journals.cambridge.org/thc2013079
Supplementary Table 4: www.journals.cambridge.org/thc2013079
CONTACT INFORMATION
Josephine Mauskopf, PhD, Vice President Health Economics (jmauskopf@rti.org), Costel Chirila, PhD, Senior Statistician, Biometrics, Catherine Masaquel, MPH, Associate Director in the Market Access and Outcomes Strategy, RTI Health Solutions, PO Box 12194, Research Triangle Park, NC, 27709
Kristina S. Boye, PhD Senior Director Global Health Outcomes, Diabetes, Lee Bowman, PhD Senior Director, Global Health Outcomes Oncology, Julie Birt, PharmD Senior Research Scientist, Global Health Outcomes, Health Technology Assessment – Center of Expertise, David Grainger, BS Global Public Policy Director, Lilly and Company, Indianapolis, Indiana, United States
CONFLICTS OF INTEREST
Josephine Mauskopf, Catherine Masaquel, Costel Chirila report payment to their institution by a contract from Eli Lilly and Company for leading the research reported in the manuscript.
Open access
