Understanding the Reasons for Differences in Relative Effectiveness

Eichler et al. (Reference Eichler, Bloechl-Daum and Abadie1) set out the methodological options and challenges in measuring relative efficacy. Their subsequent comprehensive review (Reference Eichler, Abadie and Breckenridge5) identified two reasons for an “efficacy-effectiveness gap”: biological (including genetics, age, sex, co-morbidity and baseline severity of disease) and behavioral (e.g., variation in adherence to the treatment regimen). Economists would add that any understanding of a potential “efficacy-effectiveness gap” requires translating trial outcomes into long-term health outcomes. However other factors will also play a role, notably the health system. For example, the health system and the environment impact on disease severity and co-morbidity. Adherence is not only behavioral, but also depends on the organization and resources used for managing adherence, and thus the efficiency of the healthcare system. The selection of patients for treatment and the management of patients over time will depend on incentives, organization and resources in the healthcare system. These elements may differ across jurisdictions and will impact on long-term outcomes.

Estimating Relative Effectiveness: Empirical Methods

Currently relative effectiveness, as opposed to relative efficacy, is treated in the literature and assessed by HTA and reimbursement agencies in one of three ways: (i) It is assumed to be the same as relative efficacy, and any potential “efficacy-effectiveness gap” is ignored, that is, modelling is based on clinical trial data. (ii) It is estimated using decision analytical modelling. However, this technique is typically used to project longer term health outcomes from short term or intermediate efficacy measures, or to make indirect comparisons, rather than to model the impact of the health system on the translation of the efficacy of a technology into outcomes in routine care. An intermediate case is when the relative risk reduction from a trial is applied to the absolute risk level for different groups of patients in different countries (Reference Lindgren, Jönsson and Yusuf17). However, this approach does not take into account other factors that may influence relative effectiveness, for example differences in adherence between countries. (iii) Observational studies or pragmatic clinical trials are proposed as methods to explore outcomes in routine clinical practice within a healthcare system (Reference Hollingsworth and Wildman17–Reference Hollingsworth and Wildman19). The number of observational studies being undertaken within health systems is increasing, but they can be time consuming and expensive unless relevant data are already routinely captured by a healthcare system. Analysis can be complex and results open to differing interpretations, although methods to tackle confounding and other issues are improving. Pragmatic trials also can be time consuming and expensive unless infrastructure investment is put in place and recruitment streamlined.

None of these approaches are routinely used to detect and explain differences in relative effectiveness across health systems. A review of the literature (Reference Mestre-Ferrandiz, Puig-Peiró and Towse8) found no observational studies investigating variations in relative effectiveness across Member States, although there were safety studies and studies comparing other aspects of care across several Member States. It is, therefore, not possible to address the question of differences in relative effectiveness on the basis of current evidence.

One route forward is for more observational studies or pragmatic clinical trials to be conducted across Member States. Given the cost and complexity of conducting them, however, it would make sense to explore another approach which can use existing data, and could be used to help identify whether differences in relative effectiveness were likely to be of an order to merit further study.

The Conceptual Framework: A Health Production Function Model of the Determinants of Relative Effectiveness

The relative effectiveness of a new technology is likely to depend on how it is combined with other health inputs, and a range of other variables that influence the overall relationship between inputs and health outcomes in the underlying “health production function.”

Assuming the comparator treatment is one of the inputs in the initial health production function, then relative effectiveness is the change that results when the new treatment is introduced relative to that obtained from the existing (or usual) treatment (and other inputs). In this context, “usual care” in the health system will be a key determinant of the relative effectiveness of any new treatment. The underlying research questions are: if a medicine for, say, breast cancer is granted Market Authorization by the EMA, indicating that the new technology is available in all EU countries, do we expect that its relative effectiveness will be different across them? If so, why?

One way to investigate influences on relative effectiveness is to treat this analytically as a multilevel model, separating the different levels of influence. Influences may operate on one or more of three main levels: patient, provider, and the healthcare environment and system (see Table 1): (i) “Individual” patient level factors refer to the characteristics of the patients being treated, including their demographic, genetic, and clinical characteristics. Not only may the patient mix gaining access to the technology differ but underlying population health may also differ between health systems. The same technology in the same setting may, therefore, produce different health effects depending on patient characteristics. Even if relative efficacy is the same across two health systems, if baseline risk differs then size of the absolute incremental effect measured by relative effectiveness may differ. (ii) The characteristics of the providers. The same technology may produce different health effects on patients with similar characteristics depending on how it is used and combined with other technologies in clinical practice (e.g., screening policy, or treatment setting; see Table 1). Clinical practice may vary in other ways that influence treatment outcomes (both from new technologies and comparators) for example due to economic and other incentives for providers. (iii) Healthcare environment and system factors. The availability or otherwise of technologies within the healthcare system and how technologies are used (and hence the health effects on patients) (i.e., “comparators” in Table 1) is often driven by P&R policies and clinical guidelines from regional or national decision makers. Waiting times for referral and then access to specialist care also vary between (and within) countries.

Table 1.

Overview of Factors That May Influence the Relative Effectiveness of a Medicine or Other Technology

Some determinants of outcome operate and interact at several levels. Five-year survival rate for breast cancer may differ across countries due (in part) to differences in all-cause mortality rates. These may be due to individual-level effects (e.g., smoking status), to environmental effects (e.g., taxes on cigarettes), or to health system public health measures restricting smoking. The key is to identify factors driving outcomes and ensure they are included, but only once. In this example, the variable for smoking would be most appropriately classified and included in the analysis as an individual level effect. In the accompanying article (Reference Puig-Peiro, Mason, Mestre-Ferrandiz, Towse, McGrath and Jonsson15), we provide an overview of factors affecting breast cancer outcomes identified from a literature review, grouping them according to the multilevel approach: “individual level,” “provider level,” and “environment and healthcare system level” set out in Table 1. The results are set out in Table 2 of Puig-Peiro et al. (Reference Puig-Peiro, Mason, Mestre-Ferrandiz, Towse, McGrath and Jonsson15).

The Malmquist DEA Model for Measuring Relative Effectiveness

The theory and methods for estimating health production functions have developed rapidly over the past 2 decades. Important contributions include the development of econometric approaches such as the stochastic frontier approach (Reference Zuckerman, Hadley and Iezzoni20;Reference Worthington21), and also data envelopment analysis (DEA) (Reference Hollingsworth, Dawson and Maniadakis22;Reference Hollingsworth23), including the use of Malmquist indexes which allow use of multiple input, multiple output models without prices. The latter is particularly important in a European healthcare perspective, where prices vary between countries, and often are not possible to observe and calculate in public healthcare systems. In addition, for some inputs to the health production function, such as an individual's time and effort used for prevention and treatment prices are not directly observable, as is also the case for the output health. Using index numbers that include prices for measurement of changes in total factor productivity, that is, relative effectiveness is not an option.

A further advantage of the Malmquist approach is that, once the production technology is estimated, either as a production frontier or its dual the cost frontier, one can decompose the change in two component parts: efficiency change and technical change. The methodology proposed by Färe et al. (1994) (Reference Färe, Grosskopf, Lindgren, Roos, Charnes, Cooper, Lewin and Seiford24) uses DEA to calculate distance functions to produce the Malmquist total factor productivity index, and then decomposes this into technical and efficiency components respectively. This decomposition is important because the change in relative effectiveness in a given countries is made up by a combination of changes in the frontier, and how the country is moving in relation to the frontier.

Figure 1 illustrates how a Malmquist DEA model could help identify and explain potential cross-country variations in the relative effectiveness of the introduction of a new treatment for breast cancer treatment. It can be used to study and decompose the productivity changes both in terms of technical change (best practice change, shown by a shift in the frontier) and technical efficiency change (countries moving closer to the frontier), that is, to understand how the introduction of a new technology changes outcomes, and hence to establish its relative effectiveness. This is essential for the design of policies aimed at improving relative effectiveness.

Figure 1.

Illustration of the Malmquist Index applied to the identification of possible variations in the relative effectiveness of breast cancer treatments.

For illustrative purposes, we assume just two inputs, but in practice the estimation of these models can include multiple inputs and patient attributes such as those set out in Table 1 that will impact on the patient outcomes. The horizontal axis measures the number of “physician visits” as a proxy for the intensity of monitoring and nondrug treatment, and the vertical axis measures the number of doses of “drugs” as a proxy for the intensity of drug treatment. We could also have used for example the share of visits to a specialist on the horizontal axis and an index of “vintage,” that is, year of introduction, for drugs. For illustration, we show the various combinations of the two variables that yield an 85 percent 5-year survival rate. The Production Frontier in Figure 1 is data-driven: it is the envelope of the most efficient data points observed from all countries in the group. We compare the average performance of two of the countries, A and B.

In period t, before the introduction of the new drug, efficiency is not the same in the two countries: e_A is closer to the production frontier in period t than is e_B. They use different combinations of input factors, which may reflect differences in relative prices between drugs and physician visits in the two countries. The distance function 0c/0e represents the technical inefficiency in country A and B respectively in period t, that is, how much the use of drugs and visits could have been reduced if they had applied the best practice use suggested by the production frontier at period t.

When a new drug or procedure is introduced in period t+1, the production frontier shifts inward, indicating an improvement in technical efficiency to produce the same level of health outcome with fewer of each of the inputs. The frontier may not change by the same proportion for both countries suggesting that one country may increase its efficiency in a larger proportion than the other after the introduction of the new drug.

The distance functions 0a/0f shows the new degree of inefficiency, that is, actual performance compared with best practice on the radial from the origin for the two countries, respectively.

For each observation, the change from period t to t+1 in efficiency can be computed as the ratio between the two distance functions, and the technical change as the geometric mean of the product of two ratios. These are defined in the key to Figure 1. The total productivity change, or change in relative effectiveness is the product of the two ratios. All observations can be aggregated to an overall productivity change, using the geometric mean.

The optimal mix of drugs and physician visits that would occur once the new drug was available along the production frontier in period t+1 depends on their relative prices and can be studied separately.

We show, in Online Supplementary File 1, the potential use of this approach with a very simple calculation and illustration of the technical efficiency with which combinations of breast cancer screening and treatment with trastuzumab are used in seven European countries.

Strengths and Weaknesses of the Approach

Using such an approach to empirically identify and explain differences in relative effectiveness may encounter some challenges in practice. For example, it would require data on all relevant factors to be available in a manner that facilitates cross-country comparisons. Furthermore, depending on the specific factors identified as relevant to understanding relative effectiveness in a specific disease area, relationships between explanatory variables (e.g., adherence may depend on socioeconomic status) can confound identification of their relationship to outcomes. However, such challenges are common to many studies and do not preclude useful and valuable research being undertaken.

Valid and reliable price data for (non-drug) inputs and outputs are generally not available, so techniques to estimate cross- and time-series variations in the relationship between inputs and outputs without price data, such as the approach illustrated in Figure 1, are an important step forward. They have been extensively used to understand the relative efficiency of providers (usually hospitals) within a single healthcare system and to compare the performance of healthcare systems; see Hollingsworth and Wildman (Reference Hollingsworth and Wildman25), Greene (Reference Greene26), and the EUROHOPE project (27). Jacobs et al. (Reference Jacobs, Smith and Street28) and Coelli et al. (Reference Coelli, Rao and O’Donnell29) provide a comprehensive review of the main econometric approaches for modelling health production functions.

Questions for the decision makers, based on such an analysis, should include: (i) whether a country can get closer to best practice and improve relative effectiveness; (ii) whether the likelihood of gaps between relative efficacy and relative effectiveness mean that post-launch studies are required; and (iii) whether potential differences between countries A and B mean that separate post-launch studies are needed in each.

We are not proposing econometric and economic modeling as an alternative to a prospective observational study or pragmatic trial to understand differences across healthcare systems in the relative effectiveness of a technology. Rather, we see this analytical tool as complementary: one that can use existing data (including data already available from post-launch studies) to explore the extent to which the performance of the same technology differs, or might be expected to differ, across settings as a result of differences in populations and healthcare systems.

Policy Implications

There are prima facie reasons why we might expect some convergence in relative effectiveness across the EU over time: the diffusion of medical technology is becoming global and there is a flow of information about best practice. It is also possible to observe some convergence of healthcare systems in terms of financing, organization, and performance. Yet our findings suggest it is reasonable at the moment to expect meaningful and persistent variations in relative effectiveness between and within countries. Although the literature indicates that current understanding of what might be important is limited, each case is likely to be driven by a few key factors. The health production function is a tool for identifying these at an early stage and making predictions of their likely quantitative impact. However, it can also serve as a basis for targeting observational studies and pragmatic clinical trials post launch, to test predictions and gain further insight into how to best make use of the new technology, including the potential advantages and disadvantages of using coverage with evidence development and risk sharing arrangements to assess and reward relative performance.

There are four broad policy implications from our analysis. First, at a time when we are seeing that countries increasingly look at P&R decisions and HTA assessments in other countries, the debate around the transferability of relative effectiveness evidence for P&R decisions could be better informed by an understanding of what factors in a healthcare system are likely in practice to impact on the incremental performance of a technology.

Second, the increasing demand for post-launch studies to address uncertainty about relative effectiveness of drugs needs to be put in the context of such an understanding. Studies are expensive and complex and unnecessary duplication of effort could waste time and resources and delay patient access to effective treatments. A collaborative approach between industry and governments (and in the EU across the twenty-eight Member States) is required to: (i) Limit the number of extra studies. An improved understanding of the drivers of relative effectiveness can help prioritize these studies. They can be limited to those needed to explore key expected differences between health systems that could impact patient outcomes. (ii) Collect and provide access to necessary data on the relevant contextual factors. At present, access to routine data (for example from registers) is often restricted. In Sweden, a public inquiry described the registers as “a gold mine” for collecting evidence to improve health care and health. Such registers are, however, often oriented to answering specific clinical questions and need to be augmented with a wider set of relevant factors and outcomes for the analysis of relative effectiveness.

Third, where differences in relative effectiveness reflect variation in the efficiency of healthcare systems, good practice can be identified and shared. Decision makers in poorly performing health systems, who are responsible for the procurement of new technologies and for managing healthcare providers, can learn from good practice elsewhere. Relative effectiveness is the outcome of an interaction between a technology, patients, and the health system (including the providers). The role of the latter and how to improve efficiency is often missing from debates such as that about developing an EU assessment of relative effectiveness. Failure to improve health system performance risks wasted resources or poorer health outcomes.

Fourth, if industry remuneration and regulatory benefit–risk assessment increasingly depend on the actual effectiveness of the healthcare system in using a drug, as opposed to relative efficacy at launch, then we may see an increasing use of “pay for performance” and risk sharing agreements. Such agreements are, in effect, seeking to estimate the health production function and link payment in some way to the use or outcome. Without an understanding of the contextual factors, there is a danger of companies taking the “risk” for factors that are really the responsibility of the health system and vice versa. The potential for using post-launch follow-up studies to reduce the time and cost of drug development before market authorization in some areas (cancer being an example where fast access is important and risks can be managed) needs to be informed by an understanding of the key factors driving relative effectiveness.