Highlights

What is already known:

• • Matching-adjusted indirect comparison (MAIC) methods are increasingly used in health technology assessment (HTA) submissions to adjust for population differences.

• • MAIC estimates the comparative effectiveness of interventions for the population represented by the trial with aggregate data (AgD).

What is new:

• • We present an illustration demonstrating an MAIC paradox in which the comparative effectiveness conclusions are reversed by switching the availability of IPD and AgD while adjusting the same set of effect modifiers.

• • Additionally, we examine how variations in covariate distribution overlap of effect modifiers between trials influence the estimated comparative effectiveness.

Potential impact for Research Synthesis Methods readers:

• • Through an illustrative example, we emphasize the vital importance of clearly defining the target population when applying MAIC in HTA submissions.

• • We recommend providing de-identified IPD to HTA agencies to enable a more accurate assessment of comparative effectiveness for the target population.

1 Background of MAIC

Effect modification occurs when the magnitude of the effect of a treatment on an outcome differs depending on the value of a third variable. For example, studies indicate that Black individuals may experience less favorable outcomes compared to non-Black individuals when treated with angiotensin-converting enzyme (ACE) inhibitor-based therapies.Reference Exner ^{1 ,} Reference Ogedegbe, Shah and Phillips ² In health technology assessments (HTAs), pharmaceutical companies are required to benchmark their new drugs against the prevailing standard of care for reimbursement decisions by HTA agencies.Reference Dias, Sutton, Ades and Welton ³ Nevertheless, the presence of effect modifiers can pose a unique challenge in comparing treatments when there is a lack of head-to-head trials. The traditional Bucher method,Reference Bucher, Guyatt, Griffith and Walter ⁴ which compares the relative treatment effects of two interventions assessed in two randomized trials without covariate balancing, is limited to scenarios where all effect modifiers are balanced across trial populations.Reference Phillippo, Ades, Dias, Palmer, Abrams and Welton ⁵ When individual participant data (IPD) are available for one trial while aggregate-level data (AgD) are only available for the other trial, researchers introduced population-adjusted indirect comparison (PAIC) methods to obtain unbiased estimates of comparative effectiveness, particularly when imbalances in effect modifiers exist. PAIC methods include matching-adjusted indirect comparison (MAIC),Reference Signorovitch, Wu and Yu ⁶ simulated treatment comparison (STC),Reference Ishak, Proskorovsky and Benedict ⁷ and multilevel network meta-regression (ML-NMR).Reference Phillippo, Dias and Ades ⁸

Among these methods, MAIC is becoming increasingly popular and widely used in health technology appraisals, such as the National Institute for Health and Care Excellence (NICE) in the United Kingdom.Reference Phillippo, Dias, Elsada, Ades and Welton ⁹ In a recent methodological systematic review, 88.9% (144 out of 162) of PAIC studies used MAICs.Reference Truong, Tran, Le, Pham and Vo ¹⁰ MAICs estimate a set of balancing weights for each subject in the IPD trial such that the weighted summary statistics (e.g., mean and standard deviation) of covariates in the IPD trial match the reported summaries of the same covariates in the AgD trial. Then, MAIC compares the marginal treatment effect estimated using the weighted data in the IPD trial with the marginal treatment effect reported in the AgD trial.Reference Phillippo, Ades, Dias, Palmer, Abrams and Welton ^{5 ,} Reference Phillippo, Ades, Dias, Palmer, Abrams and Welton ¹¹ For a more detailed description of the MAIC methods, readers can refer to the review paper by Jiang et al.Reference Jiang, Cappelleri, Gamalo, Chen, Thomas and Chu ¹²

Notably, the indirect comparison result is only valid with respect to the population represented by the AgD trial. However, this may not align with the population of interest for the company conducting the MAIC, which might prioritize the population represented by the IPD trial or another specific target population. This paper presents an illustrative example showing that MAICs can yield conflicting comparative effectiveness results when switching the availability of AgD and IPD between trials. This paradoxical phenomenon occurs due to differing magnitudes of effect modification for the two drugs by an effect modifier that is also imbalanced between the trial populations.

2 An illustrative example of the MAIC paradox

Consider an anchored indirect comparison between drug A (from Company A) and drug B (from Company B), each compared to a common placebo comparator C, as depicted in Figure 1. Each company has access only to IPD from its own trial and to AgD for the other company’s trial through published sources. For simplicity, we assume that race (Black versus non-Black) is the sole effect modifier, allowing both MAICs to include the “correct” effect-modifying variable. Additionally, we assume that drug A shows a stronger treatment effect among Black participants, while drug B is more effective among non-Black participants. If the AC trial includes a higher proportion of non-Black participants (among whom drug A is less effective than drug B), while the BC trial predominantly includes Black participants, we can observe a paradox: in separate MAICs, drug A outperforms drug B in the BC trial population, while drug B outperforms drug A in the AC trial population.

Figure 1

Indirect comparison of Drug A versus B in two trials. For the AC trial, we have the individual participant data (IPD). For the BC trial, we only have the aggregate level data (AgD).

Table 1 presents hypothetical data of the AC and BC trials. Let ${n}_{11}=800,{n}_{10}=400$ be the number of non-Black and Black patients in the AC trial, and ${n}_{21}=400,{n}_{20}=800$ denote the number of non-Black and Black patients in the BC trial, respectively. The MAIC performed by Company A calculates the weights ${w}_1$ for all non-Black patients and ${w}_0$ for all Black patients in the AC trial such that the weighted proportion of non-Black patients $\frac{n_{11}{w}_1}{n_{11}{w}_1+{n}_{10}{w}_0}$ matches the proportion of non-Black patients in the BC trial $\frac{n_{21}}{n_{21}+{n}_{20}}=\frac{1}{3}$ (see Table 1), subject to the constraint that weights sum to 1 (i.e., ${n}_{11}{w}_1+{n}_{10}{w}_0=1$ ). Solving the equation, we have ${w}_1=\frac{1}{3{n}_{11}}=\frac{1}{2400}$ and ${w}_0=\frac{2}{3{n}_{10}}=\frac{1}{600}$ . Based on the trial data, under the usual logit link, the estimated treatment effect (log of the odds ratio of the survival rate) for drug A versus drug C in the population of BC trial would be

$$\begin{align*}\log \left(\frac{\frac{80\times {w}_1+180\times {w}_0}{400\times {w}_1+200\times {w}_0}}{\left(1-\left(\frac{80\times {w}_1+180\times {w}_0}{400\times {w}_1+200\times {w}_0}\right)\right)}\right)-\log \left(\frac{\frac{40\times {w}_1+80\times {w}_0}{400\times {w}_1+200\times {w}_0}}{\left(1-\left(\frac{40\times {w}_1+80\times {w}_0}{400\times {w}_1+200\times {w}_0}\right)\right)}\right)=1.540.\end{align*}$$

Table 1

Results for the illustrative example. In this example, the risk difference in survival rate for Drug A versus Drug C in the AC trial is 10% for non-black patients and 50% for black patients. Treatment effect for Drug B versus Drug C in the BC trial is 40% for non-black patients and 20% for black patients

^* logOR: Log of odds Ratio.

^** Y: Outcome variable with Y = 1 indicating death and Y = 0 indicating survival.

^*** n: Sample size.

The indirect comparison of drug A versus B can then be obtained by subtracting the estimated marginal treatment effect for drug B in the BC trial from this value, which gives $1.540-1.115=0.425$ . The corresponding standard error can be calculated using the robust sandwich estimator, which provides robust results given that the weights are estimated. The 95% confidence interval can then be constructed as $\left(0.048,0.802\right)$ , indicating that drug A is statistically significantly better than drug B in the population of the BC trial.

Similarly, for the MAIC performed by Company B with IPD and AgD switched, the weights can also be determined by matching the weighted proportion of non-Black patients in the BC trial (IPD trial) to the proportion in the AC trial (AgD trial). This gives ${w}_1=\frac{2}{3{n}_{21}}=\frac{1}{600}$ and ${w}_0=\frac{1}{3{n}_{20}}=\frac{1}{2400}$ . Performing the same calculations, the estimated comparative effectiveness for drug A versus drug B in the population of the AC trial is $-0.402$ with a 95% confidence interval $\left(-0.014,-0.790\right)$ , which indicates that drug B is significantly better than drug A in the population of the AC trial.

Here, we emphasize that both conclusions—drug A being more effective than drug B or B being more effective than A—are potentially valid within the context of the specific populations considered in this example. However, without a clearly defined target population, the results of the indirect comparison lack meaningful applicability for guiding medical decisions.

In the supplementary material Section S1, we show a step-by-step derivation of the weights. In Section S2, we further explore the impact of varying proportions of non-Black participants in the IPD and AgD populations separately on the MAIC results (see Figures S1a and Figure S1b). In addition, we present an example illustrating this paradox in unanchored MAIC in Section S3.