3.1.1 Information-sharing methods

We begin by extending the notation of the standard NMA modelReference Lu and Ades ¹⁵ to describe a splitting model, not imposing any information-sharing over the basic parameters, d. We then describe alternative ISMs in the relationship they impose between the basic parameters pertaining to population of direct relevance and the basic parameters pertaining to the population of indirect relevance (i.e., $d_{1k}^{Dir}, d_{1k}^\textit{Indir}$ ).

$\underline {\text {Model 0: Splitting:}}$ Consider a set of studies comparing treatment k with a reference treatment b. Each study, i, reports only for one population, $j = \{dir, indir\}$ . Therefore, studies can be arranged as $i = 1 , ... , N_{dir} , N_{dir} + 1 , ... , N_{dir} + N_{indir}$ , with $N_{dir}$ and $N_{indir}$ being the total number of studies contributing direct and indirect information, respectively. The synthesis model for a dichotomous outcome (such as all-cause mortality considered here) takes the following form:

(1) $$ \begin{align} r_{i,k} \sim Binomial (p_{i,k}, n_{i,k}) \end{align} $$

(2) $$ \begin{align} logit(p_{i,k}) = \theta_{i,k} = \mu_{i_{b}} + \delta_{i,bk} \cdot I_{\{ k \neq b \}} \end{align} $$

(3) $$ \begin{align} \delta_{i, bk} = d_{bk}^{j} ~~~\text{(FE)} \end{align} $$

(4) $$ \begin{align} \delta_{i, bk} \sim N ( d_{bk}^{j}, {\tau^{j}}^2) ~~~\text{(RE)} \end{align} $$

(5) $$ \begin{align} d_{bk}^{j} = d_{1k}^{j} - d_{1b}^{j} \end{align} $$

(6) $$ \begin{align} d_{11}^{j} = 0 \end{align} $$

where $r_{i,k}$ , $n_{i,k}$ , and $p_{i,k}$ are the number of events, the total number of patients, and the probability of an event in study i and arm k. $\theta _{i,k}$ is the linear predictor, $\mu _{i_{b}}$ is the study-specific baseline log-odds of the outcome in the reference treatment b in trial i, and $\delta _{i,bk}$ is the study-specific Relative Treatment Effect (RTE) (log odds ratio scale) between the baseline treatment in study i and the treatment in arm k. Under a FE model, $d_{bk}^{j}$ is the population-specific RTE between treatments b and k (Equation 3). Under a RE model, the population specific parameters are the mean $d_{bk}^{j}$ and the variance ${\tau _{bk}^{j}}^2$ of the Normal distribution describing heterogeneity across the study-specific $\delta _{i, bk}^{j}$ (Equation 4). Between-trial variances are typically assumed common across comparisons (i.e., ${\tau _{bk}^{j}}^2 = {\tau ^{j}}^2$ ). Under this splitting model, the basic parameters $d_{1k}^{Dir}$ and $d_{1k}^\textit{Indir}$ are independent (no information is shared across populations), and are assigned vague prior distributions.

Note that splitting is effectively equivalent to subgroup meta-analysis and is also equivalent to a power-prior with $\alpha =0$ , a mixture prior with a weight of $0$ placed on the informative component and approximately equal to a commensurate prior with forced low precision.

$\underline{\text {Model 1 - Lumping:}}$ Lumping is implemented by extending the splitting model to assume $d_{1k}^{Dir} = d_{1k}^\textit{Indir}$ . Under random-effects, we also lump the heterogeneity parameters ( $\tau _{1k}^{Dir} = \tau _{1k}^\textit{Indir}$ ) because it is the model most commonly referred to as ‘lumping’ in policy (see NikolaidisReference Nikolaidis ⁶ for more extensive explorations).

Lumping is equivalent to a meta-analysis that does not distinguish between direct and indirect evidence. It is also equivalent to a power-prior with $\alpha = 1$ , a mixture prior with a weight of $1$ placed on the informative component and approximately equal to a commensurate prior with forced high precision.

$\underline{\text {Model 2 - Multi-level model:}}$ When the RTEs of the direct and indirect evidence sets are not expected to systematically differ, multi-level models can be applied by extending the splitting-model so that:

(7) $$ \begin{align} d_{1k}^{j} \sim N ( D_{1k}, \phi_{1k}^2 ) \end{align} $$

where comparison-specific basic parameters from the different populations are assumed to be normally distributed with a common mean $D_{1k}$ , and a between-populations standard deviation $\phi _{1k}$ . In the case of two populations, a common heterogeneity parameter across treatment comparisons (i.e., $\phi _{1k} = \phi $ ) is here used to ensure identifiability.

$\underline{\text {Model 3 - Standard informative prior distribution:}}$ This comprises of a two-step approach whereby the indirect evidence is first separately analysed using the standard NMA with vague prior distributions, and subsequently the posterior estimates of the first step are used as prior information in a standard NMA model including only the direct evidence:

(8) $$ \begin{align} d_{1k}^{Dir} \sim N (d_{1k}^\textit{Indir}, V_{1k}^\textit{Indir} ) \end{align} $$

where $d_{1k}^\textit{Indir}$ is the posterior mean obtained in step 1, and $V_{1k}^\textit{Indir}$ its corresponding variance. Under FE, this model is equivalent to lumping. Under RE, we can use either the posterior mean and its associated uncertainty for $d_{1k}^\textit{Indir}$ , that is, $\sim N (d_{1k}^\textit{Indir}, {se^2}_{1k}^\textit{Indir})$ , or its predictive distribution, that is, $\sim N (d_{1k}^\textit{Indir}, {se^2}_{1k}^\textit{Indir} + {\tau ^2}^\textit{Indir})$ . Here, we choose the latter because we want to ensure that the uncertainty that is due to heterogeneity is appropriately reflected.

$\underline {\text {Model 4 - Mixture prior distribution:}}$ The mixture prior model is an extension of the informative prior model whereby the posterior estimates of the initial analysis of the indirect evidence are combined with a non-informative prior to form comparison-specific mixture prior distribution for the basic parameters of the direct evidence:

(9) $$ \begin{align} d_{1k}^{Dir} \sim \nu \cdot N(d_{1k}^\textit{Indir}, V_{1k}^\textit{Indir}) + (1-\nu) \cdot N(0, 10^{4}), \end{align} $$

where $\nu $ is the weight placed on the informative component, reflecting the plausibility of sharing information between the direct and the indirect evidence sets. Note that, for $\nu =1$ , this is equivalent to the standard informative prior model (model 3), whilst for $\nu =0$ , this is equivalent to splitting (model 0). The parameter $\nu $ can be assumed, elicited from experts, or estimated within the model by assigning a Beta prior, when the mixture prior comprises only two components, or a Dirichlet prior when it comprises several components. Here, we choose the Beta prior approach, as it has been shown that it offers adaptive information-sharing,Reference Roever, Wandel and Friede ¹⁶ that is, encouraging information-sharing when the direct and indirect sources of evidence are similar, but discourages information when they are ‘in disagreement’.

$\underline {\text {Model 5 - Commensurate prior:}}$ Commensurate prior models, recently used to synthesise individual- and aggregate-level evidence,Reference Hong, Fu and Carlin ¹⁷ were here adapted to the case of multiple population groups. In this approach, the prior distributions for the basic parameters of the direct evidence are centred around the basic parameters of the indirect evidence and the variance of the prior distributions controls the extent of information-sharing.

(10) $$ \begin{align} d_{1k}^{Dir} \sim N (d_{1k}^\textit{Indir}, \eta_{1k}) \end{align} $$

(11) $$ \begin{align} \frac{1}{\eta_{1k}} \sim \begin{cases} N(20, 1), &\text{if } c_{1k}=0 \\ Gamma(0.1, 0.1) I (0.1, 5), &\text{if } c_{1k}=1\end{cases} \end{align} $$

(12) $$ \begin{align} \text{and} ~~ c_{1k} \sim Bernoulli(p_{1k}) \end{align} $$

where $\eta _{1k}$ are the comparison-specific variances. Comparison-specific variances imply a different extent of information-sharing (between direct and indirect evidence) across treatment comparisons which may not be plausible, so is it expected that a common variance across treatment comparisons (i.e., $\eta _{1k} = \eta $ ) would likely be preferred for both simplicity and identifiability purposes. The inverse of the variance parameter (i.e., the precision, $\frac {1}{\eta _{1k}}$ ) is assigned a ‘spike-and-slab’ hyperprior. Such a hyperprior defines two possibilities: one where precision assumes a high value—‘spike’ (defined as a normal distribution centered at 20 and with a low standard deviation of 1)—and hence strong information-sharing is forced, and another where precision assumes a very low value—‘slab’ (defined as a truncated Gamma distribution with shape and rate parameters of 0.1) —imposing minimal information-sharing.Reference Hong, Fu and Carlin ¹⁷ The occurrence of the scenarios is modelled using independent Bernoulli trials (i.e., $c_{1k} \sim Bernoulli(p_{1k})$ ), with the Bernoulli probability parameter controlling the extent of commensurability, and the strength of information-sharing between direct and indirect evidence. The probability parameter can be fixed to an arbitrary value (e.g., $p_{1k} = 0.5$ ), or assigned a vague hyper-prior, such as $p_{1k} \sim Beta (1,1)$ , in order to be estimated within the model. However, in both cases (i.e., $p_{1k}$ fixed or uncertain), adaptive information-sharing is facilitated, because the choice between the spike and the slab is regulated by $c_{1k}$ which is estimated in the model. In other words, $p_{1k}$ only specifies the value of the Bernoulli prior for $c_{1k}$ which controls the extent of information-sharing. Therefore, as also highlighted by Hong et al.,Reference Hong, Fu and Carlin ¹⁷ the approach where $p_{1k}$ is uncertain is likely to lead to excessive uncertainty. Here, we follow the approach used by Hong et al. (2018) and assume a fixed $p_{1k} = 0.5$ and a spike and slap with hyperparameters as show in Equation (11).

$\underline {\text {Model 6 - Power-prior:}}$ The power-prior, introduced by Ibrahim et al.Reference Ibrahim and Chen ¹⁸ and recently used in the NMA context by Jenkins et al.,Reference Jenkins, Martina, Dequen-O’Byrne, Abrams and Bujkiewicz ¹⁹ down-weights indirect evidence by raising its likelihood to a power $\alpha $ . Under this approach the posterior distribution of the basic parameters of the direct evidence becomes:

(13) $$ \begin{align} \pi(d_{1k}^{Dir} | D^{Dir}, D^\textit{Indir}, \alpha ) \propto \prod_{i=1}^{N_{Dir}} L(d_{1k}^{Dir} | D^{Dir} ) \cdot \underbrace{\prod_{i=1 + N_{Dir}}^{N_{Dir} + N_\textit{Indir}} L(d_{1k}^\textit{Indir} | D^\textit{Indir} )^\alpha \cdot \pi_{0}(d_{1k}^\textit{Indir})}_{\text{Power-prior}} \end{align} $$

where $D^{Dir}$ and $D^\textit{Indir}$ denote the direct and indirect data provided by the corresponding studies. $L(d_{1k}^{Dir} | D^{Dir})$ and $L(d_{1k}^\textit{Indir} | D^\textit{Indir} )^\alpha $ indicate the likelihood of the direct and the indirect evidence and $\pi _{0}(d_{1k}^\textit{Indir})$ is a vague prior for the basic parameters of the indirect evidence. The $\prod _{i=k}^{\lambda } f(i)$ denotes the product of the succeeding expression from study $i = k$ to study $i = \lambda $ . The parameter $\alpha $ regulates the influence of the indirect evidence: when $\alpha =1$ the power-prior is equivalent to lumping, while when $\alpha =0$ the approach is equivalent to splitting. The value of $\alpha $ can be arbitrarily definedReference Spiegelhalter, Abrams and Myles ²⁰ or elicited from expert opinion, but it cannot be estimated within the model without model modifications. Here, we used $\alpha $ value from $0$ to $1$ in $0.1$ increments.

3.1.1.1 Information-sharing metrics

The level of sharing imposed by each ISM is determined by comparing their posterior estimates with those obtained using the direct evidence only. In what follows, the definition of each metric is described in absolute terms. The three metrics used were:

$\underline {\text {Point Estimate Divergence:}}$ The point estimate divergence (PED) evaluates the absolute difference in the adult relative effectiveness posterior mean between splitting ( $d_{ad}^{ISM_{0}}$ ) and each of the $\nu $ alternative ISMs ( $d_{ad}^{ISM_{\nu }}$ ):

(14) $$ \begin{align} PED_{\nu} = |d_{ad}^{ISM_{\nu}} - d_{ad}^{ISM_{0}}| \end{align} $$

where a larger $PED_{\nu }$ implies a larger difference in the adult point estimate between splitting and ISM $\nu $ .

$\underline {\text {Precision Increase:}}$ To capture changes in the standard error of the relative effect estimate obtained by splitting ( $sd_{d_{ad}}^{ISM_{0}}$ ) and each of the $\nu $ ISMs ( $sd_{d_{ad}}^{ISM_{\nu }}$ ), we used precision increase (PrI), which was introduced by Jackson et al.Reference Jackson, White, Price, Copas and Riley ²¹ (note that their measure is termed borrowing of strength (BoS). However, multiple measures are used here and this measure is renamed to better reflect the underlying quantity) defined as:

(15) $$ \begin{align} PrI_{\nu} = 1 - \frac{sd_{d_{ad}}^{ISM_{\nu}}}{sd_{d_{ad}}^{ISM_{0}}} \in (-\infty, 1] \end{align} $$

Negative $PrI$ values imply that information-sharing led to increased uncertainty compared to splitting, whilst higher positive $PrI$ values indicate increasing precision gains.

$\underline {\text {Kullback--Leibler divergence:}}$ Finally, we used Kullback–Leibler (KL) divergenceReference Kullback and Leibler ²² which simultaneously considers changes in the posterior mean and standard deviation. This metric is interpreted as the information conveyed by the probability distribution of $d_{ad}^{ISM_{\nu }}$ when used to describe another probability distribution $d_{ad}^{ISM_{0}}$ , and is defined as:

(16) $$ \begin{align} D_{KL}(p,q) = \int_{}{} [ p(x) \times log(p(x)) - p(x) \times log(q(x)) ] ~d(x) \end{align} $$

where p is the target distribution (here $d_{ad}^{ISM_{0}}$ ), and q the distribution we use to describe it (here $d_{ad}^{ISM_{\nu }}$ ).

Higher KL values indicate that the amount of information that one distribution conveys about another is reduced and therefore reflect higher information-sharing. However, KL divergence is not symmetrical and does not trade differences in point estimates and uncertainty equally (see illustration in Supplementary Material). Hence, the KL divergence can only be used comparatively across ISMs.

3.2.1 Background and methods

We utilised the decision model developed in the original study, using quality-adjusted life years (QALYs) as the primary outcome and 2009 UK costs from the perspective of the National Health System (NHS). An annual discount rate of 3.5% was applied to both costs and outcomes. Incremental cost-effectiveness ratios (ICERs) were calculated as the ratio of incremental costs to QALYs and compared to common thresholds used for determining the cost-effectiveness of NHS resources. ²⁵

The model reflects the lifetime prognosis of sepsis to capture the costs and health consequences of sepsis in the absence of IVIG/IVIGAM. There were two distinct model components: first, a short-term decision-tree evaluated the consequences of the initial hospitalisation period following the first sepsis episode. The relative treatment effects obtained from the evidence synthesis were applied to the initial decision tree. Subsequently, conditional on survival in the short term, patients entered a long-term Markov model with two disease states (alive & dead) and annual cycles that captured the long-term consequences for sepsis survivors following the initial hospitalisation.

To understand the potential implications of information-sharing for decision-making, we applied the relative effects estimated by the various ISMs (Section 3.1) to the original cost-effectiveness model. Uncertainty was propagated to the model using probabilistic sensitivity analysis.Reference Briggs, Claxton and Sculpher ²⁶ We used a cost-effectiveness threshold of £30,000 per QALY gained to determine the probability of a treatment being cost-effective and to evaluate the value of further research.

The value of further research was quantified using value of information methods.Reference Drummond, Sculpher, Claxton, Stoddart and Torrance ^{27 ,} Reference Ades, Lu and Claxton ²⁸ Four estimates of the value of further research were calculated. The expected value of perfect information (EVPI) estimates the value of resolving all parameter uncertainty relating to the decision problem and represents an upper bound on the value of further research. The expected value of perfect parameter information (EVPPI) estimates the value of resolving uncertainty for individual parameters or groups of parameters. The expected value of sample information (EVSI) quantifies the value of particular research designs using a defined sample size, which are therefore likely to reduce, but not eliminate, uncertainty over particular parameters. Finally, the expected net benefit of sample information (ENBS) was estimated—this subtracts the cost of sampling from the EVSI. These estimates were scaled to the total population expected to benefit from the research over the intervention’s expected lifetime and presented in monetary terms to facilitate comparison with research costs. Further details are available in Soares et al. (Chapter 6, Appendix 5).Reference Soares, Welton, Harrison, Peura and Shankar ⁸