What is already known?

• • Meta-analysis methods are widely used to combine effect sizes across studies, typically within a traditional frequentist framework.

• • These methods face challenges in hypothesis testing because the cumulative nature of meta-analyses inherently induces multiple testing issues.

• • Bayes factors provide an alternative that directly quantify the evidence between hypotheses, and allow for natural evidence accumulation.

What is new?

• • The article compares Bayes factor testing with classical significance testing in meta-analyses, clarifying their conceptual and methodological differences.

• • It presents five Bayes factor models for evidence synthesis, illustrated using the standard single-effect-size meta-analysis setup.

• • The article discusses prior specification, including priors for the (nuisance) between-study heterogeneity.

• • It highlights the link between Bayes factors and e-values as a means for flexible classical error control in cumulative meta-analyses.

• • All methods are implemented in the R package BFpack.

Potential impact for RSM readers

• • The overview aims to guide researchers in selecting suitable evidence synthesis methods and promote flexible, statistically robust Bayesian approaches for hypothesis testing in (cumulative) meta-analyses.

3.1. Traditional meta-analysis models

3.1.1. Common effect model

Under the CE model, the key parameter, denoted by $\theta $ , is assumed to be common under all studies. In order for this assumption to hold, the conditions under which the data are collected under every study need to be (practically) identical, such as replication studies in psychologyReference Schmidt⁴² or series of randomized clinical trials by the same researchers, for patients from the same population, and testing exactly the same treatment.Reference Borenstein, Hedges, Higgins and Rothstein⁴³

For study i, for $i=1,\ldots ,k$ , we denote the estimated effect size by $y_i$ , its (assumed known) standard error by $\sigma _i$ , and the sample size in the study by $n_i$ . A Gaussian (normal) error is assumed for the study-specific estimate and the corresponding standard error resulting in the following synthesis model:

(2) $$ \begin{align} \mathcal{M}_{CE}: y_i \sim N(\theta,\sigma^2_i), \text{ for study } i=1,\ldots,k. \end{align} $$

Under the CE model, we consider the most common test in statistical practice of a precise null hypothesis, which assumes that the mean effect is zero, against a two-sided null hypothesis test, which assumes that the mean effect is nonzero, i.e.,

$$ \begin{align*} \mathcal{H}_0&:\theta = 0\\ \mathcal{H}_1&:\theta \not = 0. \end{align*} $$

3.1.2. Random effects model

Under the RE model, the effects $\theta _i$ are assumed to be heterogeneous across studies. This heterogeneity may be caused by (slightly) different conditions under which the data were collected across studies or (slightly) different populations that were considered under the different studies. The RE model is generally the preferred model since researchers deem the assumption of a CE unrealistically restrictive in most applications. Moreover, the RE model is also often preferred, because it allows for drawing inference for the distribution of true effects whereas the CE model is restricted to drawing inference to only the included studies in a meta-analysis.

A normal distribution is assumed for the study-specific effects, where the mean $\mu $ quantifies the average (global) effect across studies and the standard deviation $\tau $ quantifies the between-study heterogeneity in true effect size. Similar to the CE model, normally distributed errors are assumed for the study-specific estimates. The RE model can then be formulated as

(3) $$ \begin{align} \mathcal{M}_{RE}:\left\{\begin{array}{ccl} y_i &\sim & N(\theta_i,\sigma_i^2)\\ \theta_i &\sim & N(\mu,\tau^2). \end{array} \right. \end{align} $$

In this model, the study-specific true effects (i.e., $\theta _i$ ) are often treated as nuisance parameters which can be integrated out. The marginalized RE model can then be equivalently written as

(4) $$ \begin{align} \mathcal{M}_{RE}: y_i \sim N(\mu,\sigma_i^2+\tau^2),\text{ with } \tau^2>0. \end{align} $$

Under the RE model, we generally test whether the global effect $\mu $ is zero or not, i.e.,

(5) $$ \begin{align} \mathcal{H}_0&:\mu = 0\\ \nonumber \mathcal{H}_1&:\mu \not = 0. \end{align} $$

3.1.3. Fixed effects models

Similarly to the RE model, and unlike the CE model, the fixed effects (FE) model also assumes heterogeneous effects across studies. However, rather than specifying a distribution for the between-study heterogeneity, whose parameters are estimated from the data, an FE approach is considered without a multilevel structure. The parameters of interest are the true effect sizes of the studies. Although less common, the FE model has been used in meta-analytic applicationsReference Rice, Higgins and Lumley^4, Reference Klugkist and Volker^23, Reference Kuiper, Buskens, Raub and Hoijtink^34, Reference Wonderen, Zondervan-Zwijnenburg and Klugkist^35, Reference Van Lissa, Clapper and Kuiper⁴⁴ and when aggregating evidence across respondents.Reference Stephan, Weiskopf, Drysdale, Robinson and Friston^45– Reference Klaassen, Zedelius, Veling, Aarts and Hoijtink^47, Footnote ⁱⁱ Moreover, the FE model can also be seen as a CE model that is extended to a meta-regression model by the inclusion of a moderator that has a unique value for each study. According to this meta-regression model, each study also has its own true effect size.

Again, Gaussian errors are assumed for the study-specific effect size estimates. The FE model can then be formulated as

(6) $$ \begin{align} \mathcal{M}_{FE} : y_i \sim N(\theta_i,\sigma_i^2). \end{align} $$

Because of the absence of a parameter for a global effect, the null hypothesis assumes that all study-specific effects are zero or not,Reference Rice, Higgins and Lumley⁴ i.e.,

$$ \begin{align*} \mathcal{H}_0&:\theta_1=\dots=\theta_k=0,\\ \mathcal{H}_1&:\text{not } \mathcal{H}_0, \end{align*} $$

where the alternative assumes that at least one constraint under $\mathcal {H}_0$ does not hold. Hence, this null hypothesis fundamentally differs from the null hypothesis under the CE and RE models where the average effect across studies is assumed to be zero, while in the FE model, the null assumes that all study-specific effects are assumed to be zero. Consequently, the null is extended with every newly included study.

It has been argued that an FE model has the advantage that it can be used when the study designs and/or measurement levels of the key variables (highly) vary across studies.Reference Klugkist and Volker^23, Reference Kuiper, Buskens, Raub and Hoijtink³⁴ The argument is that we are not combining effect sizes, which may then have (highly) different scales, but rather combining statistical evidence regarding the hypotheses when computing Bayes factors. However, the relative evidence between hypotheses as quantified by the Bayes factor is directly affected by the observed effect size and its uncertainty (via the likelihood). Therefore, the appropriateness of combining evidence from different studies with highly different designs, e.g., studies with reported effect sizes based on both dichotomous and continuous outcomes, or studies with an experimental design or observational design, should be carefully assessed by substantive experts. Moreover, prior specification may be more challenging when considering effect sizes having fundamentally different scales.

3.2. Hybrid effects model

To apply a traditional meta-analysis model, a dichotomous decision needs to be made whether to assume between-study homogeneity (i.e., the CE model) or between-study heterogeneity (i.e., the RE model, which is much more common than the FE model). This comes down to the following model selection problem:

(7) $$ \begin{align} \mathcal{M}_{CE}&:\tau^2 = 0\\ \nonumber \mathcal{M}_{RE}&:\tau^2> 0. \end{align} $$

Although the Q-testReference Cochran⁴⁸ can be used for testing whether the data are homogeneous, it is not recommended for model selection since it may have low statistical power depending on the number of studies included in the meta-analysis, the sample size of the studies, and the true between-study heterogeneity.Reference Borenstein, Hedges, Higgins and Rothstein^43, Reference Viechtbauer⁴⁹ This implies potentially large error rates when choosing either one of the two possible models.

Specifically, when an incorrect CE model is employed, the standard error of the key parameter will be underestimated. In a classical significance test, this would result in inflated type I error rates, and in a Bayesian evidence synthesis, this would result in an overestimation of the evidence for the true hypothesis. On the other hand, when an incorrect RE model is employed, the standard error of the key parameter will be overestimated unless $\tau ^2$ is estimated as zero. In a classical significance test, this results in an underpowered test, and in Bayesian evidence synthesis, this would result in an underestimation of the evidence for the true hypothesis. Thus, when there is considerable statistical uncertainty regarding the true underlying model and there are no theoretical reasons for favoring one model over the others, it is useful to employ a statistical model that encompasses both the CE and RE models to avoid a potentially error-prone dichotomous decision resulting in unreliable quantifications of the statistical evidence.Reference Gronau, Heck, Berkhout, Haaf and Wagenmakers^50, Reference Aert and Mulder⁵¹ We shall refer to the class of models that encompasses both the CE model and the RE model as hybrid effects models. To our knowledge, two hybrid effects models have been proposed in the literature, which we discuss below. Appendix A discusses some statistical differences.

3.2.1. Marginalized random-effects meta-analysis (marema) model

The marginalized random-effects meta-analysis (marema) modelReference Aert and Mulder⁵¹ is closely related to the RE model with the exception that it also allows for the possibility of excessive between-study homogeneity implying less variability across studies than would be expected by chance. The marema model can be written as

(8) $$ \begin{align} \mathcal{M}_\textit{marema}: y_i \sim N(\mu,\sigma_i^2+\tau^2),\text{ with } \tau^2> -\sigma_{\min}^2, \end{align} $$

where the lower bound of $\tau ^2$ depends on the smallest sampling variance of the included studies, $\sigma _{\min }^2=\min _i \{\sigma ^2_i\}$ . Thus, under the marema model, $\tau ^2$ can attain negative values. A negative $\tau ^2$ implies that the between-study heterogeneity is smaller than expected by chance.

Although this property may seem unnatural at first sight, this setup has various advantages.Reference Nielsen, Smink and Fox^52, Reference Mulder and Fox⁵³ First, the model allows a simple check of whether between-study heterogeneity is present via the posterior probability that $\tau ^2>0$ holds (this simple Bayesian measure can also be used as an alternative to the Q-test as it does not rely on large sample theory). Second, the model allows noninformative improper priors for $\tau ^2$ (both for estimation, e.g., whether $\tau ^2>0$ , and for testing the global mean $\mu $ using a Bayes factor; see Section 4). Thereby, the model simplifies the (challenging) choice of the prior for $\tau ^2$ . Third, the model enables researchers to check for extreme between-study homogeneity (less than expected by chance), which may indicate strong correlation between studies, extreme bias, or potential fraud.Reference Ioannidis, Trikalinos and Zintzaras⁵⁴ This can be checked via the posterior probability that $\tau ^2<0$ . Fourth, as mentioned earlier, the models avoid the need to make a dichotomous decision between the CE model and the RE model but instead naturally balance between these models depending on the between-study heterogeneity that is present. Finally, note that negative variances are very common in latent variable models (such as the RE model). In the factor analysis literature, these are known as “Heywood cases,” which often indicate model misspecification.Reference Kolenikov and Bollen⁵⁵ In our current setup, this implies that the RE model would be inappropriate given the available data. Marginalized RE models have also been advocated for various other statistical problems.Reference Mulder and Fox^56– Reference Nielsen, Smink and Fox⁵²

Under the marema model, the hypothesis test on the global effect will be the same as under the RE model, i.e.,

$$ \begin{align*} & \mathcal{H}_0:\mu = 0,\\ & \mathcal{H}_1:\mu \not= 0. \end{align*} $$

3.2.2. Bayesian model-averaged meta-analysis model

The second hybrid model incorporates the statistical uncertainty regarding the true meta-analysis model via a weighted average of all models under consideration using Bayesian model averaging (BMA), a common approach in Bayesian statistics.Reference Hoeting, Madigan, Raftery and Volinsky⁵⁸ A Bayesian model-averaged meta-analysis modelReference Gronau, Heck, Berkhout, Haaf and Wagenmakers⁵⁰ can be obtained by averaging over the CE model and the RE model according to

(9) $$ \begin{align} \mathcal{M}_{BMA}: y_i \sim p_{CE}\times \mathcal{M}_{CE} + (1-p_{CE}) \times \mathcal{M}_{RE} \end{align} $$

using prespecified prior probabilities for the CE model and RE model, i.e., $\text {Pr}(\mathcal {M}_{CE})=p_{CE}$ and $\text {Pr}(\mathcal {M}_{RE})=1-p_{CE}$ . Moreover, each of the two model parts, $\mathcal {M}_{CE}$ and $\mathcal {M}_{RE}$ , need to be split regarding the absence and presence of the respective key parameters, i.e., $\theta $ and $\mu $ , resulting in four model parts: $\mathcal {M}_{CE} \, \&\, \theta = 0$ , $\mathcal {M}_{CE} \, \&\, \theta \not = 0$ , $\mathcal {M}_{RE} \, \&\, \mu = 0$ , and $\mathcal {M}_{RE} \, \&\, \mu \not = 0$ . Typically, equal prior probabilities of $\frac {1}{4}$ are chosen for these four models.Reference Gronau, Heck, Berkhout, Haaf and Wagenmakers⁵⁰ Under the BMA approach, the hypothesis test of interest would be formulated as

$$ \begin{align*} \mathcal{H}_0&:(\mathcal{M}_{CE}\, \&\, \theta = 0) \text{ or } (\mathcal{M}_{RE}\, \&\, \mu = 0)\\ \mathcal{H}_1&:(\mathcal{M}_{CE}\, \&\, \theta \not= 0) \text{ or } (\mathcal{M}_{RE}\, \&\, \mu \not= 0). \end{align*} $$

The BMA approach has also been extended to include (sub)models that correct for publication bias.Reference Maier, Bartoš and Wagenmakers⁵⁹

4.1. Prior sensitivity

The choice of the prior for the average effect, which is unique under $\mathcal {H}_1$ (as it is fixed under $\mathcal {H}_0$ ), is particularly important. The sensitivity of the Bayes factor to this prior can be understood from the definition of the Bayes factor in (1). Under $\mathcal {H}_0$ , the average effect is assumed to be fixed at zero, and thus, the marginal probability of the data in the available studies quantifies how likely the data were to be observed under the assumption that the effect is zero. Under $\mathcal {H}_1$ , the effect is assumed to be unknown and our belief about its magnitude is reflected in the prior distribution. Thus, the marginal probability of the data is equal to a weighted average of the likelihood of the data weighted according to the specified prior.

In order for the marginal probability under $\mathcal {H}_1$ to be meaningful, and to ensure that the resulting Bayes factor is meaningful, the prior should correspond to realistic “weights” on the possible (nonzero) effect sizes. For this reason, an extremely vague prior for the average effect should not be used, such as a normal prior with a very large standard deviation. Such a prior would place a relatively large weights at unrealistically large effect sizes, resulting in extremely small marginal likelihoods under $\mathcal {H}_1$ for typical (“small” to “large”) effect sizes, thereby heavily biasing the evidence in favor of the null.Footnote ⁱⁱⁱ This is illustrated in Figure 3 (left panel) when testing the average effect under an RE model for the meta-analysis of McNeely et al.Reference McNeely, Campbell and Ospina³⁷ when placing a normal prior on $\mu $ with a mean of zero and we let the prior standard deviation gradually increase to very large values. As the prior standard deviation of the normal prior increases to unrealistically large effect sizes, the Bayes factor gradually decreases toward zero implying that the evidence in favor of $\mathcal {H}_0$ keeps increasing. Furthermore, the right panel of Figure 3 illustrates that when using a prior standard deviation of 5, the prior places lower weights around the likelihood (which is concentrated around $\hat {\mu }=0.416$ ), than when using a prior standard deviation of 1. Thus, the marginal likelihood of $\mathcal {H}_1$ is lower when a prior standard deviation of 5 is used, as can also be seen from the lower Bayes factor in the left panel in Figure 3.

4.2. Priors for the average effect

The prior for the (average) effect in the case of a standardized mean difference can be specified based on different considerations. Because we are testing whether the average effect is zero or not, a natural choice would be to center the prior at zero so that negative effects are equally likely as positive effects, and so that small effects are on average more likely than large effects before observing the data. Moreover, as the distribution of the observed effect given the unknown true effect is also normal, a (conjugate) normal prior would be a natural choice. The choice of the prior standard deviation is particularly important as was illustrated from Bartlett’s paradox (Figure 3). A standard deviation of 1 is the default in the R packages RoBMAReference Bartoš and Maier⁶⁰ and BFpackReference Mulder, Williams and Gu³⁶ implying fairly large effects to be plausible. In metaBMA, the more heavy tailed Cauchy prior with scale $0.707$ is the default.

For a log odds ratio as an effect size measure (which lies on an approximate normal scale), one could start with placing priors on the success probabilities in the two groups (e.g., treatment and control), which are then transformed to the log odds ratio. As a default choice, proper independent uniform priors can be specified for the success probabilities which assume that all success probabilities are equally likely a priori. By transforming these to the log odds ratio, an approximate t distribution having a scale of 2.35 and 13 degrees of freedom is obtained (Appendix B). This is the default in BFpack. Another option could be to start with a prior for a mean effect, e.g., $\mathcal {N}(0,1)$ prior, and convert this prior to the log odds scale the transformation formulas of Borenstein et al. (Chapter 7).Reference Borenstein, Hedges, Higgins and Rothstein⁴³ This is the default in RoBMA and in this case the scale of the prior distribution is approximately the same regardless of whether the standardized mean difference or log odds ratio is the effect size measure of interest. In metaBMA, the default scale of the prior distributions is also adjusted to the effect size measure: a Cauchy prior with scale 1.283 is used, because the distribution of log odds ratio is approximately 1.81 times as wide as that of the standardized mean difference. The use of conversion formulas may be particularly useful when the scales of the outcome variables varied across studies in the same meta-analysis. Synthesizing evidence from highly heterogeneous outcomes having different measurement levels is only recommended of course when this is substantively meaningful.

Pearson correlation coefficients are commonly meta-analyzed after applying Fisher’s z transformation.Reference Fisher⁶⁶ Fisher’s z transformed correlations are approximately normally distributed. For a Fisher transformed correlation, one can specify a prior for the correlation in the interval $(-1,1)$ , which can then be transformed by applying Fisher’s z transformation. A natural proper noninformative choice for a correlation would be to use a uniform prior in the interval $(-1,1)$ .Reference Jeffreys^11, Reference Mulder and Gelissen^67, Reference Mulder⁶⁸ After applying a parameter transformation, this corresponds to a logistic prior distribution with a scale of 0.5 for Fisher’s z transformed correlation (Appendix B). This is the current default in BFpack. Alternatively, one could again use the conversion formulas from Borenstein et al.,Reference Borenstein, Hedges, Higgins and Rothstein⁴³ which is the default in RoBMA. In metaBMA, a Cauchy prior with a scale of 0.354 is the default to take into account that the distribution of Fisher’s z transformed correlations in relation to the distribution of standardized mean differences.

As a general default choice for the prior, that is independent of the effect size measure, a unit-information prior can also be specified, which contains the information of a single observation.Reference Röver, Bender and Dias⁶⁹ By construction, the amount of prior information is then relative to the amount of information in the sample instead of being based on contextual information about the key parameter. Therefore, this prior can be used as a general default. Note that the evidence as quantified by the well-known Bayesian information criterion (BIC)Reference Schwarz⁷⁰ also behaves as an approximate Bayes factor based on a unit-information prior.Reference Raftery^71, Reference Kass and Wasserman⁷² The information of one observation depends on whether between-study heterogeneity is present or not. Under the CE model, where between-study heterogeneity is absent, the unit-information prior follows a normal distribution having a mean of 0 and a prior variance that is equal to the error variance rescaled to the total sample size, i.e., $\mathcal {N}(0,N/\sum _i 1/\sigma _i^2)$ , where $N=\sum _i n_i$ denotes the total sample size across studies. Under the RE model and the hybrid models, a conditional prior for the average effect $\mu $ is required conditional on $\tau ^2$ , which follows a normal $\mathcal {N}(0,N/\sum _i(\sigma ^2_i+\tau ^2)^{-1})$ distribution.Footnote ^iv This prior needs to be combined with a prior for $\tau ^2$ (discussed later) to construct a joint prior for $\mu $ and $\tau ^2$ under $\mathcal {H}_1$ .(Note that such prior dependency between $\mu$ and ${\tau}^2$ is common in the objective Bayesian literature for Bayes factor testing.Reference Liang, Paulo, Molina, Clyde and Berger^65, Reference Zellner^73, Reference Bayarri, Berger, Forte and García-Donato⁷⁴)

When viewing a prior of the average effect as a population distribution of effect sizes from which the unknown effect size of a current meta-analysis is “drawn,” one can use the estimated distribution of effect sizes from published research to create an empirically informed prior. This has been done for meta-analyses of binary data with rare events,Reference Günhan, Röver and Friede⁷⁵ for medicine and its subfields,Reference Bartoš, Gronau, Timmers, Otte, Ly and Wagenmakers⁷⁶ and for binary trial data and time-to-event data,Reference Bartoš, Otte, Gronau, Timmers, Ly and Wagenmakers⁷⁷ for example. Depending on the availability of relevant published effect sizes given the meta-analysis at hand, such an approach may be reasonable. A prior for the average effect has also been elicited for a meta-analysis using external knowledge regarding the effect size at hand.Reference Gronau, Van Erp, Heck, Cesario, Jonas and Wagenmakers⁷⁸

4.3. Priors for the between-study heterogeneity

Under the RE model and the hybrid models, a prior for the between-study heterogeneity also needs to be chosen.Reference Röver^62, Reference Pateras, Nikolakopoulos and Roes^79, Reference Spiegelhalter, Abrams and Myles⁸⁰ As this is a common nuisance parameter under both hypotheses, $\mathcal {H}_1$ and $\mathcal {H}_0$ , the Bayes factor for testing the global effect is considerably less sensitive to the exact choice of this prior.Reference Kass and Raftery¹² This feature is advantageous, as specifying an informative prior for the between-study heterogeneity can be challenging due to its less intuitive interpretation. Interestingly, it is also possible to employ a noninformative improper prior for this common nuisance parameter,Footnote ^v allowing a default Bayes factor test.

As summarized by Röver,Reference Röver⁶² there is an extensive literature on noninformative (“objective” and “default”) priors for the between-study heterogeneity $\tau ^2$ . For researchers with a less mathematical statistical background, it will be difficult to choose one specific noninformative prior based on the available theoretical and statistical arguments. Moreover, noninformative priors have often been assessed based on their implied behavior in Bayesian estimation problems rather than their implied behavior in hypothesis testing using Bayes factors. To keep the discussion as concise as possible, we restrict ourselves to certain noninformative priors rather than providing a complete assessment of all possible priors.

One important criterion for a noninformative improper prior is whether the resulting Bayes factor is well-defined. This is the case when the marginal likelihoods are finite. In an estimation problem, this is equivalent to the important criterion of that the posterior is proper. Table 3 provides three possible noninformative improper priors for the between-study variance when testing the global mean, including the minimal number of studies for a well-defined Bayes factor. The uniform prior on $\sqrt {\tau ^2}$ or the Berger–Deely priorReference Berger and Deely⁶³ may be preferred as Bayes factor testing will already be possible when only two studies are available. From these two priors, the Berger–Deely prior is conjugate under the RE model, and therefore a more natural choice. Moreover, the Berger–Deely prior naturally extends to the marema modelFootnote ^vi (the Berger–Deely prior is the default in BFpack; Table 2). Furthermore, the scale invariant prior, $1/\tau ^2$ , which is also Jeffreys’ prior, is not recommended for lower level variances such as the between-study heterogeneity.Reference Gelman⁸² This prior may result in infinite marginal likelihoods (and improper posteriors in Bayesian estimation) when the between-study heterogeneity in the data is very low, and is therefore not recommended. Finally, we note that proper approximations of noninformative improper priorsReference Pateras, Nikolakopoulos and Roes^79, Reference Spiegelhalter⁸³ should be used with care, as they may unduly be highly noninformative.Reference Gelman^82, Reference Berger⁸⁴

Table 3 Three possible noninformative priors for the between-study heterogeneity $\tau ^2$ and the number of required studies to obtain a finite Bayes factor.

Empirically informed priors have also been proposed for the between-study heterogeneity.Reference Turner, Jackson, Wei, Thompson and Higgins^6, Reference Rhodes, Turner and Higgins^25, Reference Pullenayegum^85, Reference Van Erp, Verhagen, Grasman and Wagenmakers⁸⁶ These empirically informed priors have mostly been used for estimation problems, although there is also literature where these priors have also been used for Bayes factor testing in meta-analyses.Reference Bartoš, Otte, Gronau, Timmers, Ly and Wagenmakers^77, Reference Gronau, Van Erp, Heck, Cesario, Jonas and Wagenmakers⁷⁸ A complicating factor which is often overlooked is that the between-study variance under $\mathcal {H}_1$ will never be larger than the between-study variance under $\mathcal {H}_0$ as the mean is restricted under $\mathcal {H}_0$ . For this reason, it would be preferred to incorporate this in the informative priors (by ensuring that $\tau ^2$ is stochastically larger under $\mathcal {H}_0$ than under $\mathcal {H}_1$ a priori). To our knowledge, no priors have been proposed so far that abide this property. In the end, the suitability of informed priors would need to be carefully assessed depending on the meta-analysis at hand.

Currently, the R package “BFpack” supports the use of noninformative improper priors for the between-study heterogeneity. Since the R packages “metaBMA” and “RoBMA” perform BMA between the CE model ( $\tau ^2 = 0$ ) and the RE model ( $\tau ^2> 0$ ), calculating Bayes factors is necessary to determine the posterior model weights. This test treats $\tau ^2$ as a parameter of interest rather than a nuisance parameter, precluding the use of noninformative improper or arbitrarily vague priors. These two packages do allow a noninformative improper prior for $\tau ^2$ when solely working under the RE model (implying that the CE model is disregarded and BMA is not used). Based on our experience, the “bayesmeta” package does not support noninformative improper priors for the nuisance parameter when testing the overall effect.

4.4. Final remarks on prior specification

On a more theoretical note, it has been argued that priors should result in Bayes factors that are information consistent.Reference Liang, Paulo, Molina, Clyde and Berger^65, Reference Gronau, Ly and Wagenmakers^87, Reference Mulder, Berger, Peña and Bayarri⁸⁸ Information consistency implies that the evidence for the alternative should go to infinity when the estimated effect goes to plus or minus infinity. Roughly speaking, Bayes factors are not information consistent when the (marginal) prior for the mean effect has thinner tails than the (integrated) likelihood as a function of the mean effect (after integrating out the nuisance parameter $\tau ^2$ ). As the integrated likelihood follows an approximate Student t distribution under the RE model, information consistency is assured when using a prior with thicker tails, such as a Cauchy prior. As was shown by Mulder et al.,Reference Mulder, Berger, Peña and Bayarri⁸⁸ a normal marginal prior (having very thin tails) can be detrimental when the effect size is extremely large (such as standardized effect sizes of 10) causing the relative evidence to be approximately 1 (suggesting equal evidence for the null and alternative). For the CE and FE models, the likelihood as a function of the mean has a Gaussian shape, which already has thin tails. Therefore, information inconsistency would not even occur when using a normal prior. Based on our experience, information consistency is mainly a theoretical property and not a practical one as effect sizes are generally not that extreme such that an information inconsistent Bayes factor would result in conflicting behavior. Therefore, information consistency may not be a serious concern when choosing priors in general.

Generic approximate Bayesian approaches are also available which avoid manual prior specification, such as the BICReference Schwarz^70, Reference Raftery⁷¹ or fractional/intrinsic Bayes factors,Reference Berger and Pericchi^89– Reference Mulder⁹¹ or approximations thereof.Reference Gu, Mulder and Hoijtink⁹² These methods have, for instance, been been applied for FE meta-analyses.Reference Kuiper, Buskens, Raub and Hoijtink^34, Reference Wonderen, Zondervan-Zwijnenburg and Klugkist^35, Reference Van Lissa, Clapper and Kuiper⁴⁴ Implicitly, these approximations abide the principle of minimally informative priors, comparable to the unit-information prior. To simplify the interpretation of the evidence however, it may be preferred to only use these approximate methods for hypothesis testing problems which are not supported by the available Bayesian meta-analysis software (e.g., due to the formulated hypotheses, the statistical models, or the research designs).

Finally, Appendix C presents a small simulation on the sensitivity of the Bayes factor to the prior of the nuisance between-study heterogeneity. As shown, the Bayes factor is quite robust to the exact choice.

5.1. Evidence synthesis via (regular) Bayesian updating

Under the CE model, the RE model, and the hybrid models, evidence updating is done in a similar manner as regular Bayesian updating in estimation. In Bayesian estimation, we need to update the posterior when a new study is reported. The posterior based on the previous studies becomes the prior, which is then multiplied (“combined”) with the likelihood of the new study to obtain the new posterior via Bayes’ theorem. When testing hypotheses, we update the Bayes factor based on the previous $k-1$ studies with the Bayes factor for the new k-th study using the posteriors based on the previous studies under the hypotheses as prior for computing the marginal likelihoods. Under $\mathcal {H}_1$ , this can be written as

(10) $$ \begin{align} B_{10}(y_{1:(k+1)}) &= ~B_{10}(y_{k+1}|y_{1:k}) \times B_{10}(y_{1:k}) \end{align} $$

(11) $$ \begin{align} &~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~= \frac{p(y_{k+1}|y_{1:k},\mathcal{H}_1)}{p(y_{k+1}|y_{1:k},\mathcal{H}_0)} \,\times~ \frac{p(y_{1:k}|\mathcal{H}_1)}{p(y_{1:k}|\mathcal{H}_0)}. \end{align} $$

This follows from basic probability calculus.Reference Ly, Etz, Marsman and Wagenmakers²² Consequently, we can also write the Bayes factor for all $k+1$ studies according to

(12) $$ \begin{align} B_{10}(y_{1:(k+1)}) = B_{10}(y_{k+1}|y_{1:k})\times\cdots\times B_{10}(y_{2}|y_{1})\times B_{10}(y_{1}). \end{align} $$

Rather than using this updating scheme explicitly, meta-analysts will likely compute the Bayes factor based on the $k+1$ studies “from scratch” because R packages generally only have functions for computing marginal likelihoods and Bayes factors for a given set of studies. Computing Bayes factors for a given set of studies is generally done by computing the marginal likelihoods using numerical algorithms (e.g., based on bridge samplingReference Bennett⁹³ using the R package “bridgesampling,”Reference Gronau, Singmann and Wagenmakers⁹⁴ or importance sampling,Reference Meng and Wong⁹⁵ as used in the R package “BFpack,”Reference Mulder, Williams and Gu³⁶ for instance). When no nuisance parameters are present (as in the CE model) or when the prior of the key parameter is independent of the prior of the nuisance parameter, for instance, it is also possible to compute the Bayes factor using the Savage–Dickey density ratio.Reference Dickey⁹⁶ This quantity is relatively easy to compute from MCMC output (e.g., using StanReference Carpenter, Gelman and Hoffman⁹⁷ or JAGS,Reference Plummer⁹⁸ for example). Here, we briefly explain this as it may give readers some extra intuition since viewing the behavior of Bayes factors as marginal likelihoods may be less intuitive.

The Savage–Dickey density ratio is defined by evaluating the posterior density of $\theta $ under the unconstrained hypothesis $\mathcal {H}_1$ , at the null value divided by the unconstrained prior density at the null valueReference Dickey⁹⁶:

$$\begin{align*}B_{01}(y_{1:k}) = \frac{p(\theta=0|\mathcal{H}_1,y_{1:k})} {p(\theta=0|\mathcal{H}_1)}. \end{align*}$$

Thus, we can compute the Bayes factor in favor of $\mathcal {H}_0$ by simply computing the posterior of $\theta $ (which is obtained from an estimation step) at zero divided by the chosen prior for $\theta $ at zero. The posterior density at the null value can easily be obtained from MCMC output with Bayesian software (e.g., Stan or JAGS). The prior density at the null value generally has an analytic form when the prior belongs to a common family of probability distributions. Consequently, if the density at zero increases from prior to posterior, then there is evidence in favor of the null value, and vice versa (Figure 4). In the case, a normal prior is considered, the Bayes factor has an analytic expression and thus numerical algorithms can be avoided (Appendix D).

Figure 4 Evidence for $\mathcal {H}_0$ based on a Savage–Dickey density ratio. In case of posterior 1, there is evidence for $\mathcal {H}_0$ , and in case of posterior 2, there would be evidence for $\mathcal {H}_1$ .

5.2. Evidence synthesis via the product Bayes factor

Under the FE model (6), each study is assumed to have a unique effect size, which are not linked via a distribution on a lower level (as in an RE model). Therefore, the Bayes factor for testing the null hypothesis in (12) can be simplified as the product of the Bayes factors of the separate studies (Appendix D):

(13) $$ \begin{align} B_{10}(y_{1:k}) = B_{10}(y_{1})\times \cdots \times B_{10}(y_{k}), \end{align} $$

which is sometimes called the “product Bayes factor.”Reference Van Lissa, Clapper and Kuiper⁴⁴ Consequently, when a new study is reported, we can simply multiply the current Bayes factor with the Bayes factor of the new study, and thus, (10) can be simplified as

(14) $$ \begin{align} B_{10}(y_{1:(k+1)}) = B_{10}(y_{k+1})\times B_{10}(y_{1:k}). \end{align} $$

The Bayes factor for the new study can be computed using the prior for the study-specific effect $\theta _k$ rather than requiring the posterior based on the previous studies as in regular Bayesian updating. As no nuisance parameters are present, the individual Bayes factors can also be computed using Savage–Dickey density ratios.

6.1. Evidence monitoring

Monitoring the evidence has great potential to minimize research waste by helping researchers, practitioners, patients, and funders to make more informed decisions to start new studies depending on the available statistical evidence regarding the (non)existence of certain effects.Reference Lau, Antman, Jimenez-Silva, Kupelnick, Mosteller and Chalmers^99– Reference Clarke, Brice and Chalmers¹⁰¹ A new study is most likely not initiated if there is, for instance, already overwhelming evidence that a treatment is (or is not) beneficial based on a meta-analysis. On the other hand, when the meta-analytic p-value is close to the significance threshold or equivalently the null value is very close to the boundary of its CI, a meta-analyst may be motivated to initiate another study to clarify whether the population mean is significantly different from zero (a message that is echoed in Fergusson et al.,Reference Fergusson, Glass and Hutton¹⁰² for example). Various retrospective meta-analyses showed that many studies were still performed while satisfactory evidence was already available, suggesting inefficient use of existing evidence. This also implies that even a meta-analysis conducted only once can implicitly be cumulative, since many included studies may have been initiated or designed in light of previous results.Reference Ter Schure and Grünwald¹³ Together, these observations have led to a call for more evidence-based research.Reference Lund, Brunnhuber and Juhl¹⁰³

Meta-analysts generally use classical p-value testing to test hypotheses that by default does not take the inherent sequential nature of published studies into account. A consequence of this is that it may result in inflated type I error rates if initiating a new study is based on (repeatedly) testing hypotheses based on evidence of published studies.Reference Ter Schure and Grünwald¹³ As discussed by Higgins et al.,Reference Higgins, Whitehead and Simmonds³ in order to properly control the type I error rate in a classical sequential analysis, restrictive decision rules are necessary, including a prespecified maximal amount of information when the sequential analysis need to stop permanently. When this bound is exceeded it is no longer allowed to collect more information and “it is unclear how to proceed.”Reference Higgins, Whitehead and Simmonds³ Still, we cannot claim there is evidence in favor of the null in this case, and to our knowledge no sequential procedures are available for equivalence testing to achieve this. These considerations highlight the complexity of controlling classical error control rates in meta-analyses that get updated. This is particularly problematic as many null hypotheses may in fact be true in applied research.Reference Johnson, Payne, Wang, Asher and Mandal²⁷

Using the Bayes factor, on the other hand, one can monitor and update the evidence as new studies become available in a flexible manner. Depending on the acquired evidence (e.g., using Figure 1), a meta-analyst can decide to initiate a new study or not without requiring corrections for multiple testing. The meta-analyst’s interpretation of the prior should be taken into account here. If a purely subjective Bayesian approach is considered, which is the case when the Bayes factor is based on informative priors that accurately reflects the meta-analyst’s prior beliefs, adding more studies in combination with multiple testing is not a problem.Reference De Heide and Grünwald¹⁸ This dates back to the work on subjective Bayesian statistics.Reference De Finetti^104, Reference Savage¹⁰⁵ Loosely speaking, it can be stated that Bayes factors generally have good performance (including accurate type I error rates) in the region where the prior is concentrated regardless of the exact rule that is used to decide to start a new study or not (e.g., see also Raftery & GillReference Raftery and Gill¹⁰⁶ and Kass & RafteryReference Kass and Raftery¹²).

When using noninformative, vague, or default priors, which may be chosen out of convenience (e.g., to avoid careful (potentially time-consuming) specification of informative priors), the story becomes more nuanced.Reference De Heide and Grünwald^18, Reference Hendriksen, Heide and Grünwald⁴⁰ On the one hand, it can be argued that when using noninformative priors for the between-study heterogeneity when testing the global mean, the Bayes factor may show good performance regardless of the true value or the nuisance parameter. This is because the exact choice of the prior for the nuisance parameter generally does not have a large effect on the Bayes factor (Section 4.3). On the other hand, recent developments on e-value theory have shown that Bayes factors may only control the exact classical error rates, depending on the choice of the prior for the nuisance parameter(s).

6.2. Bayes factors and e-value theory

In the last decade, there has been a considerable development of statistical theory on so-called e-values (“e” for expectation or expected value). For a recent overview of the statistical foundations, see.Reference Ramdas and Wang³² This theory provides critical conditions which need to hold in order to control classical type I error rates in sequential data designs in a flexible manner allowing optional stopping/continuation without requiring corrections as in classical p-value testing. Interestingly, there is a close relationship between e-values and Bayes factors.Reference Grünwald, Heide and Koolen³⁹ Before discussing this, we first give the general definition of an e-value.

Let E be nonnegative statistical quantity of the relative evidence in favor of an alternative hypothesis against a null hypothesis. Specifically, $E=1$ implies that there is no evidence toward any of the hypotheses, while $E>1$ implies that there is evidence in favor of the alternative $\mathcal {H}_1$ . Note that the Bayes factor for $\mathcal {H}_1$ against $\mathcal {H}_0$ also has this interpretation (Figure 1). Now E is called an e-value, if the expected evidence (or “average evidence”) never points toward the alternative when $\mathcal {H}_0$ is true. Mathematically, this can be written as:

(15) $$ \begin{align} \mathbb{E}_{\mathcal{H}_0}\{E\} \le 1. \end{align} $$

Thus, rather than restricting the chance of incorrectly falling in the rejection region, as in classical testing, e-value theory restricts the average evidence not pointing toward the wrong hypothesis when $\mathcal {H}_0$ is true. This is a considerably stronger requirement, implying a more conservative test.Reference Ramdas and Wang³²

If condition (15) holds, it can be shown that the reciprocal of an E value, say, $p_E=1/E$ , behaves as a conservative p-value, implying that the probability that $1/E$ is smaller than a prespecified significance threshold, say, $\alpha $ , is maximally $\alpha $ under the null, i.e.,

$$\begin{align*}\text{P}_{\mathcal{H}_0}(p_E \le \alpha)\le \alpha. \end{align*}$$

This is a direct consequence of the definition of an e-value (15) using Markov’s inequality. Because “ $p_E \le \alpha $ ” is equivalent to “ $E \ge 1/\alpha $ ,” we can therefore use the reciprocal of a significance level to construct significance thresholds for E values that ensure that type I error rates are never exceeded (Table 4). As e-values are very conservative, smaller thresholds have also been advocated.Reference Ramdas and Wang^32, Reference Shafer¹⁰⁷ A similar argument was made by RoyallReference Royall¹⁰⁸ in the context of likelihood ratios, and by Benjamin et al.Reference Benjamin, Berger and Johannesson¹⁰⁹ for Bayes factors. Furthermore, using theory on super-martingales, we can extend the concept of e-values for a single experiment to sequences of e-values, referred to as e-processes, which ensure evidence monitoring as new studies are reported.Reference Ramdas, Ruf, Larsson and Koolen^110– Reference Ville¹¹³ Using e-values for testing, the type I error rate will always be controlled under optional stopping/continuation.

Table 4 Linking thresholds for Bayes factors to common significance levels via $\alpha =1/B_{10}=B_{01}$ .

Although the reciprocal of the e-value behaves as a (conservative) p-value, the reverse argument does not hold. The reciprocal of a p-value, i.e., 1/p, where a smaller p, and thus a larger $1/p$ , also implies more evidence against the null, disastrously violates the e-value criterion in Equation (15) as the expected (average) evidence against the null, while the null is true, is in fact $\infty $ (Appendix E).

When no nuisance parameters are present—as in the CE and FE models—the Bayes factor is always an e-value (the proof is given in Appendix E). This is the case regardless of the prior that is specified for the effect(s) under $\mathcal {H}_1$ . Now, let us assume that after k studies, we observe a meta-analytic Bayes factor of, say, $B_{10}(y_{1:k})=22$ under the CE model. As noted above, we can use the reciprocal of a classical significance level, such as $\alpha =0.05$ , to construct rejection thresholds for the Bayes factors, say $1/0.05=20$ (Table 4). Importantly, this error-based threshold also does not need to be chosen prior to the test, as with classical significance testing,Reference Lakens, Adolfi and Albers¹¹⁴ but it can be even chosen post-hoc depending on the observed e-value.Footnote ^vii, Reference Ramdas and Wang³² As the observed Bayes factor of 22 exceeds this threshold, we can safely reject the null hypothesis without creating inflated type I error rates regardless of the intermediate choices that have been made to initiate new studies based on previous outcomes. For a classical test, this is generally not the case. In fact, if nothing is known about the motivations of research groups that initiated the past studies based on findings in earlier studies (implying that we do not know how much of our $\alpha$ was already “spent”Reference Ter Schure and Grünwald¹³), it will be practically impossible to guarantee a type I error rate of maximally 0.05 if a traditional significance test would be executed. Using a Bayes factor approach on the other hand, it is even possible to initiate a new study with the aim to obtain more decisive evidence against the null, again without jeopardizing inflated type error rates. This is a consequence of Bayes factors being independent on the sampling plan. On the other hand, it is generally known that p-values depend on the sampling plan.Reference Wagenmakers¹¹⁵

When nuisance parameters are present—as in RE and hybrid models—the choice of the prior for these parameters plays a crucial role in determining whether the e-value criterion is satisfied.Reference De Heide and Grünwald^18, Reference Hendriksen, Heide and Grünwald⁴⁰ The reason is that condition (15) needs to hold regardless of the true value of the nuisance parameter. As shown by Hendriksen et al.,Reference Hendriksen, Heide and Grünwald⁴⁰ so-called “group invariant” priors are generally recommended. For the between-study heterogeneity, however, the noninformative improper scale invariant prior for $\tau ^2$ is equal to the Jeffreys prior $1/\tau ^2$ , which is generally not recommended for lower level variances due to possible infinite marginal likelihoods, as was also explained in Section 4.3.

To give the reader an indication of whether the noninformative priors in Table 3 and the informative inverse gamma prior result in a Bayes factor that abides the e-value criterion, Appendix F provides numerical estimates of the expected Bayes factor under the null for different true values of the between-study heterogeneity. These results indicate that a uniform prior on $\tau ^2$ may abide the criterion, while the other two noninformative priors only result in very slight violations. The largest violations are observed for the informative $IG(1,0.15)$ prior. Because this informative prior is concentrated around small values (the prior mode of $\tau $ equals $0.075$ ), the e-value condition would only hold under for small $\tau $ values.

8.1. Example 1: Statistical learning of people with language impairment

The first example is a meta-analysis presented by Lammertink et al.Reference Lammertink, Boersma, Wijnen and Rispens¹¹⁸ on the difference in sequential statistical learning ability between people with and without specific language impairment. Sequential statistical learning refers to the ability to learn structures in text by, for instance, listening to people having a conversation. The goal of the meta-analysis was to assess whether people with a language impairment scored differently on statistical learning than people without such an impairment. Ten effect sizes were included in the meta-analysis. Hedges’ g standardized mean differences were reported for each study, where a larger Hedges’ g is indicative that people without a language impairment outperformed those with an impairment. Data of this meta-analysis are presented in a forest plot in the left panel of Figure 6.

Figure 6 Forest plot and the results of Bayesian updating for the meta-analysis of Lammertink et al. (2017). Estimates of the mean effect size of the Bayesian methods were obtained using a normal prior with mean zero and standard deviation of 1,000 for the mean effect.

We used the proposed normal prior with mean 0 and standard deviation 1 for the mean effect since the standardized mean difference was the effect size measure in this meta-analysis. The Berger–Deely prior was used for the between-study heterogeneity in the RE and marema models. For metaBMA, the prior $\mathcal {IG}(1,0.15)$ was used for $\tau $ since this is the default in this R package and metaBMA does not allow noninformative improper priors.

Lammertink et al.Reference Lammertink, Boersma, Wijnen and Rispens¹¹⁸ opted for an RE model even though $\tau ^2$ was estimated to be zero and the null hypothesis of no between-study variance was not rejected with the Q-test (Q(9) = 10.126, p = 0.340) in the frequentist test. This indicated that there was insufficient evidence to reject a CE model against an RE model. Also, the hybrid models showed no strong evidence for the presence of between-study heterogeneity with a posterior probability of 0.570 for $\tau ^2> 0$ under the marema model, and the BMA model resulted in a posterior probability of 0.355 for an RE model.

The first two columns of Table 5 show the Bayes factors of $\mathcal {H}_1$ vs. $\mathcal {H}_0$ and posterior probabilities for $\mathcal {H}_1$ . Based on all ten studies, all methods provided very strong to extreme support for $\mathcal {H}_1$ as all Bayes factors were large and the posterior probabilities of $\mathcal {H}_1$ were close to one. Hence, there is convincing evidence to conclude that people with a language impairment score differently on statistical learning than people without such an impairment. Note here that the posterior probabilities can also be viewed as conditional error probabilities.Reference Hoijtink, Mulder, Lissa and Gu¹¹⁹ For example, if one would conclude that $\mathcal {H}_1$ is true under the RE model, there would be a probability of 0.007 to draw the wrong conclusion given the available data.

Table 5 Bayes factors (BF $_{10}$ ) and posterior probabilities ( $PHP(\mathcal {H}_1)$ ) for testing $\mathcal {H}_1$ versus $\mathcal {H}_0$ based on all available studies.

The right panel of Figure 6 shows the Bayes factors using a sequential updating approach where a study’s publication year determined the order of the studies. The horizontal gray dashed lines indicate the different categories of evidence for the Bayes factor (Figure 1) and the dotted lines show the common significance thresholds (Table 4). These results show that $\mathcal {H}_0$ could “safely” be rejected at $\alpha = 0.001$ under the CE model and BMA after these ten studies regardless of the decisions were made to start each of these studies. For the RE and marema model, the evidence exceeds the threshold $\alpha = 0.01$ after ten studies. Also note in the right panel of Figure 6 that under the FE model, the evidence for $\mathcal {H}_1$ decreased at some point. This can be explained by obtaining evidence in favor of the null in studies 6–8, illustrating that the evidence under the FE model can be highly sensitive to the estimated effects of individual studies.

We also ran several sensitivity analyses to examine whether the results are robust to the prior of the nuisance parameter. These sensitivity analyses showed that the results hardly changed when different prior distributions were used. Details of these sensitivity analyses are reported in Appendix I.

8.2. Example 2: Exercising after a breast cancer surgery

The second example is a meta-analysis by McNeely et al.Reference McNeely, Campbell and Ospina³⁷ on the incidence of seroma when patients start exercising within or after three days following a breast cancer surgery. Five studies are included in this meta-analysis where patients were assigned to an early or delayed exercise condition in each study. The outcome variable was the occurrence of seroma. Thus, a log odds ratio was the effect size measure of interest. A log odds ratio larger (smaller) than one indicates that seroma is more (less) likely to appear in this early period compared to the delayed exercise condition. The data of this meta-analysis are presented in the forest plot in the left panel of Figure 7.

Figure 7 Forest plot and the results of Bayesian updating for the meta-analysis of McNeely et al. (2010). Estimates of the mean effect size of the Bayesian methods were obtained using a normal prior with mean zero and standard deviation of 1,000 for the mean effect.

Using uniform priors on the occurrence of seroma in both conditions implies an approximate Student t prior with a scale of 2.35 and 13 degrees of freedom for log odds ratio. Again, the Berger–Deely prior was used for the between-study heterogeneity in the RE and marema models, and the empirically informed inverse-gamma prior was used in the BMA model. McNeely et al.Reference McNeely, Campbell and Ospina³⁷ used an RE model in their meta-analysis even though the null hypothesis of no between-study heterogeneity was not rejected with the Q-test (Q(4) = 7.765, p= 0.101). This test was likely to be underpowered since there were only five studies included in this meta-analysis. The posterior probabilities of $\tau ^2> 0$ under the marema model were equal to 0.918, implying considerable evidence for positive between-study variance. Under the BMA model, a more conservative outcome was obtained yielding a posterior probability for the RE model of 0.649. Note that the outcome under the marema model gives a more rigid indication of the support for RE while the BMA model behaves a smoother (Section 7).

The Bayes factors and posterior probabilities for $\mathcal {H}_1$ under the different models are reported in the last two columns of Table 5. The results show inconclusive results with only mild evidence in the direction of $\mathcal {H}_0$ , except under the CE model (which most likely was incorrectly specified given the support for $\tau>0$ ). Moreover, the right panel in Figure 7 shows that the differences between the different models were small in the sequential updating approach. These results were in line with those of the frequentist meta-analysis (last two rows for the forest plot in Figure 7) where the null hypothesis of no average effect was rejected under the CE model ( $z=2.259$ , $p=0.024$ ) but not under the RE model ( $z=1.300$ , $p=0.194$ ). Due to the absence of evidence, more studies are needed to draw a more reliable conclusion about the effect of exercising on the incidence of seroma. Importantly, the type I error rate would not be at stake when using the Bayes factor as a test statistic in follow-up tests of this meta-analysis. For a classical test, this would have been problematic however. For example, under the RE model, a nonsignificant effect is obtained based on a significance level of 0.05, implying that the complete type I error probability of 0.05 has now been “spent,”Reference Ter Schure and Grünwald¹³ and therefore it would be “unclear how to proceed.”Reference Higgins, Whitehead and Simmonds³

We also studied the extent to which the results were sensitive to using different prior distributions for the nuisance parameter. These sensitivity analyses showed that the results hardly changed when different prior distributions were used. Details about these sensitivity analyses are reported in Appendix I.