What is already known?

• • The I-squared index is widely used in relation to the identification or description of heterogeneity in meta-analysis.

• • The role of I-squared has been questioned, partly because it is often misinterpreted: It measures inconsistency in findings, which is not the same as variability in effect sizes.

What is new?

• • We provide a technical overview and our reflections on the I-squared index, including its definition, computation, and interpretation, and we offer a new definition that might be particularly useful when designing simulation studies.

Potential impact for RSM readers

• • Given the routine use of the I-squared index in diverse research fields, it is important that users have a proper understanding of what it can, and cannot, portray.

2.1 Standard (Q-based) definition of ${\boldsymbol{I}}^{\boldsymbol{2}}$

Perhaps the most familiar definition of the ${I}^2$ index is based on the $Q$ statistic, which is another commonly employed statistic used to examine the variability among the results included in the meta-analysis. In this approach, ${I}^2$ is defined as

(1) $$\begin{align}{I}^2=\frac{Q-\left(k-1\right)}{Q}\times 100\%,\end{align}$$

where $k$ is the number of studies and $Q$ is the ${\chi}^2$ statistic proposed by Cochran and defined as followsReference Cochran¹⁸:

(2) $$\begin{align}Q=\sum \limits_i{w}_i{\left({y}_i-{\overline{y}}_w\right)}^2.\end{align}$$

Here, ${y}_i$ is the result from the $i$ ^th study; ${w}_i=1/{\sigma}_i^2$ is the weight assigned to the result from the $i$ ^th study, with ${\sigma}_i^2$ being the within-study variance reflecting the sampling error in ${y}_i$ and typically assumed to be known in meta-analysis; and ${\overline{y}}_w={\sum {w}_i{y}_i} / {\sum {w}_i}$ is an inverse-variance-weighted average of the study-specific estimates. Under the simplifying assumption that weights are known and identical for every study, the $Q$ statistic follows a chi-squared distribution with $k-1$ degrees of freedom.Reference Hoaglin¹⁹ Although the $Q$ statistic has been widely used as a hypothesis test to detect heterogeneity among the individual results, it suffers from low statistical power in most meta-analytic applications and therefore can easily lead to misleading conclusions.Reference Borenstein, Hedges, Higgins and Rothstein^2, Reference Borenstein²⁰

Because the value of $Q$ can be smaller than its degrees of freedom, the ${I}^2$ index defined in (1) can be negative. Therefore, in practice, ${I}^2$ is customarily truncated at 0%, such that negative numbers (indicating less variation than would be expected by chance alone) are not allowed.

2.2 Conceptual definition of ${\boldsymbol{I}}^{\boldsymbol{2}}$

A broader definition of ${I}^2$ relates to the underlying parameters rather than to the test statistic $Q$ . Although the following definition appears to be parametric on the surface, it is not strictly parametric, as we shall discuss. We define a quantity ${\iota}^2$ as

(3) $$\begin{align}{\iota}^2=\frac{\tau^2}{\tau^2+{\sigma}^2}\times 100\%,\end{align}$$

where ${\tau}^2$ is the between-study variance parameter and ${\sigma}^2$ is a measure of within-study variance (i.e., the sampling error in estimation of ${y}_i$ ). We use a Greek letter here to distinguish the parametric form, ${\iota}^2$ , from a statistic computed from data, for which we use ${I}^2$ .

Numerous estimators of between-study variance, ${\tau}^2$ , have been described in the meta-analytic literature.Reference Veroniki, Jackson and Viechtbauer²¹ A moment-based estimate, ${\widehat{\tau}}_{DL}^2$ , introduced by DerSimonian and Laird, was popular for many years, although others are now considered preferable.Reference Langan, Higgins and Simmonds^22, Reference Langan, Higgins and Jackson²³ The moment-based estimate is given byReference DerSimonian and Laird²⁴

$$\begin{align*}{\widehat{\tau}}_{DL}^2=\max \left(\frac{Q-\left(k-1\right)}{\sum {w}_i-\frac{\sum {w}_i^2}{\sum {w}_i}},0\right).\end{align*}$$

The within-study variance quantity ${\sigma}^2$ is a clearly defined parameter only if all studies have the same within-study variance. Otherwise, ${\sigma}^2$ needs to be interpreted as a “typical” within-study variance across the studies included in the meta-analysis. Different choices are available for estimating such a typical variance, including the arithmetic mean, the median, and the geometric mean of the ${\sigma}_i^2$ . HT proposed the quantity:

(4) $$\begin{align}{\widehat{\sigma}}_{HT}^2=\frac{\left(k-1\right)\sum {w}_i}{{\left(\sum {w}_i\right)}^2-\sum {w}_i^2}\end{align}$$

with ${w}_i=1/{\sigma}_i^2$ .Reference Higgins and Thompson³ Takkouche, Cadarso-Suárez, and Spiegelman (TCS) usedReference Takkouche, CadarsoSurez and Spiegelman⁵

(5) $$\begin{align}{\widehat{\sigma}}_{TCS}^2=k\times Var\left({\overline{y}}_w\right).\end{align}$$

Since $Var\left({\overline{y}}_w\right)=1/\sum {w}_i$ and ${w}_i=1/{\sigma}_i^2$ , it can be seen that ${\widehat{\sigma}}_{TCS}^2$ is the harmonic mean of the ${\sigma}_i^2$ values.

Given that there are several options to estimate the between-study variance ${\tau}^2$ and several options to estimate a typical within-study variance ${\sigma}^2$ , numerous values of ${I}^2$ can be derived using Equation (3). The $Q$ -based definition of ${I}^2$ in Equation (1) corresponds to the choice of the DerSimonian and Laird estimator, ${\widehat{\tau}}_{DL}^2$ , and the use of ${\widehat{\sigma}}_{HT}^2$ from Equation (4). We refer to this as the “method-of-moments” (MM) definition:

(6) $$\begin{align}{I}_{MM}^2=\frac{{\widehat{\tau}}_{DL}^2}{{\widehat{\tau}}_{DL}^2+{\widehat{\sigma}}_{HT}^2}=\frac{\frac{Q-\left(k-1\right)}{\sum {w}_i-\frac{\sum {w}_i^2}{\sum {w}_i}}}{\left(\frac{Q-\left(k-1\right)}{\sum {w}_i-\frac{\sum {w}_i^2}{\sum {w}_i}}\right)+\left(\frac{\left(k-1\right)\sum {w}_i}{{\left(\sum {w}_i\right)}^2-\sum {w}_i^2}\right)}=\frac{Q-\left(k-1\right)}{Q}.\kern0.6em\end{align}$$

2.3 Heuristic definition of ${\boldsymbol{I}}^{\boldsymbol{2}}$

Noting the broad interpretation of ${I}^2$ as the proportion of total variation in observed results that arises from between-study heterogeneity,Reference Higgins and Thompson^3, Reference von Hippel²⁵ a further option to define ${I}^2$ is as follows:

(7) $$\begin{align}{I}^2=\frac{{\widehat{\tau}}^2}{Var\left({y}_i\right)}\times 100\%.\end{align}$$

The denominator in (7) is just the sample variance of the study estimates. Under the usual additive heterogeneity model, rather than a multiplicative heterogeneity model,Reference Mawdsley, Higgins, Sutton and Abrams²⁶ this can be regarded as conceptually equivalent to the sum of the between-study ( ${\tau}^2$ ) and within-study ( ${\sigma}^2$ ) variance components in the denominator of Equation (3).Reference von Hippel²⁵

2.4 General definition of ${\boldsymbol{I}}^{\boldsymbol{2}}$

The conceptual definition presented in Equation (3) is analogous to using weighted sums of squares as follows:

(8) $$\begin{align}{\iota}^2=\frac{WS{S}_b}{WS{S}_b+ WS{S}_w}\times 100\%.\end{align}$$

In this expression, $WS{S}_b=\sum {w}_i{\left({\theta}_i-\mu \right)}^2$ is the between-study weighted sum of squares and quantifies the dispersion of the effect parameters from each study ( ${\theta}_i$ ) from the overall mean $\mu$ . The term $WS{S}_w=\sum {w}_i{\left({y}_i-{\theta}_i\right)}^2$ is the within-study weighted sum of squares, which accounts for the dispersion of the estimates ( ${y}_i$ ) from the effect parameters ${\theta}_i$ for each study.

We can verify the conceptual equivalence of (8) with (3) and (7) by noting that

(9) $$\begin{align}WS{S}_{\mathrm{total}}&=\sum {w}_i{\left({y}_i-\mu \right)}^2 \nonumber\\&=\sum {w}_i{\left({y}_i-{\theta}_i+{\theta}_i-\mu \right)}^2 \nonumber\\&=\sum {w}_i{\left({y}_i-{\theta}_i\right)}^2+\sum {w}_i{\left({\theta}_i-\mu \right)}^2+2\sum {w}_i\left({y}_i-{\theta}_i\right)\left({\theta}_i-\mu \right),\end{align}$$

where $\sum {w}_i{\left({\theta}_i-\mu \right)}^2$ corresponds to $WS{S}_b$ and $\sum {w}_i{\left({y}_i-{\theta}_i\right)}^2$ corresponds to $WS{S}_w$ . The final term, $2\sum {w}_i\left({y}_i-{\theta}_i\right)\left({\theta}_i-\mu \right)$ , represents residual covariation between the within- and between-study variance components. When within-study variances are uncorrelated with effect sizes, this covariation will be zero and the $WS{S}_{\mathrm{total}}$ is equivalent to the sum of $WS{S}_b$ and $WS{S}_w$ .

The general definition of ${I}^2$ presented in Equation (8) provides a framework to specify the “true” value of the index ( ${\iota}^2$ ) in simulation studies. We used this definition for the study presented in the next section.

3.1 Application to two real data sets

Simulation studies indicate that alternatives to ${\widehat{\tau}}_{DL}^2$ have better properties,Reference Langan, Higgins and Simmonds^22, Reference Langan, Higgins and Jackson²³ so the question arises as to whether using an alternative estimator for ${\tau}^2$ in (3) would be preferable for computing ${I}^2$ . In addition, since different options are available for estimating ${\sigma}^2$ , it is not immediately clear which should be used. In a later section, we examine some of these issues in a simulation study. Here, we illustrate the impact that different choices can have through application to two examples, using the metafor R package for the calculations.Reference Viechtbauer²⁷

The first example is a set of 13 clinical trials of Bacillus Calmette–Guerin (BCG) vaccine to prevent tuberculosis,Reference Colditz, Brewer and Berkey²⁸ using data presented by Berkey and colleagues.Reference Berkey, Hoaglin, Mosteller and Colditz²⁹ The effect measure for this example is the log risk ratio comparing the risk of tuberculosis in the vaccinated versus unvaccinated groups. In Table 1, we list different values of ${I}^2$ for different estimators of ${\tau}^2$ and ${\sigma}^2$ as well as using the simple sample variance of the ${y}_i$ as the denominator (Equation (7)). A key point we seek to illustrate here is that ${I}^2$ values range markedly, from a minimum of 47.4% to a maximum of 94.2%. The usual value, based on the DerSimonian–Laird estimator of ${\tau}^2$ ( ${\widehat{\tau}}_{DL}^2$ ) and the HT “typical variance” formula ( ${\widehat{\sigma}}_{HT}^2$ ), is 92.1%.

Table 1 Different values of ${\boldsymbol{I}}^{\boldsymbol{2}}$ obtained using the different definitions, with different choices of estimators for ${\boldsymbol{\sigma}}^{\boldsymbol{2}}$ and ${\boldsymbol{\tau}}^{\boldsymbol{2}}$ , applied to studies of BCG vaccine

Note: The bold entry is the “standard” choice ( ${\boldsymbol{I}}_{\boldsymbol{MM}}^{\boldsymbol{2}}$ ) as introduced by Higgins and Thompson.

HT, Higgins and Thompson; TCS, Takkouche, Cadarso-Suárez, and Spiegelman; DL, DerSimonian–Laird; HS, Hunter–Schmidt; SJ, Sidik–Jonkman; MLE, maximum likelihood; REML, restricted maximum likelihood; EB, empirical Bayes; PM, Paule–Mandel.

The second example is a set of results from 19 studies examining how teachers’ expectations influence their pupils’ intelligence quotient (IQ) scores, a phenomenon sometimes labeled as the “Pygmalion effect” using data provided by Raudenbush (Table 2).Reference Raudenbush³⁰ The effect measure for this example is the standardized mean difference. Data were obtained from the metadata R package.Reference White, Nobles, Senior, Hamilton and Viechtbauer³¹ Again, ${I}^2$ values range notably, from a minimum of 9.8% to a maximum of 76.1%, with the usual value using ${\widehat{\tau}}_{DL}^2$ and ${\widehat{\sigma}}_{HT}^2$ being ${I}_{MM}^2$ = 49.8%. The question then arises as to whether any of these variants is “more correct” than others, and we turn to this question in the following section by conducting a small simulation study.

Table 2 Different values of ${\boldsymbol{I}}^{\boldsymbol{2}}$ obtained using different definitions, with different choices of estimators for ${\boldsymbol{\sigma}}^{\boldsymbol{2}}$ and ${\boldsymbol{\tau}}^{\boldsymbol{2}}$ , applied to studies on the “Pygmalion effect”

Note: The bold entry is the “standard” choice ( ${\boldsymbol{I}}_{\boldsymbol{MM}}^{\boldsymbol{2}}$ ) as introduced by Higgins and Thompson.

HT, Higgins and Thompson; TCS, Takkouche, Cadarso-Suárez, and Spiegelman; DL, DerSimonian–Laird; HS, Hunter–Schmidt; SJ, Sidik–Jonkman; MLE, maximum likelihood; REML, restricted maximum likelihood; EB, empirical Bayes; PM, Paule–Mandel.

3.2 A simulation study

3.2.2 Simulation results

Table 3 shows the results of the simulation. Weighted sums of squares $WS{S}_b$ and $WS{S}_w$ (see Equation (8)) yielded values of 0.001 and 0.004, respectively, the former corresponding to the preset known value of the between-study variance component and the latter defining the true value of the within-study variance component (this coincides with the harmonic mean of the vector of five preset ${\sigma}_i^2$ values), leading to a true parameter value of ${\iota}^2=20.0\%$ . Among the methods we compared, the formulae proposed by HT and by Takkouche and colleagues both yielded the correct average values of 0.004 for within-study variation. Conversely, using the median (0.005) or the arithmetic mean (0.005) led to the overestimation of within-study variation, resulting in an underestimation of ${\iota}^2$ (by 3.3% and 2.7%, respectively). Lastly, the method using the total sampling variance of the observed effect estimates also underestimated the true value of ${I}^2$ with an average value of 17.4%. Therefore, the different methods proposed to quantify a representative value of the within-study variation across studies, ${\sigma}^2$ , showed some discrepancies that affected the estimation of ${I}^2$ using Equation (3), with only the Takkouche and the HT methods performing close to the general formula presented in Equation (8). The lesson from our simulations is that only these two approaches should be considered (from those compared) for summarizing within-study variances across studies. Comparisons of these two methods in less favorable simulated scenarios can be found elsewhere.Reference Böhning, Lerdsuwansri and Holling^34, Reference Mittlbock and Heinzl³⁵ Our simulations involved large numbers of studies so as to ensure that the between-study variance was well estimated. Further simulation studies might elucidate preferred estimators of between-study variance, although we would expect findings to align with existing recommendations for the choice of between-study variance.Reference Langan, Higgins and Simmonds^22, Reference Langan, Higgins and Jackson²³

Table 3 Results from the simulation study

WSS, weighted sums of squares; HT, Higgins and Thompson; TCS, Takkouche, Cadarso-Suárez, and Spiegelman (harmonic mean); Med, median; AM, arithmetic mean.

5.1 What is a large value of ${\boldsymbol{I}}^{\boldsymbol{2}}$ ?

It is tempting to seek ranges or thresholds for the interpretation of ${I}^2$ : What is a large amount of inconsistency? The original paper proposing the index proposed that “mild heterogeneity might account for less than 30 per cent of the variability in point estimates, and notable heterogeneity substantially more than 50 per cent.” The use of the word “heterogeneity” in these proposals, while correct, may have led readers to believe the descriptors “mild” and “notable” related to the amount of heterogeneity rather than the amount of inconsistency. This is unfortunate. The arguments were based informally on the connection between heterogeneity and inconsistency for situations in which the chi-squared test for heterogeneity achieves statistical significance (at a 5% level) for around 10–30 studies, with meta-analyses of randomized trials very much in mind.Reference Higgins and Thompson³ The benchmarks did not directly consider the absolute magnitude of heterogeneity, as measured by ${\tau}^2$ .

The most widely cited paper about ${I}^2$ proposed “naive categorisation of values for ${I}^2$ would not be appropriate for all circumstances, although we would tentatively assign adjectives of low, moderate, and high to ${I}^2$ values of 25%, 50%, and 75%.”Reference Higgins, Thompson, Deeks and Altman⁴ Again, these proposals were intended to apply only to randomized trials. Despite the caveats, these benchmarks have become widely (mis)used and applied to situations far beyond meta-analyses of trials.

In an attempt to curb the overzealous use of the thresholds, the Cochrane Handbook for Systematic Reviews of Interventions in 2008 proposed overlapping ranges, again intended for use only with randomized trials: 0% to 40% might not be important; 30% to 60% may represent moderate heterogeneity; 50% to 90% may represent substantial heterogeneity; 75% to 100% may represent considerable heterogeneity.Reference Higgins, Thomas and Chandler⁵⁷ These suggestions were accompanied by clear guidance that “the importance of the observed value of ${I}^2$ depends on (i) magnitude and direction of effects and (ii) strength of evidence for heterogeneity (e.g., P-value from the chi-squared test, or a confidence interval for ${I}^2$ ),” advice which has again largely been ignored in practice.

Although these benchmarks can be helpful as a very rough guide, they should not be used to categorize meta-analysis data sets or to decide on whether, or how, to combine the results. We particularly discourage the use of thresholds for ${I}^2$ to decide between adopting fixed-effect(s) or random-effects meta-analysis models, because such decisions should be made on the basis of the question being asked rather than characteristics of the data.Reference Borenstein, Hedges, Higgins and Rothstein⁵⁸ Empirical investigations of typical values of ${I}^2$ can provide some useful pointers as to how a particular meta-analysis data set compares with others in related areas. Rhodes and colleagues reanalyzed 9,895 meta-analysis data sets from Cochrane reviews and fitted distributions to them.Reference Rhodes, Turner and Higgins⁵⁹ For meta-analyses of trials with binary outcomes, the median value of ${I}^2$ was 22% for log odds ratios, with interquartile range (IQR) 12% to 39%. For continuous outcomes meta-analysis, the median was 40% for standardized mean differences, with IQR 15% to 73%. Such a difference between binary and continuous outcomes has been noted by others.Reference Alba, Alexander, Chang, Maclsaac, DeFry and Guyatt⁶⁰ Differences have also been reported depending on the choice of effect measure within categoricalReference Rhodes, Turner and Higgins^59, Reference Papageorgiou, Tsiranidou, Antonoglou, Deschner and Jager⁶¹ and continuousReference Huedo-Medina, Sánchez-Meca, Marín-Martínez and Botella^54, Reference Rhodes, Turner and Higgins⁵⁹ outcomes, with, for example, inconsistency being higher for risk differences than for odds ratios or risk ratios.

Application of ${I}^2$ to results other than treatment effects from randomized trials can yield extreme results. An overview of 134 meta-analyses of prevalence studies found a median ${I}^2$ value of 96.9% (IQR: 90.5% to 98.7%).Reference Migliavaca, Stein and Colpani⁶² An overview of 138 reliability generalization meta-analyses found a median ${I}^2$ value of 93.2% (IQR: 88.8% to 96.4%) when integrating untransformed alpha coefficients, with similar results for the transformations typically used in this field.Reference López-Ibáñez, López-Nicolás, Blázquez-Rincón and Sánchez-Meca⁶³ These very high values reflect the higher precision with which these quantities are estimated in relation to the underlying heterogeneity. For example, studies of prevalence can be very large, being based on large surveys or databases. In this situation, even trivially small differences in prevalence can lead to very large values of ${I}^2$ . The utility of the ${I}^2$ index is therefore questionable in such situations, although as a statistic it cannot be said to be misleading or wrong.