Four ways to ruin a perfectly good meta-analysis

In science, the large-scale randomized controlled trial is considered the gold standard for estimating effects. But as such trials are expensive and time consuming, new research typically begins with small-scale studies which may be subsequently aggregated using meta-analysis. Relatively late in the game a randomized controlled trial may be run to provide the most definitive evidence on the subject, but in many cases the meta-analysis, for better or worse, will provide the last word on a subject. In those relatively rare instances where a large-scale randomized trial follows a meta-analysis, an opportunity emerges to compare the results obtained by the two methods. Most of the time the results are found to be congruent (Cappelleri et al. 1996; Villar and Carroli 1995). But there have been notable exceptions. In the medical field LeLorier et al. (1997) matched twelve large randomized controlled trials with nineteen meta-analyses and found several instances where a statistically significant result obtained by one method was paired with a nonsignificant result obtained by the other. Given the way in which decisions about new treatments are made in medicine, these authors concluded that if there had been no randomized controlled trials, the meta-analyses would have led to the adoption of an ineffective treatment in 32% of cases and to the rejection of a useful treatment in 33% of cases.