Meta-analyses are powerful instruments for policy-making. Handle with caution

We appreciate that in their reply to our recent commentary van Oosterhout and colleagues (Reference van Oosterhout, Smit, Krabbendam, Castelein, Staring and van der Gaag2016) recalculated the data on metacognitive training (MCT) and now acknowledge that MCT exerts a significant (small to medium) effect for positive symptoms and delusions (although not for jumping to conclusions). This is very similar to results from a new meta-analysis (Eichner & Berna, Reference Eichner and Bernain press) (Hedges’ g = 0.34 for positive symptoms; g = 0.41 for delusions; g = 0.84 for acceptance), who used a larger body of studies than previous meta-analyses.

We thank the authors for this, and also wish to clarify some of the points they raised.

1. (1) The authors comment that the study by So et al. (Reference So, Chan, Chong, Wong, Lo, Chung and Chan2015) was brought to their attention at a point when their meta-analysis was already accepted for publication. The paper was indeed sent to the authors several days after the manuscript was formally accepted, but before it was (electronically) published. We can understand the authors’ hesitation to update the meta-analysis, especially when this may have meant jeopardizing an article in a top-tier journal. This is perhaps an ethical dilemma. Its inclusion, however, would have shifted some results to significance (see above) and accordingly, may have changed their inferences substantially.

2. (2) In general, one of the recurring controversies around meta-analyses is that studies are excluded based on the (sometimes idiosyncratic) criteria of authors, which can result in very different findings between meta-analyses on the same topic (Murray, Reference Murray2014). As an illustration of this, a recent meta-analysis on CBTp by the same group (Van der Gaag et al. Reference Van der Gaag, Valmaggia and Smit2014) arrived at very favorable conclusions – much in contrast to some other meta-analyses on CBTp that came out around the same time (Lynch et al. Reference Lynch, Laws and McKenna2010; Jauhar et al. Reference Jauhar, McKenna, Radua, Fung, Salvador and Laws2014; Mehl et al. Reference Mehl, Werner and Lincoln2015). The authors of the latest one (Mehl et al. Reference Mehl, Werner and Lincoln2015) indicate that the meta-analysis by van der Gaag et al. excluded studies that may have painted a very different picture.

3. (3) While MCT integrates many CBT principles, we believe that it is not adequate to equate MCT+ – a hybrid of MCT and CBTp – with CBTp. It is not until recently that pioneers of CBTp have started to challenge cognitive biases directly. Readers are kindly referred to studies written by founders of modern CBTp (Garety, Kuipers, Fowler, and Freeman) who developed a special reasoning training, which partly uses exercises derived from MCT/MCT+ (Waller et al. Reference Waller, Emsley, Freeman, Bebbington, Dunn, Fowler, Hardy, Kuipers and Garety2015), and combined it with CBTp. If challenging cognitive biases was already an essential component of CBTp, why then a supplementary training?

4. (4) In their letter, van Oosterhout et al. claim that patients with high baseline paranoia scores generally benefit most from metacognitive intervention and contrast the effect size of the three studies with the highest baseline delusions scores, including the one by So et al. (Reference So, Chan, Chong, Wong, Lo, Chung and Chan2015), against studies with the lowest delusions scores. Our point is not that patients with delusions should be completely excluded from group treatment, since the treatment is designed specifically to address delusions. Rather, we believe that groups consisting mostly of individuals with very severe delusions may be difficult to engage. Therefore, a group should not only consist of patients that display moderate to severe delusions, as in the original study by the authors (van Oosterhout et al. Reference van Oosterhout, Krabbendam, de Boer, Ferwerda, van der Helm, Stant and van der Gaag2014). In these cases, an individual approach like CBTp or individualized MCT should be adopted (in fact, the cited study by So et al. suggesting large effects in favor of MCT used an individualized protocol, and thus does not serve as counter-evidence).

As conclusions drawn from meta-analyses are often prematurely accepted as facts by both lay and expert audiences, and influence policy-making (inclusion or exclusion of therapies from guidelines), meta-analyses can cause much more harm than any single original study. While we agree that more quality research is warranted, we maintain that the existing evidence suggests that MCT is an effective tool in the treatment of delusions (we are prepared that this may well change with new studies, for better or worse). We thank van Oosterhout et al. for their courage in acknowledging that MCT exerts a significant effect on positive symptoms and delusions; however, their meta-analysis is already out and the damage done.