Those worried about the cognitive bias modification (CBM) field being affected by ever-moving goal posts may have thought their concerns confirmed by Grafton and colleagues’ re-analysis of the meta-analysis by Cristea and colleagues.Reference Cristea, Kok and Cuijpers^1, Reference Grafton, MacLeod, Rudaizky, Holmes, Salemink and Fox² The paper concludes with the suggestion that we should only call CBM CBM if it is successful. To provide a treatment-inspired analogue: ‘This? No, this is just water, it's only homoeopathy if it works’.

It seems that we witness an almost prototypical disagreement between experimentalists and treatment evaluationists about which question to ask and which data to include. Importantly, the two author groups appear quite agreed that the answer to the question ‘whether assigning an anxious individual to engage in a CBM procedure will result in direct symptom reduction’ would be ‘not likely’. Perhaps Grafton and colleagues had better direct their critical attention towards the work by ‘field insiders’ in which CBM is quite consistently touted as a treatment, not to mention the apparent push for clinical dissemination and premature commercial exploitation. Thus, the question meta-analysed by Cristea and colleagues, authors specialising in meta-analytical evaluation of (proposed) treatments, appears perfectly legitimate.

Grafton and colleagues’ exposé on the correct question to meta-analyse, reads uncomfortably like a perceived-damage-containing mission. The discomfort is aggravated by the presented re-analysis, applying dichotomising and partly mystifying criteria to distil a subset of eligible studies from those selected for the original meta-analysis. Specifically, the requirements for a study to pass ‘Criterion 3: effect size computed by Cristea et al reflects legitimate emotional vulnerability assessment’ (Table 1, p. 268) remain unknown, as do the rules governing the final dividing criterion ‘intended CBM procedure successfully induced the process of bias modification’ (p. 268).

One could attempt to reconstruct the criterion rules from the tables provided, but it matters little. The analysis by Grafton and colleagues is flawed in a manner that must have escaped the attention of authors, reviewers and editors alike, even after Cristea and colleagues pointed it out in their commentary.Reference Cristea, Kok and Cuijpers³ To be very explicit: Grafton et al meta-analysed the study effect-size estimates calculated by Cristea et al.

In their original paper,Reference Cristea, Kok and Cuijpers¹ Cristea and colleagues state clearly that (a) for studies reporting multiple symptom outcomes, these were averaged into a single effect-size estimate (p. 8), and (b) effect-size estimates reflect symptoms assessed post-training, excluding assessments following a stressor procedure (p. 9).

Based on the narrative, it appears that criterion 3 has to do with each study either (a) assessing symptoms on trait (rather than state) measures, yet effect-size estimates averaging across state and trait measures were analysed, or (b) employing a post-training stressor procedure, yet symptoms assessed preceding such stressor procedures were analysed. Surely, we are not to assume reliable retro-active impact of unannounced stressors, nor that excluding studies with state measures only, results in adjustment of state measures retained for other studies. Therefore, we must conclude that this small yet crucial detail has gone unnoticed.

A meta-analysis by Grafton and colleagues, assessing evidence for their hypotheses, could perfectly exist alongside the meta-analysis by Cristea and colleagues. The currently presented re-analysis, however, does not convince.