Selective exclusion of studies or outcomes from meta-analyses based on post-hoc criteria or applied without transparency or a solid theoretical justification is a pernicious practice that distorts results, ^{Reference Page, McKenzie, Kirkham, Dwan, Kramer and Green1} thus usually considered a grave error. Yet Grafton and colleagues seem to do exactly that with our meta-analysis. ^{Reference Cristea, Kok and Cuijpers2} They misconstrue it as investigating whether cognitive bias modification (CBM) alters ‘emotional vulnerability’, a vague concept of uncertain clinical relevance. Instead, as evident throughout, we examined post-intervention anxiety and depression outcomes. Using our data for anxiety outcomes at post-test, they employ three arbitrary filters to selectively exclude studies and subsequently perform a strictly qualitative and unclear classification of the remaining ones.

The most conspicuous filter is the exclusion of ten studies for measuring ‘resting mood state’ instead of ‘emotional vulnerability’. Eight of these used the State–Trait Anxiety Inventory (STAI)-State, one a specific phobia inventory, ^{Reference Harris and Menzies3} another social anxiety measures. ^{Reference Heeren, Reese, McNally and Philippot4} Hence, we are at a loss as to what the authors mean, as all of these measure symptoms of ‘anxiety’. Moreover, all our effect size calculations were, as described, at post-test. If for instance authors of a trial would use the STAI-State at both post-test and after a so-called stressor task, we only considered the former. So, in this sense everything was ‘resting mood’. Furthermore, if Grafton et al deemed state anxiety measures as improper, they should have also excluded them from effect size calculations for the other included trials. Their analysis nonetheless retained five other studies that solely used the STAI-State. ^{Reference Hirsch, Mathews and Clark5,Reference Murphy, Hirsch, Mathews, Smith and Clark6} Finally, Grafton et al do not substantiate their reanalysis with any actual data analysis, except for tallying findings as yes or no. We undertook this task for them. Presuming they intended to exclude state anxiety, we recalculated effect size for the 12 remaining studies measuring anxiety. The eight where bias change occurred resulted into a small Hedges g of 0.38, virtually identical to our original findings ^{Reference Cristea, Kok and Cuijpers2} (Duval-Tweedie publication bias adjusted g = 0.28). We conducted meta-regression analyses combining bias change with other significant moderators of outcome. ^{Reference Cristea, Kok and Cuijpers2} Bias change no longer predicted outcomes (Table 1).

Table 1

Meta-regression analysis for bias change, alone and in combination with other significant predictors of outcome ^{Reference Cristea, Kok and Cuijpers2}

Bias change
Alone	And participant compensation	And delivery	And impact factor
b = 0.42, P = 0.032	b = 0.37, P = 0.18	b = 0.25, P = 0.50	b = 0.24, P = 0.38

More generally, the claim that CBM should be assessed for effectiveness only in the presence of change in its postulated mechanisms conflicts with the current standards for evaluating psychotherapies. For instance, the effectiveness of cognitive–behavioural therapy ^{Reference Cuijpers, Berking, Andersson, Quigley, Kleiboer and Dobson7} is not restricted to trials where dysfunctional thoughts were successfully changed. Process variables are commonly conjectural, unclear, multiple, confounded with outcome measures, assessed in miscellaneous ways, and produce contradictory results. Even when a hypothesised process changes in a trial, it does not follow this is indeed a mechanism of change. ^{Reference Kazdin8} For CBM, the nature and direction of bias change needed to engender symptom change have been targets of speculation and debate, ^{Reference Koster and Bernstein9,Reference MacLeod and Mathews10} although posited as self-evident facts by Grafton et al.

Ultimately, CBM researchers should decide at which table they want to sit. If CBM is cast as a laboratory development, encouraging but as yet inconsequential for clinical practice, exploring procedures to modify assumed processes is an adequate goal. Conversely, if – as repeatedly claimed ^{Reference MacLeod and Mathews10} – CBM is a promising psychotherapy for use on patients and in clinical trials, it should comply with the same standards as all psychotherapies. These standards involve evaluating effectiveness on clinically relevant outcomes, using all available evidence, as we did, ^{Reference Cristea, Kok and Cuijpers2} and cannot hinge on whether or not purported processes have changed. Grafton et al summarily gloss over other serious problems we evidenced, such as lack of effects for clinical samples, pervasive publication bias and low study quality. Vague and debatable distinctions qualitatively applied post-hoc to a subset of the available data cannot substitute for modest if extant symptom change.