Flint and colleagues make use of a powerful statistical approach to suggest that there is significant positive bias in the expected distribution of published trials of psychotherapy (Flint et al. Reference Flint, Cuijpers, Horder, Koole and Munafò2014). They draw a parallel with the bias we know exists in the drug treatment literature and imply that it is again publication bias that explains the finding. In other words, withholding publication has deflated the number of non-significant results.

Indeed, for drug trials conducted by companies (the majority), we know this is the case because the results of negative trials are submitted to the regulatory authorities and can be identified if they are not published. Much has been made of this in previous reviews (Turner, Reference Turner2013).

However, publication bias is not necessarily the explanation for the findings in psychotherapy trials. We know that the quality of most psychotherapy trials falls far short of what is normal in drug trials (Huhn et al. Reference Huhn, Tardy, Spineli, Kissling, Förstl, Pitschel-Walz, Leucht, Samara, Dold, Davis and Leucht2014). Thus, a variety of biases operate to inflate the number of significant results in psychotherapy trials before publication. These include strong allegiance bias, single blind designs and a flexible approach to statistical treatment of outcomes. The result may well be the distorted distribution of results that Flint and colleagues identify in the published literature. A simple prediction is that significant numbers of unpublished negative studies simply do not exist for psychotherapies.

In conclusion, if bias distorts the generation of data rather than simply its reporting, we are presumably uncertain what a true effect size is for psychotherapy.