Featured in The New York Times, Congratulations. Your Study Went Nowhere. Published on September 24, 2018.

Evidence-based medicine (EBM) is the cornerstone of modern clinical practice. Health care professionals trust in clinical trials to provide evidence on how to treat illnesses. But is the evidence base that EBM depends on as solid as we think?

Unfortunately, many clinical trials, particularly those without statistically significant findings, are never published, which leads to an overestimate of how effective treatments actually are. While this study publication bias is now well-known, it is actually only one of many biases that can distort apparent efficacy.

A play in four acts

In our recent publication, we illustrate the cumulative impact of four major biases, using the evidence base for depression treatments as a case study. Antidepressant trials are particularly suited for this, because pharmaceutical companies have to preregister trials that they want to use to obtain US marketing approval for a drug with the Food and Drug Administration (FDA). This means we can get information about all premarketing trials for antidepressants, even ones that were not published. Thanks to the FDA records, we started with a complete sample of 105 antidepressant trials.

Act I: Study publication bias

Fifty percent of the original trials had statistically significant findings for the primary outcome (positive trials), while the other half did not (negative trials). The figure shows that nearly all of the positive trials (in green), but only half of the negative trials (in red), were actually published. This is a classic example of study publication bias.

Act II: Outcome reporting bias

When we looked more closely, we found that ten of the negative trials actually reported positive results. When an article omits negative outcomes or switches primary and secondary outcomes (for example when the former is positive, but not the latter), this is called outcome reporting bias.

Act III: Spin

Even when trials do report non-significant results, interpretation of results can still be biased. Spin occurs when authors conclude that the treatment is effective despite non-significant results on the primary outcome, for instance by highlighting results in specific subgroups. Among the remaining 15 negative trials, most abstracts contained some degree of spin (e.g., concluding that the treatment was at least numerically better than placebo). Because of these three biases, only 5% of published trials unambiguously reported that the antidepressant was not better than placebo.

Act IV: Citation bias

And then, in the final act, we found that those few trials that clearly report negative results were cited less than half as often as trials reporting positive results. This citation bias further reduces the visibility of negative findings.

The cumulative impact of these reporting and citation biases is that treatments appear on the face of it to be much more effective than they actually are. Health care professionals, researchers, and policy makers need to be aware that similar processes are presumably at work in many medical fields. We hope that initiatives such as obligatory prospective trial registration will improve the evidence base of the future.

The University Medical Center Groningen also made an animation on these four biases that can be used as educational material.

Learn more about Ymkje Anna de Vries and Jojanneke Bastiaansen’s Open Access article, The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression, published in Psychological Medicine.