We were interested to read June's Psychological Medicine in which one of the methodological problems found in psychiatric case-control studies was examined (Schwartz & Susser, Reference Schwartz and Susser2011). The authors gave a hypothetical example of the use of ‘supernormal’ well controls where a certain disorder was an exclusion criterion for controls but not for cases, creating a difference in the prevalence of this disorder between the groups. This disorder had its own strong relationship with the exposure and overwhelmed any which may have existed between the exposure and the intended outcome, leading to misleading results. They point out that this practice has no conceivable benefits and should be discouraged.

Schwartz & Susser cite our work quantifying the prevalence of this and other problems in the general psychiatric literature for the years 2001 and 2002 (Lee et al. Reference Lee, Bindman, Ford, Glozier, Moran, Stewart and Hotopf2007). We found 32/408 (8%) of the studies were guilty of using supernormal controls and that the problem could not be excluded in a further 145 (36%). This left 231 (57%) of the studies reporting applying the same recruitment criteria to cases and controls, in accordance with best practice.

However, the particular issue of differential recruitment criteria was only the 14th most prevalent methodological problem of 17 examined. The most prevalent problems were poor reporting of the sampling of participants and poor descriptions of the cases, including whether incident cases were recruited. In general, non-reporting of key methodological issues was at least as great a problem as the reporting of poor methodology.

Since we collected our data, the STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) Statement has been published (von Elm et al. Reference von Elm, Altman, Egger, Pocock, Gotzsche and Vandenbroucke2007). It is a consensus statement documenting best practice of reporting observational epidemiology and is now officially adopted by at least 106 biomedical journals (STROBE, 2011), and four of the six journals we examined. The two journals which neither refer to the Statement in their instructions to authors nor refer to the Uniform Requirements published by the International Committee of Medical Journal Editors (which themselves refer to the Statement) are the British Journal of Psychiatry and Psychological Medicine, ironically the very journals which published these two methodological articles.

We suggest it is of particular importance for more general journals to insist on the use of guidelines of this kind because the existence of the powerful biases which may overwhelm true findings in case-control studies may not be acknowledged by all researchers in all domains of study. It is hoped that better reporting will lead to better methodology and therefore more valid results in observational epidemiology, including case-control studies in mental health research.