Skip to main content
×
×
Home

Replication and contradiction of highly cited research papers in psychiatry: 10-year follow-up

  • Aran Tajika (a1), Yusuke Ogawa (a1), Nozomi Takeshima (a1), Yu Hayasaka (a1) and Toshi A. Furukawa (a1)...
Abstract
Background

Contradictions and initial overestimates are not unusual among highly cited studies. However, this issue has not been researched in psychiatry.

Aims

To assess how highly cited studies in psychiatry are replicated by subsequent studies.

Method

We selected highly cited studies claiming effective psychiatric treatments in the years 2000 through 2002. For each of these studies we searched for subsequent studies with a better-controlled design, or with a similar design but a larger sample.

Results

Among 83 articles recommending effective interventions, 40 had not been subject to any attempt at replication, 16 were contradicted, 11 were found to have substantially smaller effects and only 16 were replicated. The standardised mean differences of the initial studies were overestimated by 132%. Studies with a total sample size of 100 or more tended to produce replicable results.

Conclusions

Caution is needed when a study with a small sample size reports a large effect.

    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Replication and contradiction of highly cited research papers in psychiatry: 10-year follow-up
      Available formats
      ×
      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Replication and contradiction of highly cited research papers in psychiatry: 10-year follow-up
      Available formats
      ×
      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Replication and contradiction of highly cited research papers in psychiatry: 10-year follow-up
      Available formats
      ×
Copyright
Corresponding author
Dr Aran Tajika, Department of Health Promotion and Human Behaviour, Kyoto University Graduate School of Medicine/School of Public Health, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501, Japan. Email: aran.tajika28@gmail.com
Footnotes
Hide All

Declaration of interest

None.

Footnotes
References
Hide All
1 US National Library of Medicine. Fact Sheet. MEDLINE, PubMed, and PMC (PubMed Central): How Are They Different? US National Library of Medicine, 2015 (http://www.nlm.nih.gov/pubs/factsheets/dif_med_pub.html).
2 Ioannidis, JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA 2005; 294: 218–28.
3 Pereira, TV, Horwitz, RI, Ioannidis, JP. Empirical evaluation of very large treatment effects of medical interventions. JAMA 2012; 308: 1676–84.
4 Nieminen, P, Rucker, G, Miettunen, J, Carpenter, J, Schumacher, M. Statistically significant papers in psychiatry were cited more often than others. J Clin Epidemiol 2007; 60: 939–46.
5 Hunt, GE, Cleary, M, Walter, G. Psychiatry and the Hirsch h-index: the relationship between journal impact factors and accrued citations. Harv Rev Psychiatry 2010; 18: 207–19.
6 Trikalinos, TA, Churchill, R, Ferri, M, Leucht, S, Tuunainen, A, Wahlbeck, K, et al. Effect sizes in cumulative meta-analyses of mental health randomized trials evolved over time. J Clin Epidemiol 2004; 57: 1124–30.
7 Hamilton, M. A rating scale for depression. J Neurol Neurosurg Psychiatry 1960; 23: 5662.
8 Kay, SR, Fiszbein, A, Opler, LA. The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophr Bull 1987; 13: 261–76.
9 Ioannidis, JP. Why most published research findings are false. PLoS Med 2005; 2: e124.
10 Ho, PM, Peterson, PN, Masoudi, FA. Evaluating the evidence: is there a rigid hierarchy? Circulation 2008; 118: 1675–84.
11 Chinn, S. A simple method for converting an odds ratio to effect size for use in meta-analysis. Stat Med 2000; 19: 3127–31.
12 Cohen, J. Statistical Power Analysis in the Behavioral Sciences. Erlbaum, 1988.
13 Higgins, J, Green, S. Cochrane Handbook for Systematic Reviews of Interventions (version 5.1.0). Cochrane Collaboration, 2011.
14 In t’ Veld, BA, Ruitenberg, A, Hofman, A, Launer, LJ, van Duijn, CM, Stijnen, T, et al. Nonsteroidal antiinflammatory drugs and the risk of Alzheimer's disease. N Engl J Med 2001; 345: 1515–21.
15 Breitner, JC, Baker, LD, Montine, TJ, Meinert, CL, Lyketsos, CG, Ashe, KH, et al. Extended results of the Alzheimer's disease anti-inflammatory prevention trial. Alzheimers Dement 2011; 7: 402–11.
16 Street, JS, Clark, WS, Gannon, KS, Cummings, JL, Bymaster, FP, Tamura, RN, et al. Olanzapine treatment of psychotic and behavioral symptoms in patients with Alzheimer disease in nursing care facilities: a double-blind, randomized, placebo-controlled trial. The HGEU Study Group. Arch Gen Psychiatry 2000; 57: 968–76.
17 De Deyn, PP, Carrasco, MM, Deberdt, W, Jeandel, C, Hay, DP, Feldman, PD, et al. Olanzapine versus placebo in the treatment of psychosis with or without associated behavioral disturbances in patients with Alzheimer's disease. Int J Geriatr Psychiatry 2004; 19: 115–26.
18 Tohen, M, Jacobs, TG, Grundy, SL, McElroy, SL, Banov, MC, Janicak, PG, et al. Efficacy of olanzapine in acute bipolar mania: a double-blind, placebo-controlled study. The Olanzipine HGGW Study Group. Arch Gen Psychiatry 2000; 57: 841–9.
19 McIntyre, RS, Cohen, M, Zhao, J, Alphs, L, Macek, TA, Panagides, J. A 3-week, randomized, placebo-controlled trial of asenapine in the treatment of acute mania in bipolar mania and mixed states. Bipolar Disord 2009; 11: 673–86.
20 Marshall, M, Lockwood, A, Bradley, C, Adams, C, Joy, C, Fenton, M. Unpublished rating scales: a major source of bias in randomised controlled trials of treatments for schizophrenia. Br J Psychiatry 2000; 176: 249–52.
21 Wood, L, Egger, M, Gluud, LL, Schulz, KF, Juni, P, Altman, DG, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 2008; 336: 601–5.
22 Savovic, J, Jones, HE, Altman, DG, Harris, RJ, Juni, P, Pildal, J, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med 2012; 157: 429–38.
23 Cumming, G. Replication and p intervals: p values predict the future only vaguely, but confidence intervals do much better. Perspect Psychol Sci 2008; 3: 286300.
24 Ioannidis, JP. Why most discovered true associations are inflated. Epidemiology 2008; 19: 640–8.
25 Leucht, S, Hierl, S, Kissling, W, Dold, M, Davis, JM. Putting the efficacy of psychiatric and general medicine medication into perspective: review of meta-analyses. Br J Psychiatry 2012; 200: 97106.
26 Schulz, KF, Chalmers, I, Hayes, RJ, Altman, DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408–12.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

The British Journal of Psychiatry
  • ISSN: 0007-1250
  • EISSN: 1472-1465
  • URL: /core/journals/the-british-journal-of-psychiatry
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
Type Description Title
PDF
Supplementary materials

Tajika et al. supplementary material
Supplementary Table S2

 PDF (86 KB)
86 KB
PDF
Supplementary materials

Tajika et al. supplementary material
Supplementary Table S1

 PDF (239 KB)
239 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 14 *
Loading metrics...

Abstract views

Total abstract views: 81 *
Loading metrics...

* Views captured on Cambridge Core between 2nd January 2018 - 20th April 2018. This data will be updated every 24 hours.

Replication and contradiction of highly cited research papers in psychiatry: 10-year follow-up

  • Aran Tajika (a1), Yusuke Ogawa (a1), Nozomi Takeshima (a1), Yu Hayasaka (a1) and Toshi A. Furukawa (a1)...
Submit a response

eLetters

No eLetters have been published for this article.

×

Reply to: Submit a response


Your details


Conflicting interests

Do you have any conflicting interests? *