Skip to main content

Biases in research: risk factors for non-replicability in psychotherapy and pharmacotherapy research

  • F. Leichsenring (a1), A. Abbass (a2), M. J. Hilsenroth (a3), F. Leweke (a1), P. Luyten (a4) (a5), J. R. Keefe (a6), N. Midgley (a7) (a8), S. Rabung (a9) (a10), S. Salzer (a11) (a12) and C. Steinert (a1)...

Replicability of findings is an essential prerequisite of research. For both basic and clinical research, however, low replicability of findings has recently been reported. Replicability may be affected by research biases not sufficiently controlled for by the existing research standards. Several biases such as researcher allegiance or selective reporting are well-known for affecting results. For psychotherapy and pharmacotherapy research, specific additional biases may affect outcome (e.g. therapist allegiance, therapist effects or impairments in treatment implementation). For meta-analyses further specific biases are relevant. In psychotherapy and pharmacotherapy research these biases have not yet been systematically discussed in the context of replicability. Using a list of 13 biases as a starting point, we discuss each bias's impact on replicability. We illustrate each bias by selective findings of recent research, showing that (1) several biases are not yet sufficiently controlled for by the presently applied research standards, (2) these biases have a pernicious effect on replicability of findings. For the sake of research credibility, it is critical to avoid these biases in future research. To control for biases and to improve replicability, we propose to systematically implement several measures in psychotherapy and pharmacotherapy research, such as adversarial collaboration (inviting academic rivals to collaborate), reviewing study design prior to knowing the results, triple-blind data analysis (including subjects, investigators and data managers/statisticians), data analysis by other research teams (crowdsourcing), and, last not least, updating reporting standards such as CONSORT or the Template for Intervention Description and Replication (TIDieR).

Corresponding author
*Address for correspondence: F. Leichsenring, DSc, Department of Psychosomatics and Psychotherapy, Justus-Liebig-University Giessen, Ludwigstr. 76, Giessen, Germany. (Email:
Hide All
† This paper is dedicated to my late teacher and friend Willi Hager.
Hide All
Antman, EM, Lau, J, Kupelnick, B, Mosteller, F, Chalmers, TC (1992). A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. Journal of the American Medical Association 268, 240248.
Asendorpf, J, Conner, M, De Fruyt, F, De Houwer, J, Denissen, J, Fiedler, K, Fiedler, S, Funder, DC, Kliegel, R, Nosek, BA, Perugini, M, Roberts, BW, Schmitt, M, van Aken, MAG, Weber, H, Wicherts, JM (2016). Recommendations for increasing replicability in psycholgy. In Methodological Issues and Strategies in Clinical Research, 4th edn (ed. Kazdin, A.), pp. 607622. American Psychological Association Washington: DC, US.
Baardseth, TP, Goldberg, SB, Pace, BT, Wislocki, AP, Frost, ND, Siddiqui, JR, Lindemann, AM, Kivlighan, DM III, Laska, KM, Del Re, AC, Minami, T, Wampold, BE (2013). Cognitive-behavioral therapy versus other therapies: redux. Clinical Psychology Review 33, 395405.
Barkham, M, Shapiro, DA, Hardy, GE, Rees, A (1999). Psychotherapy in two-plus-one sessions: outcomes of a randomized controlled trial of cognitive-behavioral and psychodynamic-interpersonal therapy for subsyndromal depression. Journal of Consulting and Clinical Psychology 67 201211.
Borenstein, M, Hedges, LV, Higgins, JPT, Rothstein, HR (2011). Introduction to Meta-analysis. Wiley: Chichester, UK.
Brunswik, E (1955). Representative design and probabilistic theory in a functional psychology. Psychological Review 62, 193217.
Carey, B (2015). Psychology's fears confirmed: rechecked studies don't hold up. New York Times, 27 August 2015.
Chan, AW, Hrobjartsson, A, Haahr, MT, Gotzsche, PC, Altman, DG (2004). Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. Journal of the American Medical Association 291, 24572465.
Clark, DM, Salkovskis, PM, Hackmann, A, Middleton, H, Anastasiades, P, Gelder, M (1994). A comparison of cognitive therapy, applied relaxation and imipramine in the treatment of panic disorder. British Journal of Psychiatry 164, 759769.
Cohen, J (1988). Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum: Hillsdale.
Cohen, J (1990). Things I have learned (so far). American Psychologist 45, 13041312.
Cottraux, J, Note, ID, Boutitie, F, Milliery, M, Genouihlac, V, Yao, SN, Note, B, Mollard, E, Bonasse, F, Gaillard, S, Djamoussian, D, Guillard Cde, M, Culem, A, Gueyffier, F (2009). Cognitive therapy versus Rogerian supportive therapy in borderline personality disorder. Two-year follow-up of a controlled pilot study. Psychotherapy and Psychosomatics 78, 307316.
Cuijpers, P (2016). Are all psychotherapies equally effective in the treatment of adult depression? The lack of statistical power of comparative outcome studies. Evidence-Based Mental Health 19, 3942.
Cuijpers, P, Berking, M, Andersson, G, Quigley, L, Kleiboer, A, Dobson, KS (2013 a). A meta-analysis of cognitive-behavioural therapy for adult depression, alone and in comparison with other treatments. Canadian Journal of Psychiatry 58, 376385.
Cuijpers, P, Huibers, M, Ebert, DD, Koole, SL, Andersson, G (2013 b). How much psychotherapy is needed to treat depression? A metaregression analysis. Journal of Affective Disorders 149, 113.
Cuijpers, P, Karyotaki, E, Andersson, G, Li, J, Mergl, R, Hegerl, U (2015). The effects of blinding on the outcomes of psychotherapy and pharmacotherapy for adult depression: a meta-analysis. European Psychiatry 30, 685693.
Cuijpers, P, Sijbrandij, M, Koole, SL, Andersson, G, Beekman, AT, Reynolds, CF III (2013 c). The efficacy of psychotherapy and pharmacotherapy in treating depressive and anxiety disorders: a meta-analysis of direct comparisons. World Psychiatry 12, 137148.
Cuijpers, P, Smit, F, Bohlmeijer, E, Hollon, SD, Andersson, G (2010). Efficacy of cognitive-behavioural therapy and other psychological treatments for adult depression: meta-analytic study of publication bias. British Journal of Psychiatry 196, 173178.
Davidson, JR, Foa, EB, Huppert, JD, Keefe, FJ, Franklin, ME, Compton, JS, Zhao, N, Connor, KM, Lynch, TR, Gadde, KM (2004). Fluoxetine, comprehensive cognitive behavioral therapy, and placebo in generalized social phobia. Archives of General Psychiatry 61, 10051013.
Doshi, P, Dickersin, K, Healy, D, Vedula, SS, Jefferson, T (2013). Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ (Clinical Research Edition) 346, f2865.
Dragioti, E, Dimoliatis, I, Evangelou, E (2015). Disclosure of researcher allegiance in meta-analyses and randomised controlled trials of psychotherapy: a systematic appraisal. BMJ Open 5, e007206.
Driessen, E, Hollon, SD, Bockting, CL, Cuijpers, P, Turner, EH (2015). Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? a systematic review and meta-analysis of US National Institutes of Health-Funded Trials. PLoS ONE 10, e0137864.
Dush, DM, Hirt, ML, Schroeder, H (1983). Self-statement modification with adults: a meta-analysis. Psychological Bulletin 94, 408422.
Elkin, I, Parloff, MB, Hadley, SW, Autry, JH (1985). NIMH treatment of depression collaborative research program. Archives of General Psychiatry 42, 305316.
Elkin, I, Shea, MT, Watkins, JT, Imber, SD, Sotsky, SM, Collins, JF, Glass, DR, Pilkonis, PA, Leber, WR, Docherty, JP, et al. (1989). National institute of mental health treatment of depression collaborative research program. General effectiveness of treatments. Archives of General Psychiatry 46, 971983.
Eysenck, HJ (1978). An exercise in mega-silliness. American Psychologist 33, 517.
Falkenström, F, Markowitz, JC, Jonker, H, Philips, B, Holmqvist, R (2013). Can psychotherapists function as their own controls? Meta-analysis of the crossed therapist design in comparative psychotherapy trials. Journal of Clinical Psychiatry 74, 482491.
Foa, EB, Rothbaum, BO, Riggs, DS, Murdock, TB (1991). Treatment of posttraumatic stress disorder in rape victims: a comparison between cognitive-behavioral procedures and counseling. Journal of Consulting and Clinical Psychology 59, 715723.
Fugelsang, JA, Stein, CB, Green, AE, Dunbar, KN (2004). Theory and data interactions of the scientific mind: evidence from the molecular and the cognitive laboratory. Canadian Journal of Experimental Psychology 58, 8695.
Gerber, AJ, Kocsis, JH, Milrod, BL, Roose, SP, Barber, JP, Thase, ME, Perkins, P, Leon, AC (2011). A quality-based review of randomized controlled trials of psychodynamic psychotherapy. American Journal of Psychiatry 168, 1928.
Giesen-Bloo, J, Arntz, A (2007). Questions concerning the randomized trial of schema-focused therapy vs transference-focused psychotherapy – Reply. Archives of General Psychiatry 64, 610611.
Giesen-Bloo, J, van Dyck, R, Spinhoven, P, van Tilburg, W, Dirksen, C, van Asselt, T, Kremers, I, Nadort, M, Arntz, A (2006). Outpatient psychotherapy for borderline personality disorder: randomized trial of schema-focused therapy vs transference-focused psychotherapy. Archives of General Psychiatry 63, 649658.
Gilboa-Schechtman, E, Foa, EB, Shafran, N, Aderka, IM, Powers, MB, Rachamim, L, Rosenbach, L, Yadin, E, Apter, A (2010). Prolonged exposure versus dynamic therapy for adolescent PTSD: a pilot randomized controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry 49, 10341042.
Higgins, JP, Altman, DG, Goetzsche, PC, Juni, P, Moher, D, Oxman, AD (2011). The cochrane statistical methods group. The Cochrane Collaboration's tool for assesingrisk of bias in randomized trials. British Medical Journal 343, d5928.
Hoffmann, TC, Glasziou, PP, Boutron, I, Milne, R, Perera, R, Moher, D, Altman, DG, Barbour, V, Macdonald, H, Johnston, M, Lamb, SE, Dixon-Woods, M, McCulloch, P, Wyatt, JC, Chan, AW, Michie, S (2014). Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ (Clinical Research Ed.) 348, g1687.
Horowitz, M, Kaltreider, N (1979). Brief therapy of the stress response syndrome. Psychiatric Clinics of North America 2, 365377.
Hsu, L (1989). Random sampling, randomization, and equivalence of contrasted groups in psychotherapy outcome research. Journal of Consulting and Clinical Psychology 57, 131137.
Ioannidis, JP (2005 a). Contradicted and initially stronger effects in highly cited clinical research. JAMA 294, 218228.
Ioannidis, JP (2005 b). Why most published research findings are false. PLoS Medicine 2, e124.
Ioannidis, JP, Allison, DB, Ball, CA, Coulibaly, I, Cui, X, Culhane, AC, Falchi, M, Furlanello, C, Game, L, Jurman, G, Mangion, J, Mehta, T, Nitzberg, M, Page, GP, Petretto, E, van Noort, V (2009). Repeatability of published microarray gene expression analyses. Nature Genetics 41, 149155.
Jacobson, NS (1991). To be or not to be behavioral when working with couples. What does it mean? Journal of Familiy Psychology 4, 436445.
Kazdin, AE (1994). Methodology, design, and evaluation in psychotherapy research. In Handbook of Psychotherapy and Behavior Change (ed. Bergin, A. E. and Garfield, S. L.), pp. 1971. Wiley: New York.
Keller, MB, Ryan, ND, Strober, M, Klein, RG, Kutcher, SP, Birmaher, B, Hagino, OR, Koplewicz, H, Carlson, GA, Clarke, GN, Emslie, GJ, Feinberg, D, Geller, B, Kusumakar, V, Papatheodorou, G, Sack, WH, Sweeney, M, Wagner, KD, Weller, EB, Winters, NC, Oakes, R, McCafferty, JP (2001). Efficacy of paroxetine in the treatment of adolescent major depression: a randomized, controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry 40, 762772.
Leichsenring, F, Leibing, E (2003). The effectiveness of psychodynamic therapy and cognitive behavior therapy in the treatment of personality disorders: a meta-analysis. American Journal of Psychiatry 160, 12231232.
Leichsenring, F, Leweke, F, Klein, S, Steinert, C (2015 a). The empirical status of psychodynamic psychotherapy – an update: Bambi's alive and kicking. Psychotherapy and Psychosomatics 84, 129148.
Leichsenring, F, Luyten, P, Hilsenroth, MJ, Abbass, A, Barber, JP, Keefe, JR, Leweke, F, Rabung, S, Steinert, C (2015 b). Psychodynamic therapy meets evidence-based medicine: a systematic review using updated criteria. Lancet Psychiatry 2, 648660.
Leichsenring, F, Salzer, S, Beutel, ME, Herpertz, S, Hiller, W, Hoyer, J, Huesing, J, Joraschky, P, Nolting, B, Poehlmann, K, Ritter, V, Stangier, U, Strauss, B, Stuhldreher, N, Tefikow, S, Teismann, T, Willutzki, U, Wiltink, J, Leibing, E (2013). Psychodynamic therapy and cognitive-behavioral therapy in social anxiety disorder: a multicenter randomized controlled trial. American Journal of Psychiatry 170, 759767.
Leichsenring, F, Steinert, C, Hoyer, J (2016). Psychotherapy versus pharmacotherapy of depression: What's the evidence? Zeitschrift fur Psychosomatische Medizin und Psychotherapie 62, 190195.
Le Noury, J, Nardo, JM, Healy, D, Jureidini, J, Raven, M, Tufanaru, C, Abi-Jaoude, E (2015). Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence. BMJ (Clinical Research Ed.) 351, h4320.
Lieb, K, Osten-Sacken, J, Stoffers-Winterling, J, Reiss, N, Barth, J (2016). Conflicts of interest and spin in reviews of psychological therapies: a systematic review. BMJ Open 6, e010606.
Luborsky, L, Diguer, L, Seligman, D, Rosenthal, R, Krause, E, Johnson, S, Halperin, G, Bishop, M, Berman, J, Schweizer, E (1999). The researcher's own allegiances: a ‘wild’ card in comparison of treatment efficacy. Clinical Psychology: Science and Practice 6, 95106.
MacCoun, R, Perlmutter, S (2015). Hide resuls to seek the truth. Nature 526, 187189.
Marcus, DK, O'Connell, D, Norris, AL, Sawaqdeh, A (2014). Is the Dodo bird endangered in the 21st century? A meta-analysis of treatment comparison studies. Clinical Psychology Review 34, 519530.
McKay, KM, Imel, ZE, Wampold, BE (2006). Psychiatrist effects in the psychopharmacological treatment of depression. Journal of Affective Disorders 92, 287290.
Meehl, PE (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald and the slow progress of soft psycholgy. Journal of Consulting and Clinical Psychology 46, 806834.
Mellers, B, Hertwig, R, Kahneman, D (2001). Do frequency representations eliminate conjunction effects? An exercise in adversarial collaboration. Psychological Science 12, 269275.
Miller, LE, Stewart, ME (2011). The blind leading the blind: use and misuse of blinding in randomized controlled trials. Contemporary Clinical Trials 32, 240243.
Miller, S, Wampold, B, Varhely, K (2008). Direct comparisons of treatment modalities for youth disorders: a meta-analysis. Psychotherapy Research 18, 514.
Milrod, B, Chambless, DL, Gallop, R, Busch, FN, Schwalberg, M, McCarthy, KS, Gross, C, Sharpless, BA, Leon, AC, Barber, JP (2016). Psychotherapies for panic disorder: a tale of two sites. Journal of Clinical Psychiatry 77, 927935.
Moher, D, Hopewell, S, Schulz, KF, Montori, V, Gotzsche, PC, Devereaux, PJ, Elbourne, D, Egger, M, Altman, DG (2010). CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. British Medical Journal (Clinical Research Edition) 340, c869.
Moher, D, Shamseer, L, Clarke, M, Ghersi, D, Liberati, A, Petticrew, M, Shekelle, P, Stewart, LA (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews 4, 1.
Munder, T, Brutsch, O, Leonhart, R, Gerger, H, Barth, J (2013). Researcher allegiance in psychotherapy outcome research: an overview of reviews. Clinical Psychology Review 33, 501511.
Munder, T, Fluckiger, C, Gerger, H, Wampold, BE, Barth, J (2012). Is the allegiance effect an epiphenomenon of true efficacy differences between treatments? a meta-analysis. Journal of Counseling Psychology 59, 631637.
Munder, T, Gerger, H, Trelle, S, Barth, J (2011). Testing the allegiance bias hypothesis: a meta-analysis. Psychotherapy Research 21, 670684.
Nuzzo, R (2015). How scientists fool themselves – and how they can stop. Nature 526, 182185.
Open Science Collaboration (2015). Psychology. Estimating the reproducibility of psychological science. Science 349, aac4716.
Owen, J, Drinane, JM, Idigo, KC, Valentine, JC (2015). Psychotherapist effects in meta-analyses: how accurate are treatment effects? Psychotherapy (Chic) 52, 321328.
Perepletchikova, F, Treat, AT, Kazdin, AE (2007). Treatment integrity in psychotherapy research: analysis of studies and examination of the associated factors. Journal of Consulting and Clinical Psychology 75, 829841.
Popper, KR (1959). The Logic of Scientific Discovery. Basic Books: New York.
Rothstein, HR, Sutton, AJ, Borenstein, M (2005). Publication bias. In Publication Bias in Meta-analysis: Prevention, Assessment and Adjustment (ed. Rothstein, H. R., Sutton, A. J. and Borenstein, M.), pp. 277302. Wiley & Sons: New York.
Silberzahn, R, Uhlmann, EL (2015). Many hands make tight work. Nature 526, 189191.
Smith, ML, Glass, GV, Miller, TI (1980). The Benefits of Psychotherapy. John Hopkins University Press: Baltimore.
Snyder, DK, Wills, RM (1989). Behavioral vs. insight-oriented marital therapy: effects on individual and interspousal functioning. Journal of Consulting and Clinical Psychology 57, 3946.
Snyder, DK, Wills, RM, Grady-Fletcher, A (1991). Long-term effectiveness of behavioral versus insight-oriented marital therapy: a 4-year follow-up study. Journal of Consulting and Clinical Psychology 59, 138141.
Spiwak, M (2016). Nothing but reviews in mind [Nichts als Gutachten im Kopf]. Die Zeit 32, 3132.
Stangier, U, Schramm, E, Heidenreich, T, Berger, M, Clark, DM (2011). Cognitive therapy vs interpersonal psychotherapy in social anxiety disorder: a randomized controlled trial. Archives of General Psychiatry 68, 692700.
Tajika, A, Ogawa, Y, Takeshima, N, Hayasaka, Y, Furukawa, TA (2015). Replication and contradiction of highly cited research papers in psychiatry: 10-year follow-up. British Journal of Psychiatry 207, 357362.
Thoma, NC, McKay, D, Gerber, AJ, Milrod, BL, Edwards, AR, Kocsis, JH (2012). A quality-based review of randomized controlled trials of cognitive-behavioral therapy for depression: an assessment and metaregression. American Journal of Psychiatry 169, 2230.
Tolin, DF (2010). Is cognitive-behavioral therapy more effective than other therapies? A meta-analytic review. Clinical Psychology Review 30, 710720.
Turner, EH, Matthews, AM, Linardatos, E, Tell, RA, Rosenthal, R (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine 358, 252260.
Walker, E, Nowacki, AS (2011). Understanding equivalence and noninferiority testing. Journal of General Internal Medicine 26, 192196.
Wampold, BE, Imel, ZE (2015). The Great Psychotherapy Debate: the Evidence for what Makes Psychotherapy Work. Routledge: New York.
Wampold, BE, Mondin, GW, Moody, M, Stich, F, Benson, K, Ahn, H (1997). A meta-analysis of outcome studies comparing bona fide psychotherapies: empirically, ‘All must have prizes’. Psychological Bulletin 122, 203215.
Watzke, B, Rüddel, H, Jürgensen, R, Koch, U, Kriston, L, Grothgar, B, Schulz, H (2012). Longer term outcome of cognitive-behavioural and psychodynamic psychotherapy in routine mental health care: randomised controlled trial. Behaviour Research and Therapy 50, 580587.
Woeller, W, Leichsenring, F, Leweke, F, Kruse, J (2012). Psychodynamic psychotherapy for posttraumatic stress disorder related to childhood abuse. Principles for a treatment manual. Bulletin of the Menninger Clinic 76, 6993.
Yeaton, WH, Sechrest, L (1981). Critical dimensions in the choice and maintenance of successful treatments: strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology 49, 156167.
Yeomans, F (2007). Questions concerning the randomized trial of schema-focused therapy vs transference-focused psychotherapy. Archives of General Psychiatry 64, 610611.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Psychological Medicine
  • ISSN: 0033-2917
  • EISSN: 1469-8978
  • URL: /core/journals/psychological-medicine
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Altmetric attention score

Full text views

Total number of HTML views: 33
Total number of PDF views: 326 *
Loading metrics...

Abstract views

Total abstract views: 1698 *
Loading metrics...

* Views captured on Cambridge Core between 13th December 2016 - 20th July 2018. This data will be updated every 24 hours.