Skip to main content Accessibility help
Hostname: page-component-59b7f5684b-s82fj Total loading time: 0.474 Render date: 2022-10-04T04:20:23.495Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "useRatesEcommerce": false, "displayNetworkTab": true, "displayNetworkMapGraph": false, "useSa": true } hasContentIssue true

Biases in research: risk factors for non-replicability in psychotherapy and pharmacotherapy research

Published online by Cambridge University Press:  13 December 2016

F. Leichsenring*
Department of Psychosomatics and Psychotherapy, Justus-Liebig-University Giessen, Giessen, Germany
A. Abbass
Department of Psychiatry, Dalhousie University, Centre for Emotions and Health, Halifax, NS, Canada
M. J. Hilsenroth
The Derner Institute of Advanced Psychological Studies, Adelphi University, NY, USA
F. Leweke
Department of Psychosomatics and Psychotherapy, Justus-Liebig-University Giessen, Giessen, Germany
P. Luyten
Faculty of Psychology and Educational Sciences, University of Leuven, Klinische Psychologie (OE), Leuven, Belgium Research Department of Clinical, Educational and Health Psychology, University College London, London, UK
J. R. Keefe
Department of Psychology, University of Pennsylvania, Philadelphia, PA, USA
N. Midgley
The Anna Freud Centre, London, UK Research Department of Clinical, Educational and Health Psychology, UCL, London, UK
S. Rabung
Department of Psychology, Alpen-Adria-Universität Klagenfurt, Universitätsstr, Klagenfurt, Austria Department of Medical Psychology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
S. Salzer
Clinic of Psychosomatic Medicine and Psychotherapy, Georg-August-Universität Goettingen, Göttingen, Germany International Psychoanalytic University (IPU), Berlin, Germany
C. Steinert
Department of Psychosomatics and Psychotherapy, Justus-Liebig-University Giessen, Giessen, Germany
*Address for correspondence: F. Leichsenring, DSc, Department of Psychosomatics and Psychotherapy, Justus-Liebig-University Giessen, Ludwigstr. 76, Giessen, Germany. (Email:


Replicability of findings is an essential prerequisite of research. For both basic and clinical research, however, low replicability of findings has recently been reported. Replicability may be affected by research biases not sufficiently controlled for by the existing research standards. Several biases such as researcher allegiance or selective reporting are well-known for affecting results. For psychotherapy and pharmacotherapy research, specific additional biases may affect outcome (e.g. therapist allegiance, therapist effects or impairments in treatment implementation). For meta-analyses further specific biases are relevant. In psychotherapy and pharmacotherapy research these biases have not yet been systematically discussed in the context of replicability. Using a list of 13 biases as a starting point, we discuss each bias's impact on replicability. We illustrate each bias by selective findings of recent research, showing that (1) several biases are not yet sufficiently controlled for by the presently applied research standards, (2) these biases have a pernicious effect on replicability of findings. For the sake of research credibility, it is critical to avoid these biases in future research. To control for biases and to improve replicability, we propose to systematically implement several measures in psychotherapy and pharmacotherapy research, such as adversarial collaboration (inviting academic rivals to collaborate), reviewing study design prior to knowing the results, triple-blind data analysis (including subjects, investigators and data managers/statisticians), data analysis by other research teams (crowdsourcing), and, last not least, updating reporting standards such as CONSORT or the Template for Intervention Description and Replication (TIDieR).

Review Article
Copyright © Cambridge University Press 2016 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)


† This paper is dedicated to my late teacher and friend Willi Hager.


Antman, EM, Lau, J, Kupelnick, B, Mosteller, F, Chalmers, TC (1992). A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. Journal of the American Medical Association 268, 240248.CrossRefGoogle Scholar
Asendorpf, J, Conner, M, De Fruyt, F, De Houwer, J, Denissen, J, Fiedler, K, Fiedler, S, Funder, DC, Kliegel, R, Nosek, BA, Perugini, M, Roberts, BW, Schmitt, M, van Aken, MAG, Weber, H, Wicherts, JM (2016). Recommendations for increasing replicability in psycholgy. In Methodological Issues and Strategies in Clinical Research, 4th edn (ed. Kazdin, A.), pp. 607622. American Psychological Association Washington: DC, US.CrossRefGoogle Scholar
Baardseth, TP, Goldberg, SB, Pace, BT, Wislocki, AP, Frost, ND, Siddiqui, JR, Lindemann, AM, Kivlighan, DM III, Laska, KM, Del Re, AC, Minami, T, Wampold, BE (2013). Cognitive-behavioral therapy versus other therapies: redux. Clinical Psychology Review 33, 395405.CrossRefGoogle ScholarPubMed
Barkham, M, Shapiro, DA, Hardy, GE, Rees, A (1999). Psychotherapy in two-plus-one sessions: outcomes of a randomized controlled trial of cognitive-behavioral and psychodynamic-interpersonal therapy for subsyndromal depression. Journal of Consulting and Clinical Psychology 67 201211.CrossRefGoogle ScholarPubMed
Borenstein, M, Hedges, LV, Higgins, JPT, Rothstein, HR (2011). Introduction to Meta-analysis. Wiley: Chichester, UK.Google ScholarPubMed
Brunswik, E (1955). Representative design and probabilistic theory in a functional psychology. Psychological Review 62, 193217.CrossRefGoogle Scholar
Carey, B (2015). Psychology's fears confirmed: rechecked studies don't hold up. New York Times, 27 August 2015.Google Scholar
Chan, AW, Hrobjartsson, A, Haahr, MT, Gotzsche, PC, Altman, DG (2004). Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. Journal of the American Medical Association 291, 24572465.CrossRefGoogle ScholarPubMed
Clark, DM, Salkovskis, PM, Hackmann, A, Middleton, H, Anastasiades, P, Gelder, M (1994). A comparison of cognitive therapy, applied relaxation and imipramine in the treatment of panic disorder. British Journal of Psychiatry 164, 759769.CrossRefGoogle ScholarPubMed
Cohen, J (1988). Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum: Hillsdale.Google Scholar
Cohen, J (1990). Things I have learned (so far). American Psychologist 45, 13041312.CrossRefGoogle Scholar
Cottraux, J, Note, ID, Boutitie, F, Milliery, M, Genouihlac, V, Yao, SN, Note, B, Mollard, E, Bonasse, F, Gaillard, S, Djamoussian, D, Guillard Cde, M, Culem, A, Gueyffier, F (2009). Cognitive therapy versus Rogerian supportive therapy in borderline personality disorder. Two-year follow-up of a controlled pilot study. Psychotherapy and Psychosomatics 78, 307316.CrossRefGoogle ScholarPubMed
Cuijpers, P (2016). Are all psychotherapies equally effective in the treatment of adult depression? The lack of statistical power of comparative outcome studies. Evidence-Based Mental Health 19, 3942.CrossRefGoogle ScholarPubMed
Cuijpers, P, Berking, M, Andersson, G, Quigley, L, Kleiboer, A, Dobson, KS (2013 a). A meta-analysis of cognitive-behavioural therapy for adult depression, alone and in comparison with other treatments. Canadian Journal of Psychiatry 58, 376385.CrossRefGoogle ScholarPubMed
Cuijpers, P, Huibers, M, Ebert, DD, Koole, SL, Andersson, G (2013 b). How much psychotherapy is needed to treat depression? A metaregression analysis. Journal of Affective Disorders 149, 113.CrossRefGoogle Scholar
Cuijpers, P, Karyotaki, E, Andersson, G, Li, J, Mergl, R, Hegerl, U (2015). The effects of blinding on the outcomes of psychotherapy and pharmacotherapy for adult depression: a meta-analysis. European Psychiatry 30, 685693.CrossRefGoogle ScholarPubMed
Cuijpers, P, Sijbrandij, M, Koole, SL, Andersson, G, Beekman, AT, Reynolds, CF III (2013 c). The efficacy of psychotherapy and pharmacotherapy in treating depressive and anxiety disorders: a meta-analysis of direct comparisons. World Psychiatry 12, 137148.CrossRefGoogle ScholarPubMed
Cuijpers, P, Smit, F, Bohlmeijer, E, Hollon, SD, Andersson, G (2010). Efficacy of cognitive-behavioural therapy and other psychological treatments for adult depression: meta-analytic study of publication bias. British Journal of Psychiatry 196, 173178.CrossRefGoogle Scholar
Davidson, JR, Foa, EB, Huppert, JD, Keefe, FJ, Franklin, ME, Compton, JS, Zhao, N, Connor, KM, Lynch, TR, Gadde, KM (2004). Fluoxetine, comprehensive cognitive behavioral therapy, and placebo in generalized social phobia. Archives of General Psychiatry 61, 10051013.CrossRefGoogle ScholarPubMed
Doshi, P, Dickersin, K, Healy, D, Vedula, SS, Jefferson, T (2013). Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ (Clinical Research Edition) 346, f2865.Google Scholar
Dragioti, E, Dimoliatis, I, Evangelou, E (2015). Disclosure of researcher allegiance in meta-analyses and randomised controlled trials of psychotherapy: a systematic appraisal. BMJ Open 5, e007206.CrossRefGoogle ScholarPubMed
Driessen, E, Hollon, SD, Bockting, CL, Cuijpers, P, Turner, EH (2015). Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? a systematic review and meta-analysis of US National Institutes of Health-Funded Trials. PLoS ONE 10, e0137864.CrossRefGoogle ScholarPubMed
Dush, DM, Hirt, ML, Schroeder, H (1983). Self-statement modification with adults: a meta-analysis. Psychological Bulletin 94, 408422.CrossRefGoogle Scholar
Elkin, I, Parloff, MB, Hadley, SW, Autry, JH (1985). NIMH treatment of depression collaborative research program. Archives of General Psychiatry 42, 305316.CrossRefGoogle ScholarPubMed
Elkin, I, Shea, MT, Watkins, JT, Imber, SD, Sotsky, SM, Collins, JF, Glass, DR, Pilkonis, PA, Leber, WR, Docherty, JP, et al. (1989). National institute of mental health treatment of depression collaborative research program. General effectiveness of treatments. Archives of General Psychiatry 46, 971983.CrossRefGoogle ScholarPubMed
Eysenck, HJ (1978). An exercise in mega-silliness. American Psychologist 33, 517.CrossRefGoogle Scholar
Falkenström, F, Markowitz, JC, Jonker, H, Philips, B, Holmqvist, R (2013). Can psychotherapists function as their own controls? Meta-analysis of the crossed therapist design in comparative psychotherapy trials. Journal of Clinical Psychiatry 74, 482491.CrossRefGoogle ScholarPubMed
Foa, EB, Rothbaum, BO, Riggs, DS, Murdock, TB (1991). Treatment of posttraumatic stress disorder in rape victims: a comparison between cognitive-behavioral procedures and counseling. Journal of Consulting and Clinical Psychology 59, 715723.CrossRefGoogle ScholarPubMed
Fugelsang, JA, Stein, CB, Green, AE, Dunbar, KN (2004). Theory and data interactions of the scientific mind: evidence from the molecular and the cognitive laboratory. Canadian Journal of Experimental Psychology 58, 8695.CrossRefGoogle ScholarPubMed
Gerber, AJ, Kocsis, JH, Milrod, BL, Roose, SP, Barber, JP, Thase, ME, Perkins, P, Leon, AC (2011). A quality-based review of randomized controlled trials of psychodynamic psychotherapy. American Journal of Psychiatry 168, 1928.CrossRefGoogle ScholarPubMed
Giesen-Bloo, J, Arntz, A (2007). Questions concerning the randomized trial of schema-focused therapy vs transference-focused psychotherapy – Reply. Archives of General Psychiatry 64, 610611.CrossRefGoogle Scholar
Giesen-Bloo, J, van Dyck, R, Spinhoven, P, van Tilburg, W, Dirksen, C, van Asselt, T, Kremers, I, Nadort, M, Arntz, A (2006). Outpatient psychotherapy for borderline personality disorder: randomized trial of schema-focused therapy vs transference-focused psychotherapy. Archives of General Psychiatry 63, 649658.CrossRefGoogle ScholarPubMed
Gilboa-Schechtman, E, Foa, EB, Shafran, N, Aderka, IM, Powers, MB, Rachamim, L, Rosenbach, L, Yadin, E, Apter, A (2010). Prolonged exposure versus dynamic therapy for adolescent PTSD: a pilot randomized controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry 49, 10341042.CrossRefGoogle ScholarPubMed
Higgins, JP, Altman, DG, Goetzsche, PC, Juni, P, Moher, D, Oxman, AD (2011). The cochrane statistical methods group. The Cochrane Collaboration's tool for assesingrisk of bias in randomized trials. British Medical Journal 343, d5928.CrossRefGoogle Scholar
Hoffmann, TC, Glasziou, PP, Boutron, I, Milne, R, Perera, R, Moher, D, Altman, DG, Barbour, V, Macdonald, H, Johnston, M, Lamb, SE, Dixon-Woods, M, McCulloch, P, Wyatt, JC, Chan, AW, Michie, S (2014). Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ (Clinical Research Ed.) 348, g1687.Google ScholarPubMed
Horowitz, M, Kaltreider, N (1979). Brief therapy of the stress response syndrome. Psychiatric Clinics of North America 2, 365377.Google Scholar
Hsu, L (1989). Random sampling, randomization, and equivalence of contrasted groups in psychotherapy outcome research. Journal of Consulting and Clinical Psychology 57, 131137.CrossRefGoogle ScholarPubMed
Ioannidis, JP (2005 a). Contradicted and initially stronger effects in highly cited clinical research. JAMA 294, 218228.CrossRefGoogle ScholarPubMed
Ioannidis, JP (2005 b). Why most published research findings are false. PLoS Medicine 2, e124.CrossRefGoogle ScholarPubMed
Ioannidis, JP, Allison, DB, Ball, CA, Coulibaly, I, Cui, X, Culhane, AC, Falchi, M, Furlanello, C, Game, L, Jurman, G, Mangion, J, Mehta, T, Nitzberg, M, Page, GP, Petretto, E, van Noort, V (2009). Repeatability of published microarray gene expression analyses. Nature Genetics 41, 149155.CrossRefGoogle ScholarPubMed
Jacobson, NS (1991). To be or not to be behavioral when working with couples. What does it mean? Journal of Familiy Psychology 4, 436445.CrossRefGoogle Scholar
Kazdin, AE (1994). Methodology, design, and evaluation in psychotherapy research. In Handbook of Psychotherapy and Behavior Change (ed. Bergin, A. E. and Garfield, S. L.), pp. 1971. Wiley: New York.Google Scholar
Keller, MB, Ryan, ND, Strober, M, Klein, RG, Kutcher, SP, Birmaher, B, Hagino, OR, Koplewicz, H, Carlson, GA, Clarke, GN, Emslie, GJ, Feinberg, D, Geller, B, Kusumakar, V, Papatheodorou, G, Sack, WH, Sweeney, M, Wagner, KD, Weller, EB, Winters, NC, Oakes, R, McCafferty, JP (2001). Efficacy of paroxetine in the treatment of adolescent major depression: a randomized, controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry 40, 762772.CrossRefGoogle ScholarPubMed
Leichsenring, F, Leibing, E (2003). The effectiveness of psychodynamic therapy and cognitive behavior therapy in the treatment of personality disorders: a meta-analysis. American Journal of Psychiatry 160, 12231232.CrossRefGoogle ScholarPubMed
Leichsenring, F, Leweke, F, Klein, S, Steinert, C (2015 a). The empirical status of psychodynamic psychotherapy – an update: Bambi's alive and kicking. Psychotherapy and Psychosomatics 84, 129148.CrossRefGoogle ScholarPubMed
Leichsenring, F, Luyten, P, Hilsenroth, MJ, Abbass, A, Barber, JP, Keefe, JR, Leweke, F, Rabung, S, Steinert, C (2015 b). Psychodynamic therapy meets evidence-based medicine: a systematic review using updated criteria. Lancet Psychiatry 2, 648660.CrossRefGoogle ScholarPubMed
Leichsenring, F, Salzer, S, Beutel, ME, Herpertz, S, Hiller, W, Hoyer, J, Huesing, J, Joraschky, P, Nolting, B, Poehlmann, K, Ritter, V, Stangier, U, Strauss, B, Stuhldreher, N, Tefikow, S, Teismann, T, Willutzki, U, Wiltink, J, Leibing, E (2013). Psychodynamic therapy and cognitive-behavioral therapy in social anxiety disorder: a multicenter randomized controlled trial. American Journal of Psychiatry 170, 759767.CrossRefGoogle ScholarPubMed
Leichsenring, F, Steinert, C, Hoyer, J (2016). Psychotherapy versus pharmacotherapy of depression: What's the evidence? Zeitschrift fur Psychosomatische Medizin und Psychotherapie 62, 190195.CrossRefGoogle ScholarPubMed
Le Noury, J, Nardo, JM, Healy, D, Jureidini, J, Raven, M, Tufanaru, C, Abi-Jaoude, E (2015). Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence. BMJ (Clinical Research Ed.) 351, h4320.Google ScholarPubMed
Lieb, K, Osten-Sacken, J, Stoffers-Winterling, J, Reiss, N, Barth, J (2016). Conflicts of interest and spin in reviews of psychological therapies: a systematic review. BMJ Open 6, e010606.CrossRefGoogle ScholarPubMed
Luborsky, L, Diguer, L, Seligman, D, Rosenthal, R, Krause, E, Johnson, S, Halperin, G, Bishop, M, Berman, J, Schweizer, E (1999). The researcher's own allegiances: a ‘wild’ card in comparison of treatment efficacy. Clinical Psychology: Science and Practice 6, 95106.Google Scholar
MacCoun, R, Perlmutter, S (2015). Hide resuls to seek the truth. Nature 526, 187189.CrossRefGoogle Scholar
Marcus, DK, O'Connell, D, Norris, AL, Sawaqdeh, A (2014). Is the Dodo bird endangered in the 21st century? A meta-analysis of treatment comparison studies. Clinical Psychology Review 34, 519530.CrossRefGoogle ScholarPubMed
McKay, KM, Imel, ZE, Wampold, BE (2006). Psychiatrist effects in the psychopharmacological treatment of depression. Journal of Affective Disorders 92, 287290.CrossRefGoogle ScholarPubMed
Meehl, PE (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald and the slow progress of soft psycholgy. Journal of Consulting and Clinical Psychology 46, 806834.CrossRefGoogle Scholar
Mellers, B, Hertwig, R, Kahneman, D (2001). Do frequency representations eliminate conjunction effects? An exercise in adversarial collaboration. Psychological Science 12, 269275.CrossRefGoogle ScholarPubMed
Miller, LE, Stewart, ME (2011). The blind leading the blind: use and misuse of blinding in randomized controlled trials. Contemporary Clinical Trials 32, 240243.CrossRefGoogle ScholarPubMed
Miller, S, Wampold, B, Varhely, K (2008). Direct comparisons of treatment modalities for youth disorders: a meta-analysis. Psychotherapy Research 18, 514.CrossRefGoogle ScholarPubMed
Milrod, B, Chambless, DL, Gallop, R, Busch, FN, Schwalberg, M, McCarthy, KS, Gross, C, Sharpless, BA, Leon, AC, Barber, JP (2016). Psychotherapies for panic disorder: a tale of two sites. Journal of Clinical Psychiatry 77, 927935.CrossRefGoogle ScholarPubMed
Moher, D, Hopewell, S, Schulz, KF, Montori, V, Gotzsche, PC, Devereaux, PJ, Elbourne, D, Egger, M, Altman, DG (2010). CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. British Medical Journal (Clinical Research Edition) 340, c869.CrossRefGoogle ScholarPubMed
Moher, D, Shamseer, L, Clarke, M, Ghersi, D, Liberati, A, Petticrew, M, Shekelle, P, Stewart, LA (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews 4, 1.CrossRefGoogle ScholarPubMed
Munder, T, Brutsch, O, Leonhart, R, Gerger, H, Barth, J (2013). Researcher allegiance in psychotherapy outcome research: an overview of reviews. Clinical Psychology Review 33, 501511.CrossRefGoogle ScholarPubMed
Munder, T, Fluckiger, C, Gerger, H, Wampold, BE, Barth, J (2012). Is the allegiance effect an epiphenomenon of true efficacy differences between treatments? a meta-analysis. Journal of Counseling Psychology 59, 631637.CrossRefGoogle ScholarPubMed
Munder, T, Gerger, H, Trelle, S, Barth, J (2011). Testing the allegiance bias hypothesis: a meta-analysis. Psychotherapy Research 21, 670684.CrossRefGoogle ScholarPubMed
Nuzzo, R (2015). How scientists fool themselves – and how they can stop. Nature 526, 182185.CrossRefGoogle Scholar
Open Science Collaboration (2015). Psychology. Estimating the reproducibility of psychological science. Science 349, aac4716.CrossRefGoogle ScholarPubMed
Owen, J, Drinane, JM, Idigo, KC, Valentine, JC (2015). Psychotherapist effects in meta-analyses: how accurate are treatment effects? Psychotherapy (Chic) 52, 321328.CrossRefGoogle ScholarPubMed
Perepletchikova, F, Treat, AT, Kazdin, AE (2007). Treatment integrity in psychotherapy research: analysis of studies and examination of the associated factors. Journal of Consulting and Clinical Psychology 75, 829841.CrossRefGoogle ScholarPubMed
Popper, KR (1959). The Logic of Scientific Discovery. Basic Books: New York.Google Scholar
Rothstein, HR, Sutton, AJ, Borenstein, M (2005). Publication bias. In Publication Bias in Meta-analysis: Prevention, Assessment and Adjustment (ed. Rothstein, H. R., Sutton, A. J. and Borenstein, M.), pp. 277302. Wiley & Sons: New York.CrossRefGoogle Scholar
Silberzahn, R, Uhlmann, EL (2015). Many hands make tight work. Nature 526, 189191.CrossRefGoogle ScholarPubMed
Smith, ML, Glass, GV, Miller, TI (1980). The Benefits of Psychotherapy. John Hopkins University Press: Baltimore.Google Scholar
Snyder, DK, Wills, RM (1989). Behavioral vs. insight-oriented marital therapy: effects on individual and interspousal functioning. Journal of Consulting and Clinical Psychology 57, 3946.CrossRefGoogle Scholar
Snyder, DK, Wills, RM, Grady-Fletcher, A (1991). Long-term effectiveness of behavioral versus insight-oriented marital therapy: a 4-year follow-up study. Journal of Consulting and Clinical Psychology 59, 138141.CrossRefGoogle ScholarPubMed
Spiwak, M (2016). Nothing but reviews in mind [Nichts als Gutachten im Kopf]. Die Zeit 32, 3132.Google Scholar
Stangier, U, Schramm, E, Heidenreich, T, Berger, M, Clark, DM (2011). Cognitive therapy vs interpersonal psychotherapy in social anxiety disorder: a randomized controlled trial. Archives of General Psychiatry 68, 692700.CrossRefGoogle ScholarPubMed
Tajika, A, Ogawa, Y, Takeshima, N, Hayasaka, Y, Furukawa, TA (2015). Replication and contradiction of highly cited research papers in psychiatry: 10-year follow-up. British Journal of Psychiatry 207, 357362.CrossRefGoogle ScholarPubMed
Thoma, NC, McKay, D, Gerber, AJ, Milrod, BL, Edwards, AR, Kocsis, JH (2012). A quality-based review of randomized controlled trials of cognitive-behavioral therapy for depression: an assessment and metaregression. American Journal of Psychiatry 169, 2230.CrossRefGoogle ScholarPubMed
Tolin, DF (2010). Is cognitive-behavioral therapy more effective than other therapies? A meta-analytic review. Clinical Psychology Review 30, 710720.CrossRefGoogle ScholarPubMed
Turner, EH, Matthews, AM, Linardatos, E, Tell, RA, Rosenthal, R (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine 358, 252260.CrossRefGoogle ScholarPubMed
Walker, E, Nowacki, AS (2011). Understanding equivalence and noninferiority testing. Journal of General Internal Medicine 26, 192196.CrossRefGoogle ScholarPubMed
Wampold, BE, Imel, ZE (2015). The Great Psychotherapy Debate: the Evidence for what Makes Psychotherapy Work. Routledge: New York.Google Scholar
Wampold, BE, Mondin, GW, Moody, M, Stich, F, Benson, K, Ahn, H (1997). A meta-analysis of outcome studies comparing bona fide psychotherapies: empirically, ‘All must have prizes’. Psychological Bulletin 122, 203215.CrossRefGoogle Scholar
Watzke, B, Rüddel, H, Jürgensen, R, Koch, U, Kriston, L, Grothgar, B, Schulz, H (2012). Longer term outcome of cognitive-behavioural and psychodynamic psychotherapy in routine mental health care: randomised controlled trial. Behaviour Research and Therapy 50, 580587.CrossRefGoogle Scholar
Woeller, W, Leichsenring, F, Leweke, F, Kruse, J (2012). Psychodynamic psychotherapy for posttraumatic stress disorder related to childhood abuse. Principles for a treatment manual. Bulletin of the Menninger Clinic 76, 6993.CrossRefGoogle Scholar
Yeaton, WH, Sechrest, L (1981). Critical dimensions in the choice and maintenance of successful treatments: strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology 49, 156167.CrossRefGoogle ScholarPubMed
Yeomans, F (2007). Questions concerning the randomized trial of schema-focused therapy vs transference-focused psychotherapy. Archives of General Psychiatry 64, 610611.CrossRefGoogle ScholarPubMed
Cited by

Save article to Kindle

To save this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the or variations. ‘’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Biases in research: risk factors for non-replicability in psychotherapy and pharmacotherapy research
Available formats

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

Biases in research: risk factors for non-replicability in psychotherapy and pharmacotherapy research
Available formats

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

Biases in research: risk factors for non-replicability in psychotherapy and pharmacotherapy research
Available formats

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *