Falk Leichsenring, Professor
15 August 2012

In a further comment to our meta-analysis [1] Kliem and colleagues reported "still some methodological discrepancies in the update of a meta-analysis" [2]. In research, however, the methods to be applied always depend on the question that is addressed. This question may be more or less narrow or wide. In psychotherapy research, especially in research on CBT, often narrow questions are addressed, e.g. "is treatment X effective in disorder Y"? However, under certain conditions, broad questions of research are more appropriate [3]. In our meta-analysis on LTPP, we examined such a broad question. We examined whether different forms of LTPP are superior to various other forms of psychotherapy in (different) complex mental disorders (personality disorders, chronic disorders, multiple disorders). This question serves as a good example for what is called "clinical diversity" or "clinical heterogeneity" [4, p. 276]. Thus,some heterogeneity between studies is to be expected. Furthermore, the aimof our meta-analysis was to make inferences beyond the observed sample of studies. For these reasons a random-effects model is more appropriate thena fixed-effects model [5]. Whereas the fixed effects model assumes that there is a true effect that is the same in every study, the random effectsmodel assumes that the effects assessed in the different studies are not identical, but follow some distribution [4, p. 280]. Thus, different questions of research are assessed: Whereas the fixed model addresses the question "what is the best estimate of the intervention effect?", the random effects model addresses the question "what is the average intervention effect?" [4, p. 281]. Applying a random effects model by the use of MetaWin 2.0 [6] we found a mean between-group effect size of d=0.54for overall outcome in favor of LTPP. For all random effects models, MetaWin yielded insignificant values for Q (p<0.09). For this reason, there was no need to exclude a study or to perform a sensitivity analysis.Kliem et al., however, question that the test for heterogeneity provided by MetaWin is appropriate [2]. As the statistical power for tests of heterogeneity is often low, especially if the number of studies is small, Takkouche et al. propose to use random effects models routinely and to quantify heterogeneity by means of Ri, i.e. the proportion of the total variance that is due to between-study variance [6]. Ri is assessed as Ri =tau-square/(tau-square + (S x var (beta))), with tau = between-study variance, S = number of studies and var(beta) = total variance. Small values of Ri (<0.40) indicate lack of heterogeneity [6,7]. To address the question of heterogeneity in our meta-analysis this way, we assessed Ri. According to the results, Ri was 0.09 for overall outcome, that is only 9% of the total variance was due to variance between studies. For target problems, symptoms, and personality functioning, Ri was low as wellwith values of 0.04, 0.08, and 0.08, respectively. Only for social functioning, the proportion of variance was somewhat higher but still in the range of low heterogeneity (Ri=0.24). According to these data, there is no indication of significant heterogeneity in effect sizes between the studies included in our random-effects meta-analysis. As a consequence, there is no need to exclude any study or to perform additional sensitivityanalysis. Furthermore, as noted above, some heterogeneity (maybe even including outliers) is in the nature of things when addressing a broad question of clinical diversity. The results of our meta-analysis stand as they were reported: LTPP is significantly superior to shorter forms of interventions in complex mental disorders [1].

1. Leichsenring, F Rabung, S. Long-term psychodynamic psychotherapy in complex mental disorders: update of meta-analysis. Brit J Psychiatry 2011; 199: 15-22.2. Kliem, S, Beller, J, Kroeger, C. Still some methodological discrepancies in the update of meta-analysis. Letter, Brit J Psychiatry, 2012. 3. Gotzsche PC. Why we need a broad perspective on meta-analysis. Brit MedJ 2000; 21: 585-6.4. Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, available from www.cochrane-handbook.org.4. Hedges LV, Vevea JL. Fixed- and random-effects models in meta-analysis.Psychol Meth 1998; 3: 486-504.5. Rosenberg MS, Adams DC, Gurevitch J. MetaWin. Statistical software for meta-analysis. Version 2.0. Sinauer Associates, 1999.6. Takkouche, B, Cadarso-Suarez, C, Spiegelmann, D. Evaluation of old and new tests of heterogeneity in epidemiologic meta-analysis. Am J Epidemiol 1999, 150, 206-215.7. Etminan M, Takkouche B, Isorna FC, Samii A. Risk of ischaemic stroke inpeople with migraine: systematic review and meta-analysis of observationalstudies. BMJ 2005, 330(7482):63.

... More

Conflict of interest: None declared

Soeren Kliem, Ph.D. Student
07 June 2012

In the latest issue of the British Journal of Psychiatry we pointed out some methodological discrepancies [1] regarding the meta-analysis "Long-term psychodynamic psychotherapy in complex mental disorders: updateof a meta-analysis" of Drs. Leichsenring and Rabung [2]. We want to thankDr. Leichsenring for replying to our objections [3]. Whereas some of our suggestions have been incorporated (see the correction), others have been refuted. First Leichsenring [3], for example, claimed that our re-analysis involvedthe calculation of a fixed effects model and that the discrepancies between their and our results were based on this modeling choice. We, however, did not calculate a fixed effects meta-analysis, as stated in ourletter, but rather used a random effects meta-analysis, as did Drs. Leichsenring and Rabung [2]. How then might the discrepancies be explained? One possible cause is that our re-analysis was conducted with the R package "metafor", but Drs. Leichsenring and Rabung instead utilized the MetaWin software. Thus we re-analyzed the meta-analysis this time using MetaWin and found that the "Qt-value" of MetaWin now approximately equals the "Qt-value" reported by Drs. Leichsenring and Rabung (pp. 19). The "Qt-value" reported by Drs. Leichsenring and Rabung as a test for heterogeneity, however, is not the standard test for heterogeneity, but adds the variance of the true effects (Tau^2) to the sample variances. This might correspond to a "test for heterogeneity in a random effects setting" but is essentially meaningless because it will not be significantin almost all practical situations. That is, the random effects model's raison detre is to explain all of the variance between effect sizes in therandom effects distribution (We thank Dr. Viechtbauer, the author of the Rpackage metafor, and Dr. Rosenberg, one of the authors of the MetaWin software for pointing this out to us; Personal Communications). Furthermore, it should always be examined whether the variance between studies might be explained by a mixed-effects analysis or by an outlier analysis. This is especially important if the usual test for heterogeneity("is the variance of the random effects distribution larger than zero?") is significant, as in our re-analysis [1]. Just because one uses a random effects model and therefor implicitly states that the variation is just random error doesn't make it so; the variation could also be due to moderator variables or outlying studies. We examined just that and found that one study was an outlier. In order to make our results reproducible we uploaded the dataset and our commented R script to the website "https://www.tu-braunschweig.de/Medien-DB/psychologie/e-letter/letter.pdf". In this script we further show that we could reproduce Drs. Leichsenring's and Rabung's [2] results if we added the variance of the true effects (Tau^2) to the variance usually analyzed by the test for heterogeneity (note that some results may differ slightly due to rounding errors). The output of our analysis with the MetaWin program is also included.

To substantiate our claim that one study is indeed an outlier we additionally used the influence function in the metafor package to calculate numerous case diagnostics indicating the influence of one study at a time on the model fit. The results indicated the respective study [4]to be clearly an outlier. The externally studentized residual of this study, for example, was 3.33 suggesting this study within a sample size of10 studies to be an outlier. The Funnel-Plot caclulated with metafor strongly seconded this result, for both see the attached R-code.If Drs. Leichsenring and Rabung [2] used the "random effects test for heterogeneity" for the other models as well and did not examine whether a mixed-effects model or study outliers might account for the variance, it must be assumed that the other results are also potentially faulty (pp.19). Therefore, all analyses should be re-calculated, and if necessary, corrected. The corrections of Dr. Leichsenring should, preferably, also be published with reproducible code.

Second, we would like to clarify our Bayesian analysis. Dr. Leichsenring [3] correctly writes: "Results of Bayesian meta-analyses depend largely on the specification of prior assumptions on the treatment effect and between-trial variance. Since Kliem et al did not provide any information about the assumptions of their analyses, it is impossible to interpret the presented result reasonably" (pp. 430). We used vague priorswhich minimize the impact of the prior information on the results, as is the standard convention in Bayesian analysis. Concretely we employed a uniform prior distribution for Tau(>0) and N(0,1000) prior distributionfor My. The main usefulness of choosing these priors is that the results of the Bayesian analysis become comparable with classical analyses. Consequently the Bayesian results replicated our other results obtained with more classical meta-analytic techniques. The usefulness of the Bayesian approach lays in the fact that direct probability statements can be made about the possible magnitude of the effect size.

Lastly, we nonetheless would like to thank Dr. Leichsenring for his effort in discussing the meta-analysis. In our view, the scientific foundation of long-term psychodynamic psychotherapy is an important research area. With their meta-analysis Drs. Leichsenring and Rabung [2] enabled important scientific discourse as we with our letters currently conduct.

Literature:

1. Kliem S, Beller J, Kroeger C. Methodological discrepancies in the update of a meta-analysis. Br J Psychiatry 2012; 200: 429.2. Leichsenring F, Rabung S. Long-term psychodynamic psychotherapy in complex mental disorders: update of a meta-analysis. Br J Psychiatry 2011;199: 15-22.3. Leichsenring F. Methodological discrepancies in the update of a meta-analysis. Author?s reply. Br J Psychiatry 2012; 200: 429-430.4. Bateman A, Fonagy P. The effectiveness of partial hospitalization in the treatment of borderline personality disorder, a randomized controlled trial. Am J Psychiatry 1999; 156: 1563-9.

Attachments: https://www.tu-braunschweig.de/Medien-DB/psychologie/e-letter/letter.pdf

... More

Conflict of interest: None declared

Soeren Kliem, Ph.D. Student
07 March 2012

Drs Leichsenring and Rabung1 reported that long-term psychodynamic psychotherapy (LTPP) is superior to less intensive forms of psychotherapy in complex mental disorders. Based on 10 trials, they found an overall effectiveness of ES = 0.55. In the aforementioned study, we found several methodological discrepancies:

First, it seems surprising that the Q-test indicated no significant unexplained variance, as the between group effect size of one of the primary studies2 (ES = 1.76) is quite outstanding in figure 2. To shed light on this issue, we recalculated the overall effect size using a random effects meta-analysis based on the values from figure 2. Our meta-analysis replicated Leichsenring and Rabung1 in the main parts. In contrast to Leichsenring and Rabung, however, we found significant unexplained variance (Q = 25.33, df = 9, p = 0.003) and a larger overall confidence interval of 0.29-0.82 (in contrast to 0.41-0.67 as reported by Leichsenring and Rabung). Additionally, computing an outlier analysis a significantly outlying study effect2 size was found (p < .001). Including the moderator considering the impact of this study yields an effect size of ES = 0.44 (95% CI 0.27-0.61, p < .001). The moderator effect, interpreted as the difference between the effect of the outlying study and the grand mean, was 1.32 (95% CI 0.57 to 2.07, p < .001). After removing the outlying study there was no significant unexplained variance (Q = 11.56, df = 8, p = .172).

Second, we calculated the fail safe-N according to Rosenthal3; 16 non-published studies with ES = 0 had to be included in the analysis to change the results of the meta-analysis (ES = 0.44) from significant to non-significant (ES < .16). As 16 is below 55 (5K + 10), the effect cannot be regarded as robust.

Lastly, to gain better insight into the interpretation of the overalleffect size as small, medium or large, we calculated a bayesian meta-analysis following Higgins, Thompson and Spiegelhalter4. The bayesian analysis essentially replicated the findings of our random effects meta-analysis. Additionally we found the probability, of the overall effect size to be small (ES < .5), at 72.5%. Thus, in contrast to Leichsenringand Rabung1, we found that the overall effect size may only be regarded as small in contrast to medium or large.

Therefore, we would greatly appreciate caution against a conclusion that the overall effectiveness of LTPP for treating complex mental disorders should now be considered as definitely proven.

1. Leichsenring F, Rabung S. Long-term psychodynamic psychotherapy incomplex mental disorders: update of a meta-analysis. Br J Psychiatry 2011;199: 15-222. Bateman A, Fonagy P. The effectiveness of partial hospitalization in the treatment of borderline personality disorder, a randomized controlled trial. Am J Psychiatry 1999; 156: 1563-9. 3. Rosenthal R. The file drawer problem and tolerance for null results. Psychol Bull 1979; 86: 638-41.

4. Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. J R Statist Soc A 2009; 172:137-59.

... More

Conflict of interest: None declared

Falk Leichsenring, Professor of Psychotherapy Research
09 February 2012

Coyne [1] criticizes our recent meta-analysis of long-term psychodynamic psychotherapy [2]. However, Coyne just reiterates the arguments he or other critics already put forward against our 2008 meta-analysis [3] two years ago [4, 5]. We repeatedly responded to this criticism in detail [2, 6, 7]. Coyne is fully aware of this, but neither acknowledges nor even refers to our prior response to criticism. We showedin detail, for example, how we calculated effect sizes and that statistical power was not lower than in many studies of CBT [2, 6, 7].

The purpose of our update [2] was to take several criticismsput forward against our 2008 meta-analysis into account. In distinction tothe 2008 meta-analysis, we (1) only included controlled trials, (2) exclusively calculated between-group effect sizes and (3) included only active comparators as comparison conditions. Coyne fails to mention these essential differences compared to the 2008 meta-analysis. Instead of this he incorrectly claims that we updated our meta-analysis by only adding "one study" [1].

It is a pity that Coyne did not report our prior response tothese same criticisms that appeared 2 years ago. Thus, readers again are left to decide from Coyne's side of the argument without other available information. Coyne's exaggerated claims against the efficacy of LTPP are repeated in two essentially redundant criticisms in two journals two yearsapart [1, 4].

An earlier unedited version of his letter was unfortunately published on 12 January and subsequently retracted by the publishers [8]. In this earlier version, Coyne spread false information about the activities of the senior author, Dr Leichsenring. No public-domain evidence for these assertions was supplied, but as Coyne spread this falseinformation for several weeks, his assertions require a comment. Coyne stated:

"Leichsenring serves as representative for the interests of long-term psychodynamic therapy on the Wissenschaftlicher Beirat Psychotherapie (WBP), a body whose judgments about the scientific status of psychotherapies are accepted by German insurance companies as the basisfor reimbursement. Currently, LTPP does not receive reimbursement, based on WBP's finding of a lack of evidence for the added expense of longer-term treatment.... It is a pity that Leichsenring and Rabung did not report Leichsenring's activities so that readers could decide for themselves whether this agenda might have influenced Leichsenring and Rabung's exaggerated claims for the efficacy of LTPP that are repeated in two essentially redundant reviews of poor quality studies."

Dr Leichsenring is indeed a member of the Wissenschaftliche Beirat Psychotherapie (WBP). The members of the WBP are independent experts in psychotherapy research. They are not paid for working with the WBP. The WBP is the official organization of Germany that evaluates the scientific status of psychotherapeutic methods by the use of detailed methodological criteria (www.wbp.de). Based on this evaluation, the WBP gives a recommendation whether or not a method of psychotherapy can be recommended for training in psychotherapy. The WBP, however, gives no statements referring to reimbursement.

For this reason, the following statements by Coyne are simply and categorically false.

a)It is not true that Dr Leichsenring is "representative of the interests of long-term psychodynamic psychotherapy on the WBP". Like all members of the WBP, Dr Leichsenring is an independent expert in psychotherapy research.

b) It is false that the WBP's judgments "are accepted by German insurance companies as the basis for reimbursement." Whether or nota treatment is reimbursed by insurance companies is based on decisions of another autonomous organization, the "Gemeinsame Bundesausschuss" (GBA; www.g-ba.de).

c)And last, but not least, it is false that "Currently, LTPP does not receive reimbursement, based on the WBP's finding of a lack of evidence for the added expense of longer-term treatment." In fact, LTPPas well as long-term CBT (!) do receive reimbursement by German insurance companies and the WBP has acknowledged (both short-term and long-term) psychodynamic psychotherapy as scientific and efficacious and recommended it for training (www.wbp.de).

1.Coyne, J., Flawed meta-analysis of long-term psychodynamic therapy in jama reappers in bjp with one study added. British Journal of Psychiatry 2012, eLetter.

2.Leichsenring, F., Rabung, S., Long-term psychodynamic psychotherapy in complex mental disorders: update of meta-analysis. British Journal of Psychiatry 2011. 199 p. 15-22.

3.Leichsenring, F., Rabung, S, The effectiveness of long-term psychodynamic psychotherapy: a meta-analysis Journal of the AmericanMedical Association (JAMA) 2008. 300 p. 1551-1564.

4.Bhar, S.S., et al., Is longer-term psychodynamic psychotherapy more effective than shorter-term therapies? Review and critique of the evidence. Psychotherapy and Psychosomatics, 2010. 79: p. 208-216.5.Thombs, B.D., M. Bassel, and L.R. Jewett, Analyzing effectiveness of long-term psychodynamic psychotherapy. Letter to the Editor. JAMA, 2009 301: p. 930.

6.Leichsenring, F. and S. Rabung, The effectiveness of long-term psychodynamic psychotherapy - reply. Journal of the American Medical Association, 2009. 301: p. 932-933.

7.Leichsenring, F. and S. Rabung, Double standards in psychotherapy research. Psychotherapy and Psychosomatics, 2011. 80: p. 48-51.8. British Journal of Psychiatry. Re: Coyne, J., Flawed meta-analysis of long-term psychodynamic therapy in jama reappers in bjp with one study added. British Journal of Psychiatry 2012. eLetter.

... More

Conflict of interest: None declared

British Journal of Psychiatry British Journal of Psychiatry
09 February 2012

A longer version of this letter (Coyne, 'Flawed meta-anaysis...') was erroneously published online on 12 January 2012 and was withdrawn by the publishers on 6 February 2012.

Conflict of interest: None declared