Hostname: page-component-f7d5f74f5-g2fc4 Total loading time: 0 Render date: 2023-10-02T19:31:42.244Z Has data issue: false Feature Flags: { "corePageComponentGetUserInfoFromSharedSession": true, "coreDisableEcommerce": false, "coreDisableSocialShare": false, "coreDisableEcommerceForArticlePurchase": false, "coreDisableEcommerceForBookPurchase": false, "coreDisableEcommerceForElementPurchase": false, "coreUseNewShare": true, "useRatesEcommerce": true } hasContentIssue false

Problem-solving therapy for adult depression: An updated meta-analysis

Published online by Cambridge University Press:  01 January 2020

Pim Cuijpers*
aDepartment of Clinical, Neuro and Developmental Psychology, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, The Netherlands
Leonore de Wit
aDepartment of Clinical, Neuro and Developmental Psychology, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, The Netherlands
Annet Kleiboer
aDepartment of Clinical, Neuro and Developmental Psychology, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, The Netherlands
Eirini Karyotaki
aDepartment of Clinical, Neuro and Developmental Psychology, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, The Netherlands
David D. Ebert
bClinical Psychology and Psychotherapy, Institute for Psychology, Friedrich-Alexander University Erlangen-Nuremberg, Erlangen, Germany
*Corresponding author at: Department of Clinical, Neuro and Developmental Psychology, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, Van der Boechorststraat 1, 1081 BT Amsterdam, The Netherlands. E-mail address: (P. Cuijpers).



Problem-solving therapy (PST) is one of the best examined types of psychotherapy for adult depression. No recent meta-analysis has examined the effects of PST compared to control groups or to other treatments. We wanted to verify whether PST is effective, whether effects are comparable to those of other treatments, and whether we could identify the possible sources of high heterogeneity that was found in earlier meta-analyses.


We conducted systematic searches in bibliographical databases, including PubMed, PsycInfo, Embase and the Cochrane database of randomized trials.


We included 30 randomized controlled trials on PST (with 3530 patients), in which PST was compared to control conditions, with other therapies, and with pharmacotherapy. We could compare these 30 trials on PST also with 259 trials on other psychotherapies for adult depression. The effect size of PST versus control groups was g = 0.79 (0.57–1.01) with very high heterogeneity (I2 = 84; 95% CI: 77–88). The effect size from the 9 studies with low risk of bias was g = 0.34 (95% CI: 0.22–0.46) with low heterogeneity (I2 = 32; 95% CI: 0–68), which is comparable to the effects of other psychotherapies. PST was a little more effective than other therapies in direct comparisons, but that may be explained by the considerable number of studies with researcher allegiance towards PST. In meta-regression analyses of all controlled studies, no significant difference between PST and other therapies was found.


PST is probably an effective treatment for depression, with effect sizes that are small, but comparable to those found for other psychological treatments of depression.

Copyright © European Psychiatric Association 2018

1 Introduction

In the past decades dozens of psychological interventions for the treatment of depression have been developed. However, only a relatively small number of these treatments have been tested in ten or more randomized trials. The therapies that have been well-examined include cognitive behavior therapy [Reference Cuijpers, Sijbrandij, Koole, Andersson, Beekman and Reynolds1, Reference Churchill, Hunot, Corney, Knapp, McGuire and Tylee2], interpersonal psychotherapy [Reference Cuijpers, Geraedts, van Oppen, Andersson, Markowitz and van Straten3, Reference Cuijpers, Donker, Weissman, Ravitz and Cristea4], behavioral activation [Reference Cuijpers, van Straten and Warmerdam5, Reference Ekers, Richards and Gilbody6], brief psychodynamic therapy [Reference Driessen, Hegelmaier, Abbass, Barber, Dekker and Van7, Reference Leichsenring and Rabung8] and non-directive counseling Reference Cuijpers, Driessen, Hollon, van Oppen, Barth and Andersson[9]. When considering the several hundreds of randomized controlled and comparative trials that have examined psychotherapies for depression, it has become clear that these therapies have modest but clinically relevant effects Reference Cuijpers, Turner, Koole, van Dijke and Smit[10]. Furthermore, this area of research has shown that there are no major differences between the effects of these treatments Reference Barth, Munder, Gerger, Nuesch, Trelle and Znoj[11], that these treatments can be delivered effectively in individual, group, guided self-help and internet-delivered format [Reference Cuijpers, van Straten and Warmerdam12Reference Cuijpers, Donker, Johansson, Mohr, van Straten and Andersson14], and that they are effective in several different target groups, such as older adults [Reference Cuijpers, van Straten, Smit and Andersson15, Reference Cuijpers, van Straten and Smit16], children and adolescents Reference Weisz, Kuppens, Eckshtain, Ugueto, Hawley and Jensen-Doss[17], postpartum depression Reference Cuijpers, Brannmark and van Straten[18], college students Reference Cuijpers, Cristea, Elbert, Koot, Auerbach and Bruffaerts[19], and patients with comorbid general medical disorders Reference van Straten, Geraedts, Verdonck-de Leeuw, Andersson and Cuijpers[20].

One of the interventions that has been tested as a treatment of adult depression in a considerable number of randomized trials is problem-solving therapy (PST). PST was developed in the 1970s as one of the first treatments of depression Reference D’zurilla and Goldfried[21], and was first tested in the 1980s [Reference Nezu22, Reference Nezu and Perri23]. PST focuses on training in adaptive problem-solving attitudes and skills and is aimed at reducing and preventing psychopathology, and at enhancing positive well-being by helping individuals cope more effectively with stressful problems in daily life Reference Bell and D’Zurilla[24].

A decade ago we conducted a meta-analysis of trials testing the effects of PST for adult depression Reference Cuijpers, van Straten and Warmerdam[25]. The 13 trials that were included in this meta-analysis pointed at a large effect of PST versus control groups (d = 0.83), and some indications were found that PST was more effective than other therapies, although that was based on a small set of studies. Since this meta-analysis, several other meta-analyses have been published [Reference Bell and D’Zurilla24, Reference Malouff, Thorsteinsson and Schutte26, Reference Kirkham, Choi and Seitz27]. However, the last comprehensive meta-analysis aimed at including all trials on PST for depression was published in 2009 Reference Bell and D’Zurilla[24]. One meta-analysis was published later Reference Kirkham, Choi and Seitz[27], but that was only aimed at studies among older adults. Because the number of trials on PST has more than doubled since then, a new meta-analysis is much needed. Furthermore, none of the comprehensive meta-analyses examined the effects of the quality of the included trials on the outcomes, nor was publication bias well examined.

Another reason why a new meta-analysis of PST is useful, is that in our first meta-analysis a very high level of heterogeneity was found, indicating that the effect sizes varied strongly across studies Reference Cuijpers, van Straten and Warmerdam[25]. It was not possible to identify possible causes of this heterogeneity in subgroup and meta-regression analyses. It is important, however, to know under which conditions PST has large or small effects. Because the number of trials is now considerably larger than 10 years ago, it may be possible to better identify possible causes of heterogeneity.

We conducted a new meta-analysis of PST for adult depression. We wanted to examine the overall effects of PST, the level of heterogeneity and to conduct subgroup and meta-regression analyses to examine potential causes of this heterogeneity. Furthermore, we wanted to examine the relative effects of PST compared with other psychological therapies. We wanted to examine this from trials directly comparing PST with other therapies, but also by comparing the effects found in trials comparing PST with control groups, with the effects found for other therapies that are compared with control groups.

2 Methods

2.1 Identification and selection of studies

We used an existing database of studies on the psychological treatment of depression. This database has been described in detail elsewhere Reference Cuijpers, van Straten, Warmerdam and Andersson[28], and has been used in a series of earlier published meta-analyses Reference Cuijpers[29]. For this database we searched four major bibliographical databases (PubMed, PsycInfo, Embase and the Cochrane Library) by combining terms (both index terms and text words) indicative of depression and psychotherapies, with filters for randomized controlled trials. The full search string for one database (PubMed) is given in Appendix A in Supplementary material. We also searched a number of bibliographical databases to identify trials in non-Western countries Reference Cuijpers, Karyotaki, Reijnders, Purgato and Barbui[30], because the number of trials on psychological treatments in these countries is growing rapidly. Furthermore, we checked the references of earlier meta-analyses on psychological treatments of depression. The database is continuously updated and was developed through a comprehensive literature search (from 1966 to January 1st, 2017). All records were screened by two independent researchers and all papers that could possibly meet inclusion criteria according to one of the researchers were retrieved as full-text. The decision to include or exclude a study in the database was also done by the two independent researchers, and disagreements were solved through discussion.

We included studies that were: (a) a randomized trial (b) in which PST (c) for adult depression was (d) compared with a control group (waiting list, care-as-usual, placebo, other inactive treatment) or another treatment (psychological or pharmacological). Depression could be established with a diagnostic interview or with a score above a cut-off on a self-report measure. Co-morbid mental or somatic disorders were not used as an exclusion criterion. Studies on inpatients were excluded. We also excluded maintenance studies, aimed at people who had already recovered or partly recovered after an earlier treatment. We considered an intervention to be PST when problem-solving was the core element of the intervention that was meant to reduce depression. Other techniques were allowed when they were aimed at supporting or strengthening the problem-solving element. If other techniques were clearly considered to be separate elements, we did not consider it PST.

In addition to the main analyses in which we focused on the studies on PST, we also wanted to compare the effects of PST with the effects found for other psychological treatments of adult depression. For this comparison we selected trials from our database in which other types of psychotherapy for depression were compared with a control condition, with the same in- and exclusion criteria as for the studies on PST.

2.2 Quality assessment and data extraction

We assessed the validity of included studies using four criteria of the ‘Risk of bias’ assessment tool, developed by the Cochrane Collaboration Reference Higgins, Altman, Gøtzsche, Jüni, Moher and Oxman[31]. This tool assesses possible sources of bias in randomized trials, including the adequate generation of allocation sequence; the concealment of allocation to conditions; the prevention of knowledge of the allocated intervention (masking of assessors); and dealing with incomplete outcome data (this was assessed as positive when intention-to-treat analyses were conducted, meaning that all randomized patients were included in the analyses). Assessment of the validity of the included studies was conducted by two independent researchers, and disagreements were solved through discussion.

We also coded participant characteristics (depressive disorder of scoring high on a self-rating scale; recruitment method; target group); characteristics of the psychotherapies (treatment format; number of sessions); and general characteristics of the studies (type of control group; country where the study was conducted). Treatment format was coded as individual, group or guided-self help (including internet-based guided self-help).

We distinguished three types of PST Reference Cuijpers, van Straten and Warmerdam[25]: (1) Extended PST, which does not only focus on the problem-solving skills themselves, but also on changing those attitudes or beliefs that may inhibit or interfere with attempts to engage in the remaining problem-solving tasks. It is typically conducted in a group format of 10 or more sessions. We (arguably) considered an intervention as extended PST when it had 10 sessions or more. (2) Brief PST, which was originally developed for primary care (PST-PC), focuses on the core elements of problem-solving and can be used by trained nurses. We considered an intervention as brief PST when it had 9 sessions or less. (3) Self-examination therapy (SET) is aimed at determining the major goals in their life, investing energy only in those problems that are related to what matters and learning to accept those situations that cannot be changed. Problem-solving skills are the core element of this approach. SET is typically used in a guided-self-help format. We considered an intervention as SET when it was based on self-examination therapy Reference Bowman, Ward, Bowman and Scogin[32] and was conducted in guided self-help format.

In the studies comparing PST with other therapies we also examined researcher allegiance, using the methods we have described before Reference Cuijpers, Driessen, Hollon, van Oppen, Barth and Andersson[9]. We coded that researcher allegiance was in favor of PST (against the alternative therapy) if: (1) PST was the only therapy mentioned in the title; (2) PST was explicitly mentioned as the main experimental intervention in the introduction section of the study; (3) the alternative therapy was explicitly described as a control condition included to control for the non-specific components of the comparison therapy; or (4) there was an explicit hypothesis that PST was expected to be more effective than the alternative therapy.

2.3 Outcome measures

For each comparison between PST (or another type of psychotherapy) and a control condition, the effect size indicating the difference between the two groups at post-test was calculated (Hedges’ g). Effect sizes of 0.8 can be assumed to be large, while effect sizes of 0.5 are moderate, and effect sizes of 0.2 are small Reference Higgins, Thompson, Deeks and Altman[33]. Effect sizes were calculated by subtracting (at post-test) the average score of the psychotherapy group from the average score of the control group, and dividing the result by the pooled standard deviation. Because some studies had relatively small sample sizes we corrected the effect size for small sample bias Reference Duval and Tweedie[34]. If means and standard deviations were not reported, we used the procedures of the Comprehensive Meta-Analysis software (see below) to calculate the effect size using dichotomous outcomes; and if these were not available either, we used other statistics (such a t-value or p-value) to calculate the effect size.

In order to calculate effect sizes we used all measures examining depressive symptoms (such as the Beck Depression Inventory/BDI Reference Beck, Ward, Mendelson, Mock and Erbaugh[35]; the BDI-II Reference Beck, Steer and Brown[36]; or the Hamilton Rating Scale for Depression/HAMD-17 Reference Hamilton[37]).

2.4 Meta-analyses

To calculate pooled mean effect sizes, we used the computer program Comprehensive Meta-Analysis (version 3.3070; CMA). Because we expected considerable heterogeneity among the studies, we employed a random effects pooling model in all analyses.

Numbers-needed-to-be-treated (NNT) were calculated using the formulae provided by Furukawa Reference Furukawa[38], in which the control group’s event rate was set at a conservative 19% (based on the pooled response rate of 50% reduction of symptoms across trials in psychotherapy for depression) Reference Cuijpers, Karyotaki, Weitz, Andersson, Hollon and van Straten[39]. As a test of homogeneity of effect sizes, we calculated the I2-statistic, which is an indicator of heterogeneity in percentages. A value of 0% indicates no observed heterogeneity, and larger values indicate increasing heterogeneity, with 25% as low, 50% as moderate, and 75% as high heterogeneity Reference Higgins, Thompson, Deeks and Altman[33]. We calculated 95% confidence intervals around I2 Reference Ioannidis, Patsopoulos and Evangelou[40], using the non-central chi-squared-based approach within the heterogi module for Stata Reference Orsini, Bottai, Higgins and Buchan[41]. We conducted sensitivity analyses excluding potential outliers. These were defined as studies of which the 95% CI of the effect size did not overlap with the 95% CI of the pooled effect size.

We conducted subgroup analyses according to the mixed effects model, in which studies within subgroups are pooled with the random effects model, while tests for significant differences between subgroups are conducted with the fixed effects model. For continuous variables, we used meta-regression analyses to test whether there was a significant relationship between the continuous variable and effect size, as indicated by a Z-value and an associated p-value. Multivariate meta-regression analyses, with the effect size as the dependent variable, were conducted in CMA.

We tested for publication bias by inspecting the funnel plot on primary outcome measures and by Duval and Tweedie’s trim and fill procedure Reference Duval and Tweedie[34], which yields an estimate of the effect size after the publication bias has been taken into account (as implemented in CMA). We also conducted Egger’s test of the intercept to quantify the bias captured by the funnel plot and to test whether it was significant.

3 Results

3.1 Selection and inclusion of studies

After examining a total of 18,500 abstracts (14,290 after removal of duplicates), we retrieved 2,092 full-text papers for further consideration. We excluded 1,827 of the retrieved papers. The PRISMA flowchart describing the inclusion process, including the reasons for exclusion, is presented in Fig. 1. A total of 30 studies on PST met inclusion criteria for this meta-analysis (Table 1).

Another 264 studies (with 342 comparisons between a treatment and a control group) on psychological treatments other than PST were included (for the comparison of effect sizes of PST with those of other psychological treatments). This makes a total of 290 studies that were included in the analyses.

3.2 Characteristics of included studies

Selected characteristics of the included studies are presented in Table 1. In the 30 included studies on PST (with 12 comparisons between PST and another therapy condition, and 27 comparisons between PST and a control condition), a total of 3,530 patients participated (PST conditions = 1,576, other therapy conditions = 542, control conditions = 1,412). Participants were recruited through: (a) announcements in local newspapers and other media (12 studies), (b) referrals from health services (eight studies), (c) other recruitment strategies (such as screening at general medical services; nine studies), and one study did not report the recruitment strategy. In total, 14 studies were targeted at adults, eight were targeted at elderly adults, two were targeted at ethnic minorities, four were targeted at adults with a general medical disorder, one at adult caregivers, and one was targeted at women with postpartum depression. Brief PST was used in 15 studies, Extended PST was used in 10 studies, and Self-Examination PST was used in five studies).

In three of the comparisons between PST and another therapy condition, cognitive behavior therapy was used as the comparator, two used behavioural activation therapy, one used case management, three used non-directive supportive therapy, one used referral, one used reminiscence, one used social skills training, and one used stress management. Seven studies used a group treatment format, 16 studies utilized individual treatment, five used a guided self-help treatment format, and two studies used another format. The number of treatment sessions ranged from four to 12. For the control group, 12 studies used a waiting list, seven studies used care-as-usual, five studies used antidepressant medication, three studies used placebo, and one study used another control group. 18 studies were conducted in North America, 11 were conducted in Europe, and one was conducted in Africa.

Selected characteristics of the 264 studies on psychotherapies other than PST (with 342 comparisons between treatment and control groups) are presented in Appendix B in Supplementary material and the references for these studies are given in Appendix C in Supplementary material.

3.3 Risk of bias

The risk of bias in most of the studies on PST was considerable. A total of 14 of the 30 studies reported an adequate sequence generation (47%). Thirteen studies reported allocation to conditions by an independent (third) party (43%). Ten studies reported using blinded outcome assessors (33%), and 16 used only self-report outcomes. In 16 studies intent-to-treat analyses (completeness of follow-up data) were conducted (53%). Only seven studies (23%) met all quality criteria. 12 studies (40%) met two or three of the criteria and the 11 remaining studies met no or only one criterion.

Fig. 1 Flowchart for the inclusion of studies.

3.4 Effects of PST compared with control groups

We could compare the effect of PST with control groups in 27 comparisons. The overall effect size was g = 0.79 (95% CI: 0.57–1.01), which corresponds with a NNT of 3.64. Heterogeneity was very high (I2 = 84; 95% CI: 77–88). The results of these analyses are summarized in Table 2. The effect sizes for each study are presented in the forest plot in Fig.2.

In three studies ([Reference Nezu22, Reference Nezu and Perri23, Reference Choi, Marti, Bruce and Hegel42]; see Table 1), two types of therapies that met our definition of PST were compared with the same control group. Because these effect sizes were not independent of each other, they may have artificially reduced heterogeneity and influenced the effect sizes. We conducted two separate analyses to examine this. In the first analysis we only included the largest effect size from each study, and in the second only the smallest effect size. As can be seen in Table 2, the effect sizes and levels of heterogeneity were comparable to those in the main analyses. When we limited the effect sizes to those based on the HAM-D and the BDI (the most used instruments), we found that the effect sizes were larger than those found in the overall analyses, especially for the HAM-D. Levels of heterogeneity remained comparable.

We found strong indications for publication bias (Fig. 3). Duvall and Tweedie’s trim and fill procedure suggested that 10 studies were missing and after imputation of these studies the effect size dropped from g = 0.79 to g = 0.39 (95% CI: 0.14–0.64). Egger’s test of the intercept also pointed at considerable publication bias (intercept: 3.32; 95% CI: 1.46–5.17; p < 0.001).

In order to generate the best estimate of the true effect size, we selected the studies with low risk of bias (9 comparisons; g = 0.34; 95% CI: 0.22–0.46; NNT = 9.49; I2 = 32; 95% CI: 0–68), and adjusted that for publication bias. The resulting effect size was g = 0.28; 95% CI: 0.14–0.42; number of imputed studies: 2), which corresponds with a NNT of 11.77.

Table 1 Selected characteristics of studies examining problem solving therapy for adult depression (N = 3.357).

Note: Abbr.: Abbreviated; ADM: Anti-depressant Medication; BA: Behavioral Activation; Brf: Brief; CAU: Care as Usual; CBT: Cognitive Behavioral Therapy; Clin: participants recruited in a clinical setting; Comm: participants recruited in a community setting; CT: Cognitive Therapy; Diagn: Diagnosis; EU: Europe; Ext: Extended; ftf: Face-to-face; GP: General Practitioner; Grp: group format; Gsh: guided self-help format; Ind: individual format; MDD: Major Depressive Disorder; Med. Dis.: Medical Disorder; Nsess: Number of sessions; Pf: Problem-Focused; PPD; Post-Partum Depression; Recr: Recruitment; RoB: Risk of Bias; SET: Self-Examination Therapy; SST: Social Skills Training; SUP: Nondirective Supportive Therapy; tel: Telephone; WL: Waitlist.

a In this column a positive (+) or negative (−) sign is given for four quality criteria of the study, respectively: allocation sequence; concealment of allocation to conditions; blinding of assessors; and intention-to-treat analyses. Sr in the third criterion indicates that only self-report measures (and no assessor) were used.

Fig. 2 Forrest plot of effect sizes from randomized controlled trials on problem-solving therapy for adult depression.

Only three studies (4 comparisons) examined the effects of PST versus control groups at the longer term, one (with two comparisons) at 6 months and two at 12 months follow-up. However, only one of these studies had low risk of bias. The pooled effect size was g = 0.35 (95% CI: 0.14–0.56; I2 = 18; 95% CI: 0–73; NNT = 9.19).

3.5 Subgroup and meta-regression analyses

In order to examine possible sources of heterogeneity, we conducted a series of subgroup analyses (Table 2). We found several significant differences between subgroups. Studies in which patients were recruited from the community resulted in significantly higher effect sizes than studies among patients from clinical samples and patients that were recruited in other ways (p = 0.001). Group treatments resulted in higher effect sizes than individual, and guided self-help treatments (p = 0.002). Studies conducted in North America had considerably higher effect sizes than studies from Europe (p = 0.03). Studies with a waiting list control condition had higher effect sizes than studies with a care-as-usual or another control group (p = 0.004), and studies with low risk of bias had lower effect sizes than studies with high risk of bias (p < 0.001). We found no indication that target group, diagnosis or type of PST were significantly related to the effect size.

Because there were several studies with very high effect sizes, which may have distorted these subgroup analyses, we repeated the analyses while excluding studies with effect sizes larger than g = 1.5. We found that the results were overall very comparable, and all significant subgroup analyses remained significant, except for the difference between effect sizes in North-American and Europe.

We also conducted a multivariate regression analysis (with the effect size as dependent variable) in which all variables of the subgroup analyses were simultaneously entered as predictors. All variables were entered as categorical variables, except risk of bias that was entered as a continuous variable. We also added the number of treatment sessions as continuous variable. In these analyses none of the predictors remained significantly associated with the effect size.

Fig. 3 Funnel plot of standard error by Hedges g.

In order to avoid overfit of the meta-regression model, we repeated this meta-regression analysis, with a (manual) stepwise backward elimination of the least significant predictor until only significant predictors remained in the model. This analysis resulted in only one significant predictor, namely risk of bias (coefficient: −0.26, SE: 0.09, p = 0.008). Higher risk of bias resulted in larger effect sizes.

3.6 Direct comparisons of PST versus other treatments

We could compare PST with another type of psychotherapy in 12 comparisons. We found a significant effect size indicating superior effects of PST compared with other therapies at post-test, with g = 0.18 (95% CI: 0.01–0.35; NNT = 19.00). Heterogeneity was low (I2 = 27; 95% CI: 0–63).

Because the number of studies was so small we only conducted two subgroup analyses. In the first subgroup analyses we examined whether the effects in studies with high risk of bias were different from those with low risk of bias. The effect sizes in the two groups of studies did not differ significantly, possibly because of low statistical power related to the small number of studies with low risk of bias. Both subgroups resulted in a small, non-significant differential effect size.

In the second subgroup analysis we compared studies with researcher allegiance to those without. The difference between the subgroups was not significant, but the five studies without allegiance did not result in a significant difference between PST and other therapies.

We could compare PST with antidepressant medication in 6 comparisons. We found no significant difference between the two (g = 0.09; 95% CI: −0.13–0.30). Because of the small number of studies we did not conduct further analyses.

3.7 Controlled trials on PST versus controlled trials on other psychotherapies

We compared the 27 comparisons between PST and a control condition to the 342 comparisons of other psychotherapies versus control conditions, and found no significant difference (other therapies: g = 0.69; 95% CI: 0.64–0.74; I2 = 75; 95% CI: 72–78; NNT = 4.25; p for difference with PST: 0.50).

Then we conducted a multivariate meta-regression analysis in which we included type of therapy as well as other characteristics of the participants, the therapies and the studies as predictor in the model. The results are presented in Table 3 (full model). As can be seen, there was no indication that the effect size found for PST differed from the effect sizes of other therapies after adjustment for the other characteristics of the participants, therapies and studies.

We conducted again another multivariate metaregression analysis with a (manual) stepwise backward elimination of the least significant predictor until only significant predictors remained in the model (except type of therapy that was not removed from the model, because this was the variable of interest). The same variables that were significant in the full model were also significant in the parsimonious model, and there was no indication that PST differed significantly from the other types of therapy.

4 Discussion

In this updated meta-analysis of PST for adult depression we found an overall large effect of PST compared to control conditions. However, a better estimate of the true effect size, based on studies with low risk of bias and after adjustment for publication bias pointed at a small effect of PST. This is in line with research on other treatments of depression, such as cognitive behavior therapy Reference Cuijpers, Sijbrandij, Koole, Andersson, Beekman and Reynolds[1] and interpersonal psychotherapy Reference Cuijpers, Donker, Weissman, Ravitz and Cristea[4], where overall large effects are found but much smaller ones in high-quality studies and after adjustment for publication bias [Reference Cuijpers, van Straten, Bohlmeijer, Hollon and Andersson43, Reference Cuijpers, Smit, Bohlmeijer, Hollon and Andersson44].

Direct comparisons of PST with other types of psychotherapy showed a small significant effect in favor of PST. However, in a considerable number of studies (more than half of the 11 trials) we found indications for researcher allegiance in favor of PST. These findings should therefore be considered with caution. A comparison of trials on PST versus control groups and the large number of trials on other therapies versus control groups, also indicated no significant difference between PST and other therapies. Furthermore, no differential effect of any of the other types of therapy was found, suggesting that all therapies are equally or about equally effective. This is also in line with several previous large meta-analyses showing no or only small differences between therapies for depression [Reference Barth, Munder, Gerger, Nuesch, Trelle and Znoj11, Reference Cuijpers, van Straten, Warmerdam and Andersson28].

Table 2 Effects of problem-solving therapy compared with control groups and other treatments: Hedges’ g.

Abbreviations: ADM: Anti-Depressant Medication; BDI: Beck Depression Inventory; CI: Confidence interval; GSH: Guided Self-Help; HAMD: Hamilton Rating Scale for Depression; Ncomp: number of comparisons; NNT: Numbers-needed-to-treat; PST: Problem Solving Therapy; RoB: Risk of Bias.

a according to the random effects model.

b The p-values in this column indicate whether the difference between the effect sizes in the subgroups is significant.

c Outliers were: Alexopoulos, 2016; Barret, 2001; Dowrick, 2000; Gellis, 2010; one of the comparisons from Nezu et al., 1986; one of the comparisons from Nezu et al., 1989; Vazquez, 2013; Williams et al., 2000.

We also found that the set of high-quality studies resulted in a low level of heterogeneity, although there was still some uncertainty around this low level. But overall this set of high-quality studies was consistent in pointing a small effect size, with no large deviations from that. Most of the heterogeneity was introduced by the studies with low or uncertain quality, which resulted in levels of heterogeneity of up to 80%. This in line with our previous meta-analysis on problem-solving Reference Cuijpers, van Straten and Warmerdam[25], where we also found a high level of heterogeneity, but could not find explanations for that in subgroup analyses. The number of high-quality studies was too small to examine whether their characteristics differed from studies of lower-qualities.

We found no significant difference between the three types of PST that we examined. However, the extended PST with 10 or more sessions did result in a considerably larger effect size, although that difference was not significant. The difference between extended and the other two types of PST may not have reached significance because of low statistical power. We had certainly too little power to examine a possible difference between the three types in high-quality studies. So it remains uncertain whether there is difference between these types of PST.

In the draft NICE treatment guidelines for depression [45] PST is not considered as a separate type of treatment and is included in the broader “family” of cognitive behavioral therapies, together with rational emotive behavior therapy and third-wave therapies. Although we do think that PST can be seen as a member of a broader “family” of cognitive behavioral therapies, we think this can be easily disputed. Cognitive restructuring is typically considered to be the core of cognitive behavioral therapies or at least one of the core elements. In PST, however, problem solving skills are the core of the treatment, and cognitive restructuring is not included in most of the PST manuals. We think therefore, that the NICE guidelines have missed an opportunity to examine the contribution of PST to treatments of depression. Because PST is included in cognitive behavioral therapy there is a risk that clinicians will be inclined not to use it anymore, while it still is one of the best examined therapies for depression.

Table 3 Standardized regression coefficients of characteristics of studies on problem solving therapy and other psychological treatments of adult depression: Multivariate metaregression analyses.

Abbreviations: BAT: Behavioral Activation Therapy; CBT: Cognitive Behavior Therapy; Coeff: Regression Coefficient; HAMD: Hamilton Rating Scale for Depression; IPT: Interpersonal Psychotherapy; p: this p-value indicates whether the regression coefficient of the subgroup differs significantly from the reference group; PST: Problem Solving Therapy; Ref: Reference group; SE: Standard Error.

This study had several limitations. The most important one is that we could include only a relatively small sample of studies, and less than 10 studies of high-quality. This resulted in low power to examine the effects and moderators of the effects of PST. Another limitation is that hardly any study examined the long-term effects of PST. We also found considerable indications for publication bias. And in the studies comparing PST with other psychotherapies we found researcher allegiance in more than half of the studies, affecting the value of these studies.

Despite these limitations we can conclude that PST is probably an effective treatment of depression, with effect sizes that are small, but comparable to those found for other psychological treatments of depression.

Conflicts of interest



No grant from funding agencies in the public, commercial, or not-for-profit sectors was received for this study.



Appendix A. Supplementary data

Supplementary data associated with this article can be found, in the online version, at


Cuijpers, P.Sijbrandij, M.Koole, S.L.Andersson, G.Beekman, A.T.Reynolds, C.F. 3rd The efficacy of psychotherapy and pharmacotherapy in treating depressive and anxiety disorders: a meta-analysis of direct comparisons. World Psychiatry 12(June (2))2013; 137-48.CrossRefGoogle ScholarPubMed
Churchill, R.Hunot, V.Corney, R.Knapp, M.McGuire, H.Tylee, al.A systematic review of controlled trials of the effectiveness and cost-effectiveness of brief psychological treatments for depression. Health Technol Assessm 2001;5(35):1-173.Google Scholar
Cuijpers, P.Geraedts, A.S.van Oppen, P.Andersson, G.Markowitz, J.C.van Straten, A.Interpersonal psychotherapy for depression: a meta-analysis. Am J Psychiatry 168(June (6))2011; 581-92.CrossRefGoogle ScholarPubMed
Cuijpers, P.Donker, T.Weissman, M.M.Ravitz, P.Cristea, I.A.Interpersonal psychotherapy for mental health problems: a comprehensive meta-analysis. Am J Psychiatry 2016;173(7):680-7.CrossRefGoogle ScholarPubMed
Cuijpers, P.van Straten, A.Warmerdam, L.Behavioral activation treatments of depression: a meta-analysis. Clin Psychol Rev 27(April (3))2007; 318-26.CrossRefGoogle ScholarPubMed
Ekers, D.Richards, D.Gilbody, S.A meta-analysis of randomized trials of behavioural treatment of depression. Psychol Med 2008;38(05):611-23.CrossRefGoogle ScholarPubMed
Driessen, E.Hegelmaier, L.M.Abbass, A.A.Barber, J.P.Dekker, J.J.M.Van, al.The efficacy of short-term psychodynamic psychotherapy for depression: a meta-analysis update. Clin Psychol Rev 42(August)2015; 1-15.CrossRefGoogle ScholarPubMed
Leichsenring, F.Rabung, S.Effectiveness of long-term psychodynamic psychotherapy: a meta-analysis. JAMA 2008;300(13):1551-65.CrossRefGoogle ScholarPubMed
Cuijpers, P.Driessen, E.Hollon, S.D.van Oppen, P.Barth, J.Andersson, G.The efficacy of non-directive supportive therapy for adult depression: a meta-analysis. Clin Psychol Rev 32(June (4))2012; 280-91.CrossRefGoogle ScholarPubMed
Cuijpers, P.Turner, E.H.Koole, S.L.van Dijke, A.Smit, F.What is the threshold for a clinically relevant effect? The case of major depressive disorders. Depression Anxiety 31(May (5))2014; 374-8.CrossRefGoogle ScholarPubMed
Barth, J.Munder, T.Gerger, H.Nuesch, E.Trelle, S.Znoj, al.Comparative efficacy of seven psychotherapeutic interventions for patients with depression: a network meta-analysis. PLoS Med 2013;10(5):e1001454.CrossRefGoogle ScholarPubMed
Cuijpers, P.van Straten, A.Warmerdam, L.Are individual and group treatments equally effective in the treatment of depression in adults?: a meta-analysis. Eur J Psychiatry 2008;22(1):38-51.CrossRefGoogle Scholar
Cuijpers, P.Donker, T.van Straten, A.Li, J.Andersson, G.Is guided self-help as effective as face-to-face psychotherapy for depression and anxiety disorders? A systematic review and meta-analysis of comparative outcome studies. Psychol Med 40(December (12))2010; 1943-57.CrossRefGoogle ScholarPubMed
Cuijpers, P.Donker, T.Johansson, R.Mohr, D.C.van Straten, A.Andersson, G.Self-guided psychological treatment for depressive symptoms: a meta-analysis. PLoS One 2011;6(6):e21274.CrossRefGoogle ScholarPubMed
Cuijpers, P.van Straten, A.Smit, F.Andersson, G.Is psychotherapy for depression equally effective in younger and older adults? A meta-regression analysis. Int Psychogeriatr 21(February (1))2009; 16-24.CrossRefGoogle ScholarPubMed
Cuijpers, P.van Straten, A.Smit, F.Psychological treatment of late-life depression: a meta-analysis of randomized controlled trials. Int J Geriatr Psychiatry 21(December (12))2006; 1139-49.CrossRefGoogle ScholarPubMed
Weisz, J.R.Kuppens, S.Eckshtain, D.Ugueto, A.M.Hawley, K.M.Jensen-Doss, A.Performance of evidence-based youth psychotherapies compared with usual clinical care: a multilevel meta-analysis. JAMA Psychiatry 70(July (7))2013; 750-61.CrossRefGoogle ScholarPubMed
Cuijpers, P.Brannmark, J.G.van Straten, A.Psychological treatment of postpartum depression: a meta-analysis. J Clin Psychol 64(January (1))2008; 103-18.CrossRefGoogle ScholarPubMed
Cuijpers, P.Cristea, I.A.Elbert, D.D.Koot, H.M.Auerbach, R.P.Bruffaerts, al.Psychological treatment of depression in college students: a metaanalysis. Depression Anxiety 2015;33(5):400-14.CrossRefGoogle ScholarPubMed
van Straten, A.Geraedts, A.Verdonck-de Leeuw, I.Andersson, G.Cuijpers, P.Psychological treatment of depressive symptoms in patients with medical disorders: a meta-analysis. J Psychosom Res 69(July (1))2010; 23-32.CrossRefGoogle ScholarPubMed
D’zurilla, T.J.Goldfried, M.R.Problem solving and behavior modification. J Abnorm Psychol 1971;78(1):107-26.CrossRefGoogle ScholarPubMed
Nezu, A.M.Efficacy of a social problem-solving therapy approach for unipolar depression. J Cons Clin Psychol 1986;54(2):196-202.CrossRefGoogle ScholarPubMed
Nezu, A.M.Perri, M.G.Social problem-solving therapy for unipolar depression: an initial dismantling investigation. J Cons Clin Psychol 1989;57(3):408-13.CrossRefGoogle ScholarPubMed
Bell, A.C.D’Zurilla, T.J.Problem-solving therapy for depression: a meta-analysis. Clin Psychol Rev 2009;29(4):348-53.CrossRefGoogle ScholarPubMed
Cuijpers, P.van Straten, A.Warmerdam, L.Problem solving therapies for depression: a meta-analysis. Eur Psychiatry 22(January (1))2007; 9-15.CrossRefGoogle ScholarPubMed
Malouff, J.M.Thorsteinsson, E.B.Schutte, N.S.The efficacy of problem solving therapy in reducing mental and physical health problems: a meta-analysis. Clin Psychol Rev 2007;27(1):46-57.CrossRefGoogle ScholarPubMed
Kirkham, J.G.Choi, N.Seitz, D.P.Meta-analysis of problem solving therapy for the treatment of major depressive disorder in older adults. Int J Geriatr Psychiatry 2016;31(5):526-35.CrossRefGoogle ScholarPubMed
Cuijpers, P.van Straten, A.Warmerdam, L.Andersson, G.Psychological treatment of depression: a meta-analytic database of randomized studies. BMC Psychiatry 2008;8(1):36.CrossRefGoogle ScholarPubMed
Cuijpers, P.Four decades of outcome research on psychotherapies for adult depression: an overview of a series of meta-analyses. Can Psychol 2017;58(1):7-19.CrossRefGoogle Scholar
Cuijpers, P.Karyotaki, E.Reijnders, M.Purgato, M.Barbui, C.Psychotherapies for depression in low- and middle-income countries: a meta-analysis. World Psychiatry 2017 accepted for publication.Google Scholar
Higgins, J.P.T.Altman, D.G.Gøtzsche, P.C.Jüni, P.Moher, D.Oxman, al.The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011; 343:d5928.CrossRefGoogle ScholarPubMed
Bowman, V.Ward, L.C.Bowman, D.Scogin, F.Self-examination therapy as an adjunct treatment for depressive symptoms in substance abusing patients. Addict Behav 21(February)1996; 129-33.CrossRefGoogle ScholarPubMed
Higgins, J.P.T.Thompson, S.G.Deeks, J.J.Altman, D.G.Measuring inconsistency in meta-analyses. BMJ 2003;327(7414):557-60.CrossRefGoogle ScholarPubMed
Duval, S.Tweedie, R.Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 2000;56(2):455-63.CrossRefGoogle ScholarPubMed
Beck, A.T.Ward, C.H.Mendelson, M.Mock, J.Erbaugh, J.An inventory for measuring depression. Arch Gen Psychiatry 1961; 4:561-71.CrossRefGoogle ScholarPubMed
Beck, A.T.Steer, R.A.Brown, G.K.BDI-II. Beck depression inventory second edition. Manual 1996, Psychological Corporation San Antonio.Google Scholar
Hamilton, M.A rating scale for depression. J Neurol Neurosurg Psychiatr 1960; 23:56-62.CrossRefGoogle Scholar
Furukawa, T.A.From effect size into number needed to treat. Lancet 1999;353(9165):1680.CrossRefGoogle ScholarPubMed
Cuijpers, P.Karyotaki, E.Weitz, E.Andersson, G.Hollon, S.D.van Straten, A.The effects of psychotherapies for major depression in adults on remission, recovery and improvement: a meta-analysis. J Affect Disord 159(April)2014; 118-26.CrossRefGoogle ScholarPubMed
Ioannidis, J.P.A.Patsopoulos, N.A.Evangelou, E.Uncertainty in heterogeneity estimates in meta-analyses. BMJ 2007;335(7626):914-6.CrossRefGoogle ScholarPubMed
Orsini, N.Bottai, M.Higgins, J.Buchan, al.Heterogi: Stata module to quantify heterogeneity in a meta-analysis. Statistical Software Components [Internet] 2006 Available from: Scholar
Choi, N.G.Marti, C.N.Bruce, M.L.Hegel, M.T.Depression in homebound older adults: problem-solving therapy and personal and social resourcefulness. Beh Ther 44(September)2013; 489-500.CrossRefGoogle ScholarPubMed
Cuijpers, P.van Straten, A.Bohlmeijer, E.Hollon, S.D.Andersson, G.The effects of psychotherapy for adult depression are overestimated: a meta-analysis of study quality and effect size. Psychol Med 40(February (2))2010; 211-23.CrossRefGoogle ScholarPubMed
Cuijpers, P.Smit, F.Bohlmeijer, E.Hollon, S.D.Andersson, G.Efficacy of cognitive-behavioural therapy and other psychological treatments for adult depression: meta-analytic study of publication bias. Br J Psychiatry 196(March (3))2010; 173-8.CrossRefGoogle ScholarPubMed
National Guideline Alliance. Depression in adults: treatment and management; Full guideline; draft for consultation; National Institute for Health and Care Excellence, Version 1.0. London, UK, NICE, 2017.Google Scholar
Alexopoulos, G.S.Raue, P.Arean, P.Problem-solving therapy versus supportive therapy in geriatric major depression with executive dysfunction. Am J Geriatr Psychiatry 11(February)2003; 46-52.CrossRefGoogle ScholarPubMed
Alexopoulos, G.S.Raue, P.J.Kiosses, D.N.Mackin, R.S.Kanellopoulos, D.McCulloch, al.Problem-solving therapy and supportive therapy in older adults with major depression and executive dysfunction: effect on disability. Arch Gen Psychiatry 2011; 68:33-41.CrossRefGoogle ScholarPubMed
Alexopoulos, G.S.Raue, P.J.McCulloch, C.Kanellopoulos, D.Seirup, J.K.Sirey, al.Clinical case management versus case management with problem-solving therapy in low-income, disabled elders with major depression: a randomized clinical trial. Am J Geriatr Psychiatry 2016;24(1):50-9.CrossRefGoogle ScholarPubMed
Arean, P.A.Perri, M.G.Nezu, A.M.Schein, R.L.Christopher, F.Joseph, T.X.Comparative effectiveness of social problem-solving therapy and reminiscence therapy as treatments for depression in older adults. J Cons Clin Psychol 61(December)1993; 1003-10.CrossRefGoogle ScholarPubMed
Barrett, J.E.Williams, J.W. JrOxman, T.E.Frank, E.Katon, W.Sullivan, al.Treatment of dysthymia and minor depression in primary care: a ramdomized trial in patients aged 18 to 59 years. J Fam Pract 2001; 50:405-12.Google Scholar
Bowman, D.Scogin, F.Lyrene, B.The efficacy of self-examination therapy and cognitive bibliotherapy in the treatment of mild to moderate depression. Psychother Res 1995;5(2):131-40.CrossRefGoogle Scholar
Chibanda, D.Shetty, A.K.Tshimanga, M.Woelk, G.Stranix-Chibanda, L.Rusakaniko, S.Group problem-solving therapy for postnatal depression among HIV-positive and HIV-negative mothers in Zimbabwe. J Int Assoc Provid AIDS Care 13(August)2014; 335-41.CrossRefGoogle ScholarPubMed
Dowrick, C.Dunn, G.Ayuso-Mateos, J.L.Dalgard, O.S.Page, H.Lehtinen, V.Problem solving treatment and group psychoeducation for depression: multicentre randomised controlled trial Outcomes of Depression International Network [ODIN] Group. BMJ 2000; 321:1450-4.CrossRefGoogle ScholarPubMed
Ebert, D.D.Lehr, D.Boss, L.Riper, H.Cuijpers, P.Andersson, al.Efficacy of an internet-based problem-solving training for teachers: results of a randomized controlled trial. Scand J Work Environ Health 40(November (6))2014; 582-96.CrossRefGoogle ScholarPubMed
Gellis, Z.D.Bruce, M.L.Problem solving therapy for subthreshold depression in home healthcare patients with cardiovascular disease. Am J Geriatr Psychiatry 18(June)2010; 464-74.CrossRefGoogle ScholarPubMed
Hopko, D.R.Armento, M.E.Robertson, S.M.Ryba, M.M.Carvalho, J.P.Colman, al.Brief behavioral activation and problem-solving therapy for depressed breast cancer patients: randomized trial. J Cons Clin Psychol 79(December)2011; 834-49.CrossRefGoogle ScholarPubMed
Kiosses, D.N.Arean, P.A.Teri, L.Alexopoulos, G.S.Home-delivered problem adaptation therapy (PATH) for depressed, cognitively impaired, disabled elders: a preliminary study. Am J Geriatr Psychiatry 18(November)2010; 988-98.CrossRefGoogle ScholarPubMed
Kleiboer, A.Donker, T.Seekles, W.van Straten, A.Riper, H.Cuijpers, P.A randomized controlled trial on the role of support in Internet-based problem solving therapy for depression and anxiety. Beh Res Ther 2015; 72:63-71.CrossRefGoogle ScholarPubMed
Lexis, M.A.Jansen, N.W.Huibers, M.J.Amelsvoort, L.G.Berkouwer, A.Tjin, al.Prevention of long-term sickness absence and major depression in high-risk employees: a randomised controlled trial. Occup Environ Med 2011; 68:400-7.CrossRefGoogle ScholarPubMed
Lynch, D.J.Tamburrino, M.B.Nagel, R.Telephone counseling for patients with minor depression: preliminary findings in a family practice setting. J Fam Pract 44(March)1997; 293-8.Google Scholar
Lynch, D.Tamburrino, M.Nagel, R.Smith, M.K.Telephone-based treatment for family practice patients with mild depression. Psychol Rep 94(June)2004; 785-92.CrossRefGoogle ScholarPubMed
Malouff, J.M.Lanyon, R.I.Schutte, N.S.Effectiveness of a brief group RET treatment for divorce-related dysphoria. J Rational-Emot Cogn-Beh Ther 1988;6(3):162-71.CrossRefGoogle Scholar
Mynors-Wallis, L.M.Gath, D.H.Lloyd-Thomas, A.R.Tomlinson, D.Randomised controlled trial comparing problem solving treatment with amitriptyline and placebo for major depression in primary care. BMJ 1995;310(6977):441-5.CrossRefGoogle ScholarPubMed
Mynors-Wallis, L.M.Gath, D.H.Day, A.Baker, F.Randomised controlled trial of problem solving treatment, antidepressant medication, and combined treatment for major depression in primary care. BMJ 320(January)2000; 26-30.CrossRefGoogle ScholarPubMed
Nollett, C.L.Bray, N.Bunce, C.Casten, R.J.Edwards, R.T.Hegel, al.Depression in Visual Impairment Trial (DEPVIT): a randomized clinical trial of depression treatments in people with low vision. Invest Ophthalm Visual Sci 2016;57(10):4247-54.CrossRefGoogle ScholarPubMed
Schmitt, S.G.Clinical depression: a comparative outcome study of two treatment approaches 1988, Fairleigh Dickinson University (Teaneck/Hackensack Campus) Teaneck, New Jersey.Google Scholar
Strong, V.Waters, R.Hibberd, C.Murray, G.Wall, L.Walker, al.Management of depression for people with cancer (SMaRT oncology 1): a randomised trial. Lancet 372(July)2008; 40-8.CrossRefGoogle ScholarPubMed
Teri, L.Logsdon, R.G.Uomoto, J.McCurry, S.M.Behavioral treatment of depression in dementia patients: a controlled clinical trial. J Gerontol B Psychol Sci Soc Sci 52(July)1997; 159-66.CrossRefGoogle ScholarPubMed
Unlu Ince, B.Cuijpers, P.van’t Hof, E.van Ballegooijen, W.Christensen, H.Riper, H.Internet-based, culturally sensitive, problem-solving therapy for Turkish migrants with depression: randomized controlled trial. J Med Internet Res 2013;15(10):e227.CrossRefGoogle ScholarPubMed
Vázquez González, F.L.Otero Otero, P.Torres Iglesias, A.García, E.H.Seoane, V.B.Fernández, O.D.A brief problem-solving indicated-prevention intervention for prevention of depression in nonprofessional caregivers. Psicothema 2013; 25:87-92.Google ScholarPubMed
Warmerdam, L.van Straten, A.Twisk, J.Riper, H.Cuijpers, P.Internet-based treatment for adults with depressive symptoms: randomized controlled trial. J Med Int Res 2008;10(4):e44.Google ScholarPubMed
Williams, J.W.Barrett, J.Oxman, T.Frank, E.Katon, W.Sullivan, al.Treatment of dysthymia and minor depression in primary care: a randomized controlled trial in older adults. JAMA 2000; 284:1519-26.CrossRefGoogle ScholarPubMed
Figure 0

Fig. 1 Flowchart for the inclusion of studies.

Figure 1

Table 1 Selected characteristics of studies examining problem solving therapy for adult depression (N = 3.357).

Figure 2

Fig. 2 Forrest plot of effect sizes from randomized controlled trials on problem-solving therapy for adult depression.

Figure 3

Fig. 3 Funnel plot of standard error by Hedges g.

Figure 4

Table 2 Effects of problem-solving therapy compared with control groups and other treatments: Hedges’ g.

Figure 5

Table 3 Standardized regression coefficients of characteristics of studies on problem solving therapy and other psychological treatments of adult depression: Multivariate metaregression analyses.

Supplementary material: File

Cuijpers et al. supplementary material

Cuijpers et al. supplementary material
Download Cuijpers et al. supplementary material(File)
File 157 KB
Submit a response


No Comments have been published for this article.