The introduction of a second generation of antipsychotic drug has created a large literature concerning their therapeutic value and tolerability. Recently there has been concern over the reliability of this literature. The effects of pharmaceutical industry funding, publication bias, the appropriateness of first-generation comparators (particular haloperidol) and a range of other methodological issues have come under close scrutiny. Reference Davis, Leucht and Glick1–Reference Leucht, Corves, Arbter, Engel, Li and Davis4 Two large independent and highly credible studies, Clinical Antipsychotic Trials of Antipsychotic Effectiveness (CATIE) Reference Swartz, Stroup, McEvoy, Davis, Rosenheck and Keefe5 and Cost Utility of the Latest Antipsychotic Drugs in Schizophrenia Study (CutLASS) Reference Jones, Barnes, Davies, Dunn, Lloyd and Hayhurst6 have cast serious doubt on the idea that second-generation antipsychotics are necessarily superior to older drugs in terms of either effectiveness or tolerability. The CATIE findings suggest that second-generation antipsychotics may not be similar to each other in effectiveness. Leucht and colleagues have produced a series of studies challenging the appropriateness of the outcome measures commonly used in antipsychotic trials. Their studies examine thresholds for reduction in Positive and Negative Syndrome Scale (PANSS) Reference Levine, Rabinowitz, Engel, Etschel and Leucht7,Reference Leucht, Kane, Kissling, Hamann, Etschel and Engel8 and Brief Psychiatric Rating Scale (BPRS) Reference Leucht, Kane, Kissling, Hamann, Etschel and Engel9 scores that are generally taken to indicate response or remission. They have demonstrated that changes in PANSS and BPRS scales can be translated into notional Clinical Global Impression scale severity and change (CGI–S and CGI–C) scores. These are scales that quantify clinicians’ overall impression of clinical severity and clinical change in individuals’ psychiatric condition. They are intended to introduce clinical meaning into drug trial findings. Leucht and colleagues have suggested that commonly accepted thresholds for determining a response (for example, 20% reduction in PANSS score) may generate statistically significant findings; however, the corresponding clinical improvement is small. Their method has some limitations, but it appears to generate an acceptably robust translation. This creates the opportunity to make an objective and reliable estimate of the clinical relevance of published findings.
Our study uses a systematic review methodology to assess the clinical relevance of the findings of the literature as a whole, by employing the method of Leucht and colleagues to translate PANSS and BPRS scores into CGI–S and CGI–C scores (Fig. 1). We do not seek to draw conclusions regarding the utility of antipsychotic drugs in clinical practice. We are interested in the clinical relevance of the findings in the literature, because this is the main body of scientific evidence that guides clinical prescribing practice and it is commonly used by the pharmaceutical industry to support their claims regarding the usefulness of their products.
Method
In order to systematically review the literature we drew upon Davis and colleagues’ Reference Davis, Chen and Glick10 systematic review of second-generation antipsychotic drug evaluation studies published up to (and including) 2001. We included in our search all studies that they had listed as references. We also conducted an electronic search for studies published between 2002 and 31 December 2007, using the same search terms. The inclusion criteria for our review were:
-
(a) participants: a diagnosis of schizophrenia or schizoaffective disorder;
-
(b) interventions: at least one second-generation antipsychotic drug;
-
(c) comparator: no comparator necessary;
-
(d) outcome measures: change in mean BPRS and/or PANSS score from baseline to end-point;
-
(e) antipsychotics: any antipsychotic that is or has been licensed;
-
(f) design: at least single-group pre–post design;
-
(g) reporting: published in a peer-reviewed journal, listed by Davies et al or by one of the three databases listed below. Books and conference posters were excluded; published in English, German or French; sample size for each study arm reported; available as electronic full text or as paper full text.
Davis et al applied the following search term to MEDLINE, PsycINFO and CINAHL to identify studies for their review: ((amisulpride OR aripiprazole OR clozapine OR olanzapine OR quetiapine fumarate OR remoxipride hydrochloride OR risperidone OR sertindole OR ziprasidone hydrochloride OR zotepine) AND (schizophrenia OR schizoaffective disorder) AND (BPRS OR PANSS)). We used this search term in our update.
Conversion of Positive and Negative Syndrome Scale (PANSS) and Brief Psychiatric Rating Scale (BPRS) scores to Clinical Global Impression – Change (CGI–C) scores, adapted from Leucht et al. Reference Leucht, Kane, Kissling, Hamann, Etschel and Engel8,Reference Leucht, Kane, Kissling, Hamann, Etschel and Engel9
We attempted to eliminate factors that would tend to systematically bias against positive findings for the index drug: we excluded studies that reported median scores in heterogeneous samples, as means usually show better results than medians in such samples. In other words, studies that reported results as medians were only included if they showed homogeneity with means. Studies were excluded if participants were essentially healthy at baseline, as this creates reduced scope for clinical improvement. Studies were included if the baseline mean scores were, using the PANSS, 58 and above, or using the BPRS, 31 and above, indicating significant illness. Using Leucht et al's method Reference Leucht, Kane, Kissling, Hamann, Etschel and Engel8,Reference Leucht, Kane, Kissling, Hamann, Etschel and Engel9 this corresponds to a CGI–S score of at least 3 (mildly ill). We excluded all samples that reported the use of age-adapted subtherapeutic doses as defined by the manufacturers.
In a second step, we attempted to identify studies that used a single sample and data-set in multiple publications. As far as practically possible, we used each data-set only once and removed all repeat data.
In keeping with Leucht's methodology, both PANSS and BPRS scores were adjusted to allow meaningful calculation of percentage change. The minimum possible score on PANSS (no symptoms) is 30, so this figure was subtracted to create an adjusted PANSS score. The corresponding score on BPRS is 18, which was similarly subtracted to produce an adjusted BPRS score. A mean, weighted for sample size, was calculated for aggregated BPRS and PANSS scores at baseline and follow-up. Mean change and percentage mean change were also calculated. These were calculated from adjusted scores (our figures show the unadjusted PANSS and BPRS scores as originally published). Percentage changes were calculated for: all active drugs combined; first- v. second-generation antipsychotic v. placebo; each individual drug.
The independent sample t-test was used to determine homogeneity between median and mean BPRS scores in the comparison of first- and second-generation antipsychotics. The test showed that there was homogeneity (P>0.05). Consequently, the median and mean BPRS scores were combined for further analysis. There were no medians reported in the PANSS samples. We translated all aggregated PANSS and BPRS scores into notional approximate CGI–C scores, using the graphs published by Leucht and colleagues.
Data were extracted by two of the authors. In order to estimate intercoder reliability, each of them extracted data from 15 of the studies independently. The baseline mean PANSS or BPRS scores that they had derived from the studies were compared. Of the 15 values, 14 were identical (93% agreement, intraclass correlation 0.99).
All data were expressed as a CGI–C equivalent change using the graphs produced by Leucht. We used the latest possible follow-up point in each study for the CGI–C translation in order to give the medication the longest possible time to have an effect, and thus avoid any bias against medication because of the short duration of follow-up. The figures are approximated with an accuracy margin of 0.05 as they derive from a non-linear graph. A CGI–C of –1.0 is seen as a measure for response to an antipsychotic drug equating to a PANSS score percentage change of 28 or a BRPS score percentage change of 30. The core aim of this study was to convert BPRS/PANSS scores into CGI–C equivalents and not to estimate the effect size for each drug. The available conversion tables Reference Leucht, Kane, Kissling, Hamann, Etschel and Engel8,Reference Leucht, Kane, Kissling, Hamann, Etschel and Engel9 are based on percentage reduction from baseline and CGI–C scores, and so results below are reported in these terms rather than in terms of effect sizes.
Data-loading reliability for each of the coders was assessed. All coded data from 10 studies for each coder were compared across paper and electronic recording sheets (49 data-points and 64 data-points respectively). Agreement across versions was 98% and 100% respectively.
Results
The initial search process generated 678 titles (Fig. 2). Of these 114 came from the reference list in Davis et al Reference Davis, Chen and Glick10 and 564 were identified in our 2002–2007 search. In a second stage we excluded those papers that did not fulfil our inclusion criteria or could not be obtained. We located 211 full-text articles in full-text version from the 678 titles originally generated. Of these, 91 papers were then excluded for the following reasons: wrong data format (for example scores reported graphically, not numerically), no new data reported, insufficient data/information (for example no sample size), baseline BPRS/PANSS below threshold and other reasons.
Flow diagram of studies and data-sets included.
BPRS, Brief Psychiatric Rating Scale; PANSS, Positive and Negative Syndrome Scale.
The remaining 120 studies (online supplement) had between 1 and 7 study arms and yielded 300 separate data-sets relating to a second-generation antipsychotic. In total, 98 data-sets were included in the BPRS analysis (12 different drugs) and 202 data-sets were included in the PANSS analysis (15 different drugs). A total of 22 428 participants were included in the PANSS scores analysis, and a total of 9772 in the BPRS scores analysis. There were no statistically significant differences in baseline PANSS and BPRS score between the three main groups (first-, second-generation antipsychotics, placebo).
Table 1 shows the results of the analysis of PANSS scores including the total sample size at baseline, the number of data-sets included, mean absolute PANSS scores at baseline and post-treatment, aggregated mean change in scores in absolute numbers and as percentages, and notional CGI–C scores.Table 2 shows the corresponding findings for BPRS.
Summary results Positive and Negative Syndrome Scale (PANSS): reported non-adjusted PANSS mean/median values
| Baseline sample, n | Data-sets, n | Absolute scores baseline (s.d.) | Absolute scores post-treatment (s.d.) | Mean change (s.d.) | Mean % change (s.d.)a | CGI–C equivalent changeb | |
|---|---|---|---|---|---|---|---|
| All drugs | 21 776 | 202 | 83.10 (12.44) | 67.26 (12.12) | 14.26 (9.47) | 25.93 (14.41) | – 0.9 |
| All first-generation antipsychotics | 2670 | 34 | 87.36 (11.65) | 75.24 (10.93) | 12.12 (7.57) | 20.96 (14.19) | – 0.7 |
| All second-generation antipsychotics | 17 805 | 154 | 82.50 (12.55) | 66.19 (10.78) | 15.37 (9.39) | 28.17 (13.33) | – 1.0 |
| Placebo | 1301 | 14 | 89.45 (6.34) | 82.98 (16.58) | 3.44 (5.87) | 5.48 (9.76) | +0.25 |
| Chlorpromazine | 37 | 2 | 102.64 (10.79) | 94.58 (17.11) | 8.07 (4.71) | 12.65 (7.65) | – 0.2 |
| First-generation antipsychoticc | 155 | 5 | 98.71 (13.21) | 74.75 (9.15) | 23.97 (9.55) | 33.18 (16.29) | – 1.2 |
| Haloperidol | 1970 | 23 | 86.74 (10.85) | 74.92 (10.88) | 11.82 (7.31) | 20.93 (15.13) | – 0.7 |
| Perphenazine | 458 | 3 | 84.40 (11.85) | 74.58 (10.06) | 9.84 (4.54) | 17.96 (5.86) | – 0.6 |
| Zuclopenthixol (depot) | 50 | 1 | 92.50 | 81.10 | 11.40 | 18.24 | – 0.6 |
| Amisulpride | 424 | 8 | 73.49 (15.01) | 46.95 (10.29) | 26.55 (7.50) | 63.71 (13.07) | – 2.2 |
| Aripiprazole | 798 | 8 | 92.30 (6.16) | 79.58 (6.24) | 12.72 (6.14) | 20.02 (9.85) | – 0.65 |
| Clozapine | 674 | 13 | 94.87 (8.16) | 76.60 (8.78) | 18.27 (10.08) | 25.57 (13.11) | – 0.9 |
| Olanzapine | 3785 | 42 | 90.82 (10.17) | 69.72 (6.64) | 21.10 (10.06) | 33.49 (11.97) | – 1.2 |
| Paliperidone | 733 | 6 | 93.36 (1.07) | 75.07 (2.33) | 18.29 (4.12) | 28.83 (4.14) | – 1.05 |
| Quetiapine | 704 | 7 | 79.80 (10.62) | 55.32 (19.36) | 12.00 (6.47) | 23.14 (10.03) | – 0.8 |
| Risperidone | 6589 | 60 | 81.05 (12.73) | 64.77 (11.29) | 15.10 (9.85) | 28.21 (13.97) | – 1.0 |
| Risperidone LAI (depot) | 3655 | 8 | 71.18 (13.80) | 62.44 (2.94) | 9.13 (2.09) | 21.76 (3.98) | – 0.75 |
| Second-generation antipsychoticc | 69 | 1 | 84.09 | 68.00 | 16.09 | 29.75 | – 1.1 |
| Ziprasidone | 183 | 1 | 75.40 | 68.00 | 7.40 | 16.30 | – 0.5 |
Summary results Brief Psychiatric Rating Scale (BPRS): reported non-adjusted BPRS mean/median values
| Baseline sample, n | Studies, n | Absolute scores baseline (s.d.) | Absolute scores post-treatment (s.d.) | Mean change (s.d.) | Mean % change (s.d.)a | CGI–C equivalent changeb | |
|---|---|---|---|---|---|---|---|
| All drugs | 9666 | 98 | 48.25 (9.93) | 34.66 (8.77) | 13.51 (7.30) | 46.16 (20.63) | – 1.6 |
| All first-generation antipsychotics | 2237 | 31 | 45.77 (10.19) | 35.66 (10.62) | 10.10 (5.38) | 39.90 (20.00) | – 1.3 |
| All second-generation antipsychotics | 7159 | 63 | 49.00 (9.83) | 33.91 (7.79) | 15.00 (7.20) | 49.58 (19.09) | – 1.65 |
| Placebo | 270 | 4 | 49.04 (6.44) | 46.78 (6.85) | 2.26 (3.67) | 7.34 (10.94) | +0 |
| Chlorpromazine | 186 | 3 | 59.70 (13.91) | 55.97 (13.46) | 3.73 (2.22) | 8.72 (5.15) | – 0.1 |
| First-generation antipsychoticc | 51 | 2 | 52.79 (12.18) | 36.70 (12.67) | 16.09 (4.97) | 52.27 (24.81) | – 1.7 |
| Haloperidol | 1948 | 25 | 44.06 (9.54) | 33.56 (8.89) | 10.50 (5.26) | 42.68 (18.54) | – 1.4 |
| Perphenazine | 52 | 1 | 52.00 | 40.00 | 12.00 | 35.29 | – 1.25 |
| Amisulpride | 215 | 2 | 55.42 (5.95) | 34.70 (6.53) | 20.38 (3.99) | 52.97 (2.79) | – 1.8 |
| Clozapine | 354 | 9 | 51.17 (11.22) | 39.87 (5.47) | 11.30 (6.29) | 31.37 (9.91) | – 1.05 |
| Olanzapine | 2119 | 14 | 40.41 (10.22) | 28.03 (8.69) | 12.06 (3.58) | 60.08 (17.28) | – 2.05 |
| Quetiapine | 255 | 5 | 45.46 (3.22) | 38.77 (2.81) | 6.69 (13.0) | 24.41 (30.07) | – 0.7 |
| Remoxipride | 440 | 9 | 44.78 (6.04) | 30.37 (7.16) | 14.44 (6.31) | 55.00 (23.01) | – 1.85 |
| Risperidone | 3685 | 22 | 54.22 (5.39) | 36.72 (4.84) | 17.50 (6.08) | 46.58 (10.93) | – 1.6 |
| Second-generation antipsychoticc | 26 | 1 | 34.20 | 25.30 | 8.90 | 54.94 | – 1.85 |
| Sertindole | 65 | 1 | 57.20 | 46.90 | 10.30 | 26.28 | – 0.8 |
In the PANSS sample, the CGI–C score for most of the drugs clustered around –0.5 to –1.2 (0 is no change and –1 is the threshold for ‘minimal improvement’). The exceptions were chlorpromazine with a CGI–C score of –0.2, and amisulpride with a CGI–C score of –2.2 (much improved). In the BPRS sample, CGI–C scores were larger, the majority of drugs scoring between –1 and –1.9 (minimal improvement). Sertindole was below the –1 threshold, and chlorpromazine was far below it. Only olanzapine scored better than –2 (much improved). SeeFig. 3 for a summary of the results.
Discussion
Main findings
The published trial literature consistently reports improvement in participants with schizophrenia and other psychotic illnesses when they are given antipsychotic drugs. However, our study suggests that on average the clinical significance of the reported findings is disappointingly limited. There is little difference in this regard between first- and second-generation drugs as categories, although there are differences between individual drugs.
Broadly speaking, our findings are similar to other recent studies, both with regard to the similarities between first- and second-generation antipsychotics as categories, for example, CutLASS, Reference Jones, Barnes, Davies, Dunn, Lloyd and Hayhurst6 and with regard to differences between the different second-generation antipsychotics, for example CATIE Reference Swartz, Stroup, McEvoy, Davis, Rosenheck and Keefe5 and two recent meta-analyses. Reference Leucht, Corves, Arbter, Engel, Li and Davis4,Reference Davis, Chen and Glick10 In the reported studies, amisulpride and olanzapine appear, by and large, to lead to a greater overall clinical improvement than other drugs. Ziprasidone, quetiapine, sertindole, aripiprazole and chlorpromazine appear to perform relatively poorly. However, only the most effective drugs produce a moderate improvement on CGI–C when PANSS and BPRS changes are translated into notional CGI–C scores. No drug achieved this on both the PANSS and BPRS score. Interestingly, even clozapine does not perform well in our analysis. This stands in contrast to other studies, despite including the CATIE results in our sample. However, in most, if not all other studies clozapine was used for individuals with defined therapy-resistance. Therefore it is not surprising that the clinical effects observed were only moderate.
Summary of results of all oral drugs with at least two studies or more than 100 participants in total. Ranked by Positive and Negative Syndrome Scale (PANSS) percentage reduction (28% reduction needed for ‘minimally improved’, 53% for ‘much improved’).
BPRS, Brief Psychiatric Rating Scale; AMI, amisulpride; OLA, olanzapine; SGA, all second-generation antipsychotics combined; RIS, risperidone; CLO, clozapine; QUE, quetiapine; HAL, haloperidol; FGA, all first-generation antipsychotics combined; ARI, aripiprazole; ZIP, ziprasidone; CPZ, chlorpromazine.
Comparison of first- and second-generation antipsychotics as categories shows neither class of drug achieving a mean CGI–C score of great clinical significance. Clinical Global Impression – Change scores derived from PANSS and BPRS scores show a similar pattern between drugs, but BPRS scores translate into greater clinical improvement than PANSS scores. This may be a result of the greater prominence of negative symptoms in the PANSS scale.
Drug trials are conducted using groups of participants that differ considerably from clinical populations with acute illnesses. Naturalistic studies of individuals under treatment for acute schizophrenia-spectrum disorder may show greater improvements in PANSS scores than those seen in drug trials. For example, Jäger et al Reference Jäger, Riedel, Obermeier, Schennach-Wolff, Seemüller and Messer11 conducted a naturalistic study of 280 in-patients with schizophrenia. There was a mean reduction in PANSS scores of 47% (equivalent to a CGI–C score of – 1.75) between admission and discharge. The PANSS reduction was 25% (equivalent to a CGI–C of –0.8) in the subsample of individuals with a history of multiple admissions, a result much more akin to our findings. The PANSS reduction for the total sample was unusually large compared with the drug trial literature. The authors suggested that this was the result of the individuals being treated holistically in a naturalistic study, together with the inclusion of a larger proportion of individuals with first-episode psychosis than would be possible in a drug trial. This underlines the value of naturalistic studies, where conditions and, it seems, findings correspond more closely to clinical experience.
Other explanations for our findings include the possibility that the clinical relevance of antipsychotics is in fact limited. Alternatively, symptomatology may only be a small aspect in individuals’ overall assessment of their quality of life. Reference Eack, Newhill, Anderson and Rotondi12 In addition, our method of aggregating and averaging results may hide the larger improvements that certain individuals gain from antipsychotics. With regard to first- and second-generation antipsychotics as categories, poorly performing drugs may have masked the admittedly limited effects of better performing drugs. Other reasons or combinations of reasons may also apply.
Limitations
Our study has some limitations. As is commonly the case in a systematic review, a large number of studies were excluded for a variety of reasons. Although we have not included the whole literature, it is reasonable to suppose that we have captured a very large proportion and included the most prominent and best conducted published studies. There is no reason to suppose that the exclusion of studies has created a bias to minimise the clinical effects of antipsychotic drugs. Leucht et al Reference Leucht, Kane, Kissling, Hamann, Etschel and Engel9 have pointed out that their method of converting PANSS and BPRS continuous scores into CGI categorical scores involves the translation of psychometrically validated instruments into impressionistic scales using conversion graphs that are not perfectly linear. We have attempted to partially overcome this by reporting CGI scores to two decimal places, to indicate proximity to threshold values on the CGI. We have not examined the length of studies as a factor affecting outcome, but the vast majority of studies were between 6 and 24 weeks long. There is, however, increasing evidence that much of the antipsychotic effect that can be measured with PANSS and BPRS scores occurs in the first 2 weeks of treatment. Reference Kinon, Chen, Asher-Svanum, Stauffer, Kollack-Walker and Sniadecki13 This suggests that it is unlikely that study duration accounts for our findings.
Clinical Antipsychotic Trials of Antipsychotic Effectiveness (CATIE) and CutLASS investigated effectiveness through a variety of different outcome criteria, whereas the majority of studies we included were designed to examine efficacy. However, our review included the results from CATIE and our findings are in keeping with other recent meta-analyses Reference Leucht, Corves, Arbter, Engel, Li and Davis4 that included efficacy studies. We do not believe that the limitations of our study invalidate the principal findings.
Implications
We would caution against drawing the conclusion that we have shown that antipsychotic drugs have negligible effects in clinical practice. Our data have been drawn from a literature that has well-recognised faults, such as short length of drug exposure, low dosage regimes for comparator drugs, studies conducted on participants who are sometimes only mildly ill at baseline (which limits the potential for improvement), publication bias and so on. This is a study of the literature, and we have shown that the findings of that literature are of limited clinical significance. As a consequence, caution should be exercised when drawing conclusions from this literature about the clinical usefulness of these drugs.
Our findings lend considerable support to recent proposals that higher outcome thresholds should be applied to PANSS and BPRS scores in future studies. Reference Leucht, Davis, Engel, Kissling and Kane14–Reference Opler, Yang, Caleo and Alberti17 We also support the recent suggestion to rescale the PANSS in order to make comparisons such as ours easier to calculate. Reference Leucht, Kissling and Davis18 We would further suggest that measures of clinical relevance should be included alongside measures of changes in psychopathology in all drug trials.
Acknowledgements
We would like to thank Stefan Leucht for providing us with information about his conversion graphs. We would like to thank the PANSS Institute for their advice on cut-off points.
eLetters
No eLetters have been published for this article.