The study of Scott et al (Reference Scott, Paykel and Morriss2006, this issue) is the largest multicentre pragmatic randomised controlled trial (RCT) of psychological therapy for bipolar disorders. The reliability and quality assurance procedures were well conducted. The results indicated no beneficial effects of cognitive–behavioural therapy (CBT). In contrast, four major pragmatic RCTs of structured psychological interventions reported beneficial effects (Reference Perry, Tarrier and MorrissPerry et al, 1999; Miklowitz et al, Reference Miklowitz, Simoneau and George2000, Reference Miklowitz, George and Richards2003; Lam et al, Reference Lam, Watkins and Hayward2003, Reference Lam, Hayward and Watkins2005; Colom et al, Reference Colom, Vieta and Martinez-Aran2003, Reference Colom, Vieta and Sanchez-Moreno2004). I believe it is useful to refer to some fundamental differences between the design of the Scott et al study and the other published studies that may have led to different findings.

Scott et al described their study as a ‘pragmatic’ trial and contrasted it with other trials which were called ‘efficacy’ trials. I would argue that the RCTs cited above were also pragmatic. The main outcome measures were practical issues such as duration of survival, relief of symptoms and social functioning. Drug treatment regimes were loosely defined as treatment as usual or according to some form of treatment algorithms in order to reflect clinical reality. Moreover, a heterogeneous patient population was recruited by Colom et al (31% with current DSM–IV (American Psychiatric Association, 1994) personality disorder) and by Lam et al (44.7% with a lifetime comorbid DSM–IV Axis I diagnosis).

Scientific studies, whether pragmatic or explanatory in nature, must have a design that allows systematic investigations of issues of interest, the results of which lend themselves to statistical analyses that permit valid inferences to be drawn. Scott et al recruited a mixed sample to obtain ‘a group highly representative of persons with bipolar disorder using adult mental health services’. However, this mix of patients confounded systematic investigations of issues important in the psychological treatment of bipolar disorders.

Bipolar disorder is a complex illness and various treatments are likely to be needed for different phases. The effectiveness of any treatment programme, pharmacological or psychological, depends on its ability to target selective problems in specific phases of the illness. Hence, psychological treatment studies have not mixed treatment for an acute episode with relapse prevention (Reference Frank, Kupfer and EhlersFrank et al, 1994; Reference Perry, Tarrier and MorrissPerry et al, 1999; Reference Colom, Vieta and Martinez-AranColom et al, 2003; Reference Lam, Watkins and HaywardLam et al, 2003; Reference Miklowitz, George and RichardsMiklowitz et al, 2003). When patients are in an acute episode, some of the relapse prevention techniques become secondary and more sessions are needed to complete the acute treatment. In the study of Scott et al, 32% of patients were in acute episodes. One possible explanation of the failure to find an advantageous effect of CBT is that, when patients were in an acute episode, more sessions were needed to tackle both the acute episode and to carry out systematic relapse prevention work. Indeed, the study reported that 40% of patients did not receive all the planned components of the package owing to some patients’ failure to engage in therapy and therapists having to treat the acute episode as well as carry out relapse prevention work. In our study (Lam et al, Reference Lam, Watkins and Hayward2003, Reference Lam, Hayward and Watkins2005), 16 sessions were just about enough time to carry out the planned interventions with patients not in an episode.

If the intention to recruit patients in or out of an episode was to allow the investigators the opportunity to examine the efficacy of CBT both in the acute episode and in relapse prevention, then a stratified random allocation procedure should have been used to ensure an adequately powered sample to answer both questions. In Scott et al's study, the acute episode subsample consisted of a relatively small number (24% depression, 5.5% hypomania and 3% mixed episode). Even with a depression subgroup of only 30 patients in the CBT and control groups (with or without antidepressants or mood stabilisers) respectively, it is unlikely that they would be able to conclude with confidence whether CBT had an effect on bipolar depression.

The mixing of patients in remission and in acute episodes at baseline also poses a severe challenge to the survival analyses. In essence, patients cannot relapse unless they have remitted. The way the authors tried to deal with a mixed sample of patients in or not in an episode was to allow for current mental state (euthymic, depressed or hypomanic/mixed state) as one of the covariates in the Cox model in the regression analysis. Thus, being in an episode was allowed to increase or reduce relapse time. However, allowing for mental state at baseline is not sufficient – that only allows for different rates of relapse according to mental state at baseline and does not allow for the fact that patients cannot relapse until they have recovered. In order to maintain the survival analysis framework, if a patient recovers only 1 month before the end of the study, then this patient only contributes 1 month of relapse-free ‘survival’ time. If someone remained ill throughout the study, then they would not contribute to the study at all. Although in Scott et al's study CBT did not affect time to remission and the trial was randomised, the authors should have reported the treatment effects for the acute episode and relapse prevention separately. In the survival analysis, only those who were not in an acute episode at baseline should have been included.

In Scott et al's study, 15% of patients were not prescribed mood stabilisers. Inclusion of these is at odds with other studies (Reference Frank, Kupfer and EhlersFrank et al, 1994; Reference Colom, Vieta and Martinez-AranColom et al, 2003; Reference Lam, Watkins and HaywardLam et al, 2003; Reference Miklowitz, George and RichardsMiklowitz et al, 2003). In the psychological treatment of psychosis, a diathesis–stress model is adopted. Hence psychotherapy was used as a combined treatment with medication. The authors tried to deal with the issue of whether patients were prescribed a mood stabiliser by using it as one of the covariates in the Cox model. I am not sure whether it is good practice to rely on statistical techniques to control for so many potentially confounding variables as covariates. If the question was to test whether CBT alone is efficacious, such a small number in the group without mood stabilisers was insufficient for the task. It was surprising that the patients in the CBT group were not more adherent to medication. This finding was again at odds with other studies (Reference Colom, Vieta and Martinez-AranColom et al, 2003; Lam et al, Reference Lam, Watkins and Hayward2003, Reference Lam, Hayward and Watkins2005; Reference Miklowitz, George and RichardsMiklowitz et al, 2003).

Finally, the authors reported ‘in a post hoc analysis a significant interaction between randomised treatment and number of episodes recorded at baseline assessment’ (Cox regression analysis). Again, this post hoc analysis included every patient who was recruited in the study, irrespective of whether they were in an episode or not at baseline. The authors did not report how they derived the number of previous episodes. However, with a sample of a mean age of about 40 years (s.d.=11), some of the patients may have had bipolar disorder for over 20 years. Medical notes may not be comprehensive and/or cover the whole period of illness. The number of previous episodes is likely to have to rely on the patients’ own estimates. Hence, this variable may be no more than a rough estimation. A post hoc analysis involving an interaction between treatment status and a variable of questionable accuracy weakened the case considerably. The authors then concluded that CBT was less helpful for those with more than 12 previous episodes, based on a median split. However, median splits can produce spurious results owing to an inaccurate summary of a continuous/interval variable (Reference MacCallum, Zhang and PreacherMacCallum et al, 2002). Typically a median split can lose up to 30% of the information. The authors’ finding of 12 sessions as a cut-off should not be used to make clinical recommendations.

Despite differences in theoretical frameworks and mode of delivery, there are some common features in psychological studies of relapse prevention in bipolar disorders (Reference Perry, Tarrier and MorrissPerry et al, 1999; Colom et al, Reference Colom, Vieta and Martinez-Aran2003, 2005; Lam et al, Reference Lam, Watkins and Hayward2003, Reference Lam, Hayward and Watkins2005; Miklowitz et al, Reference Miklowitz, Simoneau and George2000, Reference Miklowitz, George and Richards2003; Reference Scott, Paykel and MorrissScott et al, 2006, this issue). These include psychoeducation; promoting medication adherence; promotion of regular daily routine and sleep; monitoring mood; detection of early warnings and strategies to prevent early stages from developing into full-blown episodes; and some general coping strategies including problem-solving techniques. To date, the weight of evidence, despite Scott et al's study, is that structured psychological therapies with these common components have beneficial effects in relapse prevention. In addition, the finding of Scott et al that CBT was less effective for relapse prevention in those patients with many episodes is best regarded as tentative.