Introduction
Recent meta-analyses show that cognitive behaviour for psychosis is an effective intervention that addresses a variety of problems (Wykes et al. Reference Wykes, Steel, Everitt and Tarrier2008). However, cognitive behaviour therapy for psychosis is a complex intervention. To address the diverse and heterogeneous problems presented by people with psychosis, most manuals offer a rich package that includes a range of techniques. The original target for CBT for psychosis was distressing positive symptoms of psychosis; however, cognitive behaviour packages also typically include techniques to target outcomes such as relapse prevention, reduction in emotional disturbance and reduction in social disability (Fowler, Garety & Kuipers, Reference Fowler, Garety and Kuipers1995). Interventions also need to be adapted to address the differences in presentations at different stages of psychosis. CBT for young people with At-Risk Mental States (French & Morrison, Reference French and Morrison2004) has differences to CBT for people with treatment-resistant or relapsing psychosis (Fowler, Garety & Kuipers, Reference Fowler, Garety and Kuipers1995). Also interventions for people with early psychosis who have recovered from acute psychotic symptoms but still have social recovery problems (Fowler et al. Reference Fowler, Hodgekins, Painter, Reilly, Crane, Macmillan, Mugford, Croudace and Jones2009) will need to be different from interventions that target the problems of people with early psychosis recovering from their acute psychotic episode (Tarrier et al. Reference Tarrier, Lewis, Haddock, Bentall, Drake, Kinderman, Kingdon, Siddle, Everitt, Leadley, Benn, Grazebrook, Haley, Akhtar, Davies, Palmer and Dunn2004). This variation presents a challenge for researchers seeking to standardize therapy and assess whether therapy has been delivered competently and as prescribed in the manuals. Detailed assessment tools are required not only to assess the quality of the therapy but also to check on which specific techniques prescribed in the manuals have actually been delivered to individuals.
In this paper, we review the studies undertaken in this area by our research groups. We review assessment tools, describe their use in clinical studies and outline how such studies can provide a basis to undertake studies which seek to test hypotheses relating to understanding which techniques work for whom in psychosis. We also highlight how the use of such scales can assist training and supervision.
Definitions of competence and adherence
Treatment fidelity entails both therapist adherence; ‘the extent to which a therapist used interventions and approaches prescribed by the treatment manual and avoided the use of interventions proscribed by the manual’ (Waltz et al. Reference Waltz, Addis, Koerner and Jacobson1993), and therapist competence; the ‘level of skill…[that is]…the extent to which the therapists took the relevant aspects of the therapeutic context into account and responded to these contextual variables appropriately’ (Waltz et al. Reference Waltz, Addis, Koerner and Jacobson1993). Formal tests of treatment fidelity provide the basis for research into processes of therapy, and are the starting point for attempts to understand the mechanisms of change underpinning complex interventions such as CBT for psychosis. They provide clearer differentiation between therapeutic approaches, facilitate the interpretation of varied outcomes in the same ‘brand’ of therapy and where assessed allow inferences about the influence of the manualized treatment upon outcome. Measures of treatment fidelity are equally important for dissemination, as they provide a clear definition of the treatment under study for programmes of training and supervision.
Rigorous and independent checks of fidelity in the field of CBT for psychosis are relatively recent. The early randomized controlled trials of CBT gave priority to ensuring consistency of treatment within trials through ongoing supervision of the fidelity of the intervention. Kuipers et al. (Reference Kuipers, Garety, Fowler, Dunn, Bebbington, Freeman and Hadley1997) held peer supervision sessions between expert therapists, while Tarrier et al. (1998) had an independent rater judge whether a sample of sessions involved cognitive therapy or supportive counselling (the control condition) as did Turkington, Kingdon and Turner (Reference Turkington, Kingdon and Turner2002). The therapy provided in these trials was mainly carried out by the authors of the original manuals, and was therefore likely to be of good quality and consistent with the manual, but these approaches provided no formal way of quantifying adherence or competence. As the field has matured there has been consensus towards using the Cognitive Therapy Rating Scale (CTRS) as the formal assessment of competence and to use variants of the Cognitive Therapy for Psychosis Adherence Scale (CTPAS) to assess adherence.
Assessing therapeutic competence using the CTRS
The CTRS is the standard rating scale to assess the delivery of cognitive therapy, it was first described by Beck et al. (Reference Beck, Rush, Shaw and Emery1979) and subsequently revised by Blackburn et al. (Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001). It has 12 items that cover three general areas including the establishment and maintenance of an effective psychotherapeutic relationship (e.g. interpersonal effectiveness, pacing and eliciting emotional responses), structuring of therapy (e.g. agenda setting and checking feedback) and items the use of specific cognitive and behavioural methods (e.g. collaboration, guided discovery, eliciting key cognitions, conceptual integration and specific cognitive and behavioural techniques). These items rate core aspects of delivery of high-quality therapy which are equally relevant for cognitive behaviour therapy for psychosis as for cognitive behaviour therapy for other disorders. The focus on establishment and maintenance of a good working psychotherapeutic relationship and the collaborative guided discovery style of cognitive therapy is particularly important in cognitive therapy for psychosis and indeed achieving this in itself can often be a challenge when working with difficult to engage psychotic patients. The ratings on the CTRS are flexible, and allow the possibility of scoring therapists working with challenging cases and working with difficult client presentations in a sophisticated manner. The scale therefore has widespread applicability to rate the competence and quality of therapy delivered even in the context of challenging psychotic presentations.
The scale is rated from tape recordings or videos. Typically, in trials in addition to internal ratings used for supervision, a random sample of tapes will be sent to expert cognitive therapists and the ratings of these independent raters will be reported. The first study to formally use the CTRS was that by Sensky et al. (Reference Sensky, Turkington, Kingdon, Scott, Scott, Siddle, O'Carroll and Barnes2000). In this study, Independent raters assessed a representative sample of therapy tapes from the control and experimental conditions with the CTRS. Competence was satisfactory, and significantly more cognitive techniques were used in the experimental condition. The scale has been used in most subsequent trials both in supervision to ensure therapists are maintaining therapy quality, and in the final report to justify that trained therapists used in trials were, as a group, delivering therapy to internationally recognized standards of therapy quality. To date there have been few formal reports which have used the CTRS to examine associations between therapy quality and outcome. This may be an interesting possibility for the future especially in effectiveness trials where there may be variations in therapy quality provided.
The CTRS is a general cognitive therapy competency scale, and some have highlighted the differences between CBT for psychosis and CBT for other disorders. Haddock et al. (Reference Haddock, Devane, Bradshaw, McGovern, Tarrier, Kinderman, Baguley, Lancashire and Harris2001) adapted the scaling of the Cognitive Therapy Scale (CTS), for use in psychosis. The adapted scale, (the CTS-Psy) alters the scaling system to take account of alterations which CBT therapists working with psychotic patients may use, although the items retain the original focus on the use of key generic cognitive skills. The CTS-Psy was the main competence tool used in the Socrates study of CBT in the acute stages of first-episode psychosis (Tarrier et al. Reference Tarrier, Lewis, Haddock, Bentall, Drake, Kinderman, Kingdon, Siddle, Everitt, Leadley, Benn, Grazebrook, Haley, Akhtar, Davies, Palmer and Dunn2004). These adaptations may be useful but other researchers have retained use of the original CTRS, suggesting that the original scale already has flexibility to allow satisfactory ratings of the quality of CBT in psychosis. Both the CTS-Psy and the CTRS are applicable in psychosis.
The CTPAS
The CTRS is now probably accepted as the gold standard assessment of competence or quality of cognitive therapy. However, CBT for psychosis differs from other types of CBT in substantive ways. As noted by Startup, Jackson & Pearce (Reference Startup, Jackson and Pearce2002) the clearest way in which CBT for psychosis is different is in content. CBT for psychosis requires therapists to address psychotic symptoms and help them understand the nature of their psychotic disorder. Since CBT in general does not use these techniques it is possible to get a high score on the CTRS without actually using CBT for psychosis techniques at all. For example a skilled therapist can potentially, and appropriately, get a satisfactory score on the CTRS by simply assessing or maintaining a good collaborative psychotherapeutic relationship and assessing and formulating, or by addressing anxiety or depression, but not addressing psychotic symptoms directly. This was observed in the fidelity study used in the Tayside Fife trial of CBT for treatment resistant psychosis (Durham et al. Reference Durham, Guthrie, Morton, Reid, Treliving, Fowler and Mcdonald2003). The externally rated fidelity check in this study highlighted that while the trial therapist were rated as highly competent on the CTRS, only four of the 13 available tapes appeared to suggest the use of specific CBT for psychosis techniques as described in the Fowler, Garety & Kuipers, Reference Fowler, Garety and Kuipers1995) manual. These observations highlighted the need for the use of a specific adherence tool that rates the techniques which are specific to CBT for psychosis.
Startup, Jackson & Pearce (Reference Startup, Jackson and Pearce2002) developed the CTPAS as a measure of adherence in CBT for psychosis as part of the North Wales trial of CBT in acute psychosis (Startup, Jackson & Bendix, Reference Startup, Jackson and Bendix2004) that was based on the use of Fowler, Garety & Kuipers (Reference Fowler, Garety and Kuipers1995) the cognitive behaviour manual. The item content was based on the techniques described in the manual. The scale lists 12 therapeutic activities considered to be core components of the manual. These include:
(a) Recognizing problems: relating to promoting insight and helping the client to recognize that he/she was experiencing problems.
(b) Assessing psychotic experiences: assessing the antecedents, consequences, quality and impact of the client's psychotic experiences.
(c) Enhancing self-regulatory strategies: helping improve coping strategies.
(d) Evidence for delusional beliefs: assess the evidence for delusional beliefs.
(e) Columbo style: carefully but non-intrusively questioning about the details of delusions and hallucinations.
(f) Developing a narrative perspective: helping the client construct a narrative account of their experiences underpinning delusional ideas and developing and exploring this narrative.
(g) Verbal challenge of delusions: challenge of the clients beliefs by reasoning.
(h) Validity testing: encouraging behavioural experiments of delusional ideas or thoughts relating to hallucinations.
(i) Developing a model of psychosis: working with the client to develop a shared understanding of the nature of the clients psychotic thinking and behaviour.
(j) Normalizing: helping the client to recognize that their symptoms are similar to those of many people both non-clinical and other people with psychosis.
(k) Resolving ambivalence: helping the client resolve ambivalence about different possible courses of action.
(l) Keeping well: help the client develop strategies for the active management of his/her psychotic problems in the future.
A detailed manual provided definitions of each item, and illustrative examples of both adherent and non-adherent therapy for each item. The frequency of each item within a given session was rated on a seven-point scale. The scale was designed to be used to externally rate audio-tape recordings of therapy sessions, and therapists ratings of their own session. Startup, Jackson & Pearce (Reference Startup, Jackson and Pearce2002) reported that the original 12 item CTPAS had satisfactory internal consistency and inter-rater reliability and that principal components analysis indicated two main factors ‘focus on delusions’ and ‘focus on problems’.
Rollinson et al. (Reference Rollinson, Haig, Warner, Garety, Kuipers, Freeman, Bebbington, Dunn and Fowler2007a, Reference Rollinson, Smith, Steel, Jolley, Onwumere, Garety, Kuipers, Freeman, Bebbington, Dunn, Startup and Fowlerb) subsequently revised this scale for use in the Cognitive therapy in Prevention of Relapse Trial Psychological prevention of Relapse in Psychosis (PRP) (Garety et al. Reference Garety, Fowler, Freeman, Bebbington, Dunn and Kuipers2008). The subsequent revised version of the scale (the R-CTPAS) contained 21 items that together allow all six components of the Fowler, Garety & Kuipers (Reference Fowler, Garety and Kuipers1995) therapy model to be assessed, including items to rate work on social disability, schemas and depression and anxiety (Rollinson et al. Reference Rollinson, Haig, Warner, Garety, Kuipers, Freeman, Bebbington, Dunn and Fowler2007a, Reference Rollinson, Smith, Steel, Jolley, Onwumere, Garety, Kuipers, Freeman, Bebbington, Dunn, Startup and Fowlerb). These items reflected the specific focus on relapse prevention work that was the primary focus of the PRP trial. A revised scoring system incorporates a measure of both therapist competence and adherence. The scale manual was also extended to provide an account of each therapeutic activity, criteria for assessing whether minimum competence criteria were met and examples of therapy dialogues above and below the minimum competence threshold. Treatment fidelity and treatment process research was an important aspect of the PRP trial (Garety et al. Reference Garety, Fowler, Freeman, Bebbington, Dunn and Kuipers2008). All therapy sessions were recorded subject to participant consent. Systematic samples of session recordings of each client were sent to external raters within the trial for monitoring of drift on a regular basis throughout the trial, this provided 185 observer-rated sessions. A further sample of 36 selected sessions was then rated by external expert raters to provide a reliability check on the internal ratings. Therapists also rated their own sessions, providing self-report data on 1028 sessions pertaining to 84 of the 133 participants allocated to CBT. Psychometric analysis of the scale was carried out using observer ratings (Rollinson et al. Reference Rollinson, Haig, Warner, Garety, Kuipers, Freeman, Bebbington, Dunn and Fowler2007a, Reference Rollinson, Smith, Steel, Jolley, Onwumere, Garety, Kuipers, Freeman, Bebbington, Dunn, Startup and Fowlerb). Good inter-rater reliability was also established, both across raters on the PRP trial and with external raters scoring tapes from other. A principal components analysis suggested the presence of three subscales; engagement and assessment work, formulation and schema level work and relapse prevention work. These subscales accounted for 36.74% of the variance within the total sample, and had alpha reliability coefficients of 0.55, 0.35 and 0.5, respectively.
These subscales were subsequently used to inform a summative coding frame that collated individual items into two simple summary scores reflecting the presence of different levels of therapeutic activity – ‘engagement and assessment work’ or ‘active intervention’ identified by individual item ratings on the R-CTPAS. Satisfactory reliability was established in identifying the presence of ‘engagement and assessment’ or ‘active intervention’ between therapist self-ratings using CTPAS and external tape ratings using CTPAS. From ratings of over a thousand therapy sessions across the treated group it was possible to reliably derive a single summative score was determined for the therapy delivered for each participant–therapist dyad to reflect the content of the sessions they received over the course of therapy. This showed considerable variation in the therapy provided. Of the 102 participants for whom sufficient data were collected, 42 (41.8%) were rated as receiving a full therapeutic dose of CBT while 39 (38.24%) were rated as receiving primarily engagement and assessment work or ‘partial therapy’. A further 21 (15.8%) attended only five sessions or less so were rated as having not received a dose of therapy at all. These observations have important clinical implications. They suggest that even very high-quality cognitive therapists who can clearly demonstrate skills in the use of cognitive therapy using the CTRS may still not necessarily be able to deliver full doses of cognitive therapy techniques to a substantive minority of patients with treatment-resistant and relapsing psychosis. This variation in delivery of therapy may be an important contributor to outcome.
In summary, the development of the CTPAS has been an important and useful development. The use of this scale allows reliable identification that therapists are not just delivering generic CBT skills adequately, but as importantly that they are delivering the specific techniques prescribed in the manual. It has been shown that it is possible to make reliable ratings of tapes using this scale. It has also been shown that it is possible for therapists to rate their own sessions using this scale, and also that summative scores of sessions using therapists ratings have adequate reliability with external tape ratings. A useful practical application of the R-CTPAS has been to allow comparison of therapy content across different settings. One such study compared therapy content in the PRP research trial with that seen in routine clinical practice in three different settings both in the UK and the USA (Rollinson, et al. Reference Rollinson, Haig, Warner, Garety, Kuipers, Freeman, Bebbington, Dunn and Fowler2007a). Overall, the level of competence as rated by the R-CTPAS did not differ between the two settings. However, there was a significant tendency for sessions in the research trial to be more likely to involve schema work, to use specific relapse prevention interventions and to more clearly use formulation-based interventions. This study suggested that case managers who received training workshops and some supervision but who have not received formal training in CBT could deliver some elements of CBT for psychosis (partial therapy) but were not able to deliver some of the more active techniques (full therapy).
In summary, three separate studies have indicated that there may be variations within therapy delivered by cognitive behaviour therapists (Durham et al. Reference Durham, Guthrie, Morton, Reid, Treliving, Fowler and Mcdonald2003; Rollinson et al. Reference Rollinson, Haig, Warner, Garety, Kuipers, Freeman, Bebbington, Dunn and Fowler2007a, Reference Rollinson, Smith, Steel, Jolley, Onwumere, Garety, Kuipers, Freeman, Bebbington, Dunn, Startup and Fowlerb; Garety et al. Reference Garety, Fowler, Freeman, Bebbington, Dunn and Kuipers2008). Observations from the CTPAS showed that only in certain cases was full therapy including active cognitive techniques consistently delivered; in other cases therapists were only able to deliver a partial form of the therapy involving a more basic set of the techniques. Our hypothesis was that the subgroup which engaged with and received full CBT-P would have better outcomes than those who received partial therapy or who dropped out. We have subsequently analysed the relationship between therapy delivery and outcome using the PRP dataset taking advantage of a novel statistical approach which overcomes biases in subgroup analysis. The results were wholly consistent with this hypothesis. Treatment was effective in the PRP study if, and only if, clients received full therapy. By contrast participants who only received therapy consisting predominantly of engagement and assessment work did not benefit, and neither did those who dropped out (Dunn et al., submitted). These findings are potentially very important. They deserve replication and highlight the importance of studying therapy adherence in future studies.
Adherence scales for at risk mental states and early psychosis
A specific adherence tool has been developed for cognitive therapy for at-risk mental states (the Cognitive Therapy for at Risk Psychosis Adherence Scale (CTRPAS)). This derives from the CTPAS but is an instrument designed to assess the adherence to the CBT manual for at-risk mental states (French & Morrison, Reference French and Morrison2004). Many of the items that address specific techniques to address psychotic symptoms (normalizing, generating alternatives, etc.) have similarities to those of the CTPAS, but here the focus is on the management of at-risk mental state symptoms that are below the threshold of psychosis. The targeting of behavioural activity in safety behaviours and social isolation is clearer. This adherence scale has been used in the Early Detection and Intervention Evaluation for people at high risk of psychosis (EDIE) trial (Morrison et al. in press) and in associated trials in Australia. A development in the use of adherence ratings in the EDIE has been an attempt to reliably identify adherence using expert ratings of case notes of sessions using the CTRPAS. This may be a useful and cost effective approach to the rating of adherence as it does not require the intensive and costly demands of therapy ratings that may not be feasible for large numbers of sessions. Reliability with external tape ratings and therapist's own ratings is currently being established. Methods for rating adherence to a therapy for people who have residual social disability after first episode psychosis are also in development (Fowler et al. Reference Fowler, Hodgekins, Painter, Reilly, Crane, Macmillan, Mugford, Croudace and Jones2009). Here again the emphasis is different. Techniques to manage psychotic symptoms (including treatment-resistant and low-level residual symptoms) may be used (in particular promotion of self-regulatory strategies). However, in this therapy the priority is increasing meaningful behavioural activity. The adherence scale emphasizes management of emotional dysfunction (in particular depression and social anxiety) and draws from the CTRPAS in focusing on social isolation and safety behaviours and low-level psychotic phenomena. The expected balance of techniques rated as present is therefore quite different than for CBT for treatment-resistant psychosis. It is possible and even likely that different techniques may be associated with different outcomes future adherence studies in CBT psychosis need to carefully ensure they are assessing those domains most relevant to their primary outcome.
Conclusion
The study of adherence and competence in trials of CBT for psychosis is maturing. Initially, the main aim was simply to demonstrate that the trial was carried out with sufficiently competent therapists and could therefore be regarded as a trial of CBT for psychosis. The first step in establishing treatment fidelity was to establish the presence of the use of adequately competent cognitive therapists by experts rating samples of tapes using the CTRS. However, subsequent observations of cognitive therapy in psychosis trials (Durham et al. Reference Durham, Guthrie, Morton, Reid, Treliving, Fowler and Mcdonald2003; Rollinson Reference Rollinson, Haig, Warner, Garety, Kuipers, Freeman, Bebbington, Dunn and Fowler2007a, Reference Rollinson, Smith, Steel, Jolley, Onwumere, Garety, Kuipers, Freeman, Bebbington, Dunn, Startup and Fowlerb; Garety et al. Reference Garety, Fowler, Freeman, Bebbington, Dunn and Kuipers2008) suggest that this is not sufficient. Even high-quality cognitive therapists who meet standards on CTRS may not necessarily be able to fully deliver the prescribed techniques of CBT for psychosis with challenging non-engaging cases. Ratings of both competence scales (CTRS) and adherence scales (CTPAS) need to be undertaken. Clearly, cognitive therapy for psychosis is a complex treatment carried out with a challenging client group often including non-engaging participants. Careful assessments of treatment fidelity are important in making sense of this complexity. Only by identifying exactly which procedures are carried out with which patients can we move on from the basic outcome question. Does it work? Toward the more clinically informative questions of which techniques work for which patients under which conditions?
