Hostname: page-component-77f85d65b8-zzw9c Total loading time: 0 Render date: 2026-04-20T05:12:04.490Z Has data issue: false hasContentIssue false
  • English
  • Français

Judging clinical competence using structured observation tools: A cautionary tale

Published online by Cambridge University Press:  15 April 2019

Anthony D. Roth ,
Pamela Myles-Hooton  and
Amanda Branson
Anthony D. Roth*
Affiliation:
Research Department of Clinical, Educational and Health Psychology, University College London, London, UK
Pamela Myles-Hooton
Affiliation:
Charlie Waller Institute, University of Reading, Reading, UK
Amanda Branson
Affiliation:
Charlie Waller Institute, University of Reading, Reading, UK
*
*Corresponding author. Email: a.roth@ucl.ac.uk
Get access Rights & Permissions [Opens in a new window]

Abstract

Background:

One method for appraising the competence with which psychological therapy is delivered is to use a structured assessment tool that rates audio or video recordings of therapist performance against a standard set of criteria.

Aims:

The present study examines the inter-rater reliability of a well-established instrument (the Cognitive Therapy Scale – Revised) and a newly developed scale for assessing competence in CBT.

Method:

Six experienced raters working independently and blind to each other’s ratings rated 25 video recordings of therapy being undertaken by CBT therapists in training.

Results:

Inter-rater reliability was found to be low on both instruments.

Conclusions:

It is argued that the results represent a realistic appraisal of the accuracy of rating scales, and that the figures often cited for inter-rater reliability are unlikely to be generalizable outside the specific context in which they were achieved. The findings raise concerns about the use of these scales for making summative judgements of clinical competence in both educational and research contexts.

Keywords

assessmentclinical supervisiontherapist competencetraining

Information

Type
Main
Information
Behavioural and Cognitive Psychotherapy , Volume 47 , Issue 6 , November 2019 , pp. 736 - 744
Copyright
© British Association for Behavioural and Cognitive Psychotherapies 2019 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Beck, A. T., Rush, A. J., Shaw, B. F. and Emery, G. (1979). Cognitive Therapy of Depression. New York, USA: Guilford Press.Google Scholar
Blackburn, I. M., James, I. A., Milne, D. L., Baker, C., Standart, S., Garland, A. and Reichelt, F. K. (2001). The Revised Cognitive Therapy Scale (CTS-R): psychometric properties. Behavioural and Cognitive Psychotherapy, 29, 431–446. doi: 10.1017/S1352465801004040 CrossRefGoogle Scholar
Dimidjian, S., Hollon, S. D., Dobson, K. S., Schmaling, K. B., Kohlenberg, R. J., Addis, M. E., et al. (2006). Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the acute treatment of adults with major depression. Journal of Consulting and Clinical Psychology, 74, 658–670. doi: 10.1037/0022-006X.74.4.658 CrossRefGoogle ScholarPubMed
Elkin, I. (1999). A major dilemma in psychotherapy outcome research: disentangling therapists from therapies. Clinical Psychology: Science and Practice, 6, 10–32. doi: 10.1093/clipsy.6.1.10 Google Scholar
Jacobson, N. S. and Gortner, E. T. (2000). Can depression be de-medicalized in the 21st century: scientific revolutions, counter-revolutions and the magnetic field of normal science. Behaviour Research and Therapy, 38, 103–117. doi: 10.1016/S0005-7967(99)00029-7 CrossRefGoogle ScholarPubMed
Kuyken, W. and Tsivrikos, D. (2004). Therapist competence, comorbidity and cognitive-behavioral therapy for depression. Psychotherapy and Psychosomatics, 78, 42–48. doi: 10.1159/000172619 CrossRefGoogle Scholar
Loades, M. E. and Armstrong, P. (2016). The challenge of training supervisors to use direct assessments of clinical competence in CBT consistently: a systematic review and exploratory training study. The Cognitive Behaviour Therapist, 9, e27. doi: 10.1017/S1754470X15000288 CrossRefGoogle Scholar
McManus, F., Rakovshik, S., Kennerley, H., Fennell, M. and Westbrook, D. (2012). An investigation of the accuracy of therapists’ self-assessment of cognitive-behaviour therapy skills. British Journal of Clinical Psychology, 51, 292–306. doi: 10.1111/j.2044-8260.2011.02028.x CrossRefGoogle ScholarPubMed
Muse, K. and McManus, F. (2013). A systematic review of methods for assessing competence in cognitive-behavioural therapy. Clinical Psychology Review, 33, 484–499. doi: 10.1016/j.cpr.2013.01.010 CrossRefGoogle ScholarPubMed
Muse, K., McManus, F., Rakovshik, S. and Thwaites, R. (2017). Development and psychometric evaluation of the Assessment of Core CBT Skills (ACCS): an observation-based tool for assessing cognitive behavioural therapy competence. Psychological Assessment, 29, 542–555. doi: 10.1037/pas0000372 CrossRefGoogle Scholar
Roth, A. D. (2016). A new scale for the assessment of competences in cognitive and behavioural therapy. Behavioural and Cognitive Psychotherapy, 44, 620–624. doi: 10.1017/S1352465816000011 CrossRefGoogle ScholarPubMed
Roth, A. D. and Pilling, S. (2008). Using an evidence-based methodology to identify the competences required to deliver effective cognitive and behavioural therapy for depression and anxiety disorders. Behavioural and Cognitive Psychotherapy, 36, 129–147. doi: 10.1017/S1352465808004141 CrossRefGoogle Scholar
Waltz, J., Addis, N. E., Koerner, K. and Jacobson, N. (1991) Testing the integrity of a psychotherapy protocol: assessment of adherence and competence. Journal of Consulting and Clinical Psychology, 61, 620–530. doi: 10.1037/0022-006X.61.4.620 CrossRefGoogle Scholar
Young, J. and Beck, A. T. (1980). Cognitive Therapy Scale: Rating Manual (unpublished manuscript).10.1037/t00834-000CrossRefGoogle Scholar
Submit a response

Comments

No Comments have been published for this article.