Hostname: page-component-76fb5796d-9pm4c Total loading time: 0 Render date: 2024-04-29T10:44:04.628Z Has data issue: false hasContentIssue false
  • English
  • Français

Development and initial evaluation of a treatment integrity measure for low-intensity group psychoeducational interventions

Published online by Cambridge University Press:  28 November 2023

Jonah Gosling ,
Melanie Simmonds-Buckley ,
Stephen Kellett Open the ORCID record for Stephen Kellett [Opens in a new window] ,
Daniel Duffy  and
Katarzyna Olenkiewicz-Martyniszyn
Jonah Gosling
Affiliation:
Department of Psychology, University of Sheffield, Sheffield, UK
Melanie Simmonds-Buckley
Affiliation:
Department of Psychology, University of Sheffield, Sheffield, UK Rotherham Doncaster and South Humber NHS Foundation Trust, UK
Stephen Kellett*
Affiliation:
Department of Psychology, University of Sheffield, Sheffield, UK Rotherham Doncaster and South Humber NHS Foundation Trust, UK
Daniel Duffy
Affiliation:
Sheffield Health and Social Care NHS Foundation Trust, UK
Katarzyna Olenkiewicz-Martyniszyn
Affiliation:
Sheffield Health and Social Care NHS Foundation Trust, UK
*
Corresponding author: Stephen Kellett; Email: Stephen.Kellett@nhs.net
Get access Rights & Permissions [Opens in a new window]

Abstract

Background:

Despite the importance of assessing the quality with which low-intensity (LI) group psychoeducational interventions are delivered, no measure of treatment integrity (TI) has been developed.

Aims:

To develop a psychometrically robust TI measure for LI psychoeducational group interventions.

Method:

This study had two phases. Firstly, the group psychoeducation treatment integrity measure-expert rater (GPTIM-ER) and a detailed scoring manual were developed. This was piloted by n=5 expert raters rating the same LI group session; n=6 expert raters then assessed content validity. Secondly, 10 group psychoeducational sessions drawn from routine practice were then rated by n=8 expert raters using the GPTIM-ER; n=9 patients also rated the quality of the group sessions using a sister version (i.e. GPTIM-P) and clinical and service outcome data were drawn from the LI groups assessed.

Results:

The GPTIM-ER had excellent internal reliability, good test–retest reliability, but poor inter-rater reliability. The GPTIM-ER had excellent content validity, construct validity, formed a single factor scale and had reasonable predictive validity.

Conclusions:

The GPTIM-ER has promising, but not complete, psychometric properties. The low inter-rater reliability scores between expert raters are the main ongoing concern and so further development and testing is required in future well-constructed studies.

Keywords

AdherenceCompetenceGroup therapyImproving Access to Psychological Therapies Programme (IAPT)Psychoeducation
Type
Main
Information
Behavioural and Cognitive Psychotherapy , Volume 52 , Issue 3 , May 2024 , pp. 317 - 330
Copyright
© The Author(s), 2023. Published by Cambridge University Press on behalf of British Association for Behavioural and Cognitive Psychotherapies

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Arifin, W. A. (2018). A web-sample size calculator for reliability studies. Education in Medical Journal, 10. https://doi.org/10.21315/eimj2018.10.3.8 CrossRefGoogle Scholar
Bartlett, M. S. (1950). Tests of significance in factor analysis. British Journal of Psychology. https://doi.org/10.1111/j.2044-8317.1950.tb00285.x CrossRefGoogle Scholar
Bartko, J. J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin, 83, 762–765.CrossRefGoogle Scholar
Bastiampillai, T. J., Allison, S., Harford, P., Perry, S. W., & Wong, M. L. (2019). Has the UK Improving Access to Psychological Therapies programme and rising antidepressant use had a public health impact? Lancet Psychiatry, 6, e8–e9. https://doi.org/10.1016/S2215-0366(19)30040-9.CrossRefGoogle ScholarPubMed
Bennett-Levy, J., Richards, D. A., Farrand, P., Christensen, H., Griffiths, K. M., Kavanagh, D. J., & Proudfoot, J. (2010). Low intensity CBT interventions: a revolution in mental health care. Oxford Guide to Low Intensity CBT Interventions, 3, 18.Google Scholar
Blackburn, I.-M., James, I. A., Milne, D. L., Baker, C., Standart, S., Garland, A., & Reichelt, F. K. (2001). The revised cognitive therapy scale (CTS-R): psychometric properties. Behavioural and Cognitive Psychotherapy, 29, 431–446. https://doi.org/10.1017/S1352465801004040 CrossRefGoogle Scholar
Bobak, C., Barr, P. & O’Malley, A. (2018). Estimation of an inter-rater intra-class correlation coefficient that overcomes common assumption violations in the assessment of health measurement scales. BMC Medical Research Methodology, 18, 93. https://doi.org/10.1186/s12874-018-0550-6 CrossRefGoogle ScholarPubMed
Boyle, K., Deisenhofer, A. K., Rubel, J. A., Bennemann, B., Weinmann-Lutz, B., & Lutz, W. (2020). Assessing TI in personalized CBT: the inventory of therapeutic interventions and skills. Cognitive Behaviour Therapy, 49, 210–227. https://doi.org/10.1080/16506073.2019.1625945 CrossRefGoogle Scholar
Brown, N. W. (2018). Conducting Effective and Productive Psychoeducational and Therapy Groups: A Guide for Beginning Group Leaders. Taylor & Francis.CrossRefGoogle Scholar
Burns, P., Kellett, S., & Donohoe, G. (2016). ‘Stress Control’ as a large group psychoeducational intervention at Step 2 of IAPT services: acceptability of the approach and moderators of effectiveness. Behavioural and Cognitive Psychotherapy, 44, 431–443. https://doi.org/10.1017/S1352465815000491 CrossRefGoogle ScholarPubMed
Bulmer, M. G. (1979). Principles of Statistics. Courier Corporation.Google Scholar
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284. https://doi.org/10.1037/1040-3590.6.4.284 CrossRefGoogle Scholar
Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98. https://doi.org/10.1037/0021-9010.78.1.98 CrossRefGoogle Scholar
Clark, D., Layard, R., Smithies, R., Richards, D. R., Suckling, R., & Wright, W. (2009). Improving access to psychological therapy: initial evaluation of two UK demonstration sites. Behaviour Research and Therapy, 47, 910–920. https://doi.org/10.1016/j.brat.2009.07.010 CrossRefGoogle ScholarPubMed
Clark, D. M. (2011). Implementing NICE guidelines for the psychological treatment of depression and anxiety disorders: the IAPT experience. International Review of Psychiatry, 23, 318–327. https://doi.org/10.3109/09540261.2011.606803 CrossRefGoogle ScholarPubMed
Clark, D. M., Canvin, L., Green, J., Layard, R., Pilling, S., & Janecka, M. (2018). Transparency about the outcomes of mental health services (IAPT approach): an analysis of public data. The Lancet, 391, 679–686. https://doi.org/10.1016/S0140-6736(17)32133-5 CrossRefGoogle ScholarPubMed
Crane, R. S., Eames, C., Kuyken, W., Hastings, R. P., Williams, J. M., Bartley, T., Evans, A., Silverton, S., Soulsby, J. G., & Surawy, C. (2013). Development and validation of the Mindfulness-Based Interventions – Teaching Assessment Criteria (MBI:TAC). Assessment, 20, 681–688. doi: 10.1177/1073191113490790 CrossRefGoogle ScholarPubMed
Cristobal, E., Flavian, C., & Guinaliu, M. (2007). Perceived e-service quality: measurement validity and effects on consumer satisfaction and web site loyalty. Managing Service Quality, 17, 317–340. https://doi.org/10.1108/09604520710744326 CrossRefGoogle Scholar
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334. https://doi.org/10.1007/BF02310555 CrossRefGoogle Scholar
Cuijpers, P., Muñoz, R. F., Clarke, G. N., & Lewinsohn, P. M. (2009). Psychoeducational treatment and prevention of depression: the ‘Coping with Depression’ course thirty years later. Clinical Psychology Review, 29, 449–458. https://doi.org/10.1016/j.cpr.2009.04.005 CrossRefGoogle ScholarPubMed
Delgadillo, J., Kellett, S., Ali, S., McMillan, D., Barkham, M., Saxon, D., Donohoe, G., Stonebank, H., Mullaney, S., Eschoe, P., Thwaites, R., & Lucock, M. (2016). A multi-service practice research network study of large group psychoeducational cognitive behavioural therapy. Behaviour Research and Therapy, 87, 155–161. doi: 10.1016/j.brat.2016.09.010 CrossRefGoogle Scholar
Dolan, N., Simmonds-Buckley, M., Kellett, S., Siddell, E., & Delgadillo, J. (2021). Effectiveness of stress control large group psychoeducation for anxiety and depression: systematic review and meta-analysis. British Journal of Clinical Psychology, 60, 375–399. doi: 10.1111/bjc.12288.CrossRefGoogle ScholarPubMed
Dreyfus, H. L. (1989). The Dreyfus model of skill acquisition. In Burke, J. (ed), Competency Based Education and Training. Falmer Press Google Scholar
Dodge, Y. (2008). The Concise Encyclopedia of Statistics. Springer Science & Business Media.Google Scholar
Field, A. (2013). Discovering Statistics Using IBM SPSS Statistics. Sage.Google Scholar
Ghaemian, A., Ghomi, M., Wrightman, M. and Ellis-Nee, C. (2020). Therapy discontinuation in a primary care psychological service: why patients drop out. the Cognitive Behaviour Therapist, 13.CrossRefGoogle Scholar
Gordon, P. K. (2006). A comparison of two versions of the Cognitive Therapy Scale. Behavioural and Cognitive Psychotherapy, 35, 343–353. https://doi.org/10.1017/S1352465806003390 CrossRefGoogle Scholar
Green, H., Barkham, M., Kellett, S., & Saxon, D. (2014). Therapist effects and IAPT Psychological Wellbeing Practitioners (PWPs): a multilevel modelling and mixed methods analysis. Behavior Research and Therapy, 63, 43–54. doi: 10.1016/j.brat.2014.08.009 CrossRefGoogle ScholarPubMed
Gorusch, R. L. (1983). Factor Analysis (2nd edn). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
Hadjistavropoulos, H. D., Pugh, N. E., Hesser, H., & Andersson, G. (2017). Therapeutic alliance in internet-delivered CBT for depression or generalized anxiety. Clinical Psychology & Psychotherapy, 24, 451–461. doi: 10.1002/cpp.201 CrossRefGoogle ScholarPubMed
Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: an overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8, 23. https://doi.org/10.20982/tqmp.08.1.p023 CrossRefGoogle ScholarPubMed
Hopkins, K. D., & Weeks, D. L. (1990). Tests for normality and measures of skewness and kurtosis: their place in research reporting. Educational and Psychological Measurement, 50, 717–729. https://doi.org/10.1177/0013164490504001 CrossRefGoogle Scholar
Hulin, C., Netemeyer, R., & Cudeck, R. (2001). Can a reliability coefficient be too high? Journal of Consumer Psychology, 10, 55–58. https://doi.org/10.1207/S15327663JCP1001&2_05 CrossRefGoogle Scholar
Kellett, S., Simmonds-Buckley, M., Limon, E., Hague, J., Hughes, L., Stride, C., & Millings, A. (2021). Defining the assessment and treatment competencies to deliver low-intensity cognitive behavior therapy: a multi-center validation study. Behavior Therapy, 52, 15–27. https://doi.org/10.1016/j.beth.2020.01.006 CrossRefGoogle ScholarPubMed
Keyworth, C., Epton, T., Goldthorpe, J., Calam, R., Armitage, C. J. (2020). Acceptability, reliability, and validity of a brief measure of capabilities, opportunities, and motivations (‘COM-B’). British Journal of Health Psychology, 25, 474–501. doi: 10.1111/bjhp.12417 CrossRefGoogle ScholarPubMed
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15, 155–163. https://doi.org/10.1016/j.jcm.2016.02.012 CrossRefGoogle ScholarPubMed
Kroenke, K., Spitzer, R. L., & Williams, J. B. (2001). The PHQ-9: validity of a brief depression severity measure. Journal of General Internal Medicine, 16, 606–613. https://doi.org/10.1046/j.1525-1497.2001.016009606.x CrossRefGoogle ScholarPubMed
Kühne, F., Heinze, P. & Weck, F. (2020). Standardized patients in psychotherapy training and clinical supervision: study protocol for a randomized controlled trial. Trials, 21, 276. https://doi.org/10.1186/s13063-020-4172-z CrossRefGoogle ScholarPubMed
Landsverk, J. (2013). Reflections on TI in a dissemination and implementation framework. Clinical Psychology: Science and Practice, 20, 114. https://doi.org/10.1111/cpsp.12027 Google Scholar
Le, K. (2022). Pre-recorded lectures, live online lectures, and student academic achievement. Sustainability, 14, 2910. https://doi.org/10.3390/su14052910 CrossRefGoogle Scholar
Lee, K. M., Lee, J., Chung, C. Y., Ahn, S., Sung, K. H., Kim, T. W., … & Park, M. S. (2012). Pitfalls and important issues in testing reliability using intraclass correlation coefficients in orthopaedic research. Clinics in Orthopedic Surgery, 4, 149–155. https://doi.org/10.4055/cios.2012.4.2.149 CrossRefGoogle ScholarPubMed
Lukens, E. P., & McFarlane, W. R. (2004). Psychoeducation as evidence-based practice: considerations for practice, research, and policy. Brief Treatment and Crisis Intervention, 4, 205–225. https://doi.org/10.1093/brief-treatment/mhh019 CrossRefGoogle Scholar
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35, 382–385. https://doi.org/10.1097/00006199-198611000-00017 CrossRefGoogle ScholarPubMed
McCay, E., Carter, C., Aiello, A., Quesnel, S., Howes, C., & Johansson, B. (2016). Toward TI: developing an approach to measure the TI of a dialectical behavior therapy intervention with homeless youth in the community. Archives of Psychiatric Nursing, 30, 568–574. https://doi.org/10.1016/j.apnu.2016.04.001 CrossRefGoogle Scholar
Michie, S., Ashford, S., Sniehotta, F., Dombrowski, S., Bishop, A., & French, D. (2011). A refined taxonomy of behaviour change techniques to help people change their physical activity and healthy eating behaviours: the CALO-RE taxonomy. Psychology & Health, 26, 1479–1498. https://doi.org/10.1080/08870446.2010.540664 CrossRefGoogle ScholarPubMed
Muse, K., & McManus, F. (2013). A systematic review of methods for assessing competence in cognitive–behavioural therapy. Clinical Psychology Review, 33, 484–499. https://doi.org/10.1016/j.cpr.2013.01.010 CrossRefGoogle ScholarPubMed
Muse, K., McManus, F., Rakovshik, S., & Thwaites, R. (2017). Development and psychometric evaluation of the Assessment of Core CBT Skills (ACCS): an observation-based tool for assessing cognitive behavioral therapy competence. Psychological Assessment, 29, 542. https://doi.org/10.1037/pas0000372 CrossRefGoogle ScholarPubMed
National Collaborating Centre for Mental Health (2018). The Improving Access to Psychological Therapies Manual: Appendices and Helpful Resources. Retrieved from: england.nhs.uk/wp-content/uploads/2018/06/the-iapt-manual-v5.pdf Google Scholar
Noble, L. A., Firth, N., Delgadillo, J., & Kellett, S. (2021). An investigation of the competencies involved in the facilitation of CBT-based group psychoeducational interventions. Behavioural and Cognitive Psychotherapy, 16, 1–13. doi: 10.1017/S1352465821000084 Google Scholar
Pierson, H. M., Hayes, S. C., Gifford, E. V., Roget, N., Padilla, M., Bissett, R., & Fisher, G. (2007). An examination of the motivational interviewing treatment integrity code. Journal of Substance Abuse Treatment, 32, 11–17. https://doi.org/10.1016/j.jsat.2006.07.001 CrossRefGoogle ScholarPubMed
Perepletchikova, F. (2011). On the topic of treatment integrity. Clinical Psychology: Science and Practice, 18, 148–153. https://doi.org/10.1111/j.1468-2850.2011.01246.x Google ScholarPubMed
Perepletchikova, F., Treat, T. A., & Kazdin, A. E. (2007). Treatment integrity in psychotherapy research: analysis of the studies and examination of the associated factors. Journal of Consulting and Clinical Psychology, 75, 829. https://doi.org/10.1037/0022-006X.75.6.829 CrossRefGoogle ScholarPubMed
Perepletchikova, F., & Kazdin, A. E. (2005). Treatment integrity and therapeutic change: issues and research recommendations. Clinical Psychology: Science and Practice, 12, 365. https://doi.org/10.1093/clipsy.bpi045 Google Scholar
Polit, D. E. & Beck, C. T. (2006). Essentials of Nursing Research (6th edn). Lippincott Williams & Wilkins, Philadelphia.Google Scholar
Portney, L. G., & Watkins, M. P. (2009). Foundations of Clinical Research: Applications to Practice (vol. 892). Upper Saddle River, NJ: Pearson/Prentice Hall.Google Scholar
Power, N., Noble, L.A., Simmonds-Buckley, M., Kellett, S., Stockton, C., Firth, N., & Delgadillo, J. (2022). Associations between treatment adherence-competence-integrity (ACI) and adult psychotherapy outcomes: a systematic review and meta-analysis. Journal of Consulting and Clinical Psychology, 90, 427–445. doi: 10.1037/ccp0000736 CrossRefGoogle ScholarPubMed
Richards, D., & Whyte, M. (2009). Reach Out: National Programme Student Materials to Support the Delivery of Training for Psychological Wellbeing Practitioners Delivering Low Intensity Interventions. Rethink Mental Illness.Google Scholar
Roth, A., & Pilling, S. (2007). The Competences Required to Deliver Effective Cognitive and Behavioural Therapy for People with Depression and with Anxiety Disorder. Department of Health.Google Scholar
Shafran, R., Myles-Hooton, P., Bennett, S., & Öst, L. G. (2021). The concept and definition of low intensity cognitive behaviour therapy. Behaviour Research and Therapy, 138, 103803. doi: 10.1016/j.brat.2021.103803.CrossRefGoogle ScholarPubMed
Sharpless, B. A., & Barber, J. P. (2009). A conceptual and empirical review of the meaning, measurement, development, and teaching of intervention competence in clinical psychology. Clinical Psychology Review, 29, 47–56. https://doi.org/10.1016/j.cpr.2008.09.008 CrossRefGoogle ScholarPubMed
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86, 420. https://doi.org/10.1037/0033-2909.86.2.420 CrossRefGoogle ScholarPubMed
Southam-Gerow, M. A., & McLeod, B. D. (2013). Advances in applying TI research for dissemination and implementation science: introduction to special issue. Clinical Psychology: Science and Practice, 20, 1. https://doi.org/10.1111/cpsp.12019 Google Scholar
Spitzer, R. L., Kroenke, K., Williams, J. B., & Löwe, B. (2006). A brief measure for assessing generalized anxiety disorder: the GAD-7. Archives of Internal Medicine, 166, 1092–1097. https://doi.org/10.1001/archinte.166.10.1092 CrossRefGoogle ScholarPubMed
Tennant, P. W. G., Arnold, K. F., Ellison, G. T. H., & Gilthorpe, M. S. (2022). Analyses of ‘change scores’ do not estimate causal effects in observational data. International Journal of Epidemiology, 51, 1604–1615. https://doi.org/10.1093/ije/dyab050 CrossRefGoogle Scholar
Thompson, M. Parker, H. & Cave, C. (2021). Exploring which aspects of a low-intensity CBT intervention were found to contribute to a successful outcome from the service user point of view: a mixed methods study. Counselling and Psychotherapy Research, 22, 279–291.CrossRefGoogle Scholar
University College London (2015). National Curriculum for the Education of Psychological Wellbeing Practitioners. Department of Health.Google Scholar
Van Der Vleuten, C. P. M. (1996). The assessment of professional competence: developments, research and practical implications. Advances in Health Science Education, 1, 41–67. https://doi.org/10.1007/BF00596229 CrossRefGoogle ScholarPubMed
Wakefield, S., Kellett, S., Simmonds-Buckley, M., Stockton, D., Bradbury, A., & Delgadillo, J. (2021). Improving Access to Psychological Therapies (IAPT) in the United Kingdom: a systematic review and meta-analysis of 10-years of practice-based evidence. British Journal of Clinical Psychology, 60, 1–37. https://doi.org/10.1111/bjc.12259 CrossRefGoogle ScholarPubMed
Walter, S.D., Eliasziw, M. & Donner, A. (1998). Sample size and optimal designs for reliability studies. Statistical Methods, 17, 101–110. http://www.ncbi.nlm.nih.gov/pubmed/9463853 Google ScholarPubMed
Waltz, C. F., & Bausell, R. B. (1981). Nursing Research: Design, Statistics, and Computer Analysis. FA Davis Company.Google Scholar
Weck, F., Hilling, C., Schermelleh-Engel, K., Rudari, V., & Stangier, U. (2011). Reliability of adherence and competence assessment in cognitive behavioral therapy: influence of clinical experience. Journal of Nervous and Mental Disease, 199, 276–279. https://doi.org/10.1097/NMD.0b013e3182124617 CrossRefGoogle ScholarPubMed
White, J., & Keenan, M. (1990). Stress control: a pilot study of large group therapy for generalized anxiety disorder. Behavioural and Cognitive Psychotherapy, 18, 143–146. https://doi.org/10.1017/S0141347300018267 CrossRefGoogle Scholar
Wykes, C. (2013). Are gains made in IAPT psychoeducational groups maintained over time? A qualitative study. D Clin Psy thesis, University College London.Google Scholar
Yeaton, W. H., & Sechrest, L. (1981). Critical dimensions in the choice and maintenance of successful treatments: strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49, 156–167. https://doi.org/10.1037//0022-006X.49.2.156 CrossRefGoogle ScholarPubMed
Yusoff, M. S. B. (2019). ABC of response process validation and face validity index calculation. Education in Medicine Journal, 11, 55–61. https://doi.org/10.21315/eimj2019.11.3.6 CrossRefGoogle Scholar
Young-Southward, G., Jackson, A., & Dunan, J. (2020). Group CBT for mild to moderate depression and anxiety: an evaluation of patient satisfaction within a primary care mental health team. the Cognitive Behaviour Therapist, 13.CrossRefGoogle Scholar
Zechariah, S., Waller, J. L., Stallings, J., Gess, A. J., & Lehman, L. (2022). Inter-rater and intra-rater reliability of the INSPECT (Interactive Nutrition Specific Physical Exam Competency Tool) measured in multi-site acute care settings. Healthcare, 10, 212. doi: 10.3390/healthcare10020212 CrossRefGoogle ScholarPubMed
Supplementary material: File

Gosling et al. supplementary material

Gosling et al. supplementary material

Download Gosling et al. supplementary material(File)
File 412 KB
Submit a response

Comments

No Comments have been published for this article.