Development of a scale to assess formulation skills

The conventions for representing a case formulation symbolically in the form of a visual diagram are set out in the Supplementary material. Our study is an attempt to develop a scale for assessing a capacity to formulate in this manner. It focuses on nine dimensions of skill, as follows: (1) comprehensiveness of information pertaining to the presenting problem; (2) the relevance of information depicted in the diagram; (3) the reliable sourcing and accuracy of observations; (4) the extent to which inferences (hypotheses) are evidence-based and correctly interpreted; (5) a clear depiction of relationship and causal direction between observations (as antecedents, consequents, inhibitory, excitatory, etc.); (6) the provision of causal explanations for relationships (i.e. hypotheses); (7) the identification of contextual moderators of causal relationships; (8) the whole diagram as a coherent synthesis; and (9) the diagram prioritises aspects of the problem for intervention.

The 9-item scale (scoring range 0–18) was developed in earlier studies (Griffiths, Reference Griffiths2017; Pettman, Reference Pettman2017; Turner, Reference Turner2022) and was shown to have high internal consistency (alpha = 0.91) and moderate inter-rater reliability (0.57) when data were rated by skilled independent assessors. These are complex skills, and so we are attempting to refine the scale through further research. In this study, small modifications to the scale were agreed by the raters when working on the first four training diagrams (but not when the final test diagram was rated). Due to the guided nature of the training, the final item of the scale, synthesis and prioritisation, was not used to assess skills.

Training procedure

The training material consisted of written vignettes based on actual clients diagnosed with PTSD in a specialised NHS service. PTSD was selected for training purposes because a CBT model of this problem is widely known and informs practice (Ehlers and Clark, Reference Ehlers and Clark2000). It was a convenient vehicle for training participants to select and encode relevant information, not an attempt to teach this particular theoretical model for formulating PTSD. As with any disorder-specific model, idiosyncratic observations have to be collected and fitted to the general categories the model employs. The participants in this study were relatively naive to formulation because the majority were beginning their training as clinical psychologists (n=22). Due to COVID restrictions, the workshop was carried out online.

Participants completed two tasks in which a detailed written description of a client who was presenting with disturbing post-trauma experiences and behaviour was the basis for drawing up the formulation diagram. The first task was used for training; the second (a new case vignette) was for the evaluation of acquired skills.

The initial training tasks consisted of four steps, each followed by a test. The steps were: (1) accurate description and distinguishing observations from potential explanations (i.e. hypotheses); (2) identifying functional equivalences between observations (for both cues and behaviours); (3) recognising and representing mediators of behaviour, whether cognitive or otherwise; and (4) identifying and labelling observations understood to be acting as moderators of functional relationships that were already depicted in the participant’s diagram.

Following teaching and illustrative examples for each step, participants had to translate the formulation concept into a diagram with appropriately labelled symbol conventions (see Supplementary material for details). They were then provided with a hyperlink to a diagrammatic formulation that used our chosen conventions, and they were encouraged to compare this version with their own. (The focus here was on correct use of descriptive conventions, not to mimic an ‘ideal formulation’ of the client devised by the authors.) Participants’ diagrams could vary considerably without being penalised when rated for quality. The training procedure was intended to assist participants to independently identify and learn from their errors, mimicking a process that might otherwise take place in supervision. On completion of the training steps, a verbal description of a second test case vignette was presented, and each participant had to construct a formulation diagram using the conventions they had learned. Participants then uploaded photographs or screenshots of their diagrams and these were later independently rated by G.B. and R.H. on the items of a 9-item dimensional scale constructed for this purpose (see above and Supplementary material online for details).

At this stage of development of the scale, only percentage agreement between raters was calculated. Overall agreement (within 4 points) on summed scores was 72.7%, and ranged between 68 and 100% for individual items.