Research on the use of MSF for doctors

Ramsey et al ^{Reference Ramsey, Wenrich, Carline, Inui, Larson and LoGerfo5} published a landmark study showing that it was feasible for internal medicine physicians to obtain peer assessments about their humanistic qualities, clinical practice and communication skills. They also came to important conclusions about the reliability of MSF – for example, that 11 peer ratings were needed to ensure a reliability coefficient of 0.7 (the minimum acceptable for workplace-based assessments (WPBAs)) and that the results were not substantially affected by the relationship between the rater and the person being rated nor by the method used to select the raters. The findings of this study also suggested that a doctor's medical knowledge (determined by examination marks) was not predictive of how peers subsequently rated their interpersonal relationships or communication skills.

The finding that reliable and valid MSF questionnaires can be developed and be feasible to use for assessing doctors has been replicated across settings and specialties. ^{Reference Lipner, Blank, Leas and Fortna2,Reference Archer, Norcini, Southgate, Heard and Davies6–Reference Violato, Lockyer and Fidler9} A number of systematic reviews have also been published, all of which conclude that MSF as a method of assessing communication skills, collegiality, humanism and professionalism in doctors has high reliability, validity and feasibility. ^{Reference Evans, Elwyn and Edwards10–Reference Donnon, Ansari, Alawi and Violato15}

The mini-PAT as an MSF tool

The mini-PAT is used by the Royal College of Psychiatrists as an MSF instrument for trainees. It is well known because of its widespread use in the Foundation Programme. ^{Reference Davies, Archer, Southgate and Norcini16,Reference Carr17} The mini-PAT was derived from the Sheffield Peer Review Assessment Tool (SPRAT) following a mapping exercise against the foundation curriculum, ^{Reference Archer, Norcini, Southgate, Heard and Davies6} thus ensuring its content validity. The SPRAT contains 24 questions assessing a doctor's competencies and professional attributes, and it maps directly on to General Medical Council (GMC) standards of good medical practice, ¹⁸ again establishing its content validity. These standards include good clinical care, maintaining good medical practice, teaching and training, appraising and assessing, relationships with patients and working with colleagues. The SPRAT was the first MSF tool validated in the UK for use by paediatric consultants as part of their appraisal. ^{Reference Archer, Norcini and Davies19} It has also been shown to be reliable, needing as few as four raters to determine whether a doctor is in difficulty or not (more in borderline situations), and feasible, taking only 5–6 minutes to use with good return rates (more than 70%). ^{Reference Davis and Archer20} It can also discriminate between the more and less experienced trainee. ^{Reference Archer, McGraw and Davies21}

In developing the mini-PAT, nine questions which did not map on to the curriculum for the Foundation Programme were removed from the SPRAT. These included questions relating to the management of complex patients and leadership. One question about probity and health was added while the free-text element and six-point scale (where 1 indicates ‘very poor’ and 6 indicates ‘very good’) remained unchanged. ^{Reference Archer, Norcini, Southgate, Heard and Davies6} The resulting mini-PAT was thought to reflect the importance for foundation doctors of developing communication skills, team work and other humanistic qualities in relation to patient care in addition to their medical knowledge. ^{Reference Abdulla3}

In his critical analysis of the mini-PAT, while accepting its content validity and feasibility, Abdulla stated that it ‘lacks sufficient field evaluation and has not gone through any stringent criteria that are required for the validation of an assessment tool’. ^{Reference Abdulla3} Data on the reliability and validity of 693 mini-PAT assessments on 553 foundation year 1 and 2 (F1/F2) doctors have subsequently been published. ^{Reference Archer, Norcini, Southgate, Heard and Davies6} The mean scores of the two groups were found to be significantly different when using the same criterion standard (i.e. expectation for F2 completion), with 19.6% of F1s and 5.6% of F2s being assessed as borderline or below the expectations for F2 completion. This was used as evidence of internal standardisation and construct validity, as was the finding that the trainees scored higher in the domains of working with colleagues and relationships with patients compared with the clinical skills domains. Overall, 53% of F1 doctors and 74% of F2 doctors could have been assessed by no more than 8 assessors based on their mean scores. Factor analysis revealed that the two main factors were humanistic qualities and clinical performance. The authors concluded that the mini-PAT was a valid and reliable MSF tool for assessing foundation doctors.

Use of the mini-PAT in child and adolescent psychiatry training

In child and adolescent psychiatry, the process when using the mini-PAT is as follows: twice a year, the trainee provides contact details of between 8 and 12 co-workers who see them on a frequent basis in a range of situations. These people and the trainee then complete the mini-PAT online. Presumably based on the findings of Archer et al, ^{Reference Archer, Norcini, Southgate, Heard and Davies6} it is suggested that at least 8 forms must be completed to ensure the assessment is reliable. There is, however, no research specifically related to the mini-PAT on the minimum number of assessors required to give a valid result. ^{Reference Abdulla3} The form uses a 6-point Likert-type rating scale. Trainees are rated according to the standard expected at each stage of training. A score of 4 corresponds to the expected standard, with higher or lower scores suggesting the trainee's performance is better or worse. ^{Reference Searle, Holsgrove and Brown22} The responses are analysed centrally and a report is then sent to the trainee's educational supervisor who delivers the feedback in person. ^{Reference Holsgrove23}

Potential issues with using MSF tools

Several issues that have been identified in relation to the use of MSF tools for medical practitioners in general are also relevant to their use in child and adolescent psychiatry. One is the trainee's choice of rater. Although several authors have found that MSF assessment is not necessarily biased by allowing the doctor to select their own raters, ^{Reference Ramsey, Wenrich, Carline, Inui, Larson and LoGerfo5,Reference Violato, Marini, Toews, Lockyer and Fidler24,Reference Durning, Cation, Markert and Pangaraaro25} others have found that factors such as the seniority, gender and profession of raters can significantly influence the assessment. For example, Archer et al ^{Reference Archer, McGraw and Davies21} found that consultant raters using the SPRAT gave significantly lower mean scores to paediatric trainees than more junior doctors did; similarly, Bullock et al ^{Reference Bullock, Hassell, Markham, Wall and Whitehouse26} found that consultants and senior nurses were more likely to give ‘concern’ ratings when assessing junior doctors than were peers or administrators. Thus, there is a trend for assessors to be more critical with increasing seniority. When considering the mini-PAT, Archer et al ^{Reference Archer, Norcini, Southgate, Heard and Davies6} found that assessors' scores were affected by their occupation, the length of time the trainee had been working with them, and the working environment. They suggested standardising the number of consultants used as raters by each trainee. These findings support the need for more detailed guidance in rater selection from the Royal College of Psychiatrists. Trainees are currently only advised that raters be chosen from a broad range of co-workers. ^{Reference Fitch, Malik, Lelliott, Bhugra and Andiappan4} In addition, Abdulla ^{Reference Abdulla3} suggests that selection bias can be reduced if the list of raters is discussed and agreed on beforehand with the trainee's supervisor.

Measurement errors, such as the central tendency and halo effect, can also occur and are particularly likely when behaviours which cannot be easily observed are being assessed. ^{Reference Johnson and Bibiana27} A particular issue for non-doctor raters is knowing what standards they should expect for a doctor at that stage in their training. In an attempt to reduce measurement errors, Abdulla ^{Reference Abdulla3} suggests better education for mini-PAT raters. This could be provided by the Royal College of Psychiatrists as part of their online mini-PAT package.

It has been shown that doctors' self-assessments do not correlate well with peer or patient ratings. ^{Reference Hall, Violato, Lewkonia, Lockyer, Fidler and Toews7,Reference Thomas, Gebo and Hellmann28} Violato & Lockyer ^{Reference Violato and Lockyer29} studied psychiatrists, internal medicine physicians and paediatricians, and found that all were inaccurate in assessing their own performance. Those psychiatrists who were rated by peers to be in the bottom quartile saw themselves as ‘average’, whereas the psychiatrists in the top quartile significantly underrated themselves. This indicates that poorly performing doctors often lack insight, not always accepting negative feedback from others and querying its validity. ^{Reference Sargeant, Mann, Sinclair, van der Vleuten and Metsemakers30} Overeem et al ^{Reference Overeem, Wollersheim, Driessen, Lombarts, van de Ven and Grol31} advise that trained facilitators should encourage trainees to reflect on MSF results and help them set concrete goals for improvement. Offering coaching to help trainees identify their strengths and weaknesses may help facilitate changes in performance. ^{Reference Miller and Archer32} Making the feedback highly structured can help trainees acknowledge feedback from all sources rather than just the medical scores which they tend to value more. ^{Reference Ramsey, Wenrich, Carline, Inui, Larson and LoGerfo5,Reference Weinrich, Carline, Giles and Ramsey33–Reference Ferguson, Wakeling and Bowie35} Although taking the mean of the scores may be the most reliable approach, ^{Reference Wilkinson, Crossley, Wragg, Mills, Cowan and Wade36} attention should also be given to the free-text comments which might highlight specific performance issues and which may also make the feedback more acceptable. ^{Reference Ferguson, Wakeling and Bowie35} These findings highlight the importance of the MSF feedback process, which should include the development of a relevant action plan in collaboration with the doctor.

It has been proposed that a single, generic MSF tool be used in the UK. ^{Reference Donaldson37} Research supporting this includes Violato & Lockyer's ^{Reference Violato and Lockyer29,Reference Violato and Lockyer38} study of the use of one MSF tool for internal medicine physicians, paediatricians and psychiatrists. Although they found no specialty differences in response rates or reliability, it is of note that of the items clustered into the same four factors across the specialties, for psychiatry the most discriminating factor was communication whereas for the other two specialties the most important was patient management. By contrast, Mackillop et al ^{Reference Mackillop, Crossley, Vivekananda-Schmidt, Wade and Armitage39} evaluated the use of a generic MSF tool across specialties and concluded that, although the generic content was appropriate for most specialties, some would benefit from specialty-specific content.

Does the mini-PAT suit the needs of trainees in child and adolescent psychiatry?

In child and adolescent psychiatry, the mini-PAT is currently used to assess trainees. Although the mini-PAT has content validity for foundation doctors, having been mapped against their curriculum, this does not necessarily mean it is also a valid tool for other grades or for use across specialties. In the making of the mini-PAT, some questions were removed from the SPRAT, namely those relating to management of complex patients and leadership. ^{Reference Archer, Norcini, Southgate, Heard and Davies6} However, these items are highly relevant to trainees in child and adolescent psychiatry. Davies et al ^{Reference Davies, Archer, Bateman, Dewar, Crossley and Grant40} modified the SPRAT for trainees in histopathology following a blueprinting exercise against the histopathology curriculum to establish content validity. They concluded that specialty-specific MSF is feasible and achieves satisfactory reliability. A similar approach blueprinting the SPRAT against the child and adolescent psychiatry competency-based curriculum ⁴¹ could therefore be considered. The SPRAT also requires fewer raters than the mini-PAT in order for the results to be sufficiently reliable, ^{Reference Archer, Norcini, Southgate, Heard and Davies6} thus adding to its potential suitability for child psychiatry trainees who often work in small teams.

Alternatively, a specialty-specific MSF instrument for child and adolescent psychiatry trainees could be developed, to reflect the differences in their practice compared with other specialties and the greater importance placed on communication, interpersonal skills, emotional intelligence and relationship building. ^{Reference Fitch, Malik, Lelliott, Bhugra and Andiappan4} Tools taking these attributes into account have been developed for use with consultant psychiatrists and have been found to be feasible to use as well as being reliable and valid. ^{Reference Lelliott, Williams, Mears, Andiappan, Owen and Reading42,Reference Violato, Lockyer and Fidler43} The child and adolescent psychiatry competency-based curriculum ⁴¹ gives details of intended learning outcomes (ILOs), which are either mandatory or selective, some of which tap into these areas. The ILOs range from those that are predominantly clinical (e.g. managing emergencies (mandatory), paediatric psychopharmacology (mandatory) and paediatric liaison (selective)) to those that focus on more humanistic skills (e.g. professionalism (mandatory) and establishing and maintaining therapeutic relationships with children, adolescents and families (mandatory)). The ILO on professionalism includes: ‘practicing Child and Adolescent Psychiatry in a professional and ethical manner; child and family centred practice; understanding the impact of stigma and other barriers to accessing mental health services and inter-professional and multi-agency working’. ⁴¹ Some of the necessary associated skills which trainees are expected to attain include: supervising junior psychiatric staff, working with colleagues within the team and with other agencies to put the child's needs as central, and acting as an advocate for the child. There is scope to develop this area of the curriculum even further; the American Board of Pediatrics (ABP) published guidelines for the teaching and evaluation of professionalism in paediatric residency programmes ⁴⁴ as well as standards of professional behaviour against which paediatricians, including those in training, can be evaluated. ^{Reference Fallat and Glover45} Both are of relevance to child and adolescent psychiatrists.

If developed, a child and adolescent psychiatry specialty-specific MSF instrument would need to map on to the relevant ILOs. It could also include feedback from patients and families (which is not currently routinely collected as part of the WPBAs) to reflect the need to balance the views of the child (who is the patient) with those of their carers.