The main reasons and driving forces behind the introduction of WPBA

Despite large proportions of both trainees (43%) and trainers (44%) believing that WPBA was introduced with a view to improve training, the new framework has engendered overwhelmingly negative attitudes among both groups (Table 1). This is partly due a common perception (held by 41% of trainees and 36% of trainers) that WBPA is a politically motivated initiative with an unclear rationale and purpose.

It is interesting that only a small minority of trainees and trainers (5% and 6% respectively) feel that WPBA was introduced to overcome failures of the previous training system. Furthermore, only 9% of trainees and 8% of consultants feel that the system was introduced to improve patient care. Interestingly there is a statistically significant difference between the perceptions of the two groups (Fisher's exact value 10.618, associated P = 0.032), possibly due to the fact that a substantially larger proportion of trainers perceive the Royal College of Psychiatrists as an important driving force.

WPBA and its introduction

This overwhelmingly negative attitude is perhaps largely due to the manner in which WPBA was introduced (83% of trainees and 64% of trainers feel that it was not introduced in a well-thought-out manner) (online table DS1). It is also widely perceived as being conceptually flawed, based on scant evidence and of dubious validity as an assessment tool. Indeed, less than half of both groups expressed favourable opinions with regard to the validity of WPBA (42% of trainers, 23% of trainees). A similar picture emerges with regard to the propriety of WPBA having been rendered compulsory from the outset, this being endorsed by 46% of trainers and only 13% of trainees. This obvious lack of enthusiasm from the trainees is with little doubt related to the fact that the entire responsibility of managing one's WPBA portfolio has been placed on the shoulders of the trainee. Indeed, trainees consistently rated WPBA and its manner of introduction much worse than did trainers – and the differences between the groups are statistically significant on all parameters except one (i.e. whether WPBA is backed by good evidence (online table DS1).

Given the universal agreement that objectivity is essential to a fair, valid and reproducible system of assessment, it is interesting that the majority of trainees (81%) and trainers (71%) agree that WPBAs are not independent of assessors’ personal preferences (the difference between the two groups is statistically significant). Such subjectivity may stem from a number of factors including differences between individual trainers’ expectations, a lack of consensus among assessors, a lack of clear guidance on prescribed standards to be attained for each stage of training, inadequate training opportunities for assessors and indeed, an arguably inappropriate assumption that every assessor of any background (let alone a consultant trainer) is possessed of the ability to accurately, objectively and repeatably assess any given trainee at a level appropriate to that trainee's experience. This is possibly a fundamental flaw of any non-structured assessment process in which any given clinical scenario, circumstance and questions posed are not standardised.

Only a small minority in each group (8% and 20% of trainees and trainers respectively) feel that the introduction of WPBA has been accompanied by sufficient training, a significant shortfall given the relevance of such training to the introduction of any novel system.

Effect of WPBA

Trainees were vociferous in expressing overall dissatisfaction with the effects of WBPA, the majority holding the view that WPBA has had no real beneficial effects on supervision, training, clinical practice and confidence (online table DS2). Disappointingly a large number of trainers (from 19 to 42%) expressed no opinion on these four important parameters. Analysis of those who did however, suggests that the majority of trainers agree with the trainees in terms of perceiving little beneficial effect as a result of WPBA, although trainees rate the effects of WPBA far worse than do trainers, with differences that are statistically significant on all parameters except that of supervision (online table DS2). Given the rationale for the introduction of WPBA, the most striking feature here perhaps is that only a minority of trainees (19%) and trainers (33%) feel that WPBA has achieved its objective of improving training.

Practicalities relating to WPBAs

Over 70% of trainees feel that they encounter difficulties organising assessments, compared with 45% of trainers (online table DS3). This result however, requires interpretation in the light of the fact that the entire responsibility for maintaining a performance-based portfolio rests with the trainee, who is obliged to complete a given number of mandatory assessments over a limited time frame, an undeniably stressful task, given that non-fulfilment of these requirements carries the potential for a trainee to be failed at their Annual Review of Competence Progression. As discussed in the previous survey, ^{Reference Menon, Winston and Sullivan12} this process is therefore at considerable risk of degenerating into a tick-box/paper-pushing exercise, which nevertheless by its very nature, is likely to take priority over the acquisition of essential clinical experience, a sentiment expressed by numerous respondents.

Given that assessments being intrinsically time and resource intensive, both groups (70% of trainees, 63% of trainers) agree that without formal allocation of time for the increased levels of supervision and assessment of trainees (integral to the performance based portfolio) there will inevitably be negative implications on the time available for clinical duties.

In total 44% of trainees harbour concerns about lack of support from supervisors and colleagues and 79% have concerns about the lack of access to computer facilities in the context of assessments. We found that 32% of trainers appear to be dissatisfied over the issues surrounding support from their colleagues for WPBA, although trainers largely feel positive about ease of access to computers (56%), in sharp contrast with their trainees (79%).

Yet again, trainees portray a much less complimentary picture about the practicalities associated with assessments than do trainers, with differences that are statistically significant on all parameters with the exception of the impact on time available for clinical duties (online table DS3).

Assessment tools adopted by the Royal College of Psychiatrists

Numerous concerns were raised by trainees with regard to the assessment tools endorsed by the Royal College of Psychiatrists (online table DS4); 58% feel that they do not facilitate training, 46% feel that they are not easy to use and 32% feel that they are not relevant to psychiatry. Among trainers, however, a more encouraging picture emerges with regard to both the ease of use and relevance of the assessment tools (65% and 58% respectively of trainers expressing approval), although the overall view of whether WPBA affects training positively is not so encouraging (30%). Predictably trainees once again rated the assessment tools worse than did the trainers, the differences being statistically significant on two parameters, namely ease of use and facilitation of training (online table DS4).

Recording of assessments

A major source of frustration for trainees involved the online system for recording and storing assessments (HcAT), which was made compulsory from the outset. Immense difficulties rendered HcAT very unpopular – indeed, the majority of the trainees found it unacceptable (online table DS5b). Trainers on the other hand, expressed a considerably more positive outlook on an online system, 86%, 82% and 83% rating this as acceptable or better in terms of ease of use, reliability and time consumption respectively, the differences on all parameters between the two respondent groups being statistically significant (online table DS5a).

Trainers’ opinions were equally divided in terms of compliance and ease of completion of forms, although the majority felt that an online system would be more time intensive. This is in sharp contrast with the trainees, who were largely unimpressed with the online system of recording assessments, on the basis that in their collective opinion, recording assessments online would not render the process of WPBA any easier, compared with a paper-based system.

The trainers’ survey, however, took place some months after the trainees’, by which time HcAT had been abandoned in favour of ‘Assessments Online’, an updated system, introduced in September 2008. The new system appears to have possibly addressed some of the difficulties with HcAT, evidenced by the overall positive response from the trainers, although when asked specifically as to whether the current online system represented a major improvement over the previous system trainers were divided in their opinions, 53% perceiving no change, and the remainder an improvement.

Preferred system for recording assessments

There exists a statistically significant difference between trainees’ and trainers’ preferences with regard to the two methods of recording assessments (χ² = 41.096 with an associated P<0.001). Over half the trainees preferred a paper-based system of recording assessments (a smaller proportion prefer a combination system), whereas trainers largely preferred an online or combination system (Table 2). The major changes made to the online system during the time period between the two surveys however, arguably render invalid any comparative analysis of the attitudes of the two respondent groups.

Trainees’ perceptions of their assessors and trainers’ perceptions of themselves

For this question, responses of ‘acceptable’, ‘good’ and ‘very good’ were considered as being satisfactory, and ‘poor’ and ‘very poor’ as unsatisfactory. From the results it is evident that trainers almost unanimously rate themselves as satisfactory on the numerous parameters in this context (online table DS6a,b). A lower proportion of trainees, however, rated trainers as satisfactory on all parameters except the ability to assess accurately and impartially. Differences between the two groups are statistically significant on the parameters of accessibility/availability, the ability to provide constructive feedback, knowledge of WPBA, and understanding of HcAT, the differences being most marked on the last two parameters and such discrepancies could prove a stumbling block to the whole process at a fundamental level (online table DS6b).

Trainees’ views with regard to assessors from non-medical backgrounds (who, of relevance, have not undergone such assessments themselves) are however, not so complimentary with an overwhelming majority of trainees allocating poor scores. Worryingly, such assessors were rated very poorly on the three most important parameters, i.e. understanding of WPBA, the ability to assess accurately and impartially and the ability to provide constructive feedback. The views expressed by the consultant body in the context of the propriety of senior trainees being assessed by such individuals are interesting, with only 29% and 32% expressing disapproval at the concept of the assessors being nursing staff and social workers respectively, when one considers that consultants themselves have not been subjected to such assessments by non-medical staff.

Trainees’ and trainers’ overall perceptions about WPBA

Overall, between half and two-thirds of trainees feel that WPBA in its current form is unacceptable to them, does not accurately reflect their progress, is no better than the system adopted prior to Modernising Medical Careers, and is not the way forward and therefore should not be retained (Table 3). This contrasts with a much lower level of disapproval among the consultant body in these contexts, with the notable exception of the accuracy of reflection of trainees’ progress, in which area 46% of trainers expressed disapproval. Perhaps the most striking statistics for the latter group however, are those illustrating high levels of ambivalence as to whether WPBA in its existing form is being used appropriately, whether it represents a better system than the previous one and whether WPBA is the way forward, with 48%, 46% and 40% of trainers expressing no opinion. Despite this, interestingly, 55% of trainers feel that WPBA is acceptable to them. The two groups displayed statistically significant differences between their overall perceptions about WPBA on all the above parameters, the trainees’ collective opinion being worse than that of trainers (Table 3).

It is debatable as to whether individuals from non-medical backgrounds (who do not undergo a system of regular appraisal themselves) are qualified to assess the clinical acumen of postgraduate trainees. As with any principle, there are exceptions – few would argue that highly trained professionals such as psychotherapists and psychologists would not be qualified to assess trainees in their own areas of expertise. Apart from these exceptions however, the old adage ‘the eye does not see what the mind does not know’ would certainly apply to such assessments. Indeed, it would intuitively appear inappropriate that non-medical members of staff, with little understanding of medical training and the standards acceptable/mandatory at each level over a wide range of competencies should be involved in making formal assessments of medical personnel. Certainly there is little published evidence attesting to the efficacy, validity and propriety of doctors being assessed by non-medically qualified individuals.

Study strengths and limitations

With due consideration given to the intrinsic limitations of the study design, the remarkably high response rate from the trainees reflects the validity of the results obtained (in that they are truly representative of the views of Welsh psychiatric trainees) while also reflecting the strength of feeling among trainees on this subject. As a corollary, the response rate from trainers, although significantly lower and less impressive, is nevertheless respectable in the context of a cross-sectional survey and of sufficient magnitude to reasonably draw conclusions from. Furthermore, the trainees’ survey was conducted approximately a year after the introduction of WPBA and the trainers’ survey a few months later. Both surveys therefore involved respondents who had acquired first-hand practical experience with this system of assessment.

Questionnaire-based surveys are intrinsically prone to the limitations imposed by the narrow scope and potential inadequacy of the spectrum of selectable responses for any given question. The views of participants might therefore be inadequately expressed, because the questions and the spectrum of selectable answers might be incompatible with respondents’ personal views. These surveys sought to address this issue by giving respondents the opportunity to state, in free text, any further views they wished to express. This study design affords the advantage of a reduced level of ‘observer bias’ since the anonymity enjoyed by the respondents removes the constraints of social niceties and political correctness with regard to expressing their views freely.

Finally, the time interval between conducting the survey and publication of the findings carries the potential, albeit small, to detract from the relevance and applicability of the study results by the time of publication, although this would arguably apply to any study that finds its way into print.

Implications

Both surveys highlight concerns with regard to the current competency-based assessment framework that persist 2 years after its introduction. Dissatisfaction is widespread among trainees, the general consensus being that WPBA has proved a paper exercise no better than the previous training system that, while increasing their workload, has added little real value to their training. Trainers on the other hand, appear more diverse with regard to their views and opinions, although considerable numbers are similarly dissatisfied. Indeed, the most striking feature is that only a minority of the respondents (19% of trainees, 33% of trainers) feel that WPBA has achieved its objective of improving training.

The WPBA, in its current form, is plagued by numerous problems, including a generalised ambiguity surrounding assessments, a lack of clear and defined standards, a reluctance to award poor ratings to underperforming trainees and variability between assessors. There appears to be an almost unanimous perception that WPBA has not been accompanied by sufficient training – this is particularly surprising, considering the high profile enjoyed by this new initiative and the importance of education and training in this context. The implications are obvious when one considers that flawed assessments by incompetent and ineffectual assessors inaccurately reflect trainees’ abilities and performance, a point highlighted by the fact that trainees adjudged as competent in clinical skills at their WPBAs (and who therefore pass their Annual Review of Competence Progressions) often do not perform at levels consistent with their WPBA ‘scores’ at the Royal College's Clinical Assessment of Skills and Competences exam, where the pass rate has recently dropped to below a third. ^{Reference Sikdar14} Not uncommonly therefore, is encountered the ‘trainee with a portfolio of perfect WPBA scores, baffled by their failure to pass the Clinical Assessment of Skills and Competences exam’. ^{Reference Howard and Brittlebank15}

Structured training of assessors will be fundamental to addressing these weaknesses and to optimising the validity and objectivity of assessments. Encouragingly, the Royal College of Psychiatrists is taking measures to address concerns highlighted in recent surveys, including the provision of training for medical assessors. ¹⁶ This will, however, need to extend to assessors from all backgrounds – and undeniably, the provision of robust structured training to innumerable assessors from various disparate (non-medical) backgrounds will prove a logistic nightmare. Equally significant is current legislation that mandates the undertaking of formal training in order to maintain assessor status. ¹⁶ With little incentive for non-medical assessors to undergo such training, the implications are obvious.

Finally, ingrained perceptions are difficult to dispel – trainees (and perhaps trainers too) tend to view assessments as summative exercises rather than the formative experiences they are designed to be. It is not uncommon for trainees to view WPBAs as mini-examinations; poor ratings are therefore demoralising, so much so that trainees may defer assessments and/or preferentially seek out ‘soft’ assessors, who are more likely to be generous with their scoring. Remedying this will require a sea change at a very fundamental level for both assessors and assessees. Both groups of respondents in this survey have made a number of pertinent and relevant recommendations that merit careful consideration, coming from individuals at the ‘coal face’ with practical experience of WPBA.

The question as to whether WPBA has been incontrovertibly proven to be ‘fit for purpose’ therefore remains unanswered. The persistence of serious widespread and ongoing concerns 2 years after the introduction of WPBA (by which time one could reasonably expect that any teething problems would have been resolved) raises the stark question as to whether the fundamental problem involves the flawed implementation of an otherwise excellent system or alternatively whether WPBA, at its very heart, is fundamentally flawed. Can the ineffective original online system and the hurried implementation of WPBA be blamed entirely for the poor uptake and negative perceptions or are we ignorant, or turning a blind eye, to deeper problems intrinsic to WPBA? It is vital that relevant research in this context be supported to assess this basic question, along with ongoing stringent quality-assurance measures to constantly review and improve psychiatric training.

Appendix

Comments and criticisms by trainers.

‘It does not really inform us about how well are the trainee and training. It is just a form completing exercise.’

‘They are easy to complete but are essentially a meaningless bureaucratic exercise.’

‘I think the tools used are confusing. Despite some guidance I am not sure I am assessing in line with peers.’

‘We need a complete rethink of what we are trying to achieve.’

‘Pen and paper assessments are still much more immediate in the thick of clinical practice.’

‘The system has not be fully thought through and introduced in a chaotic manner without enough preparation. Improvements in all the above areas need to be made for it to be potentially useful.’

‘Due to the infrequency of ever being in a clinical setting at the same time as a trainee (i.e. they are rarely in out-patients or ward rounds) my knowledge of their abilities or otherwise is restricted to what they tell me they are doing and the WPBA seems to me to be a fig-leaf compared to actually seeing a trainee at work.’

‘I have attended WPBA training at Royal College of Psychiatrists tutor meetings and witnessed total disagreements between different assessors over whether a trainee (on DVD) had done well or not, with no way proposed to resolve the assessors differences. This hardly bodes well for consistency of training.’

‘ScrapWPBA and examine trainees in exams with external examiners. Trainees should keep a log book of clinical activity that is signed off. Trainees should be assessed by experienced doctors who have passed the MRCPsych; not any old MDT [multidisciplinary] member who has not been assessed themselves.’

‘May not be perfect but at least it is some sort of assessment which is more than we had before when people could just drift through training for years until problems were identified. The main problem is that WPBA only assess if trainees possess certain skills and can perform certain tasks – not whether they will actually do so in a real (i.e. not tested) environment.’

‘My concern is the tendency to turn this into a tick box exercise rather than look at the evidence base for how they should be used and their purpose. Further education of both trainers and trainees into their use and ways to improve their reliability and validity might be helpful to overcome this.’

‘Some guidance or consensuses from the college on rating trainees is necessary as in my experience trainer's have different expectations. I have been to training sessions but no consensus seems to exist as yet.’

‘No follow-up training or guidance since initial brief training. Doctors are doing the best they can but inevitably will be developing different ideas about what constitutes competency at certain levels of training.’

‘I always have to fill them in my personal time each evening as I have no time to complete during working day. Trainees are so busy trying to fill in forms, attending one course or another, or on rest periods due to EWTD [European Working Time Directive], their exposure to patients and continuity of treatment has been diluted and therefore their training is not as good. As for being a consultant after 6 years of this kind of run through training, I personally think this will be insufficient to prepare the trainees for life as a consultant.’

‘The online system is difficult to achieve when computer monitors are scarce. Guidance and training of assessors needs to improve – the current system makes results very unreliable. The short assessments are not really appropriate for senior trainees who need to be assessed on how they tackle difficult problems over time and manage a team. WPBA should be supplemented by other methods of assessment (like the old fashioned reference) which may perhaps be less reliable but might pick up issues missed by formal assessments and identify high flyers.’

‘The current system of WPBA is very difficult to bench mark against the average trainee. The level of knowledge acquired in our placement is so much greater for our trainees than for those who haven't done the placement. On this basis, it is frequently only possible to give the highest mark. The different types of assessment are also difficult to differentiate for our placement. All patients are seen at length and detailed discussion takes place with the trainee regarding each one. Trying to shoe horn what the trainee actually does (knowledge/skills attitudes acquired, etc.) into the boxes seems quite pointless and irrelevant. Of course we do it – as our excellent trainees cannot progress without it! However, it seems like a politically driven necessity introduced at a whim and without any evidence for its applicability. I'd recommend that it's replaced by something more individually tailored to both the trainee and the placement.’