In spring 2003 the Royal College of Psychiatrists changed the format of the clinical component of Part I of the membership examination (Reference Tyrer and OyebodeTyrer & Oyebode, 2004). The abandonment of the traditional ‘long case’ format in favour of the newer objective structured clinical examination (OSCE) was perhaps the most challenging alteration since the introduction of the critical review paper in Part II. This challenge extends both to trainees and to those responsible for training them — particularly those charged with providing an MRCPsych course which is up-to-date and relevant. Trainers and course organisers must themselves embrace potentially unfamiliar formats, and modify the teaching and examination preparation offered.
Naeem et al (Reference Naeem, Rutherford and Kenn2004) have described their experience in running OSCE ‘workshops’. We similarly provide regular OSCE practice in fortnightly tutorials for our Part I trainees. However, another time-honoured element in the process of preparation is the ‘mock’ examination. The South East Thames MRCPsych course has offered mock clinicals for both Part I and Part II for many years, and the current course organisers were concerned to provide a realistic mock examination in the new OSCE format in advance of its introduction in spring 2003. We have now run three consecutive mock OSCEs (a total of 78 individual candidate sittings), and our evaluation of the process reveals our approach to be successful and valued by participants.
Planning the project
Thorough planning and an enthusiastic team of organisers are essential, and planning should commence at least 6 months before the anticipated date of your mock. Box 1 shows the overall time scale for the project.
Box 1. Countdown to a mock OSCE
|
The timing should be arranged to be of most relevance to the trainees, just after they have completed the written paper of the Part I examination. Much earlier than this, and trainees still studying for the written paper might see the OSCE as too distant. Much later, and the experience might count against them, provoking anxiety while not allowing the trainees enough time to learn from it and change their practice if required.
Setting the date early is necessary to allow an appropriate venue to be identified and booked, large enough to accommodate the full circuit of at least 12 OSCE stations, with a reception area and separate large rooms for candidate, examiner and role-player briefings. Having a venue with a previous record of hosting OSCEs, perhaps for medical students, is a definite advantage.
Blueprinting
Blueprinting is the process by which the circuit of individual stations (the examples on the College website may be used as a template; http://www.rcpsych.ac.uk/traindev/exams/regulation/osce.htm) is planned to assess an adequate mix of clinical skills. In blueprinting its own examination, one assumes the College will aim to ensure candidates possess the essential skills required by junior psychiatrists, by having them demonstrate an appropriately comprehensive range under direct observation. The aim in blueprinting for a mock examination, however, is to try to predict what the College will do. This should become easier as further information enters the public domain from candidates who have sat the real OSCE.
A variety of skills — history taking, mental state examination, interpreting investigations, and so on — should be assessed across an appropriate range of conditions (such as schizophrenia, depression and dementia). Communication will be assessed in most stations, but in some it will be the principal skill being tested. In others, largely practical skills such as fundoscopy or physical examination might be central.
The circuit should consist of at least 12 stations, the number presented in the real examination. Twelve stations would allow a maximum of 12 candidates to be tested at any one sitting, with examiners and role players working without a break. If role players are simulating anger or hypomania, this would be exhausting. Providing rest stations, where candidates do not have any task for 7 min, allows either extra candidates or gaps in the stream of candidates, so that each examiner and role player has a rest at least once during the examination. A gap is essential if paired stations are used — for example, where a history is taken in one station and presented to a ‘ consultant’ in the next station along - so that no one starts on the second of the paired stations. This means reducing the candidate numbers by one, or adding an extra rest station.
Role players
Candidates are observed demonstrating their key skills in a live simulated clinical situation, so people are needed to play the parts of patients, relatives or even other health professionals, such as a consultant or community nurse. It might be tempting to use volunteers or other nonprofessionals to play the patients in the OSCE stations, because of the limited availability of (or expense of hiring) professionals. It would be equally easy to underestimate the demands of training them to portray the clinical situation accurately and respond in a realistic but - crucially - consistent fashion to the varying approaches of the candidates (Reference Wallace, Rao and HaslamWallace et al, 2002). In our view, the benefits of using experienced professionals who have had training in simulating mental illness and the experience of playing patients in OSCEs with the reliability and consistency required are inestimable.
It is important to provide the role players with detailed instructions in advance, covering their character's demographic details, full background history, context of the scenario and how to interact with the candidates. Giving any less information would risk the need for improvisation if a candidate veered off topic, and standardisation and consistency would be threatened.
Examiners
Most examiners will be volunteer consultants or specialist registrars, who will need plenty of warning in advance of the date. One examiner is needed per station, with others present as floating external examiners (as a check on marking consistency) and reserves. As they may not be College examiners themselves, and may have had little experience of sitting OSCEs, never mind marking them, they should be briefed on how to mark just before the session. This briefing should emphasise the limits of the format, especially the timing, as candidates are often asked to complete quite demanding tasks in only 7 min, and should explain that the construct — the written paragraph at the top of the mark sheet - sets out the parameters of what is to be assessed. As this is a postgraduate examination, ‘checklist’ style marking is not appropriate (Reference Hodges, Regehr and McnaughtonHodges et al, 1999). Rather, for each element of the construct listed in the mark sheet, the examiner should judge the candidate's ability as a fellow clinician, against the standard reasonably to be expected of a senior house officer 1 year into their training in psychiatry. Examiners should not examine a station directly related to their own specialty, as their specialist knowledge might tempt them to be overexacting in the standards they expected. Equally, the examiner should be comfortable with assessing the task required.
The examiners must be instructed not to interact with the candidates other than by asking their name and recording their candidate number (unless the question specifically requires it), and warned specifically against teaching during the examination. This would destroy the authenticity of the experience for the candidates, and might lead to serious problems with the timing.
Make it clear that mark sheets will be collected frequently between candidates while the OSCE is in progress. This is to avoid examiners, under pressure of time, storing unmarked sheets to ‘catch up later’, leading to inaccurate retrospective marking.
Candidates
The maximum number of candidates that can be accommodated in the OSCE format is fixed at one per station per circuit. Adding in rest stations (where no task is presented but the candidate simply rests, out of earshot of neighbouring stations) allows this number to be increased, at one per station. Two circuits may be necessary in one day for a large scheme; alternatively, neighbouring smaller schemes might wish to pool their resources in order to fill one circuit of candidates.
Candidates should be briefed when they arrive on the format and formalities of the OSCE.
On the day
As candidates, examiners and role players arrive they must be sequestered in separate rooms for their respective briefings. Role players should have the opportunity to discuss their scenario with the authors to clarify any last-minute misunderstandings about their role or the candidate task.
The timing once the OSCE commences is seamless and strict, with each 7 min station preceded by only 1 min for the candidate to reach it and read the instructions. We would strongly recommend automating the timing. A computerised timer may be made available by the venue, with amplified signals and verbal cues repeating on a cycle until the examination is over. Having a hand-bell available in case the system fails should be considered.
Completed feedback questionnaires should be gathered from all participants before they leave the building, and all the examiners should be invited to gather over coffee to provide their views on the station they have examined, how the candidates have fared and their impressions of the day overall.
Getting the results out
The approaching date of the actual examination makes it imperative to process and distribute the results as quickly as possible. Much more information should be provided than would be released after the real OSCE to maximise learning opportunity (Box 2).
Box 2. Format of results provided
|
Before release, results should be assessed for consistency across stations, and between the station and the external examiners, to make sure that no one station or examiner is unduly distorting the results.
Potential pitfalls
The precision in timing and detailed preparation this format requires make it worth considering possible problems in advance. Box 3 lists some pitfalls to be avoided.
Box 3. Pitfalls
|
Implications
Cost
Quite aside from the significant financial cost, there is also a substantial burden of time, effort and commitment to be sustained over the course of many months. In our opinion it needs a coherent group of enthusiastic organisers who get on well together and play to each others’ strengths. It would also be impossible without the support of consultants and specialist registrars giving up their time to be examiners on the day.
Educational benefits
Candidates may vary in the skills and experience they bring to this examination. Some will have sat OSCEs before, particularly younger candidates from UK medical schools. In other schemes, however, a large proportion of trainees will not have completed their initial medical training in the UK. Candidates thus may never have encountered OSCEs before, may come from very different educational cultures, and may have English as their second language.
Since a key purpose of any mock examination is simply exposure to the format for practice, it should closely resemble the actual examination. However, it can also prove valuable as a wider educational experience, and the high costs involved make it important that such benefits are not missed. It should therefore be optimally placed in the context of an MRCPsych course which provides systematic training and practice in the OSCE format well before the mock itself, but specific efforts should also be made to derive the maximum opportunity for candidates individually, and the training scheme as a whole, to learn real concrete lessons from the experience.
In the real OSCE little feedback will be given, but after the mock OSCE the organisers should provide a breakdown of scores for each station to the candidates, their consultants and their clinical tutors. Both examiners and role players can be asked to give written comments, with a space set aside on the mark sheet for this. The perspective of the role player as the test patient is valuable in providing extra feedback to candidates on their communication skills in particular. In keeping with the real OSCE, however, this should not be used to determine the overall mark.
In order to preserve authenticity, it is important not to allow extra time for writing feedback during the mock OSCE. This does put time pressure on examiners, but is better than taking time pressure off the candidates - the real OSCE will be tight, and they should not be lulled into a false sense of security. In spite of the time pressures on examiners, our experience shows that many helpful comments can be produced even in the minute available between candidates. This can be consolidated by providing time in the week after the examination for every candidate to meet a course organiser to discuss his or her performance individually, and to set specific learning objectives in time for the real OSCE.
The educational benefits, however, can spread further still. In the feedback session with the examiners immediately after the mock OSCE, it is possible not only to learn organisational lessons about the running of the examination, but also to encourage examiners to share their comments on the nature of the new format and the performance of the candidates they have witnessed. This means the priorities revealed for further training can be highlighted, both in the running of the MRCPsych course, and at the examiners’ local level, in their borough academic programmes and in their own supervision of trainees. Examiners should leave feeling more aware of the novel challenges posed by the OSCE format; our examiners discussed improving training in communication and physical examination in particular. Many determined to employ more direct observation of trainees’ clinical skills in routine supervision sessions.
Conclusions
The organisation and resources required to organise a regular mock OSCE as part of an MRCPsych course are considerable, but the benefits more than justify the effort. The OSCE is posited as providing a more valid and reliable method of screening for entry to the College, but running regular mock examinations also provides a direct means of assessing performance and influencing training at a local level. This can only produce better psychiatrists.
Declaration of interest
None.
Acknowledgements
The authors thank all those involved in organising our mock OSCEs, in particular Jane Courtney, Viki Gyetvai and all the examiners who generously gave up their time.
Open access
eLetters
No eLetters have been published for this article.