Rationale

There has been a broad call for the development of tools to assess competency of post-graduate medical trainees in the United States,Reference Steadman and Huang ⁶ Canada, ⁵ and internationally.Reference Amin, Boulet and Cook ^{7 ,} Reference Hamstra ⁸ As the Royal College of Physicians and Surgeons (RCPS) moves toward the implementation of competency-based medical education curricula for postgraduate training,Reference Frank, Snell and Sherbino ⁹ the need for tools to accurately assess clinical competencies has become paramount. Traditional knowledge-based examinations (written tests and oral exams) test trainees at a “knows how” level.Reference Miller ¹⁰ Direct evaluation of performance through simulation-based assessment provides an opportunity for simultaneous evaluation of knowledge, clinical reasoning, communication, and teamwork.Reference Epstein ¹¹ We launched a simulation-based OSCE in 2008 to answer the call for the development and implementation of competency assessments at the “shows how” level of Miller’s pyramid.Reference Boursicot, Etheridge and Setna ¹²

Trainees and faculty

The Queen’s University post-graduate EM program oversees 30 resident trainees per year; 20 Royal College of Physicians and Surgeons of Canada EM candidates (RCPS-EM) and 10 College of Family Physicians of Canada EM candidates (CFPC-EM). All RCPS-EM and CFPC-EM residents are required to participate bi-annually in a standardized OSCE held in our clinical simulation centre (CSC). Two specific faculty members with training in medical education are primarily responsible for the delivery of the simulation-based OSCE assessments, while the RCPS-EM and CFPC-EM program directors assist with debriefing trainees following their performances.

Assessment framework

All residents are presented with two or three resuscitation scenarios in sequential order and debriefed for up to 30 minutes by a faculty member immediately following their performance. The debriefing focuses on strengths, weaknesses, and any need for improvement of each individual trainee.

Faculty debriefers use a standardized Queen’s Simulation Assessment Tool (QSAT) for the resuscitation scenarios (Figure 1). This tool has been evolving since the inception of the assessment program and has been validated both locally and in a multicentre setting.Reference Dagnone, Hall and Woolfrey ^{13 ,} Reference Hall, Dagnone and Lacroix ¹⁴ The QSAT was designed as a hybrid scoring tool with four anchored domain scores (primary assessment, diagnostic actions, therapeutic actions, and communication) and an overall performance score (global assessment). Within each domain, key anchors are included, which may be modified to assist the determinations of expert evaluators related to assessment scores. A modified Delphi technique by EM and critical care physicians was used to modify the generic QSAT for each standardized scenario. The QSAT is easily modifiable for use across multiple resuscitation scenarios and remains an ongoing research focus within our program.Reference Hall, Dagnone and Lacroix ^{14 ,} Reference Hall, Pickett and Dagnone ¹⁵

Figure 1

Sample of the Queen’s Simulation Assessment Tool (QSAT) used to assess resident performance in an OSCE station and guide feedback during debriefing.

Scenarios

OSCE scenarios are carried out in a single room, designed as a simulated resuscitation bay, using a high-fidelity mannequin. All scenarios are run by a simulation-trained faculty member and an experienced simulation laboratory technician, and involve roles for the mannequin operator and standardized actors portraying a nurse and a respiratory therapist. The scenarios are pre-scripted and delivered in a standardized fashion to all residents. All resident performances are video-recorded using Kb Port ETC Pro (Kb Port, Allison Park, PA), a 3-camera system, and a cardiac monitor, with the audio recording included.

The scenarios utilized were designed, implemented, and debriefed by EM faculty with training in simulation-based instruction. The two faculty members primarily responsible for this program have 15 combined years of experience in simulation-based instruction and assessment, and Master’s degrees in medical education. One of the faculty members has also obtained additional training in simulation at Harvard University (Boston, USA). The scenarios include cardinal presentations, such as unstable arrhythmia, cardiac arrest, bradycardia, decreased level of consciousness, and hypotension. A blueprinting exercise was performed, mapping against the 7th edition of Rosen’s Emergency Medicine textbookReference Marx, Hockberger and Walls ¹⁶ to ensure a proper distribution of resuscitation cases and the assessment of multiple core competencies within EM. The standardized scenarios were designed to discriminate between residents of different levels of training by eliciting observable behaviours that traditionally are best assessed in the actual clinical environment. In a validation study of the QSAT, compiling data from sequential OSCE examinations, nine of 10 scenarios were found to have statistically significant differences in assessment scores, and thus discriminating ability, between junior and senior RCPS-EM trainees.Reference Hall, Dagnone and Lacroix ¹⁴

Debriefing and self-assessment

Faculty debriefing occurs in an adjacent private conference room after each resident has completed sequential performance of all scenarios. Performance expectations are based on resident’s post-graduate level. Trainees receive a score for each domain on the QSAT, a total score for all the domains combined, and a global assessment score for each scenario performance. QSAT scoring sheets are placed in each trainee’s portfolio as a formative objective assessment measure of their progression and are discussed during quarterly reviews with the program director.

All video-recorded performances are uploaded onto a secure server, and linked to each trainee on a Queen’s University online Moodle^TM platform (Figure 2). Each resident can access only their own video recordings using a login and password. Residents are also provided with a self-assessment scoring tool online in PDF form, which can be downloaded and utilized electronically. By allowing access to the video recording of their own performance and the opportunity to meet with faculty at a later date, all residents are provided a means for further formative feedback after delayed reflection and personal assessment.

Figure 2

A screen capture of the video recording made during the OSCE that residents access following the OSCE.

Resident perspective

By the end of their residency, Queen’s University RCSP-EM residents will have participated in 10 simulation-based OSCEs and will have logged over 150 hours in the simulation laboratory by participating in the simulation-based resuscitation curriculum at Queen’s University.Reference Dagnone, Mcgraw and Howes ¹⁷ Similarly, Queen’s University CFPC-EM residents will have participated in two simulation-based OSCEs and will have logged over 30 hours in the simulation laboratory. Simulation-based examinations are a valuable tool for the assessment of individual clinical skill acquisition by faculty at our centre, but are of equal importance to trainees, to gauge improvement and allow reflection on their progress and competency development.