INTRODUCTION
Simulation-based education (SBE) is well established in postgraduate medical education.Reference McGaghie, Issenberg and Petrusa 1 In certain situations, simulation has been shown to be superior to traditional clinical education,Reference McGaghie, Issenberg and Cohen 2 and there is growing evidence showing that SBE improves patient outcomes.Reference Orledge, Phillips and Murray 3 Emergency medicine (EM) training programs have particularly embraced SBE.Reference Ilgen, Sherbino and Cook 4 , Reference McLaughlin, Fitch and Goyal 5 There are several descriptions in the literature of the incorporation of SBE into postgraduate EM training programs.Reference Dagnone, McGraw and Howes 6 , Reference Binstadt, Walls and White 7 EM residents derive many benefits from SBE as a training modality. It provides a safe, realistic environment to hone their clinical decision-making, technical skills, to develop their leadership and collaborative abilities, and to receive immediate feedback on their performance.Reference Okuda, Bryson and DeMaria 8 The standardization, fidelity, and reproducibility of SBE also make it well suited for the assessment of clinical competence.Reference McGaghie, Issenberg and Petrusa 1 The direct evaluation of performance through simulation-based assessment provides a unique opportunity for simultaneous evaluation of knowledge, clinical reasoning, and teamwork.Reference Epstein 9 In Canada, there has been a national call for SBE to be further integrated into existing postgraduate curricula.Reference LeBlanc, McNaughton and Brydges 10 The Royal College of Physicians and Surgeons is currently implementing the CanMEDS 2015 Physician Competency Framework and the Competency By Design (CBD) initiative for all programs.Reference Frank, Snell and Sherbino 11 In this transition, simulation will likely play an even more important role as the assessment of clinical expertise moves toward competency-based assessment.Reference Boursicot, Etheridge and Setna 12
Over the past decade, there has been steady growth in the adoption of simulation within EM residency training. A survey of 156 U.S. EM residency programs conducted in 2003 found that the majority of institutions lacked a formal simulation curriculum.Reference McLaughlin, Bond and Promes 13 A subsequent study conducted in 2008 found that the majority of U.S. EM residency programs used over 10 hours per year of SBE in their curricula.Reference Okuda, Bond and Bonfante 14
A more complete understanding of how SBE is delivered in Canadian EM training programs will aid in establishing norms and may offer both insights on best practice and barriers to implementation. The purpose of this study was to establish the current state of SBE in Canadian FRCPC-EM training programs using structured surveys for learners and educators.
METHODS
From October 2015 to January 2016, we conducted two national surveys assessing the use of SBE in all Canadian FRCPC-EM programs. The study was approved by the Queen’s Health Sciences Research Ethics Board (#6016057).
Study populations
All 16 Canadian FRCPC-EM programs were surveyed, including the three separate geographic sites that exist at the University of British Columbia. At each program, a program representative (PR) and all residents in their second to fifth postgraduate year (PGY2-5) were surveyed. The PR was the FRCPC-EM Program Director, the Assistant Program Director, or the Simulation Director at that site.
Survey development
Surveys were designed to capture both PR and resident perspectives. This occurred in three phases. Phase one consisted of a literature review using the MEDLINE and PubMed databases to look for surveys of simulation activities in EM residencies. Two relevant articles were identified.Reference McLaughlin, Bond and Promes 13 , Reference Okuda, Bond and Bonfante 14 Study investigators developed survey questions using the topics and themes identified from a detailed review of each article, including the general characteristics of institutional simulation programs, the frequency of resident participation in simulation, the location and administration of simulation facilities, and perceived barriers to SBE implementation. Surveys for both PRs and residents contained dichotomous, Likert-type, rank-order, and open-ended questions.Reference Woodward 15
In phase two, study investigators convened to review the aforementioned themes and broaden the question list. Most study investigators have pursued advanced training in SBE (Harvard-Macy Institute, MA; Kingston Resuscitation Institute, ON) or in medical education in the form of a master’s degree in Medical Education (University of Dundee, Scotland) or Medical Education Leadership (University of New England, ME). Question topics included the use of SBE for interprofessional training, the role of SBE for resident performance assessment, a review of appropriate clinical content for SBE, and comfort participating in SBE.
The third phase consisted of a piloting and review process. The survey was piloted at a single institution (Queen’s University) among 12 non-EM residents and an FRCPC program director for comprehension and evidence of content-validity. An additional review was performed by an external EM education scholar for a final review and feedback.
Survey administration
A modified Dillman approach was used to recruit both residents and PRs.Reference Dillman and Christian 16 All potential participants received a letter of information prior to their enrolment in the study. Both residents and PRs were informed that personal and institution-specific data provided would be de-identified.
The PR survey (Online Appendix A) was conducted via telephone by one of two study investigators (CH or AH). Using a semi-structured format, an interviewer read the survey questions to PR participants with clarification provided as needed. PR responses were transcribed by the interviewer and then collated in Excel.
The resident survey (Online Appendix B) was offered in both paper and online versions. Prior to administration of the resident surveys, a resident champion was identified at each site. Paper copies were mailed to each site champion to distribute to all eligible residents. Completed hard copies were returned via prepaid, preaddressed packages. The online version was administered through an institutional subscription with fluidsurveys.com. Resident champions forwarded reminder emails to their fellow residents with a link to the online version of the survey at 2 and 3 weeks following the initial paper administration.
Analysis
Descriptive statistics were used to calculate rates and averages. Data sets were assessed for normality using the Shapiro-Wilk test. Normally distributed data are reported as mean ± standard deviation, and non-normally distributed data are reported as median (range). Responses to Likert questions were reported as the percentage of respondents who selected a particular response. Spearman rho analyses or Mann-Whitney U tests were used to compare reported hours of simulation training with other survey responses. Because this survey asked residents to recall the last 1 year of their training, residents in the first year of training were not included. Junior residents were defined as PGY2/3 respondents, and senior residents were defined as PGY4/5 respondents. For all statistical analyses, p ≤ 0.05 was considered significant.
RESULTS
Demographics
The resident and PR response rates were 63% (203/321) and 100% (16/16), respectively. Resident response rates declined with increasing postgraduate year, with 77% (64/83) in the PGY2 cohort, 62% (51/82) in the PGY3 cohort, 54% (42/78) in the PGY4 cohort, and 33% (26/78) in the PGY5 cohort. Additionally, 13 residents chose to not indicate their postgraduate year. Across institutions, resident response rates ranged from 16%–94%.
Program characteristics
Residency program statistics are summarized in Table 1. Based on PR reported data, all residency programs offer both manikin-based and task-trainer simulation modalities. Resident-reported exposure to both manikin-based and task trainer-based simulation was 99% and 91%, respectively. An established simulation curriculum existed at 94% of programs, with 19% of programs reporting that SBE scenarios were temporally linked to topics that were taught in the core curriculum. The frequency of simulation training sessions ranged from weekly to every 6 months, with most programs holding sessions every 1 to 2 months. Simulation-based assessment was used in 13% of the programs, although 94% of PRs indicated that they would be comfortable with incorporating simulation-based resident assessment.
Residency program characteristics
Volume and frequency of simulation training
According to PR estimates, the mean amount of annual simulation training was 42 ± 26 hours with considerable variability across programs (Figure 1). PRs reported an average of 4 hours of SBE per month (range of 0.5 to 8 hours). Thirty-seven percent of PRs indicated that their residents continued to participate in EM-based simulations during off-service rotations. Residents reported participating in significantly more SBE per month when on-service (EM rotations), as compared to when they were off-service (3 [0–40] v. 1.5 [0–20] hours, p<0.0001). PR-reported annual volume of SBE was significantly greater than resident-reported volumes (42.9 [6–110] v. 20 hours [0–150] hours, p=0.017); however, the reported volumes of monthly SBE were not significantly different between PRs and residents (4 [0.5–8] v. 3 [0–40] hours, p=0.19). Senior residents reported spending significantly less time per month in SBE than junior residents (2 [0–17] v. 3 [0–40] hours, p<0.05).
PR-reported total annual hours of simulation training by program. Programs that have been de-identified are listed in descending order of annual amount of SBE.
Resident- and PR-reported level of comfort and perceived utility of SBE are summarized in Table 2, Q1-4. Of the resident respondents, 87% indicated comfort with SBE, 60% were comfortable with using SBE for assessment, and 54% indicated that their programs’ current SBE was meeting their educational needs. In addition, 87% of residents indicated that SBE has improved their abilities to care for real patients.
The mean percentages of resident and PR responses to questions about comfort in a simulation environment and how SBE is administered. (For each question, resident data reflect answers to the aforementioned statement [a], and PR data reflect answers to the below listed statement [b])
Location of simulation administration
The PR-reported locations of SBE administration, as reported in Figure 2, are highly variable, with some programs solely using a hospital simulation lab, others using only an off-site simulation facility, and in situ simulation (ISS) utilization occupying from zero to 50% of programs’ SBE curricula. Aggregate resident responses reported that the most common location for SBE was a hospital-based simulation lab (56.1%), followed by an off-site simulation lab (26.6%), and then ISS within the emergency department (ED) (14.6%).
Institution-specific percentages of where SBE is occurring. Please note that the numbers on the y-axis correspond to the de-identified program numbers listed in Figure 1.
Resident responses to questions pertaining to ISS and unannounced simulation were variable (see Table 2, Q7-9). ISS was the preferred method of SBE for 35% of residents. A minority (20%) of residents preferred surprise/unannounced SBE compared to scheduled sessions. However, 49% of residents appreciated the potential educational value of surprise simulations. Residents were more likely to report preference for surprise simulation when they agreed that it had educational benefit (r2=0.48, p<0.001).
Barriers to simulation
PR-reported barriers to SBE at their institutions are reported in Table 3. The most common barriers were lack of protected faculty time (75%) and lack of faculty experience with simulation (56%). These were also identified as the most challenging barriers to overcome.
PR-reported barriers to SBE
Simulation content
Over 95% of resident respondents reported having participated in simulation training involving airway emergencies, shock, adult cardiac arrest, pediatric resuscitations, cardiac emergencies, and respiratory emergencies during their residency. Residents reported that some clinical subjects were less represented in their SBE curricula, with 17%–32% of residents indicating that they had received insufficient simulation exposure to pediatric resuscitation, to neonatal resuscitation, and to emergencies that were vascular, gastrointestinal, neurological, endocrine/metabolic, infectious, or obstetrical in nature. Resident responses pertaining to the amount of SBE time dedicated to specific clinical presentations are presented in Figure 3.
Resident attitudes on the amount of time that is being dedicated to specific clinical presentations in their SBE.
Interprofessional involvement
Interprofessional participation in the SBE program varied across institutions. Of the PR-reported data, 15 (94%) residency programs have involved nursing staff, 9 (56%) have involved respiratory therapists, and 6 (37%) have involved paramedics in their simulation programs. Residents reported that other health care professionals are present in 41% of their total simulation sessions. A breakdown of the specific professionals involved in resident-reported interprofessional simulations is presented in Figure 4. Of the resident respondents, 82% indicated that interprofessional involvement was helpful in their residency training, and 77% indicated a desire for more interprofessional involvement. Resident-reported agreement on the utility of interprofessional involvement in simulation was significantly correlated with a desire for more interprofessional involvement in SBE (r2=0.55, p<0.0001).
Resident-reported percentages of how commonly their SBE has interprofessional involvement (left). Of their SBE with interprofessional involvement, a breakdown of how commonly allied health professions are represented in their training is shown on the right.
DISCUSSION
Volume and frequency of simulation training
The annual volume and frequency of SBE training in Canadian EM residency programs are highly variable. Whereas some programs offer weekly simulation rounds for both of their junior and senior residents, other programs provide a simulation series once or twice a year for their residents. We noted that the estimated monthly volume of SBE was concordant between PRs and residents. However, PR estimates of annual hours of SBE training were significantly higher than resident estimates. We hypothesize that PR responses may not have factored in the variable attendance of residents due to off-service training experiences, post-night shifts, or vacation.
There is also variability in the experience of junior versus senior residents, with senior residents reporting 1 less hour of simulation per month. This may reflect a widespread provision of SBE for juniors in basic resuscitation and procedural skills and less frequent teaching of more complicated senior level SBE, increased elective time in senior years, or an overall increase in how much simulation is being offered by programs each year. Stable gains in procedural competence have been demonstrated in as little as 8 hours of SBE,Reference Langhan, Rigby and Walker 17 so this difference, if true, is likely educationally relevant.
Many residents and PRs indicated that their local simulation programs do not meet the educational needs of residents. The vast majority of residents expressed comfort participating in simulation and, perhaps more importantly, felt that their experience with SBE improved their ability to care for patients in the clinical setting. This finding is consistent with other studies that have demonstrated similar utility for simulation as a tool for deliberate practice and competency-based training.Reference McGaghie, Issenberg and Cohen 2 , Reference Okuda, Bryson and DeMaria 8 , Reference McGraw, Chaplin and McKaigney 18 , Reference Wayne, Didwania and Feinglass 19 Our findings suggest that many Canadian EM residents and PRs would favour increased SBE.
Location of simulation administration
Traditionally, SBE has been implemented within an established simulation centre away from patient care. However, ISS, which occurs in the actual patient care environment, has gained popularity as can be used to facilitate interprofessional team training,Reference Knight, Gabhart and Earnest 20 identify threats to patient safety,Reference Wheeler, Geis and Mack 21 and provide a cost-effective training location.Reference Calhoun, Boone and Peterson 22 Despite increased enthusiasm for ISS among educators, it remains an uncommon training strategy within Canadian EM residencies. Interestingly, most residents were neutral in their response to their preferred location for SBE (ISS v. simulation centre), potentially indicating an overall lack of familiarity with the training modality. With thoughtful planning, ISS can meet the residents’ desires for increased interprofessional training, because the on-duty clinical team is easily brought together eliminating the scheduling and logistical constraints associated with a simulation centre.
The application of ISS also allows for unannounced (or surprise) sessions during an ED shift. Resident attitudes towards such sessions were generally negative or neutral, although they acknowledged the potential educational benefit that could be achieved. Although unannounced ISS has a potential role for training residents as they transition to practice, these findings indicate that educators must recognize this reluctance and ensure a safe learning environment. Our findings demonstrate considerable opportunities for growth in ISS within Canadian EM residencies as a cost-effective training strategy.
Barriers to simulation
Understanding the barriers to SBE and how these barriers evolve over time is essential to keep pace with the evolving needs of learners, programs, and institutions. We found that faculty time and training are the primary obstacles to simulation use and implementation. These findings match U.S. data,Reference Okuda, Bond and Bonfante 14 which no longer report funding as the main barrier. Access to both equipment and space is now less problematic, because most training programs and institutions have successfully created environments for SBE. Educators and administrators alike must appreciate these barriers of faculty time and training as they develop institutional strategic plans. Focused efforts to encourage faculty development in SBE may serve to decrease the reliance on the existing small group of taxed simulation educators.
The issue of increasing faculty time for participation in SBE is challenging. Most EM training programs already struggle to find funding for their hardworking faculty. Hopefully, this survey and the previously mentioned evidence supporting SBE can be leveraged locally by programs and educators to advocate for SBE, while national organizations can be called upon to encourage protected time for SBE educators.
The establishment of a Canadian EM-specific SBE working group has the potential to facilitate increased sharing of curriculum, scenarios, assessment tools, and other materials between programs. Coordinated educational programs to address faculty development offer an opportunity to leverage the strengths of more experienced centres and to share successes between centres. This could increase both the efficiency and the support of simulation educators nationally.
Simulation content
It is clear that SBE needs to be carefully integrated into educational curriculum to complement clinical education.Reference McGaghie, Issenberg and Petrusa 1 Certain clinical content is more easily integrated within SBEReference Khan, Pattison and Sherwood 23 and therefore more frequently taught using simulation. In our survey, almost all residents reported participating in training involving common SBE content such as cardiac arrest and airway emergencies. In fact, 9% of residents reported that they perceived adult cardiac arrest to be receiving too much attention in their SBE. This was a surprising finding, given the large body of evidence supporting the use of SBE for the teaching of adult cardiac arrest.Reference Cheng, Lockey and Bhanji 24 Perhaps trainees do not recognize the importance of spaced repetition when practicing adult cardiac arrest algorithms, and the potential for poor skill retention and skill decay.Reference Mancini, Soar and Bhanji 25 Alternatively, trainees may recognize the importance of teaching cardiac arrest but think that the percentage of SBE time that it occupies detracts from other content that could be taught in its place. It is important to remember that the purpose of cardiac arrest training extends beyond effective CPR, as it provides a medium through which to teach transferable skills such as crisis resource management and effective leadership strategies. Ultimately, the finding that multiple trainees report that cardiac arrest is being over-taught suggests that there needs to be dialogue between residents and simulation leads, at minimum.
In contrast, neonatal resuscitation and obstetrical emergencies were underrepresented within the SBE curriculum, despite a clear desire for this experience. This may reflect a lack of access to either the appropriate manikins for such scenarios or experts to deliver this SBE content. Additionally, SBE content may reflect more common ED presentations, despite the apparent benefits of SBE for uncommon or rare presentations.Reference Orledge, Phillips and Murray 3 We hope that our survey findings can be used by EM programs and institutions to advocate for access to appropriate training equipment to conduct pediatric, neonatal, and obstetrical emergencies.
Moving forward, it may be reasonable to develop a list of core SBE content for EM. This content may become more apparent as we transition to competency-based medical education and deficits in clinical exposure become apparent with more rigorous tracking. Also, the national consensus on a simulation-based curriculum could be modelled after the Delphi process for establishment of a pediatric EM curriculum used by Bank et al.Reference Bank, Cheng and McLeod 26
Interprofessional involvement
Interprofessional simulation training improves the real-world functioning of teams in the ED, particularly team efficiency during life-threatening emergencies.Reference Capella, Smith and Philp 27 Interprofessional SBE can improve the residents’ knowledge of other health care providers’ roles, and develop communication and teamwork skills that are integral to successful resuscitation efforts.Reference Dagnone, McGraw and Pulling 28 , Reference Watters, Reedy and Ross 29 Without interprofessional colleagues alongside residents, it is difficult to train higher level competencies that involve leadership, collaboration, and crisis resource management. Respondents in this study indicated a resounding desire for increasing interprofessional involvement in their SBE activities. Although it is encouraging that 94% of programs have had nursing involvement during their simulation sessions, it is noted that other professions (respiratory therapists, paramedics) are less well represented. These data represent an opportunity for programs to advocate for increased meaningful interprofessional simulation.
LIMITATIONS
This study’s limitations predominantly relate to resident-reported data. We purposefully excluded PGY1 residents from participation in the survey. When the survey was administered, PGY1 residents had completed only 3 months of residency. The study team decided that this length of time was insufficient to provide an accurate report of their institutions’ SBE.
Among eligible residents, only 63% responded, and the response rate varied considerably across institutions, with as few as 16% of residents responding from one institution. Fortunately, the site-specific resident response rates were independent of any PR-reported measures. This institutional variability in the resident response rate likely affects the validity of the mean resident-reported percentages of where SBE is being administered. For these results, the location of administration will be distorted to proportionally reflect how simulation is being administered at institutions that had better resident response rates.
A participation bias among residents was also noted, with a stepwise decline in the response rate with advancing postgraduate year. This finding was similar to other surveys of EM residentsReference Petrosoniak, Herold and Woolfrey 30 and may be due to increasing survey fatigue with advancing training or a focus on the final board certification exams, in the case of the final year residents. This may have acted to falsely inflate resident hours of reported simulation at institutions where simulation is offered to junior residents only. Furthermore, this may have distorted the reported experience and attitudes relating to simulation content, because some content is typically taught at a senior level only.
Finally, surveys were offered in English only. Although there were no concerns about the English literacy of the respondents whose first language is French, offering a French survey may have improved the response rate.
CONCLUSION
This study characterizes SBE utilization across Canadian FRCPC-EM residency programs and identifies several gaps in EM simulation delivery. These gaps should be carefully considered prior to the establishment of national consensus on the role of SBE in EM training. As each FRCPC-EM program transitions to competency-based medical education, collaborations and working groups should consider the substantial variability in the administration of SBE and the barriers that exist to further expanding its utilization.
Acknowledgements: This study was supported by an unrestricted grant from the Kingston Resuscitation Institute, Kingston, ON.
Competing interests: None declared.
SUPPLEMENTARY MATERIAL
For supplementary material/s referred to in this article, please visit https://doi.org/10.1017/cem.2017.24
