CLINICIAN'S CAPSULE
What is known about the topic? Simulation is increasingly being integrated into postgraduate emergency medicine (EM) training, but no national curriculum or consensus on priority topics exists.
What did this study ask? What adult EM curricular content is best suited for simulation-based training, to inform a national postgraduate EM simulation curriculum?
What did this study find? A national Delphi panel identified 28 core curricular topics, categorized into eight domains, which are best suited for simulation-based training.
Why does this study matter to clinicians? Standardized curricular content provides the foundation for improved integration of simulation into postgraduate programs to supplement clinical training and optimize learning.
INTRODUCTION
The medical education community strongly supports the incorporation of simulation-based education into postgraduate training programs as it allows trainees to practice skills in the safety of a controlled environment without risk of patient harm and enables receipt of immediate performance enhancing feedback.Reference Cook, Andersen, Combes, Feldman and Sachdeva1,Reference McGaghie, Issenberg, Barsuk and Wayne2 Deliberate practice, as defined by Ericsson,Reference Ericsson3,Reference Ericsson, Krampe and Tesch-Römer4 involves focused repetitive performance of a skill, coupled with the receipt of informative feedback and a primary goal of improving performance toward a mastery standard. Simulation-based education with deliberate practice has been shown to be superior to traditional clinical education in acquiring specific procedural and resuscitation skills.Reference McGaghie, Issenberg, Cohen, Barsuk and Wayne5 Furthermore, there is a growing body of literature demonstrating that simulation-based education improves patient outcomes.Reference Zendejas, Brydges, Wang and Cook6,Reference Brydges, Hatala, Zendejas, Erwin and Cook7 In Canada, there has been a national call for simulation-based education to be integrated more thoughtfully into existing postgraduate curricula.Reference Leblanc, Bould and Sharma8
Emergency medicine (EM) training programs have embraced simulation with enthusiasm, and the extant literature supports structured integration of simulation-based education into EM training.Reference Ilgen, Sherbino and Cook9–Reference Cook, Brydges, Zendejas, Hamstra and Hatala13 A recent survey demonstrated that all Canadian EM postgraduate training programs utilize simulation-based education; however, there is great variation in the content, structure, quantity, and timing of simulation-based activities because of differences in funding, resources, barriers, and clinical contexts.Reference Russell, Hall and Hagel14 Further clarity on the best method of harnessing simulation to educate EM residents is needed. Currently, there is no standardized national EM simulation curriculum in Canada or the United States. Guidance and standardization of simulation curricular content would provide a framework on which teachers and educators in postgraduate EM programs could build a solid simulation curriculum.
In the transition to competency-based medical education in the form of Competence by Design as directed by the Royal College of Physicians and Surgeons of Canada (RCPSC),Reference Frank, Snell and Sherbino15 simulation will likely play an increasingly important role.Reference Boursicot, Etheridge and Setna16 EM has a broad and fairly inclusive set of objectives of training17 that includes many critical but rare clinical situations. If systematic training and assessment in a competency-based model require exposure to these rare events, simulation-based education would need to be harnessed as a tool for both training and assessment. Cook et al.Reference Cook, Andersen, Combes, Feldman and Sachdeva1 eloquently describe the “value proposition” of simulation, outlining the importance of balancing outcomes and cost. With substantial resources required for simulation, it is important that we prioritize key experiences, given existing resource limitations in postgraduate training. Defining the EM curricular content best suited for simulation-based training would provide the foundation for improved integration of this educational technique into postgraduate programs. Using the Delphi methodology, we sought to determine the content for such a simulation curriculum in adult EM that could be used to supplement clinical training and assessment to optimize learning.
METHODS
Study design
The present study used the Delphi technique to achieve consensus among a panel of experts to determine what adult EM curricular content is best suited for simulation-based training. For this study, simulation was defined broadly as an instructional methodology that engages learners in lifelike experiences with varying fidelity, designed to mimic real clinical encounters.Reference McGaghie, Issenberg, Cohen, Barsuk and Wayne5 The Delphi method is a research technique that draws upon the “pooled intelligence” of an expert panel to achieve consensus on a specific topic through the use of iterative rounds of questionnaires with controlled feedback.Reference de Villiers, de Villiers and Kent18–Reference Murry and Hammons20 The Delphi methodology, through the provision of expert professional judgment, can be used to generate content-related validity evidence for curricular content.Reference Delbecq, Van De Ven and Gustafson21 This study was approved by the Sunnybrook Health Sciences Centre institutional review board.
Delphi panel recruitment and sample
A Delphi group of Canadian experts in EM simulation was established. Purposive sampling was used to ensure that the appropriate experts were invited to participate. First, we emailed current program directors of all 14 RCPSC-accredited EM residency training programs in Canada to identify one or more experts in simulation-related education at their site. Second, some panellists were identified as national experts as evidenced by their role as opinion leaders in simulation, and/or EM education within organizations such as the RCPSC and the Canadian Association of Emergency Physicians, and/or their positions on national EM-related simulation and/or education committees. To increase the reliability of the Delphi group's composite judgment, our goal was to include a sample of at least 30 experts with broad expertise reflective of current knowledge and perceptions related to EM training and simulation.Reference Delbecq, Van De Ven and Gustafson21,Reference Murphy, Black and Lamping22 Sixty-six prospective panellists were sent email invitations explaining the study purpose and methodology. Membership of the Delphi panel was kept confidential.
Item generation and elimination of data redundancy
Potential curricular topics were generated from: 1) a systematic literature review; 2) a review of relevant educational documents (e.g., RCPSC, Accreditation Council for Graduate Medical Education, and the American Board of Emergency Medicine); and 3) input from Delphi panel members. Published literature on the topic of EM curricular content was systematically reviewed. Health professions databases were searched from inception until May 2015, including MEDLINE, EMBASE and Web of Science (see Appendix 1, search strategy). Articles pertaining to EM training and/or simulation were reviewed independently by two authors (NKG and CMW), and potential adult EM curricular topics were extracted and integrated into the initial item pool. Pediatric and neonatal curricular topics were not included, as they have been examined previously.Reference Bank, Cheng, McLeod and Bhanji24 The list of potential topics was then reduced by combining redundant items. This process was completed by three authors (NKG, AH, and CMW) who have clinical and pedagogical expertise in EM and/or simulation. Potential topics were reviewed independently for redundancy, and the research team met on two occasions to compare topics and establish consensus. Additionally, panellists were asked to comment on the list of potential curricular topics and identify any additional topics they felt should be included in a simulation-based EM training curriculum during round one of the Delphi process.
Item reduction
Iterative rounds of Delphi surveys were used to reduce the item pool further. To maximize response rates, the surveys were designed and distributed in line with principles outlined by Dillman's tailored design method: respondent-friendly surveys using clear and easy-to-understand language; personalized correspondence; and up to four email contracts for each Delphi round.Reference Dillman23 During each round, panellists were provided with a link to the online survey.
During the first Delphi round, participants individually rated the suitability of simulation as an educational tool for teaching each potential curricular topic. Topics were rated on a 1 (“best taught using methods other than simulation”) to 4 (“definitely should be taught using simulation”) ordinal scale used in a previous study to determine pediatric EM simulation curricular content.Reference Bank, Cheng, McLeod and Bhanji24 Participants were provided the opportunity to comment on the wording of potential topics, provide reasons for their choices, and/or suggest additional topic areas.
During the subsequent rounds, panellists re-rated the remaining curricular topics using the same four-point ordinal scale. Beyond the first rounds, participants were informed of the group mean rating and standard deviation (SD) for each topic in the proceeding round. Once again, panellists were given the opportunity to provide open-ended comments. The Delphi process continued until consensus among the expert panel was achieved using the criteria described below.
Data analysis and determining consensus
All statistical analysis was performed using SPSS, version 20.0 (IBM Corp). After each Delphi round, the mean rating ± SD and proportion of panellists rating an item within each category (1 to 4) were calculated. Panellists’ ratings were given equal weight. Three authors (NKG, AH, and CMW), blinded to the sources of the data, reviewed panellists’ ratings and qualitative comments. Using a priori criteria, a curricular topic was eliminated from subsequent Delphi rounds if its mean rating was <2. Topics were combined and/or their wording modified based on comments from the expert panel. The updated survey was redistributed for rating. The Delphi process continued in an iterative fashion until consensus was achieved. Consensus, a condition of homogeneity or consistency of opinion of the expert panellists, was quantified using Cronbach α.Reference Graham, Regehr and Wright19,Reference Bland and Altman25,Reference Diamond, Grant and Feldman26 A Cronbach α of ≥0.95 was predefined as the minimal threshold to determine when consensus had been achieved.Reference Bland and Altman25 It was predetermined that once consensus was achieved, topics rated ≥3.5 would be considered “core” curricular topics while items rated 3.0–3.5 would be considered “extended” curricular topics.
RESULTS
Forty-five experts from 13 of the 14 academic RCPSC-accredited EM training sites across Canada agreed to participate. The Delphi panel member demographics are outlined in Table 1. Of the participating experts, 45 (100%) completed the first Delphi survey, 42 (93.3%) completed round 2, and 43 (95.5%) completed round three. Across all three iterations of the Delphi process, 0.30% of items had missing ratings.
Profile of the panel of experts (N = 45) participating in the Delphi consensus process
*Panellists were instructed to self-select all non-clinical roles that may apply.
The systematic literature review identified 2,279 potential citations (981 Medline, 911 Embase, and 387 Web of Science), of which 1,533 were duplicates. The titles and abstracts of the remaining 746 citations were reviewed, and 199 relevant articles were identified. From these, as well as relevant educational documents, 2,515 potential curricular topic areas in adult EM were generated. No additional topics were suggested by the Delphi panel. By combining like items, these were reduced by the research team to 280 non-redundant curricular topics that fell into 29 curricular content domains (Appendix 2).
The flow of topics through each round of the Delphi process is outlined in Figure 1. Items with a mean rating of ≥2 were retained after each round, and the survey was updated and distributed for subsequent voting. Consensus was achieved after three rounds (Cronbach α = 0.97). Thirteen curricular content domains (dentistry, dermatology, ear nose and throat, endocrine, hematology and oncology, musculoskeletal, nephrology, ophthalmology, psychosocial, transplant medicine, minor procedures, emergency ultrasound, and urologic) were eliminated completely during the Delphi process. After the third round, 108 additional topics were eliminated as they did not achieve consensus (mean rating < 3). A total of 63 curricular topics categorized into 16 domains reached consensus for inclusion as adult EM curricular content best suited for simulation-based training, with 28 topics in eight domains considered “core” curricular topics (Table 2) and 35 topics in 14 domains being considered “extended” curricular topics (Table 3 and Appendix 2).
Overview of the Delphi process to identify curricular topics for a national simulation curriculum.
Core curricular topics (N = 28) for simulation-based training
Extended curricular topics (N = 35) for simulation-based training
DISCUSSION
With the implementation of competency by design, Canadian postgraduate EM training programs are engaging in the process of significant transformation. Simultaneously, there has been rapid, but variable, uptake of simulation-based education,Reference Russell, Hall and Hagel14 and many programs are grappling to understand how best to integrate simulation into the required training experiences and assessment processes of their revised competency-based programs.17,27 This Delphi process, which included 45 experts representing almost all RCPSC-accredited EM training programs in Canada, sought to determine the priority content for a simulation curriculum. Twenty-eight “core” topics and 35 “extended” topics were identified as curricular content best suited for simulation-based training. Understanding that resources are limited, and there is an ever-increasing requirement for efficiency within postgraduate training programs,Reference Nousiainen, Caverzagie, Ferguson and Frank28 this study is intended to guide EM teachers and educators regarding what topics should be prioritized for simulation-based education. These results can be used to inform a national simulation curriculum to supplement clinical training and optimize learning.
Curricular topics that were identified as “core” or considered of high importance for simulation-based education were primarily focused within the areas of crisis resource management, resuscitation, trauma, and procedures. The non-medical expert competencies of leadership, team skills, and effective communication and collaboration with an interprofessional team also achieved consensus as “core” topic areas. Our results are consistent with a recent study by Bank et al.Reference Bank, Cheng, McLeod and Bhanji24 who sought to identify content for a national simulation-based curriculum for pediatric EM training. The 48 “key” topics achieving consensus for pediatric EM were similarly grouped into the categories of resuscitation, trauma, procedures, and crisis resource management skills that included leadership, teamwork, and communication. The similarity of the findings further confirms that the “core” topics we identified are paramount to any EM simulation curricula. It is important to note, however, that the resulting list of “core” topics are fairly broad in scope and may be subject to varied interpretation by simulation educators. For example, the topic “altered level of consciousness and coma” may relate to a number of different clinical presentations including septic shock, head trauma, or hepatic encephalopathy. Interpretation of these results may present a challenge to educators charged with developing simulation curriculum relevant to the specific curricular needs of their own postgraduate program.
Our results are also consistent with existing literature outlining potential content for simulation training and assessment in adult EM that has emphasized resuscitation-focused content while also incorporating key non-medical expert roles.Reference Dagnone, McGraw and Howes11,Reference Binstadt, Walls and White12,Reference Hart, Bond and Siegelman29 In a recent national survey, EM residents reported experiencing simulation-based education in the aforementioned content areas and, in fact, perceived the amount of time dedicated to rarely encountered resuscitation scenarios such as obstetrical emergencies and pediatric resuscitations to be inadequate.Reference Russell, Hall and Hagel14 Motivating factors for determining content best suited to simulation-based education revolve around the desire to train for high stakes situations that are rarely encountered and the need to practice more frequently encountered clinical scenarios that expose patients to risk when performed by novices.Reference Ziv, Wolpe, Small and Glick30 Patient safety has been referred to as the principal motivator for the adoption of simulation-based education.Reference Cook, Andersen, Combes, Feldman and Sachdeva1 In an era when patient safety and quality of care have become a significant focus of medical practice, simulation offers an ideal training environment, as individuals can engage in sustained, deliberate practice and learn from errors with no risk to patient harm.Reference Rodriguez-Paz, Kennedy and Salas31
The panel also identified 32 “extended” curricular topics that are also considered to be important in EM training and should be considered for inclusion in a national curriculum based on local resources and perceived needs. These topics tended to reflect more commonly encountered high-acuity clinical scenarios such as anaphylaxis and gastrointestinal bleeding. The inclusion of “extended” topics enables programs to customize simulation-based education to serve their educational needs and local context; factors influenced by variables such as program size, clinical case mix, cultural and socioeconomic demographics of presenting patients, available simulation resources, and funding. Interestingly, despite a previous study in which trainees reported inadequate exposure to obstetrical emergencies,Reference Russell, Hall and Hagel14 topics in this domain reached consensus as “extended” topics, rather than “core.” We suspect that the Delphi panellists considered the need for unique manikins required to conduct obstetrical-related simulation training and were, therefore, hesitant to list them as “core” topics.
The exclusion of some topics was surprising, including acute coronary syndrome, environmental injuries (e.g., burns), biologic agents of warfare, maxillofacial trauma, cervical, and thoracolumbar spine injuries, as well as many specific toxic exposures and the approach to the violent and agitated patient. Many of these topics reflect infrequently experienced high-acuity scenarios and, therefore, are considered well suited for simulation-based education.Reference Ziv, Wolpe, Small and Glick30 Additionally, there is literature examining simulation-based training of some of the aforementioned topics such as de-escalation techniques and disaster management.Reference Wong, Auerbach and Ruppel32–Reference Franc, Nichols and Dong34 The exclusion of these topics is in contrast to prior simulation studies. Possible explanations for this difference may be the challenge of maintaining physical fidelity in the simulation environment or a lack of local expertise and/or resources.
The Delphi methodology is considered one of the most ideal methods for achieving informed group consensus on a complex topic. Through rigorous item generation and the provision of expert professional judgment, the Delphi methodology provides evidence of content validity of the proposed curricular topics and helps to enhance the generalizability of the results.Reference Goodman35 The Delphi method, however, is not without its limitations. First, there are no formal guidelines regarding panel selection or size. To minimize bias, we defined selection criteria a priori to ensure that the panellists had broad expertise related to EM training and simulation. We were limited, however, in that all panellists were Canadian and one RCPSC-accredited EM training program was not represented. Canadian College of Family Physicians (CCFP) EM training programs were also not specifically represented, although there is significant overlap between RCPSC and CCFP programs at many participating sites, with shared training experiences and resources. As such, the study findings may not be generalizable to these programs. Eight panellists, all of whom were program directors, did not self-identify as simulation educators. While program directors provide a unique perspective of the overarching EM curriculum, their knowledge of simulation-based education may be limited. Additionally, 69% of the Delphi panellists were male. Although this reflects the current demographics of Canadian EM physicians (69.4% male in 201836), it may introduce some bias. We aimed to include at least 30 experts, as the reliability of a panels’ aggregate judgment is minimally improved beyond 30 participants.Reference Delbecq, Van De Ve and Gustafson37 Additionally, our response rate was high across rounds. To increase methodological rigour, we followed recently proposed standardized methodological criteria for Delphi studies, including a clear objective and a priori determination of criteria for: 1) selecting Delphi panellists; 2) dropping and/or combining items; and 3) consensus.Reference Diamond, Grant and Feldman26 Finally, while this study attempted to define key content for simulation-based education in EM, it has not articulated how best to functionally integrate these topics into a competency-based curriculum to optimize training and assessment. Moving forward, it will be important to engage key stakeholders from across Canada to create a national simulation curriculum and accompanying assessment plan based on the proposed topics to ensure transferability across EM training programs. Formulation of a national working group would also help to maximize buy-in and facilitate sharing of scenarios, assessment tools, and other materials.
CONCLUSIONS
A national Delphi study enabled us to define EM curricular content best suited for simulation-based education. This will provide the foundation for improved integration of simulation into postgraduate EM training programs, to supplement clinical training and assessment and optimize learning. With the implementation of CBME and acknowledgement of the substantial resources required for simulation-based education, it will be important to prioritize these key topics when developing and revising a national EM simulation curriculum.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/cem.2019.348
Acknowledgements
Study concept and design: Kester-Greene, Walsh; acquisition, analysis, or interpretation of data: Kester-Greene, Hall, and Walsh; critical revision of the manuscript for important intellectual content: Kester-Greene, Hall, and Walsh; and final manuscript approval: Kester-Greene, Hall, and Walsh.
Competing interests
None. CMW holds a Career Development Award from the Canadian Child Health Clinician Scientist Program. No funding organization had any role in the design and conduct of the study, collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.
