Hostname: page-component-76d6cb85b7-8p85h Total loading time: 0 Render date: 2026-07-16T08:12:42.343Z Has data issue: false hasContentIssue false
  • English
  • Français

The Challenge of Mass Casualty Incident Response Simulation Exercise Design and Creation: A Modified Delphi Study

Published online by Cambridge University Press:  23 May 2023

Eric S Weinstein Open the ORCID record for Eric S Weinstein [Opens in a new window] ,
Michelangelo Bortolin Open the ORCID record for Michelangelo Bortolin [Opens in a new window] ,
Hamdi Lamine Open the ORCID record for Hamdi Lamine [Opens in a new window] ,
Teri Lynn Herbert Open the ORCID record for Teri Lynn Herbert [Opens in a new window] ,
Ives Hubloue Open the ORCID record for Ives Hubloue [Opens in a new window] ,
Sofie Pauwels Open the ORCID record for Sofie Pauwels [Opens in a new window] ,
Rita V Burke Open the ORCID record for Rita V Burke [Opens in a new window] ,
Mark X Cicero ,
Phoebe O Toups Dugas  and
Elizabeth O Oduwole
...Show all authors
Eric S Weinstein*
Affiliation:
CRIMEDIM, Center for Research and Training in Disaster Medicine, Humanitarian Aid and Global Health, Università del Piemonte Orientale, Novara, Italy Department for Sustainable Development and Ecological Transition, Università del Piemonte Orientale, Vercelli, Italy
Michelangelo Bortolin
Affiliation:
CRIMEDIM, Center for Research and Training in Disaster Medicine, Humanitarian Aid and Global Health, Università del Piemonte Orientale, Novara, Italy Department of Translational Medicine, Università del Piemonte Orientale, Novara, Italy BIDMC Disaster Medicine Fellowship, Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts USA
Hamdi Lamine
Affiliation:
CRIMEDIM, Center for Research and Training in Disaster Medicine, Humanitarian Aid and Global Health, Università del Piemonte Orientale, Novara, Italy Department for Sustainable Development and Ecological Transition, Università del Piemonte Orientale, Vercelli, Italy Faculty of Medicine, Ibn Aljazzar of Sousse, University of Sousse, Sousse, Tunisia
Teri Lynn Herbert
Affiliation:
Medical University of South Carolina Library, Charleston, South Carolina, USA
Ives Hubloue
Affiliation:
Research Group on Emergency and Disaster Medicine, Vrije Universiteit Brussel, Brussels, Belgium
Sofie Pauwels
Affiliation:
Research Group on Emergency and Disaster Medicine, Vrije Universiteit Brussel, Brussels, Belgium Department of Population and Public Health Sciences and Department of Pediatrics, USC Gehr Family Center for Health Systems Science & Innovation, Keck School of Medicine, University of Southern California, Los Angeles, California, USA
Rita V Burke
Affiliation:
Department of Population and Public Health Sciences and Department of Pediatrics, USC Gehr Family Center for Health Systems Science & Innovation, Keck School of Medicine, University of Southern California, Los Angeles, California, USA
Mark X Cicero
Affiliation:
Department of Pediatrics, Yale University School of Medicine, New Haven, Connecticut, USA
Phoebe O Toups Dugas
Affiliation:
Department of Computer Science, New Mexico State University, Las Cruces, New Mexico, USA
Elizabeth O Oduwole
Affiliation:
General Hospital, Apapa, Lagos, Nigeria
Luca Ragazzoni
Affiliation:
CRIMEDIM, Center for Research and Training in Disaster Medicine, Humanitarian Aid and Global Health, Università del Piemonte Orientale, Novara, Italy Department for Sustainable Development and Ecological Transition, Università del Piemonte Orientale, Vercelli, Italy
Francesco Della Corte
Affiliation:
CRIMEDIM, Center for Research and Training in Disaster Medicine, Humanitarian Aid and Global Health, Università del Piemonte Orientale, Novara, Italy Department of Translational Medicine, Università del Piemonte Orientale, Novara, Italy
*
Corresponding author: Eric S Weinstein, Email: eswein402@gmail.com
Rights & Permissions [Opens in a new window]

Abstract

Background:

A Mass Casualty Incident response (MCI) full scale exercise (FSEx) assures MCI first responder (FR) competencies. Simulation and serious gaming platforms (Simulation) have been considered to achieve and maintain FR competencies. The translational science (TS) T0 question was asked: how can FRs achieve similar MCI competencies as a FSEx through the use of MCI simulation exercises?

Methods:

T1 stage (Scoping Review): PRISMA-ScR was conducted to develop statements for the T2 stage modified Delphi (mD) study. 1320 reference titles and abstracts were reviewed with 215 full articles progressing for full review leading to 97 undergoing data extraction.

T2 stage (mD study): Selected experts were presented with 27 statements derived from T1 data with instruction to rank each statement on a 7-point linear numeric scale, where 1 = disagree and 7 = agree. Consensus amongst experts was defined as a standard deviation ≤ 1.0.

Results:

After 3 mD rounds, 19 statements attained consensus and 8 did not attain consensus.

Conclusions:

MCI simulation exercises can be developed to achieve similar competencies as FSEx by incorporating the 19 statements that attained consensus through the TS stages of a scoping review (T1) and mD study (T2), and continuing to T3 implementation, and then T4 evaluation stages.

Keywords

MCI simulationMCI full scale exercisetranslational sciencescoping reviewdelphi study

Information

Type
Original Research
Information
Disaster Medicine and Public Health Preparedness , Volume 17 , 2023 , e396
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.
Copyright
© Universitá degli Studi del Piemonte Orientale, 2023. Published by Cambridge University Press on behalf of Society for Disaster Medicine and Public Health, Inc.

Introduction

The complexity of an individual first responder participating in a sudden onset disaster mass casualty incident response (MCI), Reference Auf der Heide1 involves their patient care or duty skills, and the layering of their participation in their agency’s Incident Management System (IMS). 2 The first responder’s usual, daily standard operation command and control, changes once the MCI is declared. The first responder must have the skills required for MCI patient care or duty station, but they must also operate in the rarely utilized agency MCI-plan IMS. A live full-scale exercise (FSEx) examines these relationships; the communication between the first responder and the patient, and the responder’s IMS to prepare the first responder and system for an actual MCI response (Figure 1).

Figure 1.

Relationships of the first responder in an MCI.

IC = Incident Command

IMS = Incident Management System

Agency = The pre-hospital or hospital, government or non-government, agency, organization, group, hospital, or health care delivery system of the individual first responder

Jurisdiction = The lead command and control health authority of the MCI response

Direct 2-way interaction between the first responder with the patient and simultaneously with their agency incident command

Jurisdiction provides command and control to the First Responder

First Responder may provide information to the Jurisdiction Incident Command

An FSEx presents obstacles and challenges for designers, developers, planners, and exercise conductors to fulfill the mission of ensuring first responder competencies and MCI plan effectiveness (Supplemental digital content Figure 1s). 3 Staff, stuff, and structures (SSS), Reference Kaji, Koenig and Bey4 may be stretched thin to cover the daily need for medical services, let alone devote staff for training to gather appropriate observers, evaluators, volunteers to recruit/ moulage patient actors, and the technical staff to conduct an FSEx. Furthermore, government permits may be needed to alter road, highway, plane, and train transportation routes and/ or ensure that ecological and environmental concerns are considered in the FSEx – all time-consuming activities.

Travel to a conference that features an FSEx adds a budgetary constraint and creates a void in the staffing schedule, and there may be no capable replacement for the first responder. Education and training budgets must include resources to devote to training for the rare MCI, when emergency medical services (EMS) and health care facility administrators must also assure appropriate education and training for staff to achieve competencies for daily operations. Even allocating assets for the planning and execution of a valid FSEx may be difficult, regardless of regulatory expectations (e.g., nuclear, airplane, train, manufacturing, etc.). 5,6 Financial and time constraints may limit the discussion and agreement process by health authorities, regulators, first response agency, and health care facility administrators. (Supplemental Digital Content Figure 2)

This dilemma was made vivid with the Coronavirus disease-2019 (COVID-19) response worldwide. Staff and facilities struggled to acquire and maintain SSS to mount a safe COVID-19 response. There was no way to gather responsible actors, create or revise an MCI plan, or develop and execute an FSEx. Face-to-face classroom learning and travel to conferences became a public health casualty to prevent the spread of COVID-19.

Simulation and serious gaming (Simulation) for skills-based medical education has become an effective adjunct to, or has supplanted conventional methodologies, and was considered to replace face-to-face MCI response education, and training to achieve similar or equal first responder competencies. The challenge of the realism that an FSEx creates for first responders’ critical decisions has been postulated to be similar using other simulation methodologies but was not ready to be ‘taken off the shelf’ to incorporate into an MCI simulation exercise of any scale.

This study is designed to address the scope of the NO-FEAR Project (Network Of practitioners For Emergency medicAl systems and cRitical care) under work package number 57: education and training of personnel and volunteers, regarding technical and non-technical skills, teamwork, critical thinking, clinical care, incident management, and psychological support. NO-FEAR asks for new simulation tools for education and training to achieve MCI response competencies (e.g., high-fidelity and live simulation, 2-dimensional enhanced and immersive simulation software, tools to provide quantitative, and qualitative evaluation of responders’ performance during exercises). 7 The present research is aimed at describing the design needs for future MCI response training simulations through a combination of literature review and expert assessment that achieves equal or similar competencies as a FSEx. A competency component of the MCI response training simulation design is to approximate the realism of the FSEx to better prepare the first responder for an actual dynamic MCI response.

Methods

This translational science (TS) study begins with the TS question (T0) Reference Weinstein, Cuthbertson, Ragazzoni and Verde8 : How can first responders achieve similar MCI competencies as an FSEx using MCI simulation exercises?

The objective of this study produced the first stage (T1) scoping review and second stage (T2) modified Delphi study (mD) consensus statements that can be offered to formulate MCI simulation exercise guidelines in the future TS third (T3) MCI simulation exercise creation stage. The fourth TS (T4) stage that follows will study these MCI simulation exercises to determine if similar or equal FSEx competencies were obtained.

T1: Scoping review

A systematic review following the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR), 9 was conducted from August 2021 through October 2021. Scoping reviews synthesize knowledge following a defined scientific process to identify sources that can be interrogated using a defined data extraction tool to determine concepts, theories, and knowledge gaps. Reference Munn, Peters, Stern, Tufanaru, McArthur and Aromataris10 The study of MCI education, training, and exercises that lead to competencies is multi-disciplinary, creating a body of knowledge that is heterogeneous, and thus apropos for PRISMA-ScR methodology. Reference Tricco, Lillie and Zarin11

Literature search criteria

A T0 research question was developed using the Patient, Intervention, Control/ Comparison, Outcome (PICO) standard to frame the search strategy. Reference da Costa Santos, de Mattos Pimenta and Nobre12

  1. 1) Population: MCI, pre-hospital and hospital providers, simulation training exercise, or drill.

  2. 2) Intervention: Not MCI simulation training exercise or drill, individual duty station competencies.

  3. 3) Comparison: Individual intra-agency and inter-agency IMS competencies.

  4. 4) Outcomes: Intra-agency and inter-agency IMS competencies.

Literature search methods

Inclusion criteria

The search strategy included only terms relating to or describing the intervention (Table 1). The review included English-language papers published from January 1, 1990, to July 1, 2021, in these databases:

  1. 1) PubMed (National Center for Biotechnology Information, National Institutes of Health, Bethesda, Maryland, USA)

  2. 2) SCOPUS (The largest and most comprehensive abstract and citation database of peer-reviewed literature from Elsevier, Netherlands)

  3. 3) CINAHL (Cumulative Index to Nursing and Allied Health Literature, EBSCO, Elton B Stephens Company, Ipswich, Massachusetts, USA)

  4. • DTIC (Defense Technical Information Center, United States Department of Defense) database for reports and other government publications

  5. • ECRI (Emergency Care Research Institute, Plymouth Meeting, Pennsylvania, USA) Trust for published guidelines

  6. • PsycInfo (American Psychological Association, Washington District of Columbia, USA) validating surveys/questionnaires on disaster training databases.

Table 1.

T1 scoping review search terms

Finally, an ancestry search was also performed to identify additional references from the bibliography of references when appropriate.

Exclusion criteria

References from the databases that did not meet the inclusion criteria, specifically did not study, or report an MCI/ MCI exercise, were excluded.

PRISMA ScR Figure 2

2 review authors independently screened 1320 reference titles and abstracts to determine if inclusion criteria were met; any disagreement was resolved by discussion. Then each of the remaining 215 full articles was read by both authors to determine if inclusion criteria were met, and again any disagreement was resolved by discussion.

Of the remaining 97 included articles, Reference Achatz, Friemert and Trentzsch13–Reference Wilkerson, Avstreih, Gruppen, Beier and Woolliscroft110 47 discussed competencies, Reference Andreatta, Maslowski and Petty16–Reference Austin, Bastepe-Gray and Nelson19,Reference Betka, Bergren and Rowen24,Reference Carenzo, Bazurro, Colombo, Petrini and Ingrassia29–Reference Collander, Green, Millo, Shamloo, Donnellan and DeAtley39,Reference Dickerson-Young, Keilman, Yoshida, Jones, Cross and Thomas41–Reference Ferrandini Price, Escribano Tortosa and Nieto Fernandez-Pacheco44,Reference Foo, So and Lu46–Reference Hartman, Daines, Seto, Shimshoni, Feldman and LaBrunda55,Reference Koutitas, Smith and Lawrence71,Reference Mills, Dykstra and Hansen84,Reference Obaid, Bailey and Wheeler90,Reference Phattharapornjaroen, Glantz, Carlström, Dahlén Holmqvist and Khorram-Manesh91,Reference Rusling, Masin and Voss95,Reference Saber, Strout, Caruso, Ingwell-Spolan and Koplovsky96,Reference Sena, Forde, Yu, Sule and Masters99,Reference Shah, Pierce, Roblin, Walker, Sergio and Waterworks100,Reference So, Dziuban and Franks103–Reference Wilkerson, Avstreih, Gruppen, Beier and Woolliscroft110 and were split between 2 authors to extract data into an Excel database (Microsoft Corp., Redmond, Washington, USA) that was developed using themes and subthemes from MCI exercise publications to derive statements for the mD. 111–115 (Supplemental Digital Content link to Excel database)

T2: Modified Delphi study

The mD method permits experts from various locations to independently review statements to attain consensus when no consensus existed previously. 9 The first stage of the mD began after the final database was analyzed by lead authors who created initial draft statements based on the most relevant datapoints. Then an internal focus group of authors discussed and edited these statements to meet the format of Delphi statements for the Stat59 statistical analysis platform (Stat59 Services Limited, Alberta, Canada). 116 An external focus group comprised of content experts in the field of MCI simulation exercises and NO-FEAR partners was established to further discuss and edit the draft statements to be clear and concise. After a video conference moderated by the lead authors, these experts discussed the draft statements. External focus group participants performed asynchronous editing via a shared online document producing the final 27 statements for the second stage of the mD.

A list of content experts, derived from the authors of included references, academicians, and researchers studying MCI simulation exercises was created to establish the mD expert panel. Introductory emails were sent explaining the project objectives and the mD to these mD experts.

mD experts that agreed were sent an email from the Stat59 (Stat59 Services Limited, Alberta, Canada) mD organizational program on the day that the mD began with a link to the Stat59 (Stat59 Services Limited, Alberta, Canada) website consent page. Each mD expert registered an account, validated it, and were sent a new email to log into their secure webpage to begin the first mD expert consensus round. 3 days later, mD experts that had not logged into the system were asked to verify their access and log in and asked to notify the author if they had not received the introductory email, with instructions on how to ensure future emails were received.

Once the mD experts logged in, they were provided with a formal explanation of the mD methodology and informed consent was obtained. For informed consent (Supplemental Digital Content link to Stat59 Consent Page), participants were notified that they were anonymous volunteers who could withdraw at any time, that participation or withdrawal would not impact their employment, and that their data was secure (Supplemental Digital Content link to Stat59 Security Page).

The next page was the list of 27 statements that were finalized in the T2 external focus group with instruction to rank each statement on a 7-point linear numeric scale, where 1 = disagree and 7 = agree. With this initial set of statements, the mD expert was asked to answer 4 demographic questions. Consensus amongst mD experts was defined as a standard deviation ≤ 1.0.

Statements that attained consensus after this first mD expert round were included in the final report. Each mD expert received an email from the Stat59 (Stat59 Services Limited, Alberta, Canada) program after the first mD expert round and a reminder email from the author shortly afterwards to log back into their Stat59 page that showed the mean response of all the mD experts for each statement that did not attain consensus, their own response for that specific remaining statement, and were asked to reconsider their 7-point linear numeric scale for these remaining statements.

This process was repeated after the second mD expert round with statements that attained consensus included in the final report. The statements that did not attain consensus were advanced to the third and final round with the mD experts asked to reconsider these statements. This third mD expert round produced the final statements that attained consensus to add to the first and second round consensus statements in the final report. Remaining statements after this third round were the final statements that did not attain consensus.

The McLeod Health Institutional Review Board Office (Florence, South Carolina USA) has determined that this study does meet the exemption criteria found at 45 CFR 46.104(d)(2). 117

Results

As summarized in Figure 3, 35 mD experts confirmed their participation and established a unique account on the Stat59 website (Table 2). 31 completed the first mD expert round that was open from January 17, 2022, until January 30, 2022. 5 statements attained statistical significance with a standard deviation ≤ 1.0 after this first mD expert round, and achieved consensus (Table 3, first round, first section in bold). The 22 statements that did not attain statistical significance, with standard deviation > 1.0, were advanced to the second mD expert round.

Figure 3.

Modified Delphi statement creation.

Table 2.

Modified Delphi expert panel demographics

Table 3.

Statements that attained consensus

Bold T2 mDE Rounds 1 and 3

Not Bold T2 mDE Round 2

29 mD experts completed the second mD expert round that was open from January 31, 2022, to February 19, 2022. 12 of the 22 statements that advanced to the second mD expert round achieved consensus (Table 3, second round middle section). The remaining 10 statements were unable to attain consensus and advanced to the third mD expert round.

29 mD experts completed the third and final mD expert round that was open from February 23, 2022, to March 9, 2022. 2 of the remaining 10 statements achieved consensus, so a total of 19 statements achieving consensus. (Table 3 third round, last section in bold). The remaining 8 statements were unable to attain consensus after 3 T2 mD expert rounds and were not recommended for T3 consideration (Table 4).

Table 4.

Statements that did not attain consensus

Limitations

The PRISMA-ScR produced a qualitative analysis of published studies and reports. Though the search followed this process, there may have been references that were not discovered.

The Delphi method seeks to arrive at group consensus by the aggregate of a panel of experts who rate a statement on a linear numeric scale. Internal validity is largely unknown; therefore, stability of response is more accurate to determine consensus or lack of consensus.

The objective of the distribution of mD experts was to represent MCI simulation exercise designers, developers, and those that would execute an exercise. The distribution of mD experts favor resource rich countries but all experts are involved in MCI exercises in a way.

An essential component of the Central Limit Theorem is that the average of sample means will be the population mean, or if 1 finds the average of all of the standard deviations in the sample, then 1 will find the actual standard deviation for the population. 118 This will hold true regardless of whether the source population is normal or skewed, provided the sample size is sufficiently large (usually ≥ 30, the number of mD experts in this study per round averaged 29.67). 119 The application of the Central Limit Theorem to this study infers that the 19 statements that attained consensus can be recommended to assist the T3 development of guidelines for MCI simulation exercises.

Discussion

Providing MCI training can be challenging for several reasons: (1) provider schedules are often erratic and involve long hours; (2) there is a temporal dissociation between disaster response training and the application of the skills, leading to cognitive skill decay at the time the skill needs to be performed; and (3) traditional learning methods, such as didactic presentations, tabletop simulations, and FSExs require the physical presence of learners and educators at a certain place and time (synchronous learning). Reference Cicero, Whitfill and Walsh36 In response, mD experts agreed, educators should design periodic simulation training to maintain competencies of rarely used skills that will deteriorate over time.

Briggs et al. showed that heterogeneous organizations with different command structures and missions participate in the response to a disaster, and therefore a clear objective of an MCI plan is to define the IMS of a region. Reference Briggs120 An example of a 2017 FSEx objective of Regional Operability in Ohio (United States) as reported by McElroy is that ‘Participants shall identify the management structure to support effective operational coordination between all agencies and entities.’ Reference McElroy, Steinberg, Keller and Falcone83 The mD experts agreed that the simultaneous integration of patient care or duty skills (e.g., triage, utilization of resources, communication) and IMS skills should be incorporated into the exercise design to achieve the competency for the individual to recognize and assume their position in their agency MCI plan through situation awareness and critical decision-making.

As the layers of FSEx objectives are incorporated into the FSEx design, mD experts agreed that patient care or duty competencies can be evaluated using distinct separate modalities or as part of an exercise designed to include these skills with IMS skills. mD experts agreed that multiple exercise modalities should be considered to minimize time, cost, and impact to non-participants affected by the exercise. Conceptually a simulation exercise can be created to achieve one of the many IMS competency objectives achieved by an FSEx as required by health authorities, regulators, IMS partners, and stakeholders:

  1. 1) Utilize effective means of inter- and intra-agency communication through redundant modalities;

  2. 2) Achieve the appropriate continuum of patient triage and treatment from the initial evaluation to definitive care;

  3. 3) Deliver the right patient to the right alternate care facility or definitive health care facility capable of attending to the injuries of that patient;

  4. 4) Maintain accurate patient tracking leading to expedient hospital registration;

  5. 5) Manage resources utilizing the supply chain in a resource scarce environment; and

  6. 6) Manage information and media through intelligence acquisition, vetting, and transmission.

MCI simulation education and training is multimodal: lectures, readings, and individual hands-on skill sessions to meet the requirements of the individual’s MCI patient care duties. To do that, education must be active, interactive, and experiential, as well as participatory. Reference Kagawa and Selby121 The additional layers in MCI education and training acknowledge that the individual is part of the IMS. The evolution of MCI simulation exercises will appreciate these complex simultaneous actions of the individual as they manage patient care and their role in the IMS, to achieve the same competencies that the FSEx will produce. To approximate the success of the FSEx, MCI exercise simulation must depend on high physical fidelity to develop the individual’s manual abilities, as well as high conceptual fidelity to develop clinical reasoning and problem-solving skills.

Lastly, high emotional or experiential fidelity, Reference Johnson, Brindley and Gillman122 where the learner is emotionally invested in the simulation to a degree that memories of the experience are believable, will develop the individual’s retention of the material. Reference Ferrandini Price, Escribano Tortosa and Nieto Fernandez-Pacheco44 There must be emotional learning in which positive emotions under stress facilitate greater retention of data. This success is not completely based on the realism of the simulation, but on the commitment of the participants in their roles; that there is an adequate connection between those involved; and that the student manages to actively link the social, psychological and clinical experiences lived. 123 The mD experts agreed that the realism created through moulage of simulated patients and environmental special effects in any simulation setting should focus on the participant’s situation awareness and not deter from the overall exercise objectives. In 2019, Saunders et al. Reference Saunders, Davey, Bayerl and Lohrmann97 published a study that demonstrated that virtual-reality-based law enforcement trainings, either by themselves or in combination with traditional hands-on training, can be as effective as highly resource-intensive practical training sessions. 123

Individuals suspend disbelief during the FSEx to approximate the physical and emotional stress of an actual MCI (e.g., the demand of an unknown number of patients with unknown injuries, an unknown supply of SSS in a resource-scarce environment with sensory overload, and fear for their own safety, as well as a failure to accomplish their assignments, or letting their team/ agency down). A successful MCI simulation exercise would match the individual’s pressure for realism through their critical decisions suspending disbelief in the simulation environment without the sensory elements inherent in an FSEx. This can be achieved following design principles explained by Alharthi et al., along with the understanding that the designers must simulate the actual experience making the exercise practical through highly cognitive intense work and physical exertion immersing the individual in the simulation. Reference Alharthi, LaLone and Khalaf124 (Supplemental Digital Content Table 1)

To accomplish the objectives of an FSEx, controllers plan and manage exercise play, set up/ operate the exercise site, and act in the roles of organizations, agencies, or individuals that are not playing in the exercise. Reference Toups, Hamilton and Alharthi125 Controllers direct the pace of the exercise, provide key data to players, and may prompt or initiate certain player actions to ensure exercise continuity. Simulators or facilitators provide feedback and cues based on predetermined expected actions of players as well as injects in response to player’s actions that are not expected to maintain the flow of the exercise and to instruct. In addition, they issue exercise material to players as required, monitor the exercise timeline, and supervise the safety of all exercise participants and the surrounding environment.

mD experts agreed that an exercise controller should build realism based on a prepared action script to anticipate and deliver injects based on player’s actions and responses to evaluate non-technical communication skills. MCI simulation exercise creators have the challenge to integrate live or reflex injects based on player’s responses to the scenario, changes in patient’s clinical conditions or other player’s actions. An FSEx occurs in real time with all the agencies simultaneously responding; mD experts agreed that the challenge of MCI simulation exercises is the appreciation of the passage of time to accomplish an action as if in the real time of an FSEx. To achieve this level of realism, mD experts agreed that the exercise design should have modality technicians supporting the exercise to be able to recognize and address any participant struggling with the modality to promote the overall objectives.

A crucial component of any MCI simulation is the debriefing process following the activity; this provides a structured reflection for participants to analyze and self-correct their behavior, decisions, and thought processes to promote cognitive accommodation and assimilation of their learning experience into future professional practice. Reference Greco, Lewis, Sanford, Sawin and Ames52 mD experts agreed that observers and evaluators should be specific content experts external to the exercise and use validated template scoring tools to evaluate competencies of players, the exercise itself, and any documentation that is required of regulators or the health authority. These specific content experts can utilize ‘debriefing through meaningful learning,’ Reference Dreifuerst126 to provide exercise objectives education. mD experts agreed that this debrief can discover latent safety threats through a frank non-punitive discussion to uncover potential actions that may lead to medical errors. Reference Carmichael, Mastoras and Nolan30 mD experts further agreed that a semi-structured debrief based on validated formats should include all stakeholders to improve the exercise, using open-ended questions to determine MCI simulation exercise areas of improvement.

Conclusion

The modified Delphi experts agreed that the simultaneous integration of individual duty and incident management skills should be incorporated into simulation MCI exercise design to achieve competencies depending on high physical fidelity to develop the individual’s manual abilities, as well as high conceptual fidelity, to develop the individual’s clinical reasoning and problem-solving skills.

MCI simulation exercises can be developed to achieve similar competencies as FSExs incorporating the 19 statements that attained consensus through the TS stages of a scoping review (T1) and mD (T2). The TS process should continue with development of these exercises in the T3 implementation stage and then evaluated in the T4 stage.

Supplemental digital content

  1. 1) The database for this study can be found at: https://docs.google.com/spreadsheets/d/153CM_LhR8s0IRod9XbodeWK8wZN-m9NA/edit#gid=876207018

  2. 2) The Stat59 Security Page can be found at: https://www.stat59.com/about/security

  3. 3) The Stat59 Consent Page can be found at: https://www.stat59.com/projects/delphi-consent-view?pid=145.

Supplementary material

For supplementary material accompanying this paper visit https://doi.org/10.1017/dmp.2023.71

Acknowledgements

Samra Baxley, George Teo Voicescu and Dalton Weinstein for their administrative assistance; Monica Linty for her assistance with NO-FEAR; Ricardo Galesso with the External Focus Group; The modified Delphi Experts consented to be acknowledged: B Adini, E Alpert, P Andreatta, L Anthony, T Baxley, K Biggers, L Carenzo, D Cone, L Greci Cooke, M Dittmar, J Dohaney, C Evans, NP Foo, E Fragniere, P Halpern, A Hewitt, E Hsu, P Jacobson, K Johnson, D Lauwaert, S Magalini, G Mastoras, B Mills, S Morse, E Noste, M Pardo, P Pucher, A Redmond, R Ruffing, J Ryder, D Saber, P Severin, and J Tochkin.

Co-authors are DMPHP deputy editors (other co-authors are known to the DMPHP editorial staff and potential reviewers). The manuscript was blinded throughout the review process.

Author contributions

Study Concept: ESW, LR, FDC; Study Design: ESW, MB, LR; Scoping Review: ESW, MB, HL, TLH; Internal Focus Group: ESW, MB, HL, TLH, IH, SP, LR; External Focus Group: ESW, MB, HL, TLH, IH, SP, RVB, MXC, POTD, EO, LR; Primary writing: ESW; Editing, revising: ESW, HL, TLH, IH, SP, RVB, MXC, POTD, EO, LR

Competing interest

None

Funding

This NO-FEAR project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 786670 with only funds allocated for the STAT59 statistical program fee.

Ethics

The McLeod Health Institutional Review Board Office (Florence, South Carolina USA) has determined that this study does meet the exemption criteria found at 45 CFR 46.104(d)(2).

Abbreviations

CINAHL, Cumulated Index to Nursing and Allied Health (database); COVID-19, Coronavirus disease-2019 ; DTIC, Defense Technical Information Center (US Department of Defense database); EBSCO, Elton B Stephens Company; ECRI, Emergency Care Research Institute (guidelines database); ED, Emergency Department; EMS, Emergency Medical Services; FSEx, Full-Scale Exercise; IMS, Incident Management System; MCI, Sudden Onset Mass Casualty Incident Response; mD: modified Delphi Study; NO-FEAR, Network Of practitioners For Emergency medicAl systems and cRitical care; PICO, Patient, Intervention, Control/ Comparison, Outcome (framework); PRISMA-ScR, Preferred Reporting Items for Systematic reviews and Meta-Analyses, extension for Scoping Reviews; Simulation, Simulation and Serious Gaming; SSS: Staff, Stuff and Structures; Stat59, Stat59 Services Limited, Edmonton, Alberta, Canada (Statistical Analysis Platform); TS, Translational Science; WHO, World Health Organization

References

Auf der Heide, E. Disaster response: principles of preparation and coordination. St. Louis: Mosby; 1989.Google Scholar
Federal Emergency Management Agency (FEMA). National Incident Management System. https://www.fema.gov/emergency-managers/nims. Accessed November 1, 2022.Google Scholar
World Health Organization (WHO). WHO simulation exercise manual. Published February 2017. https://apps.who.int/iris/bitstream/handle/10665/254741/WHO-WHE-CPI-2017.10-eng.pdf;sequence=1. Accessed August 20, 2021.Google Scholar
Kaji, A, Koenig, KL, Bey, T. Surge capacity for healthcare systems: a conceptual framework [published correction appears in Acad Emerg Med. 2007;14(1):22.]. Acad Emerg Med. 2006;13(11):1157-1159. doi: 10.1197/j.aem.2006.06.032 CrossRefGoogle Scholar
Centers for Medicare and Medicaid Services. Emergency preparedness rule. https://www.cms.gov/Medicare/Provider-Enrollment-and-Certification/SurveyCertEmergPrep/Emergency-Prep-Rule. Accessed March 3, 2022.Google Scholar
USNRC, Federal Emergency Management Agency (FEMA). Criteria for preparation and evaluation of radiological emergency response plans and preparedness in support of nuclear power plants. https://www.fema.gov/sites/default/files/2020-06/nureg-0654_fema-rep-1_rev2_change_summary__dec_2019.pdf. Accessed March 4, 2022.Google Scholar
NO-FEAR. Network Of practitioners For Emergency medicAl systems and cRitical care. https://no-fearproject.eu/. Accessed May 6, 2021.Google Scholar
Weinstein, ES, Cuthbertson, JL, Ragazzoni, L, Verde, M. A T2 translational science modified Delphi study: spinal motion restriction in a resource-scarce environment. Prehosp Disaster Med. 2020;35(5):538-545. doi: 10.1017/S1049023X20000862 CrossRefGoogle Scholar
PRISMA. PRISMA for scoping Reviews. https://www.prisma-statement.org/Extensions/ScopingReviews. Accessed July 10, 2021.Google Scholar
Munn, Z, Peters, MDJ, Stern, C, Tufanaru, C, McArthur, A, Aromataris, E. Systematic review, or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143. doi: 10.1186/s12874-018-0611-x CrossRefGoogle ScholarPubMed
Tricco, AC, Lillie, E, Zarin, W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467-473. doi: 10.7326/M18-0850 CrossRefGoogle ScholarPubMed
da Costa Santos, CM, de Mattos Pimenta, CA, Nobre, MR. The PICO strategy for the research question construction and evidence search. Rev Lat Am Enfermagem. 2007;15(3):508-511. doi: 10.1590/s0104-11692007000300023 CrossRefGoogle ScholarPubMed
Achatz, G, Friemert, B, Trentzsch, H, et al. Terror and disaster surgical care: training experienced trauma surgeons in decision making for a MASCAL situation with a tabletop simulation game. Eur J Trauma Emerg Surg. 2020;46(4):717-724. doi: 10.1007/s00068-020-01441-x CrossRefGoogle ScholarPubMed
Alexander, AJ, Bandiera, GW, Mazurik, L. A multiphase disaster training exercise for emergency medicine residents: opportunity knocks. Acad Emerg Med. 2005;12(5):404-409. doi: 10.1197/j.aem.2004.11.025 CrossRefGoogle ScholarPubMed
Anan, H, Otomo, Y, Kondo, H, et al. Development of mass-casualty life support-CBRNE (MCLS-CBRNE) in Japan. Prehosp Disaster Med. 2016;31(5):547-550. doi: 10.1017/S1049023X16000686 CrossRefGoogle ScholarPubMed
Andreatta, PB, Maslowski, E, Petty, S, et al. Virtual reality triage training provides a viable solution for disaster-preparedness. Acad Emerg Med. 2010;17(8):870-876. doi: 10.1111/j.1553-2712.2010.00728.x CrossRefGoogle ScholarPubMed
Ashkenazi, I, Ohana, A, Azaria, B, et al. Assessment of hospital disaster plans for conventional mass casualty incidents following terrorist explosions using a live exercise based upon the real data of actual patients. Eur J Trauma Emerg Surg. 2012;38(2):113-117. doi: 10.1007/s00068-011-0154-x CrossRefGoogle ScholarPubMed
Atack, L, Bull, E, Dryden, T, Maher, J, Rocchi, M. An evaluation of learner perception of competency and satisfaction with three models of an interdisciplinary surge capacity course. J Allied Health. 2012;41(3):106-112.Google ScholarPubMed
Austin, EN, Bastepe-Gray, SE, Nelson, HW, et al. Pediatric mass-casualty education: experiential learning through university-sponsored disaster simulation. J Emerg Nurs. 2014;40(5):428-433. doi: 10.1016/j.jen.2014.05.015 CrossRefGoogle ScholarPubMed
Bartley, BH, Stella, JB, Walsh, LD. What a disaster?! Assessing utility of simulated disaster exercise and educational process for improving hospital preparedness. Prehosp Disaster Med. 2006;21(4):249-255. doi: 10.1017/s1049023x00003782 CrossRefGoogle ScholarPubMed
Bentley, S, Iavicoli, L, Boehm, L, et al. A simulated mass casualty incident triage exercise: SimWars [published correction appears in MedEdPORTAL. 2019;15:10879]. MedEdPORTAL. 2019;15:10823. doi: 10.15766/mep_2374-8265.10823 CrossRefGoogle Scholar
Berndt, H, Wessel, D, Mentler, T, Herczeg, M. Human-centered design of a virtual reality training simulation for mass casualty incidents. In: 10th International conference on virtual worlds and games for serious applications, VS-games 2018. 2018:8493427.CrossRefGoogle Scholar
Berndt, H, Wessel, D, Willer, L, Herczeg, M, Mentler, T. Immersion, and presence in virtual reality training for mass casualty incidents. In: Proceedings of the 15th ISCRAM Conference 2018. 2018(5):797-805.CrossRefGoogle Scholar
Betka, AA, Bergren, MD, Rowen, JL. Improving rural disaster response preparedness. Public Health Nurs. 2021;38(5):856-861. doi: 10.1111/phn.12924 CrossRefGoogle ScholarPubMed
Bieler, D, Franke, A, Blätzinger, M, et al. Evaluation of the terror and disaster surgical care course. Eur J Trauma Emerg Surg. 2020;46(4):709-716. doi: 10.1007/s00068-020-01418-w CrossRefGoogle ScholarPubMed
Bilek, N, Feldhofer, A, Moser, T. Virtual reality based mass disaster triage training for emergency medical services. In: IEEE Conference on virtual reality and 3D user interfaces abstracts and workshops 2021. 2021:13.CrossRefGoogle Scholar
Carenzo, L, Barra, FL, Ingrassia, PL, Colombo, D, Costa, A, Della Corte, F. Disaster medicine through Google Glass. Eur J Emerg Med. 2015;22(3):222-225. doi: 10.1097/MEJ.0000000000000229.CrossRefGoogle ScholarPubMed
Carenzo, L, Ragozzino, F, Colombo, D, Barra, FL, Della Corte, F, Ingrassia, PL. Virtual laboratory and imaging: an online simulation tool to enhance hospital disaster preparedness training experience. Eur J Emerg Med. 2018;25(2):128-133. doi: 10.1097/MEJ.0000000000000421 Google ScholarPubMed
Carenzo, L, Bazurro, S, Colombo, D, Petrini, F, Ingrassia, PL. An island-wide disaster drill to train the next generation of anesthesiologists: the SIAARTI academy experience. Disaster Med Public Health Prep. 2020;15(2):151-154. doi: 10.1017/dmp.2019.163 CrossRefGoogle ScholarPubMed
Carmichael, H, Mastoras, G, Nolan, C, et al. Integration of in situ simulation into an emergency department code orange exercise in a tertiary care trauma referral center. AEM Educ Train. 2020;5(2):e10485. doi: 10.1002/aet2.10485 Google Scholar
Castoldi, L, Greco, M, Carlucci, M, Lennquist Montán, K, Faccincani, R. Mass casualty incident (MCI) training in a metropolitan university hospital: short-term experience with MAss Casualty SIMulation system MACSIM®. Eur J Trauma Emerg Surg. 2022;48(1):283-291. doi: 10.1007/s00068-020-01541-8 CrossRefGoogle Scholar
Ceresa, IF, Savioli, G, Angeli, V, et al. Preparing for the maximum emergency with a simulation: a table-top test to evaluate bed surge capacity and staff compliance with training. Open Access Emerg Med. 2020;12:377-387. doi: 10.2147/OAEM.S267069 CrossRefGoogle ScholarPubMed
Cicero, MX, Whitfill, T, Munjal, K, et al. 60 seconds to survival: a pilot study of a disaster triage video game for prehospital providers. Am J Disaster Med. 2017;12(2):75-83. doi: 10.5055/ajdm.2017.0263 CrossRefGoogle ScholarPubMed
Cicero, MX, Whitfill, T, Overly, F, et al. Pediatric disaster triage: multiple simulation curriculum improves prehospital care providers’ assessment skills. Prehosp Emerg Care. 2017;21(2):201-208. doi: 10.1080/10903127.2016.1235239 CrossRefGoogle ScholarPubMed
Cicero, MX, Golloshi, K, Gawel, M, Parker, J, Auerbach, M, Violano, P. A tabletop school bus rollover: Connecticut-wide drills to build pediatric disaster preparedness and promote a novel hospital disaster readiness checklist. Am J Disaster Med. 2019;14(2):75-87. doi: 10.5055/ajdm.2019.0318 CrossRefGoogle ScholarPubMed
Cicero, MX, Whitfill, T, Walsh, B, et al. Correlation between paramedic disaster triage accuracy in screen-based simulations and immersive simulations. Prehosp Emerg Care. 2019;23(1):83-89. doi: 10.1080/10903127.2018.1475530 CrossRefGoogle ScholarPubMed
Claudius, I, Kaji, A, Santillanes, G, et al. Comparison of computerized patients versus live moulaged actors for a mass-casualty drill. Prehosp Disaster Med. 2015;30(5):438-442. doi: 10.1017/S1049023X15004963 CrossRefGoogle Scholar
Cole, LA, Natal, B, Fox, A, et al. A course on terror medicine: content and evaluations. Prehosp Disaster Med. 2016;31(1):98-101. doi: 10.1017/S1049023X15005579 CrossRefGoogle ScholarPubMed
Collander, B, Green, B, Millo, Y, Shamloo, C, Donnellan, J, DeAtley, C. Development of an ‘all-hazards’ hospital disaster preparedness training course utilizing multi-modality teaching. Prehosp Disaster Med. 2008;23(1):63-69. doi: 10.1017/s1049023x00005598 CrossRefGoogle ScholarPubMed
Cone, DC, Serra, J, Kurland, L. Comparison of the SALT and Smart triage systems using a virtual reality simulator with paramedic students. Eur J Emerg Med. 2011;18(6):314-321. doi: 10.1097/MEJ.0b013e328345d6fd CrossRefGoogle ScholarPubMed
Dickerson-Young, T, Keilman, A, Yoshida, H, Jones, M, Cross, N, Thomas, A. Pediatric emergency medicine disaster simulation curriculum: the 5-minute trauma assessment for pediatric residents (TRAP-5). MedEdPORTAL. 2020;16:10940. doi: 10.15766/mep_2374-8265.10940 CrossRefGoogle ScholarPubMed
Donevant, SB, Svendsen, ER, Richter, JV, et al. Designing and executing a functional exercise to test a novel informatics tool for mass casualty triage. J Am Med Inform Assoc. 2019;26(10):1091-1098. doi: 10.1093/jamia/ocz087 CrossRefGoogle Scholar
Farra, S, Hodgson, E, Miller, ET, et al. Effects of virtual reality simulation on worker emergency evacuation of neonates. Disaster Med Public Health Prep. 2019;13(2):301-308. doi: 10.1017/dmp.2018.58 CrossRefGoogle ScholarPubMed
Ferrandini Price, M, Escribano Tortosa, D, Nieto Fernandez-Pacheco, A, et al. Comparative study of a simulated incident with multiple victims and immersive virtual reality. Nurse Educ Today. 2018;71:48-53. doi: 10.1016/j.nedt.2018.09.006 CrossRefGoogle ScholarPubMed
Fletcher, L, Justice, S, Rohrig, L. Designing a disaster. J Trauma Nurs. 2015;22(1):35-40. doi: 10.1097/JTN.0000000000000098 CrossRefGoogle ScholarPubMed
Foo, NP, So, EC, Lu, NC, et al. A 36-hour unplugged full-scale exercise: closing the gaps in interagency collaboration between the disaster medical assistance team and urban search and rescue team in disaster preparedness in Taiwan. Emerg Med Int. 2021;2021:5571009. doi: 10.1155/2021/5571009 CrossRefGoogle Scholar
Gable, BD, Misra, A, Doos, DM, Hughes, PG, Clayton, LM, Ahmed, RA. Disaster day: a simulation-based disaster medicine curriculum for novice learners. J Med Educ Curric Dev. 2021;8:23821205211020751. doi: 10.1177/23821205211020751 CrossRefGoogle Scholar
Gillett, B, Peckler, B, Sinert, R, et al. Simulation in a disaster drill: comparison of high-fidelity simulators versus trained actors. Acad Emerg Med. 2008;15(11):1144-1151. doi: 10.1111/j.1553-2712.2008.00198.x CrossRefGoogle Scholar
Glow, SD, Colucci, VJ, Allington, DR, Noonan, CW, Hall, EC. Managing multiple-casualty incidents: a rural medical preparedness training assessment. Prehosp Disaster Med. 2013;28(4):334-341. doi: 10.1017/S1049023X13000423 CrossRefGoogle ScholarPubMed
Gofrit, ON, Leibovici, D, Shemer, J, Henig, A, Shapira, SC. The efficacy of integrating “smart simulated casualties” in hospital disaster drills. Prehosp Disaster Med. 1997;12(2):97-101. doi: 10.1017/S1049023X00037365 CrossRefGoogle ScholarPubMed
Greci, LS, Ramloll, R, Hurst, S, et al. vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE). Prehosp Disaster Med. 2013;28(3):215-222. doi: 10.1017/S1049023X13000083 CrossRefGoogle Scholar
Greco, S, Lewis, EJ, Sanford, J, Sawin, EM, Ames, A. Ethical reasoning debriefing in disaster simulations. J Prof Nurs. 2019;35(2):124-132. doi: 10.1016/j.profnurs.2018.09.004 CrossRefGoogle ScholarPubMed
Green, GB, Modi, S, Lunney, K, Thomas, TL. Generic evaluation methods for disaster drills in developing countries. Ann Emerg Med. 2003;41(5):689-699. doi: 10.1067/mem.2003.147 CrossRefGoogle ScholarPubMed
Gryth, D, Rådestad, M, Nilsson, H, et al. Evaluation of medical command and control using performance indicators in a full-scale, major aircraft accident exercise. Prehosp Disaster Med. 2010;25(2):118-123. doi: 10.1017/s1049023x00007834 CrossRefGoogle Scholar
Hartman, EN, Daines, B, Seto, C, Shimshoni, D, Feldman, ME, LaBrunda, M. Sort, assess, life-saving intervention, triage with drone assistance in mass casualty simulation: Analysis of educational efficacy. Cureus. 2020;12(9):e10572. doi: 10.7759/cureus.10572 Google ScholarPubMed
Heinrichs, WL, Youngblood, P, Harter, P, Kusumoto, L, Dev, P. Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II. Prehosp Disaster Med. 2010;25(5):424-432. doi: 10.1017/s1049023x00008505 CrossRefGoogle Scholar
High, EH, Lovelace, KA, Gansneder, BM, Strack, RW, Callahan, B, Benson, P. Promoting community preparedness: lessons learned from the implementation of a chemical disaster tabletop exercise. Health Promot Pract. 2010;11(3):310-319. doi: 10.1177/1524839908325063 CrossRefGoogle ScholarPubMed
Imamedjian, I, Maghraby, NHM, Homier, V. A hospital mass casualty exercise using city buses and a tent as a hybrid system for patient decontamination. Am J Disaster Med. 2017;12(3):189-196. doi: 10.5055/ajdm.2017.0273 CrossRefGoogle Scholar
Ingrassia, PL, Prato, F, Geddo, A, et al. Evaluation of medical management during a mass casualty incident exercise: an objective assessment tool to enhance direct observation. J Emerg Med. 2010;39(5):629-636. doi: 10.1016/j.jemermed.2009.03.029 CrossRefGoogle ScholarPubMed
Ingrassia, PL, Ragazzoni, L, Carenzo, L, Colombo, D, Ripoll Gallardo, A, Della Corte, F. Virtual reality, and live simulation: a comparison between two simulation tools for assessing mass casualty triage skills. Eur J Emerg Med. 2015;22(2):121-127. doi: 10.1097/MEJ.0000000000000132 CrossRefGoogle Scholar
Jacobson, PA, Severin, PN, Rumoro, DP, Shah, S. Mass-casualty training exercise using high-fidelity computerized simulators and involving time and resource limitation. Prehosp Disaster Med. 2021;36(3):313-320. doi: 10.1017/S1049023X21000327 CrossRefGoogle ScholarPubMed
Jain, TN, Ragazzoni, L, Stryhn, H, Stratton, SJ, Della Corte, F. Comparison of the Sacco triage method versus START triage using a virtual reality scenario in advance care paramedic students. CJEM. 2016;18(4):288-292. doi: 10.1017/cem.2015.102 CrossRefGoogle ScholarPubMed
Jorm, C, Roberts, C, Lim, R, et al. A large-scale mass casualty simulation to develop the non-technical skills medical students require for collaborative teamwork. BMC Med Educ. 2016;16:83. doi: 10.1186/s12909-016-0588-2 CrossRefGoogle ScholarPubMed
Jung, D, Carman, M, Aga, R, Burnett, A. Disaster preparedness in the emergency department using in situ simulation. Adv Emerg Nurs J. 2016;38(1):56-68. doi: 10.1097/TME.0000000000000091 CrossRefGoogle ScholarPubMed
Kaplan, BG, Connor, A, Ferranti, EP, Holmes, L, Spencer, L. Use of an emergency preparedness disaster simulation with undergraduate nursing students. Public Health Nurs. 2012;29(1):44-51. doi: 10.1111/j.1525-1446.2011.00960.x CrossRefGoogle ScholarPubMed
Kim, CH, Shin, SD, Park, JO, Kim, SC, Coule, PL. The effects of a community-based disaster drill of simulating Disaster Medical Assistance Team (DMAT) on the knowledge and attitudes. Ulus Travma Acil Cerrahi Derg. 2021;27(2):174-179. doi: 10.14744/tjtes.2020.93947 Google ScholarPubMed
Kim, J, Lee, O. Effects of a simulation-based education program for nursing students responding to mass casualty incidents: a pre-post intervention study. Nurse Educ Today. 2020;85:104297. doi: 10.1016/j.nedt.2019.104297 CrossRefGoogle ScholarPubMed
Kim, TE, Shankel, T, Reibling, ET, et al. Healthcare students interprofessional critical event/disaster response course. Am J Disaster Med. 2017;12(1):11-26. doi: 10.5055/ajdm.2017.0254 Google ScholarPubMed
Klima, DA, Seiler, SH, Peterson, JB, et al. Full-scale regional exercises: closing the gaps in disaster preparedness. J Trauma Acute Care Surg. 2012;73(3):592-598. doi: 10.1097/TA.0b013e318265cbb2 CrossRefGoogle ScholarPubMed
Knight, JF, Carley, S, Tregunna, B, et al. Serious gaming technology in major incident triage training: a pragmatic controlled trial. Resuscitation. 2010;81(9):1175-1179. doi: 10.1016/j.resuscitation.2010.03.042 CrossRefGoogle ScholarPubMed
Koutitas, G, Smith, KS, Lawrence, G, et al. A virtual and augmented reality platform for the training of first responders of the ambulance bus. In: ACM Int Conf Proc Series. 2019:299-302CrossRefGoogle Scholar
Kurenov, SN, Cance, WW, Noel, B, Mozingo, DW. Game-based mass casualty burn training. Stud Health Technol Inform. 2009;142:142-144. doi: 10.3233/978-1-58603-964-6-142 Google ScholarPubMed
Kyle, RR, Via, DK, Lowy, RJ, Madsen, JM, Marty, AM, Mongan, PD. A multidisciplinary approach to teach responses to weapons of mass destruction and terrorism using combined simulation modalities. J Clin Anesth. 2004;16(2):152-158. doi: 10.1016/j.jclinane.2003.09.003 CrossRefGoogle ScholarPubMed
Lee, CW, McLeod, SL, Van Aarsen, K, Klingel, M, Franc, JM, Peddle, MB. First responder accuracy using SALT during mass-casualty incident simulation. Prehosp Disaster Med. 2016;31(2):150-154. doi: 10.1017/S1049023X16000091 CrossRefGoogle ScholarPubMed
Lee, WH, Kuan, JT, Shiau, YW, et al. Designation of a new training model of a local disaster medical system with tabletop exercises. Chang Gung Med J. 2003;26(12):879-888.Google ScholarPubMed
Legemaate, GA, Burkle, FM Jr, Bierens, JJ. The evaluation of research methods during disaster exercises: applicability for improving disaster health management. Prehosp Disaster Med. 2012;27(1):18-26. doi: 10.1017/S1049023X11006789 CrossRefGoogle ScholarPubMed
Leiba, A, Goldberg, A, Hourvitz, A, et al. Who should worry for the ‘worried well?’ Analysis of mild casualties’ center drills in non-conventional scenarios. Prehosp Disaster Med. 2006;21(6):441-444. doi: 10.1017/s1049023x00004179 CrossRefGoogle ScholarPubMed
Leow, JJ, Brundage, SI, Kushner, AL, et al. Mass casualty incident training in a resource-limited environment. Br J Surg. 2012;99(3):356-361. doi: 10.1002/bjs.7762 CrossRefGoogle Scholar
Lerner, EB, Schwartz, RB, Coule, PL, Pirrallo, RG. Use of SALT triage in a simulated mass-casualty incident. Prehosp Emerg Care. 2010;14(1):21-25. doi: 10.3109/10903120903349812 CrossRefGoogle Scholar
Liu, A, Acosta, E, Cope, J, et al. The wide area virtual environment: a new paradigm for medical team training. In: Schmorrow, D, Fidopiastis, C, eds. Augmented cognition: users and contexts. AC 2018 Lecture Notes in Computer Science. New York: Springer; 2018:293-304.Google Scholar
Lowe, J, Peng, C, Winstead-Derlega, C, Curtis, H. 360 virtual reality pediatric mass casualty incident: a cross sectional observational study of triage and out-of-hospital intervention accuracy at a national conference. J Am Coll Emerg Physicians Open. 2020;1(5):974-980. doi: 10.1002/emp2.12214 CrossRefGoogle Scholar
Ma, D, Shi, Y, Zhang, G, Zhang, J. Does theme game-based teaching promote better learning about disaster nursing than scenario simulation: a randomized controlled trial. Nurse Educ Today. 2021;103:104923. doi: 10.1016/j.nedt.2021.104923 CrossRefGoogle ScholarPubMed
McElroy, JA, Steinberg, S, Keller, J, Falcone, RE. Operation continued care: a large mass-casualty, full-scale exercise as a test of regional preparedness. Surgery. 2019;166(4):587-592. doi: 10.1016/j.surg.2019.05.045 CrossRefGoogle Scholar
Mills, B, Dykstra, P, Hansen, S, et al. Virtual reality triage training can provide comparable simulation efficacy for paramedicine students compared to live simulation-based scenarios. Prehosp Emerg Care. 2020;24(4):525-536. doi: 10.1080/10903127.2019.1676345 CrossRefGoogle ScholarPubMed
Molka-Danielsen, J, Prasolova-Forland, E, Fominykh, M, Lamb, K. Use of a collaborative virtual reality simulation for multi-professional training in emergency management communications. In: IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE). Wollongong: IEEE. 2019;408-415.CrossRefGoogle Scholar
Ngo, J, Schertzer, K, Harter, P, Smith-Coggins, R. Disaster medicine: a multi-modality curriculum designed and implemented for emergency medicine residents. Disaster Med Public Health Prep. 2016;10(4):611-614. doi: 10.1017/dmp.2016.8 CrossRefGoogle ScholarPubMed
Nieto Fernández-Pacheco, A, Castro Delgado, R, Arcos González, P, et al. Analysis of performance and stress caused by a simulation of a mass casualty incident. Nurse Educ Today. 2018;62:52-57. doi: 10.1016/j.nedt.2017.12.016 CrossRefGoogle ScholarPubMed
Nilsson, H, Rüter, A. Management of resources at major incidents and disasters in relation to patient outcome: a pilot study of an educational model. Eur J Emerg Med. 2008;15(3):162-165. doi: 10.1097/MEJ.0b013e3282f4d14b CrossRefGoogle ScholarPubMed
Nilsson, H, Jonson, CO, Vikström, T, et al. Simulation-assisted burn disaster planning. Burns. 2013;39(6):1122-1130. doi: 10.1016/j.burns.2013.01.018 CrossRefGoogle ScholarPubMed
Obaid, JM, Bailey, G, Wheeler, H, et al. Utilization of functional exercises to build regional emergency preparedness among rural health organizations in the US. Prehosp Disaster Med. 2017;32(2):224-230. doi: 10.1017/S1049023X16001527 CrossRefGoogle ScholarPubMed
Phattharapornjaroen, P, Glantz, V, Carlström, E, Dahlén Holmqvist, L, Khorram-Manesh, A. Alternative leadership in flexible surge capacity — the perceived impact of tabletop simulation exercises on Thai emergency physicians’ capability to manage a major incident. Sustainability. 2020;12(15):6216. doi: 10.3390/su12156216 CrossRefGoogle Scholar
Pryor, E, Heck, E, Norman, L, et al. Integrated decision-making in response to weapons of mass destruction incidents: development and initial evaluation of a course for healthcare professionals. Prehosp Disaster Med. 2006;21(1):24-30. doi: 10.1017/s1049023x00003289 CrossRefGoogle ScholarPubMed
Pucher, PH, Batrick, N, Taylor, D, Chaudery, M, Cohen, D, Darzi, A. Virtual-world hospital simulation for real-world disaster response: design and validation of a virtual reality simulator for mass casualty incident management. J Trauma Acute Care Surg. 2014;77(2):315-321. doi: 10.1097/TA.0000000000000308 CrossRefGoogle ScholarPubMed
Riley, PW, Dalby, DJ, Turner, EA. Making acute hospital exercises more realistic without impacting on healthcare delivery. J Bus Contin Emer Plan. 2012;6(2):143-150.Google ScholarPubMed
Rusling, M, Masin, D, Voss, M, et al. Medical student coping and performance in simulated disasters. Anxiety Stress Coping. 2021;34(6):766-777. doi: 10.1080/10615806.2021.1916481 CrossRefGoogle ScholarPubMed
Saber, DA, Strout, K, Caruso, LS, Ingwell-Spolan, C, Koplovsky, A. An interprofessional approach to continuing education with mass casualty simulation: planning and execution. J Contin Educ Nurs. 2017;48(10):447-453. doi: 10.3928/00220124-20170918-05 CrossRefGoogle ScholarPubMed
Saunders, J, Davey, S, Bayerl, PS, Lohrmann, P. Validating virtual reality as an effective training medium in the security domain. In: 26th IEEE conference on virtual reality and 3D user interfaces, VR. 2019;1908-1911.CrossRefGoogle Scholar
Schulz, CM, Skrzypczak, M, Raith, S, et al. High-fidelity human patient simulators compared with human actors in an unannounced mass-casualty exercise. Prehosp Disaster Med. 2014;29(2):176-182. doi: 10.1017/S1049023X14000223 CrossRefGoogle Scholar
Sena, A, Forde, F, Yu, C, Sule, H, Masters, MM. Disaster preparedness training for emergency medicine residents using a tabletop exercise. MedEdPORTAL. 2021;17:11119. doi: 10.15766/mep_2374-8265.11119 CrossRefGoogle ScholarPubMed
Shah, VS, Pierce, LC, Roblin, P, Walker, S, Sergio, MN, Waterworks, Arquilla B., a full-scale chemical exposure exercise: interrogating pediatric critical care surge capacity in an inner-city tertiary care medical center. Prehosp Disaster Med. 2014;29(1):100-106. doi: 10.1017/S1049023X13009096 CrossRefGoogle Scholar
Silenas, R, Akins, R, Parrish, AR, Edwards, JC. Developing disaster preparedness competence: an experiential learning exercise for multi-professional education. Teach Learn Med. 2008;20(1):62-68. doi: 10.1080/10401330701798311 CrossRefGoogle Scholar
Smithers, B, Tenhunen, ML. Planning and implementing disaster drills for undergraduate nursing students. Nurs Educ Perspect. 2020;41(2):130-131. doi: 10.1097/01.NEP.0000000000000430 CrossRefGoogle ScholarPubMed
So, M, Dziuban, EJ, Franks, JL, et al. Extending the reach of pediatric emergency preparedness: a virtual tabletop exercise targeting children’s needs. Public Health Rep. 2019; 134(4):344-353. doi: 10.1177/0033354919849880 CrossRefGoogle ScholarPubMed
Start, E, Culley, JM, Tavakoli, A, Polyakova-Norwood, V. Students experience mass casualty nursing from the patient perspective in a simulated mass casualty exercise. Nurs Educ Perspect. 2021;42(3):174-176. doi: 10.1097/01.NEP.0000000000000646 CrossRefGoogle Scholar
Strout, K, Saber, DA, Caruso, LS, et al. Interprofessional mass casualty incident simulation design protocol to prepare prelicensure nursing students to respond to a disaster. Nurse Educ. 2017;42(5):E1-E4. doi: 10.1097/NNE.0000000000000365 CrossRefGoogle ScholarPubMed
Thomas, TL, Hsu, EB, Kim, HK, Colli, S, Arana, G, Green, GB. The incident command system in disasters: evaluation methods for a hospital-based exercise. Prehosp Disaster Med. 2005;20(1):14-23. doi: 10.1017/s1049023x00002090 CrossRefGoogle ScholarPubMed
Waddington, RM, Reeves, TC, Kalin, EJ, et al. Value of a ludic simulation in training first responders to manage blast incidents. Int J Gaming Comput Mediat Simul. 2013;5(2):60-72. doi: 10.4018/jgcms.2013040104 CrossRefGoogle Scholar
Whitfill, T, Auerbach, M, Diaz, MCG, et al. Cost-effectiveness of a video game versus live simulation for disaster training. BMJ Simul Technol Enhanc Learn. 2020;6(5):268-273. doi: 10.1136/bmjstel-2019-000497 CrossRefGoogle ScholarPubMed
Whitney, RE, Burke, RV, Lehman-Huskamp, K, Arora, G, Park, DB, Cicero, MX. On shaky ground: learner response and confidence after tabletop earthquake simulation. Pediatr Emerg Care. 2016;32(8):520-524. doi: 10.1097/PEC.0000000000000681 CrossRefGoogle ScholarPubMed
Wilkerson, W, Avstreih, D, Gruppen, L, Beier, KP, Woolliscroft, J. Using immersive simulation for training first responders for mass casualty incidents. Acad Emerg Med. 2008;15(11):1152-1159. doi: 10.1111/j.1553-2712.2008.00223.x CrossRefGoogle ScholarPubMed
US Fire Administration (USFA), Federal Emergency Management Agency (FEMA). Operational templates and guidance for EMS mass incident deployment. https://www.usfa.fema.gov/downloads/pdf/publications/templates_guidance_ems_mass_incident_deployment.pdf. Accessed August 20, 2021.Google Scholar
Federal Emergency Management Agency (FEMA). Homeland Security Exercise and Evaluation Program (HSEEP). https://www.fema.gov/sites/default/files/2020-04/Homeland-Security-Exercise-and-Evaluation-Program-Doctrine-2020-Revision-2-2-25.pdf. Accessed August 20, 2021.Google Scholar
Federal Emergency Management Agency (FEMA). Safe exercise best practice. https://www.fema.gov/sites/default/files/documents/fema_safe-exercise-best-practice_06072021.pdf. Accessed August 21, 2021.Google Scholar
Federal Emergency Management Agency (FEMA). Virtual exercise best practices. https://www.fema.gov/sites/default/files/documents/fema_virtual-exercise-best-practice.pdf. Accessed August 20, 2021.Google Scholar
World Health Organization (WHO). WHO simulation exercise manual. https://apps.who.int/iris/bitstream/handle/10665/254741/WHO-WHE-CPI-2017.10-eng.pdf;sequence=1. Accessed August 20, 2021.Google Scholar
Stat59 Services. Stat59. http://www.STAT59.com. Accessed December 1, 2021.Google Scholar
Office for Human Research Protections (OHRP). OHRP exemptions. https://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr-46/common-rule-subpart-a-46104/index.html. Accessed December 2, 2021.Google Scholar
Central limit theorem: Definition and examples. Statisticshowto. https://www.statisticshowto.com/probability-and-statistics/normal-distributions/central-limit-theorem-definition-examples. Accessed May 9, 2022.Google Scholar
Boston University Master of Public Health Module. The role of probability. https://sphweb.bumc.bu.edu/otlt/mph-modules/bs/bs704_probability/BS704_Probability12.html#:∼:text=The%20central%20limit%20theorem%20states,will%20be%20approximately%20normally%20distributed. Accessed April 12, 2022.Google Scholar
Briggs, SM. Regional interoperability: making systems connect in complex disasters. J Trauma. 2009;67(2 Suppl):S88-S90. doi: 10.1097/TA.0b013e3181adbcc0 Google ScholarPubMed
Kagawa, F, Selby, D. Ready for the storm: Education for disaster risk reduction and climate change adaptation and mitigation. J Educ Sustain Dev. 2012;6(2):207-217. doi: 10.1177/0973408212475200 CrossRefGoogle Scholar
Johnson, GG, Brindley, PG, Gillman, LM. Fidelity in surgical simulation: Further lessons from the STARTT course. Can J Surg. 2020;63(2):E161-E163. doi: 10.1503/cjs.017818 CrossRefGoogle ScholarPubMed
Medical Bioformatics. Virtual reality simulators. https://bioinformaticamedicablog.wordpress.com/2017/11/07/simuladores-de-realidad-virtual/. Accessed November 7, 2021.Google Scholar
Alharthi, SA, LaLone, N, Khalaf, AS, et al. Practical insights into the design of future disaster response training simulations. https://par.nsf.gov/biblio/10061145. Accessed July 7, 2021.Google Scholar
Toups, ZO, Hamilton, WA, Alharthi, SA. Playing at planning. Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play. Published online October 15, 2016. doi: 10.1145/2967934.2968089 CrossRefGoogle Scholar
Dreifuerst, KT. Getting started with debriefing for meaningful learning. Clin Simul Nurs. 2015;11(5):268-275. doi: 10.1016/j.ecns.2015.01.005 CrossRefGoogle Scholar
Figure 0

Figure 1. Relationships of the first responder in an MCI.IC = Incident CommandIMS = Incident Management SystemAgency = The pre-hospital or hospital, government or non-government, agency, organization, group, hospital, or health care delivery system of the individual first responderJurisdiction = The lead command and control health authority of the MCI response Direct 2-way interaction between the first responder with the patient and simultaneously with their agency incident command Jurisdiction provides command and control to the First Responder First Responder may provide information to the Jurisdiction Incident Command

Figure 1

Figure 2. PRISMA flow diagram.9Included References.13–110Specifically discussing competencies.16–19,24,29–39,41–44,46–55,71,84,90,91,95,96,99,100,103–110

Figure 2

Table 1. T1 scoping review search terms

Figure 3

Figure 3. Modified Delphi statement creation.

Figure 4

Table 2. Modified Delphi expert panel demographics

Figure 5

Table 3. Statements that attained consensus

Figure 6

Table 4. Statements that did not attain consensus

Supplementary material: File

Weinstein et al. supplementary material

Figure S1

Download Weinstein et al. supplementary material(File)
File 132.7 KB
Supplementary material: File

Weinstein et al. supplementary material

Figure S2

Download Weinstein et al. supplementary material(File)
File 59.8 KB
Supplementary material: File

Weinstein et al. supplementary material

Table S1

Download Weinstein et al. supplementary material(File)
File 19.2 KB