The Challenge of Mass Casualty Incident Response Simulation Exercise Design and Creation: A Modified Delphi Study

Abstract Background: A Mass Casualty Incident response (MCI) full scale exercise (FSEx) assures MCI first responder (FR) competencies. Simulation and serious gaming platforms (Simulation) have been considered to achieve and maintain FR competencies. The translational science (TS) T0 question was asked: how can FRs achieve similar MCI competencies as a FSEx through the use of MCI simulation exercises? Methods: T1 stage (Scoping Review): PRISMA-ScR was conducted to develop statements for the T2 stage modified Delphi (mD) study. 1320 reference titles and abstracts were reviewed with 215 full articles progressing for full review leading to 97 undergoing data extraction. T2 stage (mD study): Selected experts were presented with 27 statements derived from T1 data with instruction to rank each statement on a 7-point linear numeric scale, where 1 = disagree and 7 = agree. Consensus amongst experts was defined as a standard deviation ≤ 1.0. Results: After 3 mD rounds, 19 statements attained consensus and 8 did not attain consensus. Conclusions: MCI simulation exercises can be developed to achieve similar competencies as FSEx by incorporating the 19 statements that attained consensus through the TS stages of a scoping review (T1) and mD study (T2), and continuing to T3 implementation, and then T4 evaluation stages.


Introduction
The complexity of an individual first responder participating in a sudden onset disaster mass casualty incident response (MCI), 1 involves their patient care or duty skills, and the layering of their participation in their agency's Incident Management System (IMS). 2 The first responder's usual, daily standard operation command and control, changes once the MCI is declared. The first responder must have the skills required for MCI patient care or duty station, but they must also operate in the rarely utilized agency MCI-plan IMS. A live full-scale exercise (FSEx) examines these relationships; the communication between the first responder and the patient, and the responder's IMS to prepare the first responder and system for an actual MCI response ( Figure 1).
An FSEx presents obstacles and challenges for designers, developers, planners, and exercise conductors to fulfill the mission of ensuring first responder competencies and MCI plan effectiveness (Supplemental digital content Figure 1s). 3 Staff, stuff, and structures (SSS), 4 may be stretched thin to cover the daily need for medical services, let alone devote staff for training to gather appropriate observers, evaluators, volunteers to recruit/ moulage patient actors, and the technical staff to conduct an FSEx. Furthermore, government permits may be needed to alter road, highway, plane, and train transportation routes and/ or ensure that ecological and environmental concerns are considered in the FSExall time-consuming activities.
Travel to a conference that features an FSEx adds a budgetary constraint and creates a void in the staffing schedule, and there may be no capable replacement for the first responder. Education and training budgets must include resources to devote to training for the rare MCI, when emergency medical services (EMS) and health care facility administrators must also assure appropriate education and training for staff to achieve competencies for daily operations. Even allocating assets for the planning and execution of a valid FSEx may be difficult, regardless of regulatory expectations (e.g., nuclear, airplane, train, manufacturing, etc.). 5,6 Financial and time constraints may limit the discussion and agreement process by health authorities, regulators, first response agency, and health care facility administrators. (Supplemental Digital Content Figure 2) This dilemma was made vivid with the Coronavirus disease-2019 (COVID-19) response worldwide. Staff and facilities struggled to acquire and maintain SSS to mount a safe COVID-19 response. There was no way to gather responsible actors, create or revise an MCI plan, or develop and execute an FSEx. Face-to-face classroom learning and travel to conferences became a public health casualty to prevent the spread of COVID- 19. Simulation and serious gaming (Simulation) for skills-based medical education has become an effective adjunct to, or has supplanted conventional methodologies, and was considered to replace face-to-face MCI response education, and training to achieve similar or equal first responder competencies. The challenge of the realism that an FSEx creates for first responders' critical decisions has been postulated to be similar using other simulation methodologies but was not ready to be 'taken off the shelf' to incorporate into an MCI simulation exercise of any scale.
This study is designed to address the scope of the NO-FEAR Project (Network Of practitioners For Emergency medicAl systems and cRitical care) under work package number 5 7 : education and training of personnel and volunteers, regarding technical and non-technical skills, teamwork, critical thinking, clinical care, incident management, and psychological support.
NO-FEAR asks for new simulation tools for education and training to achieve MCI response competencies (e.g., high-fidelity and live simulation, 2-dimensional enhanced and immersive simulation software, tools to provide quantitative, and qualitative evaluation of responders' performance during exercises). 7 The present research is aimed at describing the design needs for future MCI response training simulations through a combination of literature review and expert assessment that achieves equal or similar competencies as a FSEx. A competency component of the MCI response training simulation design is to approximate the realism of the FSEx to better prepare the first responder for an actual dynamic MCI response.

Methods
This translational science (TS) study begins with the TS question (T0) 8 : How can first responders achieve similar MCI competencies as an FSEx using MCI simulation exercises?
The objective of this study produced the first stage (T1) scoping review and second stage (T2) modified Delphi study (mD) consensus statements that can be offered to formulate MCI simulation exercise guidelines in the future TS third (T3) MCI simulation exercise creation stage. The fourth TS (T4) stage that follows will study these MCI simulation exercises to determine if similar or equal FSEx competencies were obtained.

T1: Scoping review
A systematic review following the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR), 9 was conducted from August 2021 through October 2021. Scoping reviews synthesize knowledge following a defined scientific process to identify sources that can be interrogated using a defined data extraction tool to determine concepts, theories, and knowledge gaps. 10 The study of MCI education, training, and exercises that lead to competencies is multi-disciplinary, creating a body of knowledge that is heterogeneous, and thus apropos for PRISMA-ScR methodology. 11 Literature search criteria A T0 research question was developed using the Patient, Intervention, Control/ Comparison, Outcome (PICO) standard to frame the search strategy. 12 1) Population: MCI, pre-hospital and hospital providers, simulation training exercise, or drill. 2) Intervention: Not MCI simulation training exercise or drill, individual duty station competencies. 3) Comparison: Individual intra-agency and inter-agency IMS competencies. 4) Outcomes: Intra-agency and inter-agency IMS competencies.
Literature search methods Inclusion criteria. The search strategy included only terms relating to or describing the intervention ( Table 1). The review included English-language papers published from January 1, 1990, to July 1, 2021, in these databases:   Finally, an ancestry search was also performed to identify additional references from the bibliography of references when appropriate.
Exclusion criteria. References from the databases that did not meet the inclusion criteria, specifically did not study, or report an MCI/ MCI exercise, were excluded.
PRISMA ScR Figure 2 2 review authors independently screened 1320 reference titles and abstracts to determine if inclusion criteria were met; any disagreement was resolved by discussion. Then each of the remaining 215 full articles was read by both authors to determine if inclusion criteria were met, and again any disagreement was resolved by discussion.

T2: Modified Delphi study
The mD method permits experts from various locations to independently review statements to attain consensus when no "'disaster management"' OR "'disaster medicine"' OR "'disaster response"' OR "'disaster planning"' OR "'first responder*"' AND simulat* OR tabletop OR "'tabletop"' OR virtual OR computer OR "'artificial reality"' OR "'augmented reality"' OR gamification AND exercise* OR drill OR training OR education AND earthquake* OR terror* OR hurricane* OR "'building collapse"' OR "'bridge collapse"' OR "'train crash"' OR "'plane crash"' OR "'train derailment"' OR tornado OR "'natural disaster'*" OR cyclone* OR explosion OR wildfire OR "'airplane crash"' OR avalanche OR landslide OR mudslide OR flood* OR disaster* OR "'mass casualty"' OR "'sudden onset disaster*"' OR bioterrorism OR tsunami* OR volcano* DTIC (DoD): disaster OR "'mass casualty"' AND training OR education OR simulation OR virtual OR tabletop OR computer PubMed: disaster medicine [MeSH] OR "'disaster management"' OR "'disaster medicine"' OR "'disaster response"' OR "'first responder*"' AND computer simulation [MeSH] OR simulat* OR tabletop OR "'tabletop"' OR virtual OR computer OR "'artificial reality"' OR "'augmented reality"' OR gamification AND disasters [MeSH] OR earthquake* OR terror* OR hurricane* OR "'building collapse"' OR "'bridge collapse"' OR "'train crash"' OR "'plane crash"' OR "'train derailment"' OR tornado OR "'natural disaster*"' OR cyclone* OR explosion OR wildfire OR "'airplane crash"' OR avalanche OR landslide OR mudslide OR flood* OR disaster* OR "'mass casualty"' OR "'sudden onset disaster'*"' OR bioterrorism OR tsunami* OR volcano* CINAH: Exp MH disaster planning OR "'disaster management"' OR "'disaster medicine"' OR "'disaster response"' OR "'disaster planning"' OR "'first responder'*" AND Exp MH simulation OR Exp MH Computer simulation OR simulat* OR tabletop OR "'tabletop"' OR virtual OR computer OR "'artificial reality"' OR "'augmented reality"' OR gamification AND Exp MH disasters OR Exp MH natural disasters OR earthquake* OR terror* OR hurricane* OR "'building collapse"' OR "'bridge collapse"' OR "'train crash"' OR "'plane crash"' OR "'train derailment"' OR tornado OR "'natural disaster*"' OR cyclone* OR explosion OR wildfire OR "'airplane crash"' OR avalanche OR landslide OR mudslide OR flood* OR disaster* OR "'mass casualty"' OR "'sudden onset disaster*"' OR bioterrorism OR tsunami* OR volcan* PsycInfo: Exp MH emergency preparedness OR "'disaster management"' OR "'disaster medicine"' OR "'disaster response"' OR "'disaster planning"' OR "'first responder*"' AND Exp MH simulation OR Exp MH Computer simulation OR simulat* OR tabletop OR "'tabletop"' OR virtual OR computer OR "'artificial reality"' OR "'augmented reality"' OR gamification AND Exp MH disasters OR earthquake* OR terror* OR hurricane* OR "'building collapse"' OR "'bridge collapse"' OR "'train crash"' OR "'plane crash"' OR "'train derailment"' OR tornado OR "'natural disaster'*" OR cyclone* OR explosion OR wildfire OR "'airplane crash"' OR avalanche OR landslide OR mudslide OR flood* OR disaster* OR "'mass casualty"' OR "'sudden onset disasterr*" OR bioterrorism OR tsunami* OR volcano* ECRI: disaster OR "'mass casualty"' AND training OR education OR simulation OR virtual OR tabletop OR computer Disaster Medicine and Public Health Preparedness 3 consensus existed previously. 9 The first stage of the mD began after the final database was analyzed by lead authors who created initial draft statements based on the most relevant datapoints. Then an internal focus group of authors discussed and edited these statements to meet the format of Delphi statements for the Stat59 statistical analysis platform (Stat59 Services Limited, Alberta, Canada). 116 An external focus group comprised of content experts in the field of MCI simulation exercises and NO-FEAR partners was established to further discuss and edit the draft statements to be clear and concise. After a video conference moderated by the lead authors, these experts discussed the draft statements. External focus group participants performed asynchronous editing via a shared online document producing the final 27 statements for the second stage of the mD. A list of content experts, derived from the authors of included references, academicians, and researchers studying MCI simulation exercises was created to establish the mD expert panel. Introductory emails were sent explaining the project objectives and the mD to these mD experts.
mD experts that agreed were sent an email from the Stat59 (Stat59 Services Limited, Alberta, Canada) mD organizational program on the day that the mD began with a link to the Stat59 (Stat59 Services Limited, Alberta, Canada) website consent page. Each mD expert registered an account, validated it, and were sent a new email to log into their secure webpage to begin the first mD expert consensus round. 3 days later, mD experts that had not logged into the system were asked to verify their access and log in and asked to notify the author if they had not received the introductory email, with instructions on how to ensure future emails were received.
Once the mD experts logged in, they were provided with a formal explanation of the mD methodology and informed consent was obtained. For informed consent (Supplemental Digital Content link to Stat59 Consent Page), participants were notified that they were anonymous volunteers who could withdraw at any time, that participation or withdrawal would not impact their employment, and that their data was secure (Supplemental Digital Content link to Stat59 Security Page).
The next page was the list of 27 statements that were finalized in the T2 external focus group with instruction to rank each statement on a 7-point linear numeric scale, where 1 = disagree and 7 = agree. With this initial set of statements, the mD expert was asked to answer 4 demographic questions. Consensus amongst mD experts was defined as a standard deviation ≤ 1.0.
Statements that attained consensus after this first mD expert round were included in the final report. Each mD expert received an email from the Stat59 (Stat59 Services Limited, Alberta, Canada) program after the first mD expert round and a reminder email from  9 Included References.  Specifically discussing competencies. 16 the author shortly afterwards to log back into their Stat59 page that showed the mean response of all the mD experts for each statement that did not attain consensus, their own response for that specific remaining statement, and were asked to reconsider their 7-point linear numeric scale for these remaining statements. This process was repeated after the second mD expert round with statements that attained consensus included in the final report. The statements that did not attain consensus were advanced to the third and final round with the mD experts asked to reconsider these statements. This third mD expert round produced the final statements that attained consensus to add to the first and second round consensus statements in the final report. Remaining statements after this third round were the final statements that did not attain consensus.
The McLeod Health Institutional Review Board Office (Florence, South Carolina USA) has determined that this study does meet the exemption criteria found at 45 CFR 46.104(d)(2). 117

Results
As summarized in Figure 3, 35 mD experts confirmed their participation and established a unique account on the Stat59  (Table 2). 31 completed the first mD expert round that was open from January 17, 2022, until January 30, 2022. 5 statements attained statistical significance with a standard deviation ≤ 1.0 after this first mD expert round, and achieved consensus (Table 3, first round, first section in bold). The 22 statements that did not attain statistical significance, with standard deviation > 1.0, were advanced to the second mD expert round.
29 mD experts completed the second mD expert round that was open from January 31, 2022, to February 19, 2022. 12 of the 22 statements that advanced to the second mD expert round achieved consensus (Table 3, second round middle section). The remaining 10 statements were unable to attain consensus and advanced to the third mD expert round.
29 mD experts completed the third and final mD expert round that was open from February 23, 2022, to March 9, 2022. 2 of the remaining 10 statements achieved consensus, so a total of 19 statements achieving consensus. (Table 3 third round, last section in bold). The remaining 8 statements were unable to attain consensus after 3 T2 mD expert rounds and were not recommended for T3 consideration (Table 4).

Limitations
The PRISMA-ScR produced a qualitative analysis of published studies and reports. Though the search followed this process, there may have been references that were not discovered.
The Delphi method seeks to arrive at group consensus by the aggregate of a panel of experts who rate a statement on a linear numeric scale. Internal validity is largely unknown; therefore, stability of response is more accurate to determine consensus or lack of consensus.
The objective of the distribution of mD experts was to represent MCI simulation exercise designers, developers, and those that would execute an exercise. The distribution of mD experts favor resource rich countries but all experts are involved in MCI exercises in a way.
An essential component of the Central Limit Theorem is that the average of sample means will be the population mean, or if 1 finds the average of all of the standard deviations in the sample, then 1 will find the actual standard deviation for the population. 118 This will hold true regardless of whether the source population is normal or skewed, provided the sample size is sufficiently large (usually ≥ 30, the number of mD experts in this study per round averaged 29.67). 119 The application of the Central Limit Theorem to this study infers that the 19 statements that attained consensus can be recommended to assist the T3 development of guidelines for MCI simulation exercises.

Discussion
Providing MCI training can be challenging for several reasons: (1) provider schedules are often erratic and involve long hours; (2) there is a temporal dissociation between disaster response training and the application of the skills, leading to cognitive skill decay at the time the skill needs to be performed; and (3) traditional learning methods, such as didactic presentations, tabletop simulations, and FSExs require the physical presence of learners and educators at a certain place and time (synchronous learning). 36 In response, mD experts agreed, educators should design periodic simulation training to maintain competencies of rarely used skills that will deteriorate over time.
Briggs et al. showed that heterogeneous organizations with different command structures and missions participate in the response to a disaster, and therefore a clear objective of an MCI plan is to define the IMS of a region. 120 An example of a 2017 FSEx objective of Regional Operability in Ohio (United States) as reported by McElroy is that 'Participants shall identify the management structure to support effective operational coordination between all agencies and entities.' 83 The mD experts agreed that the simultaneous integration of patient care or duty skills (e.g., triage, utilization of resources, communication) and IMS skills should be incorporated into the exercise design to achieve the competency for the individual to recognize and assume their position in their agency MCI plan through situation awareness and critical decision-making.
As the layers of FSEx objectives are incorporated into the FSEx design, mD experts agreed that patient care or duty competencies can be evaluated using distinct separate modalities or as part of an exercise designed to include these skills with IMS skills. mD experts agreed that multiple exercise modalities should be considered to minimize time, cost, and impact to non-participants affected by the exercise. Conceptually a simulation exercise can be created to achieve one of  A competency should be for the individual to recognize and assume their duty position in their agency MCI plan. 29 6.1 0.9 Competencies of the individual at their duty station can be evaluated using distinct separate modalities or as part of an exercise designed to include these skills with IMS skills. Exercise objectives can be to discover latent safety threats (LSTs), which may contribute to medical errors. 29 5.8 1.0 MCI triage will occur during the continuum from the initial first responder assessment until arrival at the definitive health care facility. 29 6.1 0.8 Multiple exercise modalities should be considered to minimize time, cost and impact to non-participants affected by the exercise. 29 5.7 0.9 An incident management competency should include the ability to deliver the right patient to the right alternate care facility or definitive health care facility capable of attending to the injuries of that patient.  The objective of a mass casualty incident (MCI) exercise should explore gaps in the situational awareness, critical decision-making competencies of an individual and their agency. The individual will represent their agency in their jurisdiction's MCI Incident Management System (IMS).
Each interaction between the participant and simulated patient in any modality should be captured to monitor and record accomplishment of specific competencies, and to evaluate and improve the simulation exercise. Validated self-assessment pre-and post-exercise surveys should be designed to improve the education and training process leading to competencies.  Extended reality may lead to simulator sickness by stakeholders and should be considered in the exercise. At least 2 trained independent observers and evaluators at the same exercise location should be utilized to provide inter-rater consistency. Observers and evaluators with job action scripts should be able to inject or to alter the progression of the exercise based on their area of expertise. The competency of a participant's decision to place a patient into a triage category should be consistent with their agency's methodology. Disaster Medicine and Public Health Preparedness the many IMS competency objectives achieved by an FSEx as required by health authorities, regulators, IMS partners, and stakeholders: 1) Utilize effective means of inter-and intra-agency communication through redundant modalities; 2) Achieve the appropriate continuum of patient triage and treatment from the initial evaluation to definitive care; 3) Deliver the right patient to the right alternate care facility or definitive health care facility capable of attending to the injuries of that patient; 4) Maintain accurate patient tracking leading to expedient hospital registration; 5) Manage resources utilizing the supply chain in a resource scarce environment; and 6) Manage information and media through intelligence acquisition, vetting, and transmission.
MCI simulation education and training is multimodal: lectures, readings, and individual hands-on skill sessions to meet the requirements of the individual's MCI patient care duties. To do that, education must be active, interactive, and experiential, as well as participatory. 121 The additional layers in MCI education and training acknowledge that the individual is part of the IMS. The evolution of MCI simulation exercises will appreciate these complex simultaneous actions of the individual as they manage patient care and their role in the IMS, to achieve the same competencies that the FSEx will produce. To approximate the success of the FSEx, MCI exercise simulation must depend on high physical fidelity to develop the individual's manual abilities, as well as high conceptual fidelity to develop clinical reasoning and problem-solving skills.
Lastly, high emotional or experiential fidelity, 122 where the learner is emotionally invested in the simulation to a degree that memories of the experience are believable, will develop the individual's retention of the material. 44 There must be emotional learning in which positive emotions under stress facilitate greater retention of data. This success is not completely based on the realism of the simulation, but on the commitment of the participants in their roles; that there is an adequate connection between those involved; and that the student manages to actively link the social, psychological and clinical experiences lived. 123 The mD experts agreed that the realism created through moulage of simulated patients and environmental special effects in any simulation setting should focus on the participant's situation awareness and not deter from the overall exercise objectives. In 2019, Saunders et al. 97 published a study that demonstrated that virtual-reality-based law enforcement trainings, either by themselves or in combination with traditional hands-on training, can be as effective as highly resourceintensive practical training sessions. 123 Individuals suspend disbelief during the FSEx to approximate the physical and emotional stress of an actual MCI (e.g., the demand of an unknown number of patients with unknown injuries, an unknown supply of SSS in a resource-scarce environment with sensory overload, and fear for their own safety, as well as a failure to accomplish their assignments, or letting their team/ agency down). A successful MCI simulation exercise would match the individual's pressure for realism through their critical decisions suspending disbelief in the simulation environment without the sensory elements inherent in an FSEx. This can be achieved following design principles explained by Alharthi et al., along with the understanding that the designers must simulate the actual experience making the exercise practical through highly cognitive intense work and physical exertion immersing the individual in the simulation. 124 (Supplemental Digital  Content Table 1) To accomplish the objectives of an FSEx, controllers plan and manage exercise play, set up/ operate the exercise site, and act in the roles of organizations, agencies, or individuals that are not playing in the exercise. 125 Controllers direct the pace of the exercise, provide key data to players, and may prompt or initiate certain player actions to ensure exercise continuity. Simulators or facilitators provide feedback and cues based on predetermined expected actions of players as well as injects in response to player's actions that are not expected to maintain the flow of the exercise and to instruct. In addition, they issue exercise material to players as required, monitor the exercise timeline, and supervise the safety of all exercise participants and the surrounding environment. mD experts agreed that an exercise controller should build realism based on a prepared action script to anticipate and deliver injects based on player's actions and responses to evaluate nontechnical communication skills. MCI simulation exercise creators have the challenge to integrate live or reflex injects based on player's responses to the scenario, changes in patient's clinical conditions or other player's actions. An FSEx occurs in real time with all the agencies simultaneously responding; mD experts agreed that the challenge of MCI simulation exercises is the appreciation of the passage of time to accomplish an action as if in the real time of an FSEx. To achieve this level of realism, mD experts agreed that the exercise design should have modality technicians supporting the exercise to be able to recognize and address any participant struggling with the modality to promote the overall objectives.
A crucial component of any MCI simulation is the debriefing process following the activity; this provides a structured reflection for participants to analyze and self-correct their behavior, decisions, and thought processes to promote cognitive accommodation and assimilation of their learning experience into future professional practice. 52 mD experts agreed that observers and evaluators should be specific content experts external to the exercise and use validated template scoring tools to evaluate competencies of players, the exercise itself, and any documentation that is required of regulators or the health authority. These specific content experts can utilize 'debriefing through meaningful learning,' 126 to provide exercise objectives education. mD experts agreed that this debrief can discover latent safety threats through a frank non-punitive discussion to uncover potential actions that may lead to medical errors. 30 mD experts further agreed that a semi-structured debrief based on validated formats should include all stakeholders to improve the exercise, using open-ended questions to determine MCI simulation exercise areas of improvement.

Conclusion
The modified Delphi experts agreed that the simultaneous integration of individual duty and incident management skills should be incorporated into simulation MCI exercise design to achieve competencies depending on high physical fidelity to develop the individual's manual abilities, as well as high conceptual fidelity, to develop the individual's clinical reasoning and problem-solving skills.
MCI simulation exercises can be developed to achieve similar competencies as FSExs incorporating the 19 statements that attained consensus through the TS stages of a scoping review (T1) Co-authors are DMPHP deputy editors (other co-authors are known to the DMPHP editorial staff and potential reviewers). The manuscript was blinded throughout the review process.