Introduction
Hospital surge capacity is defined as a hospital’s ability to meet a sudden and unusual spike in demand for resources. While many types of events can call upon such capacity, none is more time-sensitive or potentially consequential than a large-scale, mass-casualty incident (MCI). Recent events—the 2013 Boston Marathon bombing (Boston, Massachusetts USA); the 2015 Paris (France) terrorist attacks; the 2015 Amtrak derailment (Philadelphia, Pennsylvania USA); and the 2016 mass shooting in Orlando, Florida (USA), to name just a few—are unfortunate reminders that hospitals everywhere must be prepared for sudden, unexpected, massive influxes of patients, some with critical injuries.Reference Biddinger, Baggish and Harrington 1 - Reference McCarthy 5 Increasingly over the past 15 years or more, emergency department (ED) crowding, a downstream effect of hospital crowding, has been recognized as a predominant threat to preparedness. 6 - Reference Hick, Hanfling and Burstein 9 Yet, true as it might be, crowding has only grown worse and widespread over this same interval, while MCIs have become steadily more frequent and deadly. 10 With no remedy in sight, policymakers have turned to the quality of response as a key determinant of success. With overcrowded EDs leaving little margin for error, hospitals must be prepared to immediately and expertly implement an MCI plan which rapidly transitions a hospital from normal operations to crisis standards of care. 11 - Reference Kellermann 14
Real-world MCIs usually occur without notice, and require decision making in a chaotic, information-poor environment. Yet, no-notice exercises to assess a hospital’s performance under such conditions are almost never conducted. Typical exercises (including those that fulfill Joint Commission [Oakbrook Terrace, Illinois USA] requirements) tend to be pre-announced and highly choreographed.Reference Kaji and Lewis 15 They sometimes make use of moulaged patients, which can provide a degree of realism, but can also tend to limit the size of the surge. Mass-casualty exercises rarely incorporate quantitative performance metrics and therefore may provide limited insight into a hospital’s real-world surge capacity or the effectiveness of improvement efforts.
To address these gaps, the US Department of Health and Human Services (Washington, DC USA), Office of the Assistant Secretary for Preparedness and Response (ASPR), in conjunction with the Hospital Preparedness Program (HPP), commissioned RAND Corporation (Santa Monica, California USA) to develop a low-burden, no-notice, mass-casualty exercise which could be used by any hospital to assess its crisis surge capacity in the immediate aftermath of a large-scale MCI. This report describes the product of that work, which is now publicly available for use, and the insights gained during the development and piloting process. 16 It is one of the first published accounts detailing the implementation of a no-notice exercise in a hospital setting.
Report
Development Process
Work was performed in two phases, during 2010-2012 and 2014-2015, with substantial revisions in the second phase based upon lessons learned in the first. In consultation with ASPR and thought leaders in the hospital preparedness community, the development team identified the following initial high-priority objectives: (1) Realism—The exercise should be no-notice, at least from the perspective of players. It should proceed in real-time, thereby simulating the pace and stress of a real incident. Players should have no more information available to them than they would have in a real incident. (2) Minimize Burden to Hospital Operations—The goal of realism should be balanced with the need to minimize the burden placed upon the hospital, particularly clinical personnel. Patient safety cannot be compromised, and an infeasible exercise is unlikely to be adopted. (3) Design for Use by Peer Assessors—Compared to using hospital insiders to run the exercise, peer assessors have the potential advantages of facilitating the element of surprise required for a no-notice exercise and improving objectivity. (4) Provide Immediate, High-Quality Feedback using well-defined, clinically meaningful measures of performance.
The exercise was designed and refined using an iterative design-feedback approach, repeating a cycle of soliciting input, designing or refining the exercise, and piloting. The initial prototype was informed by discussions with ASPR and a diverse group of subject matter experts in the fields of hospital preparedness, emergency management, emergency medicine, logistics, exercise design, and evaluation and measurement. Published reports of actual MCIs informed exercise parameters (eg, timeframe and rate of patient arrivals, types of injuries, and distribution of injury severity over time).Reference Kirschenbaum, Keene, O’Neill, Westfal and Astiz 17 - Reference de Ceballos, Fuentes, Diaz, Sanchez, Llorente and Sanz 22 The initial prototype, which made extensive use of printed materials, was piloted in two urban academic medical centers, one on the East Coast and another on the West Coast. In these first pilots, members of the design team ran the exercise in lieu of peer assessors. While these initial pilots confirmed that hospital players saw value in the no-notice approach, they also uncovered operational difficulties.
To address problems identified in the initial pilots, the exercise was substantially redesigned for improved usability, with most paper-based materials (eg, instructions and data collection instruments) replaced by macro-enabled Microsoft Excel workbooks (Microsoft Corp.; Redmond, Washington USA), which consolidated exercise timing, instructions, data collection, and feedback. Five additional pilot tests (three at medium-sized suburban hospitals, one at a non-university urban hospital, and one at an urban teaching hospital) were conducted after the re-design. With ongoing input from HPP and ASPR staff, further revisions were made after each pilot based upon structured feedback received from hospital players and peer assessors through post-test debriefs.
The project was approved by the RAND Human Subjects Protection Committee (institutional review board). As a condition of participation by the piloting hospitals, RAND agreed that it would not maintain records of individual hospital performance or disseminate hospital-level results.
Description of the Exercise
The final product is what HPP has termed a “surge evaluation in a box:” a freely-available toolkit designed to allow any hospital to stage a 75-minute, real-time, no-notice exercise run with minimal advanced preparation by invited peer assessors. 16 The toolkit consists of an 18-page “user manual” and five macro-enabled Excel workbooks that guide the assessment team through the process of planning and implementation.
Exercise play focuses on the actions of a hospital during the first 75 minutes of a large-scale MCI, a time during which a hospital should anticipate the worst, even as the magnitude of the event is not yet clear. Specifically, the principal “outcome” that is tested is the degree to which a hospital can maintain ED capacity for the immediate assessment and treatment of new arrivals (“immediate bed availability [IBA]”) throughout the exercise timeframe.
Exercise play is directed from the ED, but play involves a whole-hospital response. Peer assessors view the response from two vantage points, the ED and the hospital incident command center.
The mechanics of the exercise, which are described in more detail in the user manual, are as follows. Once hospital leadership decides to run the exercise, it chooses a one-week interval during which time the exercise can be run, and recruits four outside peer assessors to run it. Peer assessors are meant to be unaffiliated clinical personnel (eg, Medical Doctors/MDs, Registered Nurses/RNs, or Emergency Medical Technicians/EMTs) with sufficient experience to understand ED and hospital operations. A single member of the hospital staff is designated as a “trusted insider” to serve as a liaison between the hospital and the four-person peer assessment team, and to assist in pre-exercise planning. The trusted insider should be the only member of the hospital staff to know the exact date and time of the exercise, within the one-week window.
Prior to the day of the exercise, assessors choose roles (each of four needs to be covered) and familiarize themselves with their role-specific Excel workbook, which they are told to pre-load onto a laptop that they bring to the exercise. The assessors must also decide in advance how many “mass-casualty patients” will present to the hospital and enter that number into an Excel tool (“arrival list generator”) which automatically generates printable lists that are given to ED players as “exercise injects” at each of four 15-minute intervals during the exercise. These lists contain brief, one-line descriptions of patients arriving during the interval (eg, “Walking around, facial lacerations”). The distribution of injury severities and number of arrivals at each interval is based upon published accounts of large-scale MCIs.Reference Kirschenbaum, Keene, O’Neill, Westfal and Astiz 17 , Reference Hogan, Waeckerle, Dire and Lillibridge 18 , Reference Wightman and Gladish 20 - Reference Leibovici, Gofrit and Shapira 24 For example, patients arriving 15-30 minutes into the exercise are mostly “walking wounded,” and arrivals peak during the 30-45 minute interval.
At a pre-designated day and time, the assessment team assembles with the trusted insider just prior to exercise start. Once ready, two assessors are brought to the site of the command center (not yet activated), so that they can view the activity there from the time of initial mobilization. The other two enter the ED through the public entrance, ask to speak to the charge nurse or attending physician, and let them know that the exercise is about to begin. Once the ED staff agrees to participate (checking with supervisors as necessary), the lead assessor (“ED controller”) reads a script from his or her Excel workbook which describes the exercise rules, and when all are ready, the ED controller clicks on a “start exercise” button on the spreadsheet, initiating an automated exercise clock as he reads the scenario. The default scenario is simply this: “A bombing has occurred at a nearby building. Many casualties are expected.” It provides few details, just as few details are typically available in real life. The scenario can be changed to any other sudden event (eg, train crash or mass shooting) deemed more appropriate for the hospital setting.
Once notified, the ED is expected to recognize the potential for a mass influx of injured patients and to react pre-emptively. The second assessor is charged with watching the ED’s response and judging how well the ED players undertook critical actions, such as activating the hospital incident command system “without delay.” Such subjective assessments are recorded on a Likert-type scale embedded within one of the Excel workbooks.
Meanwhile, the ED controller, following spreadsheet prompts, collects data regarding that actual state of the ED at the start of the exercise – the total number of physical treatment spaces and the total number of registered patients (by triage severity). Using this baseline information, the spreadsheet displays a continuously updated, real-time report of the ED IBA, which decreases as new patients arrive and increases as inpatient beds are assigned to the ED. This display serves in place of the usual visual cues that would be available to players, but are not since patients are not actually moving during the exercise.
Every 15 minutes, the ED controller provides a list of injured patients arriving during the interval, and ED players are asked to triage the new arrivals and categorizing them using a Simple Triage and Rapid Treatment/START-like framework as “red, yellow, or green” (Online Supplement, Figure 1; available online only). Also at 15-minute intervals, ED players are asked whether inpatient beds have been found so that their patients can be moved from the ED to make room for new arrivals. An explicit simplification is that players are instructed to assume that red patients require intensive care unit (ICU) beds and that yellow patients require non-ICU beds. Green patients, by definition ambulatory and stable, are assumed not to occupy beds in either the ED or the inpatient units; rather, the evaluation team makes a qualitative evaluation of the adequacy of the hospital’s plan for caring for them. The ED IBA is therefore defined as the difference between the number of physical ED treatment spaces and the number of registered patients triaged as red or yellow in a 3-level system.
To free up a bed in the ED, an ED player must provide the assessor with an inpatient location specific enough “that a transporter would know where to take the patient.” Assessors are also instructed to consider the face validity of all responses. The bed assignments are expected to come from the hospital incident command center, which would only know which beds are needed by communicating with the ED regarding their baseline census and new patient arrivals.
One of the assessors based in the hospital incident command center tracks inpatient IBA (for ICU and non-ICU beds) in a way that is analogous to what is done in the ED. The other assessor considers qualitatively whether the hospital has performed a series of critical actions.
At 75 minutes, an automated pop-up appears on the exercise controller’s spreadsheet, indicating that the exercise is over. All players and the assessment team then convene in the hospital’s incident command center for a real-time “hot-wash.” The script for the hot-wash discussion is automatically generated, as are graphs that show ED arrivals, transfers out of the ED, and ED IBA versus time. Graphs can be projected so that participants can see dynamic changes in ED IBA as the scenario unfolds (Online Supplement, Figure 2; available online only). The ED IBA is figured as the net difference between the number of beds and the number of patients at any point in time. It is a function of both conditions at the start of the exercise, patient flux, and the creation of “new treatment space” in the ED, and may at times be negative (ie, more registered patients than beds). In an optimal response, the hospital would maintain a positive ED IBA (by rapidly clearing existing patients and new arrivals) for as much of the exercise as possible.
Discussion
This work demonstrates the feasibility of implementing a no-notice exercise in a hospital setting and provides a tool-kit that allows an easy way for a hospital to test its response in the immediate aftermath of a large-scale MCI. The exercise has several innovative qualities: by running in real-time and without prior notice to players, it simulates the challenges of working amid the “fog of war” – the uncertain situational awareness and communications breakdowns (both internal and external) that often accompanies real events; by enabling the collection of real-time data on the dynamic allocation of beds, the focus of assessment shifts from process (eg, did the hospital say that it would clear ICU beds?) to outcome (eg, were ICU beds available throughout the exercise?). Data collected during the exercise are fed back to participants during the post-exercise “hot-wash,” allowing for a more objective self-assessment than is typically available. The exercise is scalable, designed to be used at a hospital of any type or size, and the scenario can be customized to fit the geographic/community setting. It is publicly available and open-source, and can thus be used or adapted free of charge. The assumption behind the tool is that there are enough commonalities in preparedness goals across institutions that the duplicative effort of homegrown exercise creation can be avoided.
Design Insight: Making the Tool Work for Peer Assessors
The iterative piloting process led to a number of insights regarding exercise design that might be generalizable to future exercise development. Chief among them is that users (ie, peer assessors) cannot be expected to give exercise materials more than a cursory review in advance, and that they will absorb few details even if they try. An early paper-based version of the exercise proved unusable, partly because it relied on assessors reading a detailed user manual. Upon realizing that assessors were not following instructions, the design team’s initial reaction was to add more detail to the manual, which only exacerbated the problem. This was not a failure on the part of the highly talented and motivated group of pilot assessors. Rather, this remarkably consistent empiric finding (which parallels what is known to be true for consumer productsReference Blackler, Gomez, Popovic and Thompson 25 ) suggests that it is unrealistic to expect a group of naïve users (however elite) to absorb a complex series of instructions for running an unfamiliar exercise and to implement them correctly on the first try.
The use of peer assessors introduces an additional level of complexity: not only does the assessment team need to understand the exercise, but they must also transmit instructions to players. In initial pilot tests, this “game of telephone” – whereby peer assessors try to grasp a sequence of instructions, relay them to players, and then ensure correct implementation, all in a real-time, high-stress environment – seemed to contain an almost insurmountable number of potential failure points.
Mitigation of this problem led to many design decisions, chief among them being the use of macro-enabled Excel workbooks instead of paper-based exercise materials. The Excel tools incorporate a real-time exercise clock that leads users through a series of time-based unambiguous prompts, such as “Read Script Aloud,” “Input Data,” and “Assess,” and allows for the consolidation of instructions, script, data collection, and data analysis in a way that proved impossible with paper. The use of short verbatim scripts proved to be a workable solution to the problem of having assessors transmit instructions to players with adequate fidelity, although only after adjusting wording and graphics to help assessors overcome the tendency to paraphrase (often incorrectly or incompletely).
Balancing Authenticity with Feasibility and Usability
Since the scope of the exercise was limited to the first hour and 15 minutes of an MCI, the exercise naturally focused on certain structural elements of hospital response (eg, freeing up physical space and maintaining communications and situational awareness) rather than the other two elements of the “space, staff, stuff” trifecta, which tend to come into play later.Reference Koenig, Lim and Tsai 13 The idea of quantifying IBA was borne originally out of ASPR’s desire to create a tool that could be used by planners to assess regional surge capacity. During development, it became apparent that there were too many unknowns and (necessary) simplifications to allow results to be interpreted in absolute terms. Simplifications were made as transparently as possible, recognizing the need to balance between authenticity with feasibility and usability. For example, assessors read the following script, verbatim, to players: “For the purposes of the exercise, we will now make several simplifying assumptions:
-
– Green patients will go to auxiliary sites; they will not occupy beds in the department.
-
– Red patients require inpatient ICU beds.
-
– Yellow patients require floor beds.
-
– Red and yellow patients leave the department as soon as you identify free inpatient beds.”
In this example, there are at least three extreme simplifications: (1) that ambulatory patients with minor injuries are never brought into the main ED, and therefore do not count against the IBA; (2) that ED disposition is deterministic, a function only of initial triage category, according to a three-level system designed for prehospital care; and (3) that ED beds are freed up instantaneously, as soon as an inpatient location is identified. These simplifications were not thought to mirror reality, but were judged to be acceptable approximations within the narrow timeframe of the exercise, better than alternatives that seemed infeasibly complex or likely to place an unacceptable burden (of requiring additional players) on regular hospital operations.
Pilot tests suggested that the exercise’s quantitative focus provided a useful auditing mechanism to uncover weaknesses in the hospital response that might not otherwise be recognized. Each time a list of arriving patients was provided to ED players, players were asked “Where are they going?” Since the real-time IBA was visible to both assessors and players, the plausibility of responses could readily be ascertained. The requirement that the ED name a specific inpatient location before getting credit for clearing an ED space meant that any breakdown in communication between the ED and hospital incident command center quickly resulted in an ED patient backlog.
The pilot tests were conducted for the purpose of refining the tool rather than evaluating the hospital response, but one consistent failure point on the part of hospitals was notable: there were almost always problems in the back-and-forth information transfer between the ED and hospital incident command center. The command center rarely retained an accurate understanding of how many patients had arrived or what the ED census was, and there was often a delay in letting the ED know when inpatient locations had been found for their patients. Meanwhile, the ED often felt that they were “on their own” and had little idea what decisions were being made upstairs. Poor situational awareness appeared to be ubiquitous and did not appear to be an artifact of the testing situation or solely attributable to shortcomings of ED information systems. In fact, the goal of improving situational awareness, particularly two-way communication between the ED and the hospital incident command center, might be among the most compelling reasons for a hospital to run this particular exercise.Reference Landman, Teich and Pruitt 26 , Reference Burstein 27
Balancing the Benefits of No-Notice with the Impact on Clinical Operations
Hospital administrators were reluctant to agree to the “no-notice” requirement; several facilities declined to participate, and there was considerable pre-exercise anxiety among some who did. Their concerns were reasonable: almost by definition, any exercise meant to test the response to an unexpected event will cause some disruption to the normal workflow. Recognizing that there would be minimal tolerance for disruption of the clinical (versus administrative) workflow, the exercise was ultimately designed so that it could be run with a minimum of just two ED players. Although such bare-bones participation would seem to make the hospital’s performance unreasonably dependent upon the actions of two people, it mimics real life. A hospital that relies on quick thinking on the part or one or two clinical personnel might or might not perform well on a given day, but one that has a well-developed and widely disseminated mass-casualty response plan should perform well consistently, just as day-to-day excellence of clinical care is expected regardless of who is on duty. Based on the feedback received during the pilots, it seems likely that hospitals will see the tradeoff between disruption and value to be a reasonable one, and that institutional anxiety can be achieved by specifying that the exercise should be no-notice from the players’ perspective, but not necessarily to the highest levels of hospital leadership.
Conclusion
This work demonstrates that hospitals can implement a no-notice, mass-casualty exercise and introduces dynamic measurements of IBA as an objective way to assess preparedness. The product of this work is an exercise toolkit that is available for download and use or for others to further refine or develop. The lack of notice and real-time pace enhances realism, while the step-by-step instructions, easy-to-use data collection features, and analytics built into the Excel tools limit the costs of running the exercise and ensures that players receive quick and objective feedback on their performance. While this exercise tests only a narrow slice of preparedness, the model of developing and refining widely applicable tools in this manner holds great promise for enhancing readiness.
Acknowledgements
This work was done under contract (HHSO100201200075G) with the Department of Health and Human Services, Office of the Assistant Secretary for Preparedness and Response. Drs. Arthur Kellermann and Timothy Davis initially conceived of this project. The authors thank the many people and institutions who volunteered their time to pilot the exercise and those who helped in the development process in other ways: George Tobin and the HPP team; as well as Drs. Karen Remley, Richard Serino, Nathaniel Hupert, John Hick, Dan Hanfling, Eric Goralnick, Paul Biddinger, Nicole Lurie, Jeanne Ringel, and Jeffrey Wasserman.
Supplementary Material
To view supplementary material for this article, please visit https:/doi.org/10.1017/S1049023X17006793
