An increase in Mass Casualty Incidents (MCIs) and daily crowding in Emergency Departments (EDs) worldwide has created a growing need for translational research to support clinical decision-making. Translational research encompasses the progression of knowledge from basic science to proof-of-concept studies (T1), clinical trials (T2), implementation research (T3), and effectiveness research (T4).Reference Fort, Herr and Shaw1 In a preceding systematic review on reverse triage, we analyzed the available proof-of-concept studies in the literature (T1). The next step involves this Modified Delphi study, where expert consensus will serve as the foundation for developing an evidence-based tool that translates the existing proof-of-concept into a practical application for end-users (T2). An unexpected influx of patients can easily overwhelm the capacity and resources of an ED, or even an entire hospital, in a short period of time. Inseparably, this will be accompanied by a reduction in the quality of delivered patient care, an increased risk of errors, and a potential rise in morbidity and mortality.Reference Pollaris, De Bondt and Sabbe2 To optimize the hospitals’ surge capacity, various methods have been developed over the years, among which Reverse Triage has emerged as a promising approach. It is a process that supports clinical decision-making to rapidly increase bed capacity by identifying hospitalized patients receiving treatment for acute or acute-on-chronic medical conditions who have not yet met discharge criteria but are at low risk of experiencing a Consequential Medical Event (CME) if discharged. Following identification, a multidisciplinary assessment determines which patients should leave the hospital earlier than planned.Reference Pollaris, De Bondt and Sabbe2 This approach creates bed capacity for the reception of disaster victims. Who are, at that time, in greater need of hospital care and resources. The hospital surge capacity for standard inpatient beds can significantly be enhanced by reverse triage, if effectively implemented.Reference Kelen, McCarthy and Kraus3 Moreover, the reverse triage principle could also be used to reduce daily crowding in EDs. The inability to access inpatient beds, referred to as access block, is the main cause of ED crowding. This throughput block causes long waiting times (boarding) and unsafe conditions such as a higher incidence of adverse events and an increase in the inpatient mortality rate.Reference Taylor4 An initial base of the reverse triage selection process in MCIs was established by Kelen et al. (2006). They developed a Patient Disposition Classification Model (PDCM) based on critical interventions.Reference Kelen, Kraus and McCarthy5 Since clinical guidelines are continuously evolving and the US health care system differs significantly from its European counterpart, a European-level update and expert consensus was deemed indispensable to address this gap. The novelty of this Delphi study lies in its dual focus: the expert panel will not only critically reassess the initial PDCM for MCIs from a European perspective but will also explore its applicability in daily crowding situations.
So, the main objective of the electronic Delphi was to reach a European consensus concerning terminology, criteria, and cutoffs regarding the reverse triage selection process in MCIs and daily crowding.
Methods
The Conducting and REporting DElphi Studies (CREDES) guideline was used to conduct and report this Delphi study.Reference Jünger, Payne and Brine6
Design
To give the expert panel enough freedom to suggest ideas while maintaining the core narrative, a modified Delphi study was set up with a semi-structured first round. Unlike the traditional Delphi method, the modified Delphi study adopts a more structured approach, using data from proof-of-concept studies identified in the T1 systematic review as the foundation for discussions. A two-part design was chosen. Part 1a aimed to develop a theoretical framework of a PDCM for MCIs and crowding. The PDCM helped to categorize patients based on the risk of developing a CME following an early discharge. During part 1b the expert panel reviewed whether all critical interventions of the original American framework should be considered “critical” in MCI or crowding situations in a European health care system context and whether any critical intervention was missing. Furthermore, consensus had to be reached on the definition and terminology of each critical intervention of the final list. In part 2, the predetermined list of critical interventions was evaluated using a 10-point linear numeric scale, ranging from “Not at all likely” (score 0) to “Extremely Likely” (score 10), to assess the extent to which withholding or withdrawing any specific intervention would result in a CME or patient harm. For analytic purposes, the critical interventions were placed in three categories: major importance (score 7-10), moderate importance (score 5-6), and lower importance (score ≤ 4). Finally, the expert panel was tasked with assigning an upper limit of risk tolerance to each PDCM category using a percentage slider scale ranging from 0% to 100%. The selected percentage reflected the expert’s acceptable level of risk for a patient in that category developing a CME if discharged early during MCIs or crowding situations.
The cloud-based survey platform Qualtrics Research Core® was used to develop the survey. The final draft of each round was reviewed in depth and extensively tested for programming errors. Once the requirements were met, each panelist received a personalized link to the Qualtrics® platform. This way, each invitation could be individually tracked so server issues could immediately be dealt with. Moreover, it allowed the research team to send timely reminders and panelists to close and resume the survey at any time, on any device, without data loss. This sought to increase the ease of use and responsiveness.
Throughout the Delphi, both qualitative and quantitative questions were addressed. In part 1a, panelists were asked dichotomously (quantitative yes/no) whether they agreed with a five-category PDCM for MCIs and whether they agreed with the descriptions of these categories. If the answer was no, they were asked to clarify their answer (qualitative, free text). The same methodology was used to create a PDCM for crowding. In part 1b, panelists were also dichotomously asked whether each original critical intervention was still considered critical in MCIs and crowding situations, with the European health care system in mind. Subsequently, they were asked whether they agreed with the definition of these critical interventions (yes/no). If the answer was no, they were again asked to clarify their answer (free text). Furthermore, they were asked whether they would like to add another in-hospital intervention to the list (free text). Part 2 consisted of only quantitative questions (score on the linear numeric scale and percentage slider). The layout remained uniform throughout the rounds. Consensus was defined as ≥ 70% agreement in the dichotomous questions. Nevertheless, two similar requests for re-discussion or suggestions in the panelists’ free text responses were sufficient to take an item to the next round, regardless of consensus achieved. If there was still no consensus after four rounds and no more similar suggestions were given, a non-consensus was declared.
Selection of Experts
The expert panel was composed of European emergency physicians, who possess a great familiarity with operational decision-making in incident management and are responsible for maintaining the functionality of emergency services during periods of crowding. Experts were selected from the research network of the European Society for Emergency Medicine. Care was taken to ensure a spread between clinicians and researchers. According to Day and Bobeva (2005), the recommended size to ensure a productive and dynamic group that can produce consensus was between 10 and 15 experts.Reference Day and Bobeva7 The panel consisted of 15 experts from the following countries: Belgium, Romania, Poland, Portugal, Italy, Spain, Denmark, Czechia, the UK, Finland, Ireland, Israel, Hungary, France, and Germany. Seven emergency physicians were also academics with a PhD, seven clinicians obtained the European Master in Disaster Medicine, and one the Master in Crisis and Public Order Management. Each expert provided consent to have their full name published in the corresponding research article.
Prior Information
To enrich the scientific background knowledge regarding the topic, some key publications were shared at the start of the Delphi study. Those were the publication of the original American framework and some general publications about the normal adverse event rate after discharge.Reference Friebel, Hauck and Aylin8-Reference Satterthwaite and Atkinson12 As mentioned before, it was always the intention to critically assess, update, and alter the original framework with a European mindset. Therefore, sharing the original publication was considered to have no effect on the experts’ judgements. Additionally, study-related abbreviations and general assumptions were also disclosed (Figure 1).
Study related abbreviations and general assumptions.
Guidelines, Processing, and Synthesis
No maximum number of rounds was set a priori. Decision rules were clearly communicated at the start. All responses were processed by the research group and displayed in the next round until consensus was reached. For each round, the quantitative data from the previous round were presented. A previous consensus reached item was typeset along with its consensus percentage but without the possibility to comment. When the consensus cutoff was not met, the qualitative data from the previous round (similar comments or feedback) were presented anonymized in bullet points. Afterwards, the question was asked again, keeping in mind the comments given. For rounds 2 and 3, the additional information from the previous round was typeset in another color. The structure of the questionnaire remained consistent throughout the rounds. The Delphi study started in February 2021 with round 1 (February-March), followed by round 2 (March-April), and round 3 (June-August). The final round was initially planned as an in-person roundtable discussion in Belgium in early January 2022. However, it was canceled a few weeks prior due to stricter COVID-19 government restrictions. Consequently, the Delphi congress was rescheduled as an online event in April 2022. A final follow-up meeting was conducted in December 2022.
Statistical Analysis
The distribution of data points representing the upper limit of acceptable risk for a CME for each PDCM category was analyzed to identify potential outliers. The presence of outliers can distort the normality of the distribution and skew the data. To assess the normality of the data, the Kolmogorov-Smirnov and Shapiro-Wilk tests were employed. The degrees of freedom (df) corresponded to the number of observation points, defined by the number of experts providing votes at the time. The significance level (Type I error rate) was set at α = 0.05, with a P value < 0.05 indicating a statistically significant deviation from normality. The central tendency of data following a normal distribution is best described by the mean, whereas the central tendency of skewed data is better described by the median, along with the interquartile range (IQR). The IQR measures the spread of the middle 50% of the data, providing a sense of variability around the median. A higher IQR indicates greater variability and a less narrow consensus.
Role of the Research Team
Methodological aspects were continuously evaluated, the application of the consensus rule was monitored, and the summary statistics along with the panel input following each round were interpreted and discussed with the entire research group. Close attention was paid to ensure that the panelist’s opinions were not influenced in any way.
Strategy to Improve Response Rate
To maximize the response rate, experts were individually contacted by phone to invite them to participate in the Delphi study. The study’s objectives and process were explained, emphasizing the importance of their commitment to ensuring the validity of the results. Surveys were distributed via the cloud-based survey platform Qualtrics®, which generated personalized links for each expert. These links allowed responses to be automatically saved across devices, facilitating ease of use. Reminders were sent based on previous response times and the expected duration to complete the questionnaire in time. If responses were not forthcoming, personal contact was made by the supervisor. As outlined earlier, qualitative data from previous rounds were anonymized and presented in bullet point format. The experts’ identities were accessible only to the research team to improve response rate and address technical issues. The invitation email specified that survey progress would be automatically saved when using the personalized link. By accessing and completing the survey through this link, experts were considered to have provided implied consent. Therefore, the Research Ethics Committee advised that a comprehensive ethical review was not deemed necessary.
Results
To make the evolving of consensus transparent, a summary of each round can be found in the flowchart (Figure 2). The response rates were 100%, 93%, 87%, and 80% respectively.
Flowchart Delphi process.
Part 1a reached consensus after three rounds, resulting in a five-category PDCM to be used both for MCIs as for daily crowding (Table 1). The last critical interventions of part 1b reached consensus in the final round (round 4), resulting in a universal list of 18 critical interventions that will lead to a CME if withdrawn or withheld from the patient during an MCI or daily crowding. Starting from the original American list of 28 items, 12 were retained, of which 5 renamed; 13 were grouped into 4 new critical interventions; 3 were omitted, and 2 more critical interventions were added. More details can be found in the Online Data Supplement. Part 2 was voted in the final round, resulting in an upper limit of acceptable risk for a CME for each category of the PDCM (Table 1). In categories 1 and 2, outliers were present, and the Kolmogorov-Smirnov and Shapiro-Wilk normality tests were statistically significant (P < 0.05) for 3 out of 5 categories (Figure 3 and Table 2). Therefore, a skewed distribution can be assumed and the median with the IQR was used to summarize the central tendency of the variables. For categories 1 to 5, the median upper limit of acceptable risk (IQR) was 10% (4-17), 19% (13-20), 45% (23-51), 68% (50-77), and 90% (60-97), respectively (Table 1). Furthermore, part 2 also resulted in a 10-point linear numeric scale ranking of the 18 critical interventions, based on their likelihood of developing a CME if withdrawn or withheld. While Table 3 presents the critical interventions across all categories, those identified as having major importance were: cardiopulmonary resuscitation (10), airway management (10), breathing support (10), hemodynamic mechanical support (9), hemodynamic IV support (9), renal replacement therapy (9), vascular occlusion therapy (9), transfusion (8), urgent surgical or interventional procedures (8), burn care (8), and delivery-related support (7).
European patient disposition classification model
CME, consequential medical event; CI, critical intervention; IQR, interquartile range.
Upper limit of acceptable risk for developing a CME.
Tests of normality
* This is a lower bound of the true significance.
a Lilliefors Significance Correction.
Likelihood of developing a CME if CI withheld or withdrawn
* Median (IQR); CME, consequential medical event; CI, critical intervention.
Discussion
Reverse triage discharge protocols and PDCM have received growing attention in recent years. Numerous studies have sought to refine the American reverse triage framework for MCIs proposed by Kelen et al. (2006).Reference Kelen, Kraus and McCarthy5 However, several concerns persist. The framework developed by Kelen et al. (2006) is tailored to the US health care system, which differs substantially from the European system, and is designed exclusively for application in MCIs. To date, no universally accepted criteria for safe early discharge in daily crowding scenarios have been established. Evidence, however, suggests that early inpatient discharge can mitigate delays in ED admissions and alleviate crowding.Reference Pollaris, De Bondt and Sabbe2, Reference Caramello, Marulli and Reimondo13 Access block, the inability to access inpatient beds, remains the primary cause of ED crowding. It impairs ED responsiveness and increases the incidence of adverse events.Reference Taylor4 Additionally, crowding has been linked to higher inpatient mortality rates.Reference Richardson14 Although some studies have investigated the safety of the original reverse triage discharge protocol in a non-disaster setting, no significant changes or updates were made to the original American framework proposed in 2006.Reference Caramello, Marulli and Reimondo13, Reference Caramello, Marulli and Reimondo15 So, despite subsequent advancements, the absence of substantial updates or adaptations to account for regional health care system differences highlights the persistent limitations.
To advance knowledge in the field, one of the primary objectives was to develop a revised and European theoretical framework applicable to both MCIs and daily crowding scenarios. The Delphi expert panel determined that developing a universal PDCM suitable for both MCIs and daily crowding situations would be more effective. The model accounts for the number of patients in each category and the number of beds that need to be made available, requiring earlier-than-planned discharge from 1 or more categories. However, it is essential to recognize that the risk of developing a CME increases with each additional category of patients discharged. Consequently, the multidisciplinary risk analysis differs between MCIs and daily crowding scenarios. In disaster settings, a higher level of risk may be acceptable, provided the acute care needs of disaster victims outweighs the potential risk of CMEs for discharged inpatients. In contrast, the balance is more delicate in daily crowding situations. For example, following a mass shooting with numerous victims, discharging an elderly patient recovering from a hip replacement 1 or 2 days earlier than planned may be justified. However, discharging the same patient due to general ED crowding is more ethically complex. Furthermore, early discharges may elevate the risk of developing a CME, potentially necessitating readmission and imposing an additional burden on the already strained ED.Reference Pollaris, De Bondt and Sabbe2
The subsequent stage (T2) of the translational science process will involve the application of this evidence-based framework to develop a real-time clinical decision support tool, designed for use in both MCIs and daily crowding scenarios, and to include an evaluation through a clinical study.Reference Fort, Herr and Shaw1 A comprehensive multidisciplinary risk analysis should be integrated to minimize the occurrence of CMEs resulting from early discharge.
While the European theoretical framework for reverse triage holds promising potential, it is important to give some consideration to the data used to develop it. As noted in the results section, outliers were identified in the upper limit of risk tolerance boxplots of categories 1 and 2 (Figure 3). As they differ greatly from the overall expert opinion, a misinterpretation of the question was most likely. Despite the outliers, an overall acceptable consensus (IQR < 30%) was achieved for categories 1 and 2. For categories 3, 4, and 5, no outliers were detected, but the spread of the central portion of the data is higher. Nevertheless, an IQR > 30%, indicating a greater variability across the answers, was only noted in the upper limit of risk tolerance of category 5 (Table 1). Thus, an overall valid spread of the data across the different categories was maintained. Because the Delphi method may not be as effective when opinions are highly polarized, it could be beneficial to validate the upper limit of risk tolerance cutoffs throughout an open discussion.
Limitations
Many experts were preoccupied, managing the repercussions of the COVID-19 pandemic in their country or were responsible for the coordination of medical aid to the war in Ukraine. Therefore, multiple reminders were necessary, resulting in long round times with episodes of less focus. The Delphi technique involves controlled feedback through repeated individual questionnaires. This prevents direct discussion between experts to avoid biases, for example, dominance of the majority opinion. Nevertheless, this method may not be as effective when opinions are highly polarized. Moreover, not all European countries were represented in the expert panel. As for the fact that the UK left the EU in 2020, only 14 European countries of the 27 were invited to participate. This may affect the European generalizability of the study results.
Conclusions
The Delphi study successfully met its objectives by establishing a European consensus on the terminology, criteria, and cutoffs related to the reverse triage selection process in MCIs and crowding. This consensus led to the development of a theoretical European reverse triage framework. However, to translate this framework into actionable practice, a real-time Clinical Decision Support Tool (CDST) must be developed to facilitate the reverse triage selection process. The added value of Artificial Intelligence (AI) to the development of the CDST is particularly crucial, as AI’s evolving capabilities can significantly improve decision-making accuracy. Implications for practice and policy include the need for health care systems to adopt a standardized framework for reverse triage, enhancing their efficiency of decision-making in (daily) crisis situations. Several directions for future research can be identified. First, it is essential to test and validate the CDST throughout a multicentric study within our own health care system, across diverse hospitals with varying patient case mixes. Second, exploring the ethical implications and potential challenges associated with the practical implementation of the CDST. Third, investigating the integration of AI and the inclusion of health economic data to ensure the CDST’s adaptability and scalability across different clinical environments. Finally, conducting cross-national testing of the CDST in various European health care systems will be instrumental in refining its universal applicability.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/dmp.2025.62.
Data availability statement
The authors confirm that the data supporting the findings of this study are available within the article or its supplementary materials.
Acknowledgments
We wish to express our gratitude to all members of the Delphi panel for their commitment:
Prof. Abdo Khoury (Université de Franche-Comté, France), Dr. Luc Mortelmans (ZNA Cadix, Belgium), Dr. Jana Šeblová (University Hospital Motol, Czechia), Prof. Francesco Della Corte (Università del Piemonte Orientale, Italy), Prof. Juliusz Jakubaszko (Medical College of Bydgoszcz, Poland), Prof. Castrén Maaret (University of Helsinki, Finland), Dr. Raed Arafat (Ministry of Internal Affairs, Dept. for Emergency Situations, Romania), Prof. Pinchas Halpern (University of Tel Aviv, Israel), Dr. Zsolt Bognar (Pediatric Emergency Dept. Heim Pal National Paediatric Institute, Hungary), Dr. Eric Weinstein (Università del Piemonte Orientale, Italy), Dr. Barbara Hogan (Medical Center MVZ an der Elbe, Germany), Prof. Joost Bierens (University of Portsmouth, UK), Dr. Mick Molloy (Wexford General Hospital, Ireland), Prof. Freddy Lippert (University of Copenhagen, Denmark) and Dr. Vitor Almeida (Sociedade Portuguesa Medicina de Urgência e Emergência, Portugal).
Author contribution
Gwen Pollaris: Conceptualization, Methodology, Formal Analysis, Investigation, Writing—Original Draft, Funding acquisition; Frieda De Bondt; Conceptualization, Methodology, Formal Analysis, Writing—review & editing; Olivier Hoogmartens: Writing—review & editing; Michiel Stiers: Writing—review & editing; Marc Sabbe: Writing—review & editing, Supervision, Funding acquisition.
Funding statement
The corresponding author declares financial support was received for the research, authorship, and publication of this article. KU Leuven and the Flemish Government by the Special Research Fund project C24M/21/034.8.
Competing interests
None declared.
Ethical standard
The Research Ethics Committee advised that a comprehensive ethical review was not deemed necessary.
Open access
