The adoption of information technology (IT) has transformed health care over the years, and hospitals have come a long way from when all records were captured on paper. This has led to a positive correlation between IT adoption and patient safety outcomes. Many emergency department (ED) workflows depend on IT with demands placed on the availability, usability, and accuracy of the applications involved. Ransomware attacks are more numerous and disruptive throughout the world, and recently the health care sector has become a significant target. Reference O’Dowd1–Reference Akselrod3
On May 14, 2021, during the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, the largest cyberattack ever recorded on the Irish national health service (Health Service Executive) was discovered and the response shut down national and local systems involved in all core hospital services. 4,5 This resulted in no digital patient administration system, emergency department information system (EDIS) Symphony®, laboratory system, national radiology image access, Reference Anderson and Torreggiani6 nor Intranet/Internet access. The IT blackout confronted clinicians and administrators, normally in a near fully digitally supported working environment, with the realities of working in an ED during a digital disaster. The hospital did not have a section of its major emergency plan dedicated to such a loss of IT systems.
Most health care workers express an interest in learning to prepare for natural, pandemic, and manmade disasters. Reference Balut, Der-Martirosian and Dobalian7,Reference Dobalian, Balut and Der-Martirosian8 If a hospital is not prepared for crises and disasters, it undermines the capacity of administration and staff to safeguard the safety of patients. Reference Barten, Klokman and Cleef9 Reflection on disaster incidents can inform future disaster planning, provide an opportunity to identify lessons learned that will drive improvements in emergency management through preparedness and mitigation measures, and provide response innovations. Reference Shea, Hick, Schwedhelm and Sauer10
This descriptive case study illustrates the experience of a pediatric tertiary referral center by describing emergent interventions and the resulting impact by a time series analysis of three 6-week periods before, during, and after the cyberattack IT outage. The data analysis aspect of this study will focus on the patient flow timings (three 6-week period comparisons and the impact of the gradual restoration of each IT system) and the quality of the data captured.
Methods
Study Setting and Approach
A retrospective analysis was performed from a pediatric tertiary multi-university-affiliated teaching hospital in the Republic of Ireland, with yearly pre-pandemic ED attendances of over 39 500. Equal 6-week comparison periods were analyzed before and after, from April 2, 2021, to August 5, 2021 (18 weeks) inclusive, prior to the cyberattack digital disaster from April 2 to May 13, 2021 (Period 1), during the cyberattack from May 14 to June 24, 2021 (Period 2) when all or most IT systems were unavailable, and after from June 25 to August 5, 2021 (Period 3) when all systems were restored. Sub-analysis on patient flow during the staggered restoration of each IT system was analyzed, which included the hospital patient administration system, radiology, pathology, and finally the Symphony® (EMIS Health, UK) EDIS. The reporting guidelines for Strengthening the Reporting of Observational Studies in Epidemiology were followed. Reference Vandenbroucke, von Elm and Altman11
Data Sources, Quality, and Measures
A hastily designed and implemented Microsoft® Excel® 2016 MSO (Version 2110) spreadsheet was created by the principal investigator (MB) in the hours following the cyberattack to maintain an accurate ED census and essential data along with additional administrative personnel (a minimum of 1 dedicated Excel sheet administrator and 1 administration runner between registration, triage, and spreadsheet). Paper ED triage and clinical notes were commenced on the day of the cyberattack by adjusting an existing section of the major emergency plan used for mass casualty incidents for the initial generation of unique hospital patient identifying numbers. Further unique hospital patient identifying numbers were generated in consultation with the hospital medical record department for the remainder of the 6-week period.
Data entry points specific to the ED included existing (if known and confirmed) or new health care record (HCR) number, suspected SARS-CoV-2 status at triage, triage category (5 point scale; category 1= most critical to category 5 least critical), 12 presenting complaint, treating clinician, ED location, specialty referred to, discharge destination, and patient experience times (time points for registration, triage, clinician time seen, specialty referred to, decision to admit [DTA], and discharge time). The bespoke Excel® sheet was on a non-networked standalone desktop computer (connected to a large television in the central administration hub), staffed 24/7 by 1 to 2 newly created ED administrators (in response to the disaster). An administrator updated Excel®, in a contemporaneous fashion from written or verbal information from administrators, nurses, doctors, advanced nurse practitioners, and bed managers. All necessary updates were checked and reinforced at the increasing number (5+) of ED rounds in response to the disaster. Data for the pre (Period 1) and post (Period 3) system IT outage were extracted from the principal EDIS (Symphony® [EMIS Health, UK]).
Patient experience times were recorded throughout each period. The Excel® workbook was cross referenced (MB and HO’R) from 2 separate pre-existing paper independent primary sources containing data on admitted patients across all time periods. Additional feature engineering tasks were performed on data sources prior to consolidation into 1 data set (outlined in Supplementary Table 1). A single power outage resulted in a 1 day loss of electronic data, which occurred on May 28, 2021, culminating in a data capture issue; therefore, this date was excluded from the patient flow analysis (Supplementary Figure 1). Also excluded from the patient flow analysis were duration times less than 0 and records with missing date times.
Data quality dimensions Reference Mashoufi, Ayatollahi and Khorasani-Zavareh13–Reference Wang and Strong17 were used to compare quality between data recorded in Excel® during Period 2 with data recorded in the EDIS during Periods 1 and 3. Data quality characteristics were analyzed using 8 dimensions Reference Earley, Henderson and Management Association16 : accuracy, completeness, consistency, integrity, reasonability, timeliness, uniqueness/de-duplication, and validity. Also included were the dimensions of accessibility and security from the Wang and Strong framework. Reference Wang and Strong17
Data Analysis
Data were extracted from the EDIS using SQL Server Management Studio, version 18.10, and an analysis was performed in R version 4.1.1.
Categorical data were described using counts and percentages. Continuous data were assessed for normality, using QQ plots and the Shapiro-Wilk test. Median and interquartile range (IQR) were used for non-parametric continuous data and mean and standard deviation for parametric continuous data. Periods 1 and 3 were compared separately, using χ2 for categorical, t-test for parametric continuous variables, and Mann-Whitney for non-parametric continuous variables.
For the analysis of the gradual restoration of each IT system on ED performance, the difference between the independent groups for non-parametric data was evaluated, using the Kruskal-Wallis test. Post hoc tests Reference Siegel and Castellan18 were carried out on progress times yielding a significant result, using pairwise comparisons adjusted for multiple comparisons. A 2-tailed P value < 0.05 was considered statistically significant.
Results
The final data set consisted of 13 390 visits: Period 1 comprised 4471 visits, 4596 visits during system outage in Period 2 when all or some IT systems were unavailable, and 4323 in Period 3 when all systems were restored.
Demographics and Characteristics
A significantly younger cohort presented in Period 3 (median 4 years) compared to Periods 1 and 3 (median 6 years). No significant difference was seen for gender, triage, day of the week, and return within 7 days. There was a significant difference (P < 0.001) in patients with suspected SARS-CoV-2, 22.1% during system outage (Period 2) compared to 29.5% (Period 1) and 39.7% (Period 3). There was an increase in patients admitted from the ED in Period 2, although not statistically significant. Comparing medians, there were 29 and 67 additional admissions compared with Periods 1 and 3, respectively (Table 1).
Characteristics of patient visits, comparing data in period 1, prior to the cyberattack, period 2, during the cyberattack, and period 3, post cyberattack
a 0.2% of overall data missing
b 3 missing gender
c 0.5% of overall data missing or patient left before triage
d 4% of presenting complaints grouped into ‘Other’
e 0.2% of overall data missing
# Individual comparison using binary variable for each characteristic
Patient Flow
During Period 2, the total length of stay reduced significantly (P < 0.001) by between 19 (Period 1) and 22 minutes (Period 3). By comparison of medians, this time reduction was most noticeable from registration to triage (2 to 3 minutes shorter) and from clinician review to discharge (excluding admitted patients) (12 and 16 minutes shorter). An analysis of the return visits revealed no significant change in the total length of stay between periods. However, registration to triage was again significantly reduced.
For the most critical triage categories of 1 and 2, there was a saving of 5 to 6 minutes from registration to triage and 33 to 35 minutes from clinician review to discharge (excluding admitted patients) during system outage (Table 2).
Volume and Flow by Triage
a 1.9% of overall data missing
b 3.3% of overall data missing or patient left before triage
c 8.9% of overall data missing or patient left before triage or left before being seen by a clinician
d 6.9% of overall discharges (excluding admitted patients) missing data
e 10% of visits with a discharge group of ‘Admitted to ward’, ‘Death’, ‘PICU’ or ‘Transfer to another hospital’ missing data
f 6.9% of visits with a discharge group of ‘Admitted to ward’, ‘Death’, ‘PICU’ or ‘Transfer to another hospital’ missing data
IQR, interquartile range.
After an initial increase in time over Days 1 and 2 of the attack, during the remainder of the first week of the cyberattack (Week 1 of Period 2), all patient flow stages decreased. When the EDIS was turned back on in Week 1 of Period 3, the time for all patient flow stages increased, except triage to clinician review (Figure 1).
Median patient flow times from periods 1 to 3.
The sub-analysis of the gradual restoration of each system showed that compared to the 23 days when there were no IT systems available, patient experience times significantly decreased (P < 0.05) during Period 2. Total length of stay reduced from a median of 188 (IQR 107-292) to 166 (IQR 100-285) minutes, registration to triage reduced by 1 minute, triage to clinician review by 3 minutes, clinician review to discharge (excluding admitted patients) by 8 minutes and clinician review to DTA by 12 minutes. None of the system restorations had a greater impact on timings than the EDIS. Upon EDIS restoration, registration to triage doubled from 6 (IQR 4-11) to 12 (IQR 7-19) minutes, clinician review to discharge (excluding admitted patients) from 60 (IQR 29-122) to 81 (IQR 39-153) minutes, and the total length of stay increased significantly from 167 (IQR 98-258) to 185 (IQR 109-291) minutes (Supplementary Tables 2 and 3).
The average patient lengths of stay of Periods 1 and 3 were applied to the number of patients who presented in Period 2 and compared to the actual accumulated average: a reduction of 1149 patient hours (47.8 days) in the ED for the 6-week period during system outage was identified. Applying the same method, 230 hours (9.5 days) were saved from registration to triage and 766 hours (31.9 days) from clinician review to discharge (excluding admitted patients).
Data Quality
Analysis of data quality dimensions revealed many data quality issues with the use of the bespoke Excel®. The EDIS improved the quality in all 10 dimensions (Table 3).
Data Quality Dimensions
Return visits for 175 patients had an HCR number belonging to another patient. Using the patient’s name and date of birth, the HCR number was obtained for 93 visits, and 82 were undetermined. Missing or unreasonable duration times revealed issues with 19.7% of data recorded during Period 2 compared to 4.7% (Period 1) and 5.1% (Period 3). Missing, invalid or the reuse of HCRs was highest during Period 2, with 4.4% of records impacted. Missing timings were highest during Period 2: 3.6% for triage, 4.6% for clinician, 2.5% for discharge, and 11.3% for DTA time; 1.2% of Period 2 visits had no discharge destination recorded. Patient flow durations that were less than 0 minutes for triage to clinician review were lower during Period 2 at 0.8% compared to 2.8% (Period 1) and 2.7% (Period 3); however, all other patient stage timing inaccuracies were higher (see Table 4).
Data Quality Analysis for Missing Data and Duration Times < 0 Minutes
a as a percentage of visits excluding did not wait before triage
b as a percentage of visits excluding did not wait
c as a percentage of visits with a discharge group of ‘Admitted to ward’, ‘Death’, ‘PICU’ or ‘Transfer to another hospital’
# Included data for 28th May 2021, which was excluded from time series analysis due to significant data quality issues.
Limitations
This study provides a unique insight into ED performance when moved from high dependence on IT systems to none or partial access for 6 weeks as a result of the cyberattack 5 and back again. Limitations include the fact this is a retrospective single site study and prone to all the biases therein; however, prospective work on this subject is unlikely. Presenting complaint categorization was based on decision rules around free text; a patient may have many symptoms recorded, and only 1 was used. Data entry variations may be a contributing factor to significant differences in presenting complaint, SARS-CoV-2 streaming and increased use of “Other/Unknown” for discharge destination in Period 2. Hospital number was used to determine returns; missing and the reuse of hospital numbers impacted this calculation. Although data quality imperfections were most apparent in the timings in Period 2, this was based on 2 measures alone: missing timings and durations less than 0 minutes. An audit of each individual timestamp would not be feasible to perform. The EDIS does not have controls in place to mitigate against the recording of inaccurate timings, having similar controls to those of Excel® except for mandatory discharge times. These data discrepancies are limitations and reflect data quality results. Partial restoration of IT systems such as radiology allowed only small sample size (229 attendances) comparisons.
Discussion
This study describes the situation, response, and outcomes of the pediatric ED during a 6-week digital disaster. There are learnings present in the data and experiences during that disaster that will enhance the department’s efficiency day to day with EDIS and the preparation and response to the next digital disaster. To the best of our knowledge, no such report exists in the literature. There was no significant change to most demographics and characteristics of patient visits presented in Table 2. Significant variance was found in the recording of SARS-CoV-2 status, presenting complaint and discharge group; however, data capture issues and input value differences indicate probable cause (see Table 4). The overall ED length of stay reduced significantly in Period 2, by between 19 and 22 minutes (based on the median). Within the first week of Period 2 (cyberattack), all patient experience times had reduced. The restoration of each IT system did not have as big an impact on timings until the EDIS was restored, resulting in a significant increased patient experience times within the first week of Period 3 (except for clinician review to DTA). The 10 data quality dimensions revealed significant compromise in data quality with the use of Excel® during Period 2 compared to data recorded in the EDIS (19.7% data imperfections in Period 2, 4.7% in Period 1, and 5.1% in Period 3).
The perception amongst providers is that administration and triage processes were much faster in Period 2 and uncertainty about being better for the patient. The reduction in length of stay during system outage was most significant from registration to triage and clinician review to discharge (excluding admitted patients). These are times associated with the largest volume of patient-related data entry. The ED management team postulate that this reflects a combination of inadequate pre-existing local IT infrastructure and current EDIS workflows that were bypassed by both the simple pen and paper and human resourced administration processes. A reduced number of data points was captured electronically during Period 2, 21 data points compared to over 489 data points that are generally entered on the EDIS for each patient attendance (see Table 4). During Period 2 registration, triage and clinical assessments were paper-based and therefore portable, easily accessible, and apparently more efficient from a patient experience perspective. Dedicated 24/7 administrative staff entered information, and timings in Excel® upon verbal instruction allowed delegation of data entry from the clinician to an administrator. Electronic data take longer to record than paper, Reference Perry, Sutherland and Symington19 and the use of electronic documentation in some EDs was associated with significant increases in patient length of stay. Reference Feblowitz, Takhar and Ward20 On the use of scribes, there are conflicting reports with some finding no difference to length of stay Reference Addesso, Nimmer and Visotcky21,Reference Heaton, Castaneda-Guarderas and Trotter22 and others reporting a reduction. Reference Walker, Ben-Meir and Dunlop23–Reference Thomas, Marcum, Wagner and Kohn25 In this instance, the process of administrative staff data entry was associated with significantly improved patient experience times. Owing to the computer infrastructure limitations and EDIS interface, the real time accuracy of the EDIS timing pre- and post-system outage comes into question. Ward et al. observed that more data errors were present after EHR implementation emphasizing differences between observed and inputted electronic timestamps. Reference Ward, Froehle, Hart and Lindsell26 Dedicated administration improved the apparent currency of these timings with Excel®.
Although data were entered in a timelier manner, they were at a significant cost. The quality of data suffered with 19.7% of the records exhibiting issues in Period 2. User training and systematic safeguards can prevent dirty data in databases, Reference Vosburg and Kumar27 keeping in mind data quality dimensions during the initial setup. Reference Earley, Henderson and Management Association16,Reference Wang and Strong17 Temporary processes and infrastructure had to be put in place quickly (hours) with little time for development or training. SARS-CoV-2 infection control processes from registration were undermined. Variables such as presenting complaint and discharge destination were completed by multiple users of varying experience in a busy work environment most likely contributing to data quality issues. The reuse of HCR numbers presented a risk which could have been mitigated by applying referential integrity Reference Earley, Henderson and Management Association16 at the patient level.
Incidents of hospital-targeted cybercrime are becoming more widespread and a proactive approach should be taken to prevent an attack taking hold in the first instance. Reference Bhuyan, Kabir and Escareno28 Key cyber security principles should be put in place centered around leadership, governance, cyber security expertise, IT infrastructure, and staff awareness and training. 5,Reference Muthuppalaniappan and Stevenson29,Reference Argaw, Bempong, Eshaya-Chauvin and Flahault30 This should be incorporated by the hospital emergency planning committee into the hospital’s major emergency/disaster plan. Reference Pugh and Dameff31 If a breach occurs, it may take time to recover systems from offline backups; a business continuity plan should be implemented, including preparations for workarounds for the total loss of IT systems 5 that may be inaccessible for weeks. Continuity plans should include how and where data will be recorded to support clinical workflows. An Excel® workbook was used in the ED; the design could have been improved by following data quality dimensions and implementing backups. A considerable gap has been identified in this study, which could be filled by the pre-emptive creation of a simplified bespoke electronic patient information system that operates on standalone computers. The investment of new additional staff to the ED is critical to support the implementing and maintaining of temporary information systems and assisting new workflows.
Externally, it should be noted that cyberattack-related national public announcements did not result in the desired outcome of reducing pediatric ED attendances. This is hugely different from initial SARS-CoV-2 pandemic experience Reference McDonnell, Nicholson and Conlon32 and during the WannaCry attack across the NHS in England; visits to the ED decreased by an average of 6% per infected hospital per day. Reference Ghafur, Kristensen and Honeyford33 In this study, despite a 25% decline in attendance on the second day following the cyberattack compared to the average for all 3 periods, daily visits during system outage were on average 3% higher compared to Period 1 and 6% higher compared to Period 3, despite public announcements. Compared to matched periods in 2019, visits were higher for all 3 periods of time in 2021. The observed increase in attendances during the cyberattack period reflects the role placed upon the pediatric ED in the Irish health system for the provision of unscheduled primary and secondary care during a National Cyberattack. Reference McDonnell, Nicholson and Barrett34
Interestingly, this cyberattack made it possible to learn about the current IT system/infrastructure. Existing clinical/digital workflows are not efficiently supported by processes, and IT structures with insufficient computer hardware in key locations were identified. This affects accurate recording of patient journey times in 2 ways; a proportion of the clinician’s time is spent accessing the EDIS and, separately, the time competing clinical priorities also result in a delay in data entry due to prioritizing patient assessments and clinical care.
Conclusions
The loss of IT systems from a digital disaster for 6 weeks was uniquely associated with a reduction in ED patient experience time. This may be due in part to increased clinician administration efficiencies and administrative staff assisting with data entry; however, data quality suffered greatly. A hospital’s major emergency plan should proactively include a low-tech cyber-disaster plan, and this study uniquely provides the experience of digital disaster management and resultant outcomes in a pediatric ED.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/dmp.2023.86
Data availability statement
The underlying anonymized data will be made available upon reasonable request to the corresponding author pending the approval of the data protection office.
Acknowledgments
We greatly acknowledge and thank all hospital clinical, administration, information technology, and operational staff for their contribution during the cyberattack and the resulting recovery.
Author contribution
FL and MB conceptualized the study. MB created the Excel® file and implemented workflows with PD, CB, LM, BC, ER, KB, and DH. For the data recorded in Excel® during system outage, PD, BC, CB, KB, DH, ER, LM, and MB coordinated the data with administrators, while MB, H’OR, and FL prepared the Excel® data for analysis. FL extracted the data from the Information Technology systems, created the final data set and carried out the primary analysis. FL, MB, H’OR, PD, CB, and BC provided a substantial contribution to the design, methodology, and analysis of data in the initial stages. All authors have reviewed, provided critical feedback, contributed to the writing of the manuscript, and approved the final version.
Funding statement
This study was not funded.
Competing interests
The authors declare no conflicts of interest.
Ethical standard
This study was registered with the Children’s Health Ireland research ethics office (Communication June 28, 2021). All data processing and storage complied with GDPR and data protection policies. This service evaluation study was exempt from requiring ethical approval by the hospital research ethics office (Communication June 29, 2021).
Open access
