Introduction
It is estimated that every 42 seconds, an American will suffer a heart attack, otherwise known as myocardial infarction (MI).Reference Sanchis-Gomar, Perez-Quilis, Leischik and Lucia 1 The state of Utah alone averaged 5.4 hospitalizations per day throughout 2015 for MI. 2 With a high enough clinical suspicion for MI, health care providers will acquire an electrocardiogram (ECG) to see if this event is of the sub-type associated with ST-segment elevation. ST-segment elevation myocardial infarction (STEMI) is a time-sensitive diagnosis that requires immediate intervention. Time from onset of symptoms to definitive treatment is directly correlated to mortality, with current guidelines recommending that mechanical thrombectomy by percutaneous coronary intervention occur within 90 minutes of presentation to the hospital (“door-to-balloon time” [D2B]).Reference Bossaert, O’Connor and Arntz 3
Prehospital acquisition of ECGs is consistently associated with reduced D2B times in the treatment of STEMI due to activation of the cardiac catheterization lab (CCL) prior to the patient arriving in the emergency department (ED).Reference Brown, Mahmud, Dunford and Ben-Yehuda 4 – Reference Clark, Sejersten, Clemmensen and Macfarlane 9 The 2015 American Heart Association (Dallas, Texas USA) Guidelines for Emergency Cardiovascular Care made early acquisition of prehospital ECGs a Class I recommendation.Reference Bossaert, O’Connor and Arntz 3 , 10 The question remains, who should interpret the prehospital ECG?
Paramedic training includes basic competency interpreting 12-lead ECGs, and with sufficient training, paramedic ECG interpretation can be non-inferior to that of an emergency physician.Reference Feldman, Brinsfield, Bernard, White and Maciejko 11 However, because this level of training investment is not feasible for many Emergency Medical Services (EMS) agencies, strict reliance on paramedic interpretation is not always an option. This is particularly true for fire-based systems where many topics compete for limited training hours and funding (ie, HAZMAT or high-angle rescue). Besides reliance on paramedic interpretation, another option is strict reliance on computer interpretation. However, previous literature has shown a wide-range of accuracy of these computer algorithms,Reference Al-Akchar, Aguirre and Mahmaljy 12 – Reference Bhalla, Mencl, Gist, Wilber and Zalewski 18 and has identified common sources of computer error – namely artifact and conditions causing a wide QRS.Reference Bosson, Sanko and Stickney 19 , Reference Swan, Nighswonger, Boswell and Stratton 20 A third option is to transmit the ECG for physician interpretation. Transmission can be fraught with technical difficulties (eg, poor wireless data signal) that prevent patients from receiving the benefit of prehospital activation. Additionally, reliance on physician read increases physician workload, creates additional interruptions, and may contribute to alarm fatigue. Depending on physician coverage, patient volume, patient acuity, and a number of other factors, this model may be undesirable.
Previous literature has identified that clinical tests with binary output are difficult to use efficiently.Reference Simel, Matchar and Feussner 21 – Reference Cannesson, Le Manach and Hofer 27 A highly sensitive algorithm is likely to produce false-positives. A highly specific algorithm would not correctly identify some patients requiring intervention, preventing them from receiving the benefit of pre-arrival CCL activation. Additionally, physician review of positive results from a highly specific algorithm may be unnecessary because the post-test probability of the result is already so high.
The ideal use case combines algorithms such that instead of a binary output, it produces three tiers of probability – high, intermediate, and low.Reference Simel, Matchar and Feussner 21 , Reference Jamart 23 – Reference Cannesson 26 High and low probability cases could bypass physician overread, focusing physician attention on the intermediate “gray zone” where their training is best applied.
It is proposed that computer interpretations of STEMI can algorithmically be recognized using four simple criteria to create high and intermediate probability tiers, while deeming computer negatives as low probability. It is further hypothesized that through this combination of man and machine, that the specificity and positive predictive value of the high probability tier will be sufficient for prehospital CCL activation by paramedics without physician overread.
Methods
This study was conducted at a single urban academic center in Salt Lake City, Utah with an annual census of 88,000 patient encounters. The receiving facility ED had 67 beds in three zones, with a minimum of double attending physician coverage 24 hours per day. Patients arrived from multiple EMS agencies in the region. Prehospital medical response and transports were either entirely fire-department-based, or fire department first response with third service ambulance contracted for patient transport. Agencies in the region use the same chest pain protocol, which instructs providers to acquire a 12-Lead ECG if possible, then: “If the patient has a STEMI then transport to the closest available STEMI/PCI [percutaneous coronary intervention] receiving center (if available) and give advanced notification of ECG findings and transmission of ECG if possible.” Currently, STEMI is not clearly defined in the regional protocol, leaving room for providers to interpret the ECG themselves or utilize the computerized interpretation. There were six STEMI receiving centers in the county, and paramedics chose the closest center based on distance and knowledge of local traffic patterns. Patients without STEMI were transported to the closest appropriate facility. Prehospital providers are licensed by the Utah Bureau of Emergency Medical Services and Preparedness (Salt Lake City, Utah USA), which recognizes but does not require certification by the National Registry of Emergency Medical Technicians (Columbus, Ohio USA).
Prehospital 12-lead ECGs were acquired on ZOLL E-series monitors (ZOLL Medical Corporation; Chelmsford, Massachusetts USA) and transmitted via email attachment at the discretion of paramedics based on their own ECG interpretation or the computer interpretation of STEMI on ECG. A nurse in the ED printed the ECG and showed it to an emergency physician for a decision to activate the STEMI protocol or not. If the decision was made to activate the CCL, the ED charge nurse would then call the “STEMI nurse” via the Vocera Wi-Fi communication system (Vocera Inc.; San Jose, California USA), who served as a single point of contact for CCL activation. The interventional cardiologist would assess the patient in the ED while the CCL team prepared for the procedure. Interventional cardiologists had the option to overread the emergency physician interpretation before proceeding with the procedure, but this was left to provider preference.
Based on previously published data on sources of computer false-positives, four criteria were selected to identify likely false-positive computer interpretations of STEMI (Table 1).Reference de Champlain, Boothroyd and Vadeboncoeur 15 , Reference Bosson, Sanko and Stickney 19 , Reference Swan, Nighswonger, Boswell and Stratton 20 , Reference Pilbery, Teare, Goodacre and Morris 28 Prehospital STEMI cases were prospectively collected from May 2012 through October 2013. This collection period was based on the availability of research assistants rather than a-priori sample size calculations. All prehospital ECGs received at the STEMI receiving center with a computer interpretation of “*** Acute MI ***” were eligible for enrollment. Patients transferred between facilities already diagnosed with STEMI by a physician were excluded. Cases were entered into an Excel spreadsheet (Microsoft Inc.; Redmond, Washington USA) as they were received. Cases were then matched to ED records, and the chart was used to determine if the ED physician activated the CCL, which was used as the outcome of interest. The ECGs were interpreted and ran through each of the four criteria by a nationally registered paramedic with three years of prehospital experience and four years of experience as an ED ECG tech who was not blinded to the study hypothesis. Both CCL activation and ED diagnosis were abstracted after the ECG had been put through the algorithm.
Table 1. Algorithm Criteria and Associated Source of Computer Interpretation False-Positive
Cases meeting all four criteria were considered high probability, while those that failed any of the criteria were considered intermediate probability. Only one failed criterion was recorded per case – the one that was felt to be the major source of error by the paramedic rater. Cases that were not read by the computer as STEMI were considered low probability and were not collected in this dataset (Figure 1). For intermediate probability cases, which criterion were failed was documented. The first ED ECG was retrieved and its computer interpretation for STEMI was recorded for all cases. For cases where the prehospital ECG failed criteria solely due to either artifact or baseline wander, the ED ECG was evaluated using the algorithm in a separate secondary analysis under the assumption that the ED ECG was likely physiologically similar but of better technical quality. Cardiac catheterization outcomes (if an intervention was performed) were also recorded.
Figure 1. Overview of Algorithm.
Statistical analysis was performed in R (R core team; Vienna, Austria) using the EpiR package (Stevenson, et al; Melbourne, Australia). To test if applying the algorithm would result in a significant reduction in the number of tests requiring physician overread, McNemar’s test was used on a 2x2 table. To validate the criteria, every combination of criteria was tested as a different version of the algorithm, as well as the prehospital computer interpretation alone. Contingency tables were created to compare each version to the outcome of interest of ED physician activation of the CCL. Test characteristics of sensitivity, specificity, positive predictive value, negative predictive value, and likelihood ratios along with their 95% confidence intervals were calculated. Linear regression was used to test for association between accuracy and number of criteria used in each version of the algorithm, and logistic regression to test for association between physician CCL activation and criterion that was failed. The University of California, San Diego Human Research Protections Program (San Diego, California USA) approved the study protocol and granted waivers for informed consent and HIPAA authorization (project 170918XX). The authors have no conflicts of interest.
Results
A total of 83 cases were collected during the study period, of which 18 were lost to follow-up and two were excluded as interfacility transfers (Figure 2). The cases lost to follow-up were transmitted to the facility in error (ie, the transporting EMS unit selected the incorrect facility when transmitting the ECG and the patient never arrived at the facility). In the remaining 63 cases, there were 24 ECGs considered intermediate probability (did not meet all criteria) and 39 considered high probability (met all criteria). The most common criterion failed was the presence of baseline wander or artifact. No cases failed the heartrate cutoff.
Figure 2. ECGs Applied to the Algorithm and Activation of Cardiac Catheterization Lab.
Logistic regression of physician CCL activation on failed criterion revealed that there was no significant relationship (P = .80). If physicians only reviewed intermediate probability cases (ie, high probability cases resulted in CCL without physician review), this algorithm would reduce the number of ECGs requiring physician overread from 63 to 24, or by a factor of 0.62 (95% CI, 0.48-0.75; P <.01; Table 2). All patients that had the CCL activated received an intervention, defined as balloon angioplasty and/or stent placement.
Table 2. Reduction in Prehospital ECGs Requiring Physician Read if Only Intermediate Probability Cases were Reviewed and High Probability Cases Initiated CCL without Physician Read
Note: Reduction factor 0.62 (95% CI, 0.48-0.75; P <.01).
Abbreviations: CCL, cardiac catheterization lab; ECG, electrocardiogram.
The sensitivity, specificity, false-positive rate, and area under the curve were resulted for each combination of the four criteria, including a variation using the repeat ED ECG in the cases with prehospital ECG artifact (Table 3). The comparison between number of criteria used and false-positives showed a negative correlation (adjusted r-squared = 0.51; P <.01). Each combination of criteria was tested as a separate version of the algorithm. Because no cases failed the heartrate criterion, several combinations of criteria were effectively the same (eg, V10 and V12 differ only by the heartrate criterion, so when applied to the dataset [with no cases that failed to meet the heartrate cutoff], they produced identical results. Hence, several rows in Table 3 and points in Figure 3 appear repetitive). Additionally, the test characteristics of V14 (which uses only the heartrate criterion) could not be evaluated because it failed to classify any cases.
Table 3. Testing Algorithm Accuracy with Application of Various Criteria Component Combinations
Note: ART: artifact or baseline wander; HR: Heart rate; STE: ST-elevation <1 mm; QRS: Wide QRS; V1ED: full criteria applied to dataset that included first ED ECG; FDR: False Discovery Rate; AUC: Area Under Curve). Values in parenthesis are the 95% confidence interval around the estimate.
Figure 3. Receiver Operator Curve.
The combination of all four criteria (V1) and the version with three criteria excluding heart rate only (V4, which had an n of zero) were both accurate with a 0.0 false discovery rate (1.00 ‐ positive predictive value) as shown in Figure 4. As fewer criteria were used, the false discovery rate increased. The accuracy of computer interpretation alone (ie, zero criteria) was 0.11, as shown by the point for zero criteria labeled “V0” in Figure 4. The version of the algorithm using only heart rate criterion (V14) had no improvement in accuracy over the computer alone given the n of zero. For algorithms that used only one criterion, artifact/baseline wander had the greatest power to improve false discovery, as shown by algorithm V15.
Figure 4. Number of Criteria from the Algorithm versus False Discovery Rate.
Algorithms V1 and V4 had excellent test characteristics for ruling in STEMI, with sensitivity 0.7 (95% CI, 0.56-0.81), specificity 1.00 (0.59-1.00), positive predictive value 1.00 (0.91-1.00), and negative predictive value 0.29 (0.13-0.51). The algorithm performed well again in the secondary analysis, which replaced the prehospital ECG with the first ED ECG for cases that failed due to baseline wander or artifact (V1ED; n = 13): sensitivity 0.85 (0.74-0.94), specificity 1.00 (0.54-1.00), positive predictive value 1.00 (0.93-1.00), and negative predictive value 0.43 (0.18-0.71). The mean time from prehospital ECG acquisition to ED arrival was 21.3 minutes with a standard deviation of 9.3 minutes. A receiver operator curve comparing all algorithm versions is shown in Figure 3. V1 and V4 had the greatest area under the curve of 0.70 in the dataset without ED ECG substitution, and 0.86 in the secondary analysis with ED ECG (V1ED).
Discussion
In this small, single-center pilot study, it was shown that an algorithmic approach to identifying prehospital false-positive computer interpretation of STEMI can improve test characteristics compared to computer interpretation alone by creating multiple tiers of diagnostic probability. The high probability group demonstrated a 0.00 false-positive rate, 0.00 false discovery rate, and 1.00 specificity when using all criteria in the algorithm, compared to the 0.11 false discovery rate of the computer alone. In other words, any given computer ECG interpretation of STEMI has an 11% chance of being a false-positive, while zero percent false-positives using the algorithm in this data set were observed. The low false discovery rate is likely attributable to a small sample size. The 95% confidence interval for positive predictive value in the high probability group went as low as 0.91, which would result in a nine percent false activation rate. Comparison to previous literature is difficult, as researchers use different gold standards for their definition of true STEMI. Despite these challenges, the algorithm appears to perform robustly in the high probability group.Reference Al-Akchar, Aguirre and Mahmaljy 12 – Reference Bhalla, Mencl, Gist, Wilber and Zalewski 18
For cases that failed any criterion (intermediate probability), the most frequent reasons to fail meeting criteria were baseline wander and artifact, both of which have been implicated as common causes for prehospital false-positive computer interpretation of STEMI.Reference Sanko, Eckstein and Bosson 14 , Reference Kado, Wilson, Strom and Box 17 , Reference Bosson, Sanko and Stickney 19 , Reference Coffey, Serra, Goebel, Espinoza, Castillo and Dunford 29 While there are many challenges to acquiring high-quality ECGs in the prehospital setting, it is likely that additional training on ECG acquisition skills could improve data quality. Repeating the ECG may also have a role, as previous literature has shown that repeat prehospital ECGs increase the identification of STEMI.Reference Verbeek, Ryan, Turner and Craig 30 , Reference Tanguay, Lebon, Lau, Hébert and Bégin 31 It is unclear if this effect is from the evolution of the infarct or acquiring a higher quality ECG. When utilizing the ED ECG for these cases, the algorithm accurately classified all cases.
The logistic regression found no predictable relationship between failing any particular criterion and physician activation of CCL. If a particular criterion was predictive of a physician CCL activation, this would have suggested that criterion was a significant source of misclassification. However, because no relationship was found, it appears that no single criterion is significantly inaccurate compared to any other. This is a key reason for physicians to overread intermediate probability ECGs, as they represent the most challenging cases to interpret.
Previous attempts to merge paramedic and computer interpretation have been successful, reducing false activations to as low as 0.16 by using clinical rather than electrocardiographic criteria.Reference Wilson, Kado and Percy 16 With the lower bound of the confidence interval for positive predictive value at 0.91, this would represent at worst a false activation rate of nine percent. This suggests that ECG criteria may have a larger role than clinical criteria in accurately identifying prehospital STEMI patients.
Systems that rely on computer interpretation alone may deal with false activations ranging from 10%-50%.Reference Sanko, Eckstein and Bosson 14 – Reference Bhalla, Mencl, Gist, Wilber and Zalewski 18 Allowing paramedics to apply an algorithm for CCL activation without physician overread could dramatically decrease the costs of a STEMI program by reducing false activations that cost thousands of dollars per incident. This approach may also be appealing to EMS systems where mobile data connections used for ECG transmission are problematic, or in systems that cannot afford the additional equipment needed for ECG transmission. While hospitals will ultimately need a copy of the prehospital ECG before performing the intervention, for high probability cases, this approach would remove the time sensitive component to transmitting the ECG and allow it to occur when there is good connectivity and all critical patient interventions are already completed.
For systems that rely on transmission for physician interpretation, allowing paramedics to utilize these criteria to activate CCL resources would reduce the need for ECG overread in high probability cases. Physicians could focus instead on cases that fail criteria (ie, intermediate probability) and require their level of training to interpret. In addition to reducing D2B time for high probability cases, this would have the added benefit of reducing alarm fatigue and interruptions in a specialty where interruptions to clinical work already occur at a startlingly high frequency.Reference Westbrook, Raban, Walter and Douglas 32 – Reference Chisholm, Dornfeld, Nelson and Cordell 35 Alternatively, paramedics could activate the CCL, then transmit the ECG for physician read to potentially cancel the activation, or simply for documentation purposes. The decision whether to transmit high probability cases, allow paramedic activation of the CCL, or a combination of both, will depend heavily on regional practice variations and the relationships between EMS, the ED, and interventional cardiology.
It would also be possible to integrate these findings directly into computer algorithms, producing three tiers of probability from the computer printout directly. Paramedics could then cross-check the computer interpretation by verifying the algorithm criteria. Having paramedics and the computer follow the same algorithm would prevent paramedics from adopting an attitude of consistently assuming their device is incorrect, instead learning to verify its findings like any other piece of clinical equipment.
The benefit of implementing any of these systems will vary dramatically based on facility and regional characteristics. While shorter D2B is associated with a reduction in mortality, there has been no continued reduction demonstrated to pushing D2B below 90 minutes.Reference Menees, Peterson and Wang 36 Thus, the impact of any reduction in D2B depends on a facility’s D2B statistics prior to any process improvement. While there is no demonstrated mortality benefit below the 90-minute mark, there is signal of morbidity benefit in patient-oriented measures like rates of re-infarction.Reference Chen, Lin, Kung, Cheng and Li 37 Some evidence suggests, however, that reduced D2B is associated with increased false-positives, which could affect implementation of a system such as proposed here.Reference Fanari, Abraham and Kolm 38
Limitations
This study is limited by its small sample size and single-center design. Additionally, the data collection period was defined by the availability of research assistants rather than number of cases, making this a convenience sample. Cases were, however, collected consecutively during the study. Previous literature has used a variety of gold standards for true STEMI, including physician consensus of ECG findings, disposition to CCL, and CCL outcomes. Emergency physician activation of the CCL (patient disposition to CCL) rather than CCL outcomes was used as the outcome of interest for several reasons. For example, patients may require cardiac catheterization for diagnostic purposes, not solely interventional purposes, as in the case of cocaine-induced vasospasm. While this would represent a false-positive catheterization, the procedure was not inappropriate. Additionally, some cases of true STEMI may be deemed inappropriate for CCL based on alternate factors such as risk factors or patient choice. If, however, an ED physician did not activate the CCL but the patient was later found to have acute occlusion, this would represent a false-negative, though this did not occur in this study. While perfect accuracy would be aimed for, performing as well as a trained physician is a reasonable performance benchmark. Additionally, every patient referred to CCL by an ED physician received an intervention. If CCL outcomes were used as the outcome of interest, this would not have altered the results, but it is possible a difference would appear in a larger sample.
Additionally, this dataset only includes cases that began with prehospital computer diagnosis of STEMI. Without collecting any computer negative cases, it is impossible to meaningfully evaluate sensitivity and specificity of the computer algorithm alone. This constraint is largely because the region did not maintain any kind of STEMI database or information exchange, making it impossible to trace back cases that were false-negative in the prehospital setting after they were identified in the ED. Additionally, providers in this region frequently transmitted ECGs with age and sex, but without other patient identifiers. Even if every prehospital ECG had been included, there would have been limitations from the frequent lack of patient identifiers to find prehospital records.
For cases that were classified as intermediate probability solely due to baseline wander or artifact, the first ED ECG was examined under the assumption that the test was physiologically similar but of better technical quality. Given the time elapsed between EMS and ED ECG, it is possible that the infarct evolved making it easier for the computer to identify. Earlier in the patient interaction, these could have been falsely classified as low probability STEMIs.
No collected cases were ruled out due to heart rate. Several versions of the algorithm have the same test characteristics because of this. It is impossible to comment on the accuracy of this criterion. A zero percent false discovery rate was achieved without it, so it may or may not improve accuracy in a larger sample. Given that computer algorithms historically have had difficulty interpreting atrial fibrillation and flutter, it seems reasonable to include this criterion in future iterations of this algorithm to gather additional data. Reference Bosson, Sanko and Stickney19,Reference Swan, Nighswonger, Boswell and Stratton20
All the prehospital cardiac monitors in this study were ZOLL E-series devices (ZOLL Medical Corporation; Chelmsford, Massachusetts USA) that used the GE/Marquette 12-SL interpretation algorithm (GE Healthcare; Chicago, Illinois USA). These results may not be immediately valid when using other brands of monitor, despite most research implicating similar sources of false-positives between software algorithms.Reference Sanko, Eckstein and Bosson 14 – Reference Bhalla, Mencl, Gist, Wilber and Zalewski 18 The reported differences in baseline sensitivity and specificity would likely affect the performance of this algorithm. Any future updates to the software could also dramatically alter the accuracy of this algorithm.
While the intention is for this algorithm to eventually be applied by paramedics in the field, in this study, they were evaluated by a single paramedic/ED ECG technician. Inter-rater reliability and reproducibility on a larger scale is the next step in research. The ability of paramedics to reliably and accurately identify these criteria and classify cases using this algorithm is unknown and may affect algorithm use.
It is possible that there may be some harm from bypassing the ED to take a patient directly to the CCL. For example, a patient may require emergency or stabilizing treatment that the CCL is unprepared to provide. This is a larger question faced by any scheme intended to bypass the ED and take patients directly to the CCL to reduce D2B. Because of this, future research should record patient-oriented outcomes like mortality and rates of re-infarction to look for signal of harm or benefit.
Conclusions
This study shows STEMI ECGs can be accurately stratified to high, intermediate, and low probabilities. Activation of high probability cases by paramedics, without physician overread, would reduce the need for ECG transmission and physician interpretation. Future research should focus on additional validation of these criteria in a larger sample, with multiple paramedic reviewers, and with various brands of ECG monitors.
Conflicts of interest
none
