Introduction
Triage is a fundamental component of prehospital and emergency department (ED) patient assessment. In health systems where ED overcrowding leads to delays in physician assessment of patients, effective triage systems accurately prioritize for seriously ill and high risk patients and help predict the scope of clinical services required.Reference Beveridge 1 , Reference Innes, Stenstrom and Grafstein 2 The value of electronic triage data as ED acuity and case-mix indicators in health services utilization research is evolving.Reference Innes, Stenstrom and Grafstein 2 - Reference Dong, Bullard and Meurer 7 The Canadian Triage and Acuity Scale (CTAS) is a five-level triage score that has been validated and widely implemented in EDs across Canada and internationally as a reliable and effective triage tool.Reference Jiménez, Murray and Beveridge 3 , Reference Beveridge, Ducharme and Janes 8 - Reference Worster, Gilboy and Fernandes 16 CTAS scoring systems are available for both pediatric and adult populations. The CTAS consists of five triage levels combining severity of illness or risk with recommended fractile reponse times to medical assessment supported by a standardized patient presenting complaint list.Reference Murray, Bullard and Grafstein 17 , Reference Bullard, Unger and Spence 18 Level 1 is considered most urgent and Level 5 least urgent. Traditionally CTAS has been applied by trained triage nurses in hospital EDs.
Current evidence supports high levels of interrater reliability of CTAS between physicians, nurses, and paramedics, but has been limited to non-clinical, paper-based casesReference Manos, Petrie and Beveridge 5 , Reference Beveridge, Ducharme and Janes 8 , Reference Fernandes, McLeod and Krause 19 or triage nurse agreement in an ED setting.Reference Dong, Bullard and Meurer 10 - Reference Dong, Bullard and Meurer 12 , Reference Ng, Yen and Tsai 14 , Reference Grafstein, Innes and Westman 15 Two prospective comparisons of interrater reliability between paramedics and triage nurses published in abstract form described fair (κw=0.37) to good (κw=0.62) agreement.Reference Kramer and Feinstein 21 , Reference Dallaire, Poitras and Aubin 22
Hypothesis
Given similar training and supported by a clinical documentation tool, prehospital care providers can apply the CTAS with the same level of accuracy as triage nurses. The objective of this study is to prospectively evaluate CTAS interrater reliability between emergency medical services (EMS) providers and ED triage nurses with a system-wide implementation during routine clinical practice.
Methods
Study design/participants
We conducted a prospective, observational cohort study of all patients ≥17 years old transported to any ED in the Winnipeg Region by Winnipeg Fire and Paramedic Service (WFPS) EMS from July 1, 2014–December 31, 2014. The Winnipeg Regional Health Authority serves a metropolitan population of >700,000 citizens. An integrated network of seven EDs (one adult, one pediatric, five mixed) with an annual census of 200,000 visits, in combination with the WFPS, provide coordinated emergency medical care to the population. The WFPS operates an ALS off-line medical control paramedic service with the majority of ambulances staffed by a primary care paramedic (PCP) partnered with an advanced care paramedic (ACP). All patients assessed by both EMS and ED triage nurses were eligible for inclusion. There were no exclusion criteria. Ethics approval was obtained from the University of Manitoba Health Research Ethics Board for all aspects of this study. The Health Research Ethics Board waived the need for informed consent.
Data were extracted separately from the EMS electronic medical record (EMR) and the Emergency Department Information System (EDIS) databases into separate Microsoft Excel worksheets. The spreadsheets were then merged using the statistical software R (version 3.1.2) and RStudio (version 0.98); data linkage was accomplished using recorded EMS Run Number, Personal Health Information Number (PHIN), and date of birth, last name, and destination facility, which were recorded in both systems. All subsequent data analysis was performed in R with RStudio.
Prior to the onset of data collection, all EMS personnel in the WFPS were trained in application of the CTAS via a train-the-trainer model utilizing the same nationally standard CTAS education curriculum materials used to train WRHA ED triage nurses. Pre- and post- CTAS curriculum assessments were performed to evaluate skill acquisition as a matter of quality assurance. The WFPS follows an off-line model of medical control. During the study period in the prehospital setting, paramedics provided care according to their standard practice protocols with the addition of the CTAS assessment. EMS providers determined CTAS on-scene, CTAS prior to transport, and CTAS on arrival (CTAS arrival ) at hospital prior to their triage interaction for each patient. EMS CTAS scores were calculated with reference to a paper-based booklet and documented in an electronic medical records system on ruggedized laptops along with their standard documentation. Paramedics provided other clinical information to ED triage nurses according to current practice, but not their CTAS scores. Both nursing and paramedic CTAS scores were prospectively collected; EMS CTAS scores were not used clinically.
CTAS scores for ED triage nurses were calculated and recorded electronically as part of routine clinical practice. Initial ED CTAS scores (CTAS initial ) are generated in EDIS using computerized clinical decision support by the triage nurse. Nurses can then either accept the computer-assisted CTAS score, or override the CTAS score (CTAS final ) in order to better match their impression of patient acuity. EMS providers do not have the ability to override the CTAS score. Both the CTAS initial and CTAS final scores were analyzed.
All ED triage nurses and EMS providers were blinded to each others’ scores, which were maintained in separate electronic systems. Medical Supervisors (WFPS paramedics with an enhanced scope of practice) reinforced this blinding throughout the study period.
Outcomes/variables
Our variables of interest were the final CTAS score calculated by EMS immediately prior to their triage interaction in the ED (CTAS arrival ) compared to both the baseline CTAS score calculated by triage nurses (CTAS initial ) and the final score including nursing overrides (CTAS final ).
Our primary outcome was interrater reliability for triage nurse baseline CTAS initial and EMS CTAS arrival scores. Secondary outcomes included interrater reliability on ED triage nurse CTAS final CTAS and EMS CTAS arrival , as well as proportion of patient encounters with perfect or near-perfect agreement.
Secondary analyses
We compared the distribution of CTAS scores between nursing and EMS to identify trends in scoring. Subgroup analysis of interrater reliability between nursing and EMS at tertiary versus community ED sites was performed. Differences in admission rates between higher acuity (CTAS 1-3) and lower acuity (CTAS 4-5) patients, as assigned by nursing and EMS, were assessed.
Statistical analysis
Interrater reliability for triage nurse CTAS scores and EMS CTAS arrival scores was compared using a quadratic weighted kappa (κw) statistic for primary and secondary outcomes as well as subgroup analyses. Using the Wilcoxon Signed Rank test, CTAS distributions between nursing and EMS were examined through comparison of the proportion of scores in each category to identify statistically significant differences. The proportion of patients admitted in higher acuity (CTAS 1-3) and lower acuity (CTAS 4-5) groups, as ranked by triage nursing and EMS, were compared using the Chi Squared tests, and average length of stay for these two groups were compared using standard t-tests.
Sample size for multicategory weighted kappa was estimated as at least 50 patients per category.Reference Soeken and Prescott 20
Results
From July–December 2014, 14,378 consecutive adult patient encounters in the Winnipeg Region were evaluated. Patients were 17-105 years old and 55% female. Case-mix was representative of the ED patient population utilizing EMS transport. There were 17,328 eligible records in the EMS dataset, and 15,876 records in the EDIS dataset. The primary reason for the lower number of eligible records from EDIS was a failure on the part of triage nurses to universally record an EMS run number on the triage note. Overall, a total of 14,378 records were positively matched between the two sources.
Unweighted (k) and both linear- (kWL) and quadratic-weighted kappa (kWQ) were calculated. Kappa agreement was defined a priori as excellent (k>0.8), good (0.6<k <0.8), moderate (0.4<k<0.6), fair (0.2<k<0.4) or poor (k<0.2).Reference Kramer and Feinstein 21 Our primary outcome, interrater reliability between EMS CTAS arrival and ED CTAS initial scores (the CTAS score prior to overrides), indicated moderate agreement [κw=0.437 (p<0.001, 95% CI 0.421-0.452)]. EMS CTAS arrival and ED CTAS initial scores had an exact or within one point match 84.3% of the time. The secondary interrater reliability outcome between EMS CTAS arrival and ED CTAS final score (including nurse overrides) also showed moderate agreement [κw=0.452 (p<0.001, 95% CI 0.437-0.466)] and an exact or within one point match 86.5% of the time.
The distribution of CTAS scores is demonstrated in Figure 1. Table 1 summarizes the proportions of matched CTAS scores between EMS providers and ED triage nurses.
Frequency distributions of CTAS scores
Proportion of matched CTAS scores
It was observed that, when compared to the triage nurses, EMS tended to score patients equal or lower for CTAS 1, 2, and 3 patients, but tended to score equal or higher for CTAS 4 and 5 patients. This observed difference was examined using the Wilcoxon Signed Rank test on these two dependent subsets, which identified a significant difference not equal to zero (p<0.001).
Assessed using the Chi-Squared test, the difference in the proportion of admissions for EMS-coded high-acuity (CTAS 1-3) patients (0.284) versus triage nurse–coded high-acuity patients (0.274) was not statistically significant (p=0.102, 95% CI -0.00214-0.09236). The difference in proportion of admissions for low-acuity patients (CTAS 4-5), however, was statistically significant (0.173 vs. 0.148, p<0.05, 95% CI 0.0102-0.0396).
A subgroup analysis evaluated our primary interrater reliability outcome at tertiary university-affiliated sites versus community sites. There was a non-significant trend toward better agreement between paramedic CTAS arrival and triage nurse CTAS final evaluations in tertiary sites [kw=0.469 (95% CI 0.447-0.491)] versus [kw=0.439 (95% CI 0.421-0.457)]. There was no statistically significant difference in proportion of patients admitted to hospital categorized by paramedic versus triage nurse CTAS score when CTAS was dichotomized into high- (CTAS 1-3) and low- (CTAS 4-5) acuity groups.
Discussion
There is a growing body of literature on the inter- and intra-rater reliability of the CTAS between health care providers. Beveridge et al. published the first study showing good concordance rates between physicians and nurses for the application of the CTAS to written case reports (κw=0.8).Reference Beveridge 1 , Reference Beveridge, Ducharme and Janes 8 Fernandes et al. reproduced a high level of agreement (κw=0.79) between community and tertiary ED triage nurse scores on paper-based cases.Reference Fernandes, McLeod and Krause 19 Grafstein et al. showed that paired ED triage nurses observing a single clinical interaction could apply CTAS with similar levels of agreement in clinical practice (κw=0.75).Reference Grafstein, Innes and Westman 15 Dong et al. published a series of papers examining the interrater reliability of CTAS using eTRIAGE (an electronic CTAS decision support tool) and blinded triage nurse assessments, and demonstrated improved accuracy using electronic decision support compared to memory- or paper-based methods,Reference Dong, Bullard and Meurer 11 with good agreement between nursing scores (κw=0.66)Reference Dong, Bullard and Meurer 10 and improvement in agreement with ongoing CTAS training.Reference Dong, Bullard and Meurer 12
Manos et al. also demonstrated a high level of agreement between physicians and nurses when applying CTAS to written cases and expanded their scope to include BLS and ALS paramedics, who also showed good concordance (κw=0.77).Reference Manos, Petrie and Beveridge 5 The generalizability of these findings to clinical practice is limited by the use of paper-based cases in artificial circumstances, or calculation of CTAS scores based on observation but not participation in clinical interactions. It has been suggested that visual or non-verbal cues from individual patients in vivo, along with environmental stimuli such as departmental crowding, have significant impact on triage processes.Reference Dallaire, Poitras and Aubin 22 As such, the interrater reliability of the CTAS would be expected to vary more in clinical practice than in paper-based scenarios. Prospective evaluation of prehospital CTAS interrater reliability in practice has been limited to two studies presented as abstracts: Murray and Bondy described good interrater reliability of CTAS assessments between EMS providers and triage nurses (κw=0.61) in clinical practice, although significant variability was noted between centers (quadratic weighted kappas ranged from κw=0.7 to κw=0.47);Reference Murray and Bondy 23 Smith et al. found fair agreement (κw=0.37) between CTAS scores assigned by EMS providers and triage nurses in a prospective study of 1,131 EMS transports, although these findings may be limited by the fact that the EMS providers relied on memory, while the triage nurses used an EDIS with programmed CTAS clinical decision support.Reference Smith, Lobay and Bullard 24
Previous studies of paramedic acuity assessments in the field using non-CTAS scales or clinical judgement have also yielded mixed results. In a prospective survey with linked medical record review, Hauswald demonstrated moderate-to-fair agreement between paramedics and physicians when evaluating whether patients required ambulance transport and medical attention in the ED (κw=0.47 and κw=0.32, respectively).Reference Hauswald 25 Silvestri et al. had similar results, finding that paramedic assessment was 81% sensitive but only 34% specific for predicting requirement of ED care.Reference Silvestri, Rothrock and Kennedy 26
Our study has some limitations that warrant consideration. It has been previously noted that ED conditions, such as overcrowding and resource-intensive cases in the resuscitiation area, may influence triage nurse CTAS assignment based on their awareness of flow realities. If done indiscriminantly, this is referred to as triage drift (towards mean scores), to be contrasted by experienced nurses who sometimes use judgment to appropriately override calculated CTAS score and prioritization based on patient acuity.Reference Dong, Bullard and Meurer 10 , Reference Bjørn and Rødje 27 EMS providers need only assign a CTAS score to the individual under their care, irrespective of the state of the ED they are presenting to. Future research should include consideration of the impact of system factors on CTAS interrater reliability between EMS providers and triage nurses. While EMS and ED CTAS scores were entered and housed in independent electronic systems and maintenance of blinding was routinely reinforced, it was not possible to monitor every triage interaction and confirm blinding was maintained. There were missing data due to incomplete documentation by ED nurses (1,452/17,328 or 8.4% of potential records could not be reliably matched), which could potentially represent an underlying source of selection bias not identified.
We have demonstrated that EMS providers can assign CTAS scores with moderate interrater reliability when compared with ED triage nurses. Our results support that EMS providers can be trained and apply CTAS in their routine practice in an off-line medical control system with a moderate level of agreement compared to ED triage nurses. It is important to recognize that EMS providers are applying the CTAS to a single patient and repeating that assessment on several occasions, looking for any acuity change. They are not comparing their patient to any other, or making decisions about the level of priority. EMS providers may complete a more comprehensive assessment than the triage nurse, who must manage, triage, and prioritize all patients presenting to the triage desk and being held in the waiting room, while balancing the availability of staff and treatment spaces in the ED. It may be unreasonable to expect perfect agreement between these providers; however, one intuitive benefit in standardized definitions and descriptors of patient acuity is the potential for enhanced patient care and meaningful comparisons of data.
Conclusions
EMS providers can assign CTAS scores during routine clinical practice with moderate interrater reliability as compared to ED triage nurses. The region-wide nature of our study supports the generalizability of these results to other systems. Further research is needed to define the role of CTAS in EMS systems where alternate levels of care and/or delayed- or non-transport processes are incorporated. Future studies should also evaluate the impact of CTAS acuity change from the scene to the ED to determine whether this is a predictor of a higher risk patient than those patients who remain stable from on-scene to ED, and to characterize the impact of prehospital interventions on patient acuity.
Acknowledgements: The authors would like to thank Lori Mitchell, PhD, Researcher with the Winnipeg Regional Health Authority, for her statistical input.
Competing Interests: None declared.
