INTRODUCTION
Ankle and foot injuries are a common presentation to the pediatric emergency department (ED). When a fracture is present, it may require casting, splinting, or operative intervention. However, most injuries are sprains that require only conservative treatment.Reference Conrad and Rang 1 , Reference Boutis, Plint and Stimec 2
Unstructured clinical judgment alone is insufficient to rule out fractures.Reference Boutis, Komar and Jaramillo 3 , Reference Al Omar and Baldwin 4 Plain radiography can reliably rule out fractures but leads to radiation exposure, increased time in the ED, and higher costs to the health care system.Reference Cockshott, Jenkin and Pui 5 , Reference Gleadhill, Thomson and Simms 6 The Ottawa Ankle Rules (OAR) are a clinical decision tool developed to augment physician assessment in determining the need for radiography.Reference Stiell, Greenberg and McKnight 7
Physician use of the OAR has been shown to reduce radiography, time spent in the ED, and health care expenditures, without missing clinically relevant fractures.Reference Stiell, Greenberg and McKnight 7 - Reference Fan and Woolfrey 9 The OAR have been independently validated in both adult and pediatric populations and in diverse clinical settings worldwide.Reference Dohin and Lubanziado 10 - Reference Springer, Arciero, Tenuta and Taylor 15 In 2009, a systematic review on the use of the OAR in children aged 5-17 years demonstrated a pooled sensitivity of 98.5% and a decrease in radiography use of 24.8%.Reference Dowling, Spooner and Liang 16 Since this meta-analysis was conducted, three additional studies have been performed, with a sensitivity of 100%.Reference Gravel, Hedrei, Grimard and Gouin 17 - Reference Meena and Gangary 19
Although nearly 90% of Canadian pediatric emergency physicians report using the OAR, non-physicians are ordering almost 40% of radiographs in acute ankle injuries.Reference Dowling and Wishart 20 While some evidence has supported the use of the OAR among non-physicians in an adult population,Reference Allerston and Justham 21 - Reference Salt and Clancy 24 there is a paucity of literature supporting the use of the OAR by non-physician providers (NPPs) in children.
Karpas et al.Reference Scheaffer, Mendenhall and Ott 25 is the only study to examine the accuracy of OAR use specifically by non-physicians in children. This study used a cross-sectional design and enrolled 190 participants who had the OAR applied by a nurse. The sensitivity of the OAR for clinically important fractures was 100%.
Our objective was to build on this small literature base, to determine the sensitivity and specificity of the OAR as applied by NPPs, to detect all fractures along with a subset of clinically significant fractures in children with ankle and foot injuries.
METHODS
Study design, setting, and population
The study was conducted at the Alberta Children’s Hospital, which has 75,000 ED visits per year. Participants were prospectively enrolled between July 2012 and June 2014. Research ethics approval was obtained from the Conjoint Health Research Ethics Board at the University of Calgary.
All patients, aged 5-17 years, presenting to the Alberta Children’s Hospital with an acute ankle injury were included. Patients who had been assessed by the on-duty physician prior to an NPP assessment were excluded.
Any patients presenting for reassessment of the same injury or more than 24 hours after the injury; having open, penetrating, neurovascular, or multiple injuries; metabolic bone disease (e.g., osteogenesis imperfecta); with an inability to communicate or ambulate before the injury; or who were referred from an outside health centre with x-rays were excluded.
Patients meeting the inclusion and exclusion criteria were identified by research assistants at the time of triage and approached after the first assessment by nursing staff in the treatment area. Participation was voluntary, and informed consent and assent were obtained.
Health provider recruitment
NPPs (nurses and orthopedic technologists [OT]) were recruited to participate in the study using email, bulletin boards, and direct contact, regardless of prior knowledge or use of the OAR. Nurses included licensed practical nurses (LPNs) and registered nurses (RNs) working in the ED. A single registered OT working in the ED was recruited. Participation by NPPs was voluntary. Physician participation included a convenience sample of emergency room (ER) physicians who assessed study participants after an NPP had performed an OAR assessment. Consent to include physician assessment was implied if a physician filled out the assessment form.
Enrolment, intervention, and outcome measures
During phase one, the physicians and NPPs treated consenting patients in the usual manner. At our institution, there is a nurse-initiated protocol allowing nurses to order below-the-knee x-rays prior to physician assessment if a patient has point tenderness, provided the patients do not necessitate intravenous (IV) analgesia and no obvious deformity exists. Baseline data were collected on recruitment forms by research assistants on a convenience sample of patients (a priori n=100) who consented to inclusion (see Appendix 1).
After completion of phase one, a 20-minute learning module was presented to all recruited NPPs. This module outlined pertinent background, the relevant anatomy, the content of the OAR, and details on NPP involvement in the study. Participants submitted proof of completion in the form of a self-assessment quiz. NPPs were directly observed using the OAR by study investigators prior to applying the OAR to study patients to ensure accurate understanding and application of the decision tool. Completion of the learning module, quiz, and direct observation was voluntary but required to participate in the study.
After the teaching intervention, patient enrolment into the second phase of the study began. NPPs applied the OAR to all included patients, and research assistants filled out the appropriate recruitment form (see Appendix 2). If a radiograph was indicated based on the OAR, then patients were sent for x-rays. If not, then the patient would wait to be assessed by a physician. Physicians were free to care for the patient as they normally would.
Any patient who did not have an x-ray while in the ED obtained follow-up by telephone 7-10 days after discharge. Study investigators asked standardized questions regarding pain, activity, and further investigations/physician visits. Any patients who still had pain or had not returned to normal activities were encouraged to return to the ED for reassessment and possible radiography at the discretion of the treating physician.
Patients were selected ad hoc to undergo a second OAR assessment, by an NPP or physician, to assess inter-observer reliability. To encourage NPPs to comply with this protocol, a random draw for a gift basket was awarded at the end of enrolment. To ensure blinding between NPPs and physicians, NPPs were asked to fill out recruitment forms prior to physician assessment. Recruitment occurred seven days a week between 8:00 a.m. and midnight whenever a trained NPP was available.
Statistical analysis
The data analysis for phase one was descriptive, detailing baseline x-ray usage, as well as the ordering provider (physician, nurse, or OT).
Data analysis for phase two included determining the sensitivity and specificity of the OAR as applied by NPPs. A clinically significant fracture included any fracture other than Salter-Harris I fractures and avulsion fractures <3 mm in size. This definition is consistent with previous definitions in other pediatric OAR studies.Reference Plint, Bulloch and Osmond 11 The sample size for phase two (n=186) was calculated in a manner similar to that of previous studies.Reference Scheaffer, Mendenhall and Ott 25
NPP radiography rates were calculated as the number of positive OAR assessments divided by the total number of OAR assessments. The inter-observer agreement results were compared using the kappa (κ) statistic. Further, 95% confidence intervals (CI) were calculated for sensitivity and specificity.
RESULTS
Population
Phase one
One hundred six patients were enrolled in phase one of the study. Further, 81% (84/104) of the patients obtained a foot or ankle x-ray (data unavailable for two patients). Moreover, 61% (51/84) of the x-rays were ordered by physicians, 25% (21/84) were ordered by nurses, and 13% (11/84) were ordered by an OT (data unavailable for one patient). Most of the physicians who ordered x-rays were staff physicians (82%, 42/51). Residents (16%, 8/51) and fellows (2%, 1/51) ordered the remainder.
Phase two
Seven hundred eighty-three patients presented with ankle and/or foot injuries during phase two of the study. Further, 294 patients met one or more of the exclusion criteria. Moreover, 305 patients were eligible but were not enrolled, including 107 patients who presented when study personnel or a trained NPP was not available, 38 patients who did not consent to enrolment, 147 patients who were missed by study personnel, and 13 patients who were not enrolled for other reasons.
One hundred eighty-four patients were included in the final analysis. Most of the patients were male (94/182, 52%), and patients ranged in age from 5 to 17 years (mean 12.1 years). Further, 118 patients had only an ankle assessment, and 40 patients had a foot assessment performed only. Twenty-six patients had both ankle and foot assessments (Table 1).
Characteristics of the 184 study patients
SH=Salter-Harris.
NPPs
A total of 56 RNs and LPNs and 1 full-time OT volunteered to be part of the teaching phase. All 57 participants attended the OAR learning module, completed the self-assessment quiz, and were observed applying the OAR correctly prior to phase two. The number of patients enrolled per NPP is described in Table 2.
NPP enrolment
NPP=non-physician provider; OAR=Ottawa Ankle Rules.
Radiography
If only patients with positive rules had x-rays, NPPs would have obtained x-rays for 73% of the patients (135/184).
After being examined by an NPP, 49 patients had not had an x-ray (based on negative OAR). Physicians had obtained x-rays for 30 of these patients (30/49, 61%).
Overall, after assessment by both an NPP and a physician, 90% (165/184) of the patients had an x-ray.
In this study, 70% (128/184) of the patients had an ankle x-ray. Further, 14% of these patients (18/128) had an ankle fracture, of which eight were clinically significant (Table 1). In addition, 39% (72/184) of the patients had a foot x-ray; 21% of these patients (15/72) had a foot fracture, of which nine were clinically significant (Table 1). None of the non-significant fractures required operative intervention, but one patient did develop pressure sores because of an air cast.
Sensitivity and specificity
When NPPs applied the ankle portion of the OAR, the sensitivity was 88% (95% CI 47%-99%), and the specificity was 31% (95% CI 23%-40%) for clinically significant fractures. If all fractures were included, the sensitivity was 76% (95% CI 50%-92%), and the specificity was 31% (95% CI 23%-40%) (see Table 3).
A two-by-two table of NPP assessment for the OAR and Ottawa Foot Rule
* Inter-observer performed by a physician (MD) was positive for the Ottawa Ankle Rules (OAR).
When NPPs applied the foot portion of the OAR, the sensitivity was 100% (95% CI 56%-100%), and the specificity was 17% (95% CI 9%-29%) for clinically significant fractures. If all fractures were included, the sensitivity was 100% (95% CI 70%-100%), and the specificity was 19% (95% CI 10%-32%) (see Table 3).
In total, one clinically significant fracture was missed by an NPP. The missed fracture was a minimally displaced lateral and posterior malleolar fracture.
Inter-observer agreement
In this study, 34% (72/211) of the patient evaluations had an inter-observer assessment. Fifty-one were performed on the ankle, and 21 were performed on the foot. Further, 72% (52/72) were performed by physicians, and 28% (20/72) were completed by a different NPP. Inter-observer agreement (κ) was 0.24 for the ankle rule and 0.49 for the foot rule. Inter-observer agreements for the individual portions of the OAR are listed in Table 4.
Inter-observer Kappa values for components of the Ottawa Foot Rule
IO=inter-observer.
Follow-up
A total of 20 patients did not have an x-ray taken in the ED on the index visit. Nineteen of these patients received telephone follow-up 7-10 days after discharge. One patient was lost to follow-up. Among those reached by phone, five had a positive follow-up question (four had not returned to regular daily activities, and walking had not improved in one patient). Further, one of these patients had a follow-up x-ray, which was negative. The remaining four patients had no follow-up visits or x-rays in their medical record.
DISCUSSION
The OAR reduce radiography without missing clinically important fractures in children.Reference Dowling, Spooner and Liang 16 Many x-rays are ordered by nurses and other NPPs prior to assessment by a physician.Reference Dowling and Wishart 20 These nurse-initiated protocols may decrease the ED length of stay and help alleviate overcrowding.Reference Rowe, Willa-Roel and Guo 27 However, there is a lack of published data evaluating the accuracy of the OAR if used by NPPs in children.
In our study, NPPs detected the need for radiography in all but one clinically important ankle fracture. The single missed fracture was a minimally displaced bi-malleolar fracture. A physician inter-observer assessment was performed on this patient and was positive. In our estimation, this scenario reflects everyday clinical practice in Canada. All patients have a complete assessment by a physician, including evaluation of the need for imaging, after the initial assessment by NPP. As was the case in our study, physicians should likely capture infrequently missed fractures prior to patient discharge from the ED.
The specificity of the OAR in our study is similar to that of previous studies in children. In a study by Karpas et al.,Reference Karpas, Hennes and Walsh-Kelly 26 the specificity was 25% if nurses applied the OAR in children. A meta-analysis by Dowling et al.Reference Dowling, Spooner and Liang 16 found that the specificity of the OAR was between 7% and 50% if applied by physicians in children. This comparable specificity suggests radiography can be minimized, even in the hands of NPPs.
The primary goal of the OAR is to reduce radiography without missing clinically important fractures. If used as a screening tool prior to physician assessment, it is essential that an NPP not unnecessarily increase the use of radiography. Karpas et al.Reference Karpas, Hennes and Walsh-Kelly 26 found a nurse radiography rate of 79% and that institution of the OAR in their nurse-initiated protocol would have decreased negative films by 21%. In our study, the radiography rate by NPPs was 73%. NPPs captured all but one fracture while using 8% fewer x-rays than that of phase one. Overall, after the final assessment by a physician, the x-ray rate was 90% (up from 81% in phase one). Physicians x-rayed more than one-half of patients who had a negative OAR assessment by an NPP. Given the robust literature supporting a reduction in imaging with the use of the OAR by physicians, these data raise the question as to whether the physicians in our study were using the OAR. Failure to use the rules may be the reason for the increased x-ray usage demonstrated in our study.
Despite the good sensitivity and specificity of the OAR in our study, the inter-observer agreement was poor. This finding is in stark contrast with the study data of Karpas et al.Reference Karpas, Hennes and Walsh-Kelly 26 that found the OAR had an inter-observer agreement of 100% among nurses in children. This may be in part because most of our inter-observer assessments were performed by physicians (72%).
In our study, NPP training was thorough and included an educational session, a self-assessment quiz, and direct observation of OAR application. Enrolment forms included a visual aid of the OAR to increase knowledge retention, and investigators were available for follow-up questions and clarification if required.
Karpas et al.Reference Karpas, Hennes and Walsh-Kelly 26 used a similar training scheme but also provided a second follow-up educational session in which the investigators met one-on-one with individual nurses to review the OAR, discuss the data collection sheet, and answer questions. This additional session helped solidify understanding and familiarize nurses with the study and may have led to a better inter-observer agreement than in our study. Two parts of the OAR had very low inter-observer agreement in our study. The navicular bone and the posterior edge of the lateral malleolus had inter-observer agreement of less than 50%. These error-prone portions of the OAR would be an ideal focus for additional teaching sessions.
There are several other potential factors that might explain the disparity between our results and previous studies examining physician application of the OAR. First, we had a large number of NPPs performing assessments. Because of this, a large proportion of NPPs (36%) enrolled only a single patient in the study. NPPs also had a lag-time of up to 10 months between training and enrolment that may have affected recall. Finally, the NPPs’ diverse levels of ED work experience and previous practice with the OAR could have led to differences in familiarity with the OAR that may have affected inter-observer agreement.
Previous work involving the OAR with physicians has shown poor recall of the rule components even among those physicians who reported familiarity with the rule.Reference Brehaut, Stiell, Visentin and Graham 28 Memory difficulty was more common among older, part-time physicians who reported not applying the rule regularly.Reference Brehaut, Stiell, Visentin and Graham 28 Despite the simplicity of the rule, it is apparent that errors in its application can occur, particularly in these high-risk groups. Attempts at improved recall have included mnemonics, posters, pocket cards, digital calculator, clinical support tools, and others.Reference Gravel, Roy and Carrière 29 - Reference Silveira, Ip and Sumption 31 Given the poor inter-observer agreement in our study, it may be even more important for NPPs to support OAR use with memory tools.
Regardless, there is the potential for unacceptably high variability among NPP assessments with the OAR. While this did not negatively impact the accuracy of the OAR in our study, it could affect patient assessments in clinical encounters. Future research might explore inter-observer variability among pediatric NPPs using the OAR. A follow-up educational session and refresher training with a focus on clarifying problematic portions of the OAR in combination with more routine clinical practice with the OAR may ensure more consistent application by NPPs.
LIMITATIONS
All but one of the NPPs in our study were nurses; a single OT participated. As such, we were unable to perform subgroup analyses on the type of NPP. Our study does not answer the question of whether non-nursing groups of NPPs are able to apply the OAR accurately.
Not all patients in our study had an x-ray. Despite telephone follow-up with all but one of these patients, it is possible a fracture was missed. However, only one patient re-presented to the hospital and had a negative x-ray, so it is unlikely any significant injuries were missed.
Finally, our study size was small, and the CIs for our estimates are wide. Reported sensitivities and specificities may be over-estimates of actual clinical practice. However, our study joins a growing body of research, including a recent systematic review in adults that demonstrated that nurses can reliably apply the OAR in the ED.Reference Ho, Chau and Cheung 32
CONCLUSIONS
The sensitivity of the OAR was very good if applied by NPPs. The only clinically significant fracture that was missed by NPPs was identified during a physician assessment. More training and practice using the OAR would likely improve the inter-observer reliability of NPPs. Our data suggest that the OAR may be a useful tool for NPPs to use to determine the need for radiography prior to physician assessment.
Acknowledgements
Ms. Janie Williamson functioned as a research coordinator for this study. The Pediatric Emergency Research Team (PERT) at the Alberta Children’s Hospital consented and enrolled all patients.
Competing interests: JM secured a trainee grant from the Calgary Emergency Medicine Research Advisory Committee (EMRAC) to support this work. The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/cem.2017.399
