5 results
LO41: Competency-based learning of pediatric musculoskeletal radiographs
- K. Boutis, M. Lee, M. Pusic, M. Pecarcic, B. Carrier, A. Dixon, J. Stimec
-
- Journal:
- Canadian Journal of Emergency Medicine / Volume 20 / Issue S1 / May 2018
- Published online by Cambridge University Press:
- 11 May 2018, p. S21
- Print publication:
- May 2018
-
- Article
-
- You have access Access
- Export citation
-
Introduction: Pediatric musculoskeletal (MSK) image interpretation has been identified as a knowledge gap among emergency medicine trainees. The main objective of this study was to implement a validated on-line pediatric MSK radiograph interpretation system with a performance-based competency endpoint into pediatric emergency fellowship programs and examine the number of cases needed to achieve a competency threshold of 80% accuracy, sensitivity and specificity. We further determined proportion who successfully achieved competency in a given module and the change in accuracy from baseline to competency. Methods: This was a prospective cohort multi-centre study. There were seven MSK radiograph modules, each containing 200-400 cases (demo-https://imagesim.com/course-information/demo/). Thirty-seven pediatric emergency medicine fellows participated for 12 months. Participants did cases until they reached competency, defined as at least 80% accuracy, sensitivity and specificity. We calculated the overall and per module median number of cases required to achieve competency, proportion of participants who achieved competency, median time on case, and the mean change in accuracy from baseline to competency. Results: Overall, the median number of cases required to achieve competency was 76 (min 54, max 756). Between different body parts, there was a significant difference in the median number of cases needed to achieve competency, p <0.0001, with ankle and knee being among the most challenging modules. Proportions of those who started a module and completed it to competency varied significantly, and ranged from 32.4% in the ankle module to 97.1% in the forearm/hand, p<0.0001. The overall median time on each case was 34.1 (min 7.6, max 89.5) seconds. The overall change in accuracy from baseline to 80% competency was 13.5% (95% CI 12.1, 14.8), with the respective Cohens effect size of 1.98. The change in accuracy was different between modules, p=0.001, with post-hoc analyses demonstrating that the ankle/foot radiograph module had a greater increase in accuracy relative to elbow (p=0.009) and pelvis/femur (p=0.006). Conclusion: It was feasible for pediatric emergency medicine fellows to complete each learning pediatric MSK learning module to competency within approximately one hour, with the exception of the ankle module. Learners who completed the modules to competency demonstrated very significant increases in interpretation skill.
LO79: Climbing the learning curve teaching the pediatric emergency physician how to interpret point-of-care ultrasound images
- C. Kwan, K. Weerdenburg, M. Pecarcic, M. Pusic, M. Tessaro, H. Salehmohamed, K. Boutis
-
- Journal:
- Canadian Journal of Emergency Medicine / Volume 20 / Issue S1 / May 2018
- Published online by Cambridge University Press:
- 11 May 2018, p. S35
- Print publication:
- May 2018
-
- Article
-
- You have access Access
- Export citation
-
Introduction: Point-of-Care Ultrasound (POCUS) is rapidly being integrated into Pediatric Emergency Medicine (PEM), and image interpretation is an important component of this skill. Currently, PEM physicians often rely on case-by-case exposure and feedback by a POCUS expert physician to learn this skill; however, this may not be efficient, reliable or feasible. Thus, there is a pressing need to develop effective POCUS image interpretation learning and assessment tools. We developed an on-line learning platform that allowed for the deliberate practice of images in four POCUS applications [soft tissue, lung, cardiac and Focused Assessment Sonography for Trauma (FAST)], and determined the quantity of participant skill acquisition by deriving performance metrics and learning curves. Methods: This was a prospective cross-sectional study administered via an on-line learning and measurement platform. Images were acquired from a pediatric emergency department and each POCUS application contained 100 still/video images. Final diagnosis of each image was determined via the consensus of three PEM POCUS experts. PEM fellow and attending study participants were recruited from the USA and Canada and were required to complete the cases of at least one application. We aimed to enroll 200 participants who had to complete a minimum of 100 cases which, based on prior work, would provide sufficient raters for item analyses and comparisons between PEM attendings and fellows. To derive reference standard performance metrics and to validate image interpretations, a unique set of five PEM POCUS experts completed each application. Results: We enrolled 225 PEM physicians, 74 fellows and 151 attendings. For all applications, the Cohens d effect size was large at 0.87, and there was no difference between PEM attendings and fellows with respect to summary performance metrics (accuracy, p= 0.29; sensitivity, p=0.13; specificity, p=0.92). Final accuracy soft tissue, lung, cardiac, and FAST for all participants was 86.4%, 89.6%, 81.6%, 88.0%, respectively, and the corresponding accuracy of PEM POCUS experts for each application was 96.0%, 96.0%, 90.0%, and 93.0%. Learning curves show maximal learning gains (inflection point) up until 65 cases for soft tissue, 70 for FAST, 75 for lung, and 85 for cardiac. Conclusion: Deliberate practice of POCUS image interpretation was effective for ensuring broad domain coverage and predictable skill improvement. Specifically, there was a large learning effect after 100 case interpretations, and 65-85 case interpretations were needed to reach an accuracy threshold of approximately 85%.
LO44: Optimizing skill retention in radiograph interpretation: a multicentre randomized control trial
- K. Boutis, B. Carrier, J. Stimec, M. Pecarcic, A. Willan, M. Pusic
-
- Journal:
- Canadian Journal of Emergency Medicine / Volume 20 / Issue S1 / May 2018
- Published online by Cambridge University Press:
- 11 May 2018, p. S22
- Print publication:
- May 2018
-
- Article
-
- You have access Access
- Export citation
-
Introduction: Simulation-based education systems have increased physician skill in radiograph interpretation. However, the degree of skill retention and the factors that influence it are relatively unknown. The main objective of this research was to determine the rate and quantity of skill decay in post-graduate trainee physicians who completed a simulation-based learning intervention of radiograph interpretation. The impact of testing and refresher education on skill decay was also examined. Methods: This was a prospective, multicenter, analysis-blinded, four arm randomized control trial conducted from November 2014 to June 2016. Study interventions were administered using an on-line learning and measurement platform. Pediatric and emergency medicine residents in the United States and Canada were eligible for study participation. Participants were randomized to one of four groups. All participants completed an 80-case deliberately practiced learning set of pediatric elbow radiographs followed by an immediate 20-case post-test. Following this, Group 1 had no testing until 12 months; Groups 2, 3, and 4 had testing (20 cases without feedback) every 2 months until 12 months, but Group 3 also had refresher education (20 cases with feedback) at six months while Group 4 had refresher education at two, six, and ten months. The main outcome measure was accuracy at 12 months, adjusted for immediate post-test score, days to completion of 12 month test, and time on case. Based on prior data, we assumed the smallest important difference between groups in learning decay is 10%, a between-participant/within-group standard deviation of 17%, a type I error probability of 5%, a power of 80% and adjusted for three tests with a Bonferroni correction. For the primary analysis of Group 1 versus 2, 3, 4, this resulted in a minimal total sample size of 56, with 14 participants per group. Results: We enrolled 106 participants that completed all study interventions. The sample sizes in Groups 1, 2, 3, and 4 were 42, 22, 22, and 20 respectively. Overall, accuracy increased by 11.8% (95% CI 9.8, 13.8) with the 80-case learning set and then decreased by 5.5% (95% CI 2.5, 8.5) at 12 months. The difference in learning decay in Group 1 vs. Groups 2, 3, 4 was -8.1% (95% CI 2.5, 13.5), p=0.005. For Group 2 vs. Group 3 and 4, it was +0.8% (95% CI -7.2, 7.3), p=0.8, and between Group 3 vs. Group 4 it was +0.8% (95% CI -7.3, 10.1), p=0.8. Conclusion: Skill decay was significantly reduced by testing with 20 cases every two months. Refresher education had no additional effect to testing on reducing learning decay.
MP20: ImageSim - performance-based medical image interpretation learning system
- K. Boutis, M. Pecarcic, M. Pusic
-
- Journal:
- Canadian Journal of Emergency Medicine / Volume 20 / Issue S1 / May 2018
- Published online by Cambridge University Press:
- 11 May 2018, p. S47
- Print publication:
- May 2018
-
- Article
-
- You have access Access
- Export citation
-
Introduction: Medical images (e.g. radiographs) are the most commonly ordered tests in emergency medicine. As such, emergency medicine physicians are faced with the task of learning the skill of interpreting these images to an expert performance level by the time they provide opinions that guide patient management decisions. However, discordant interpretations of these images between emergency physicians and expert counterparts (e.g. radiologists) is a common cause of medical error. In pediatrics, this problem is even greater due to the changing physiology with age. Methods: ImageSim (https://imagesim.com/) is an evidence-based on-line learning platform derived and validated over an 11 year period (https://imagesim.com/research-and-efficacy/). This learning system incorporates the concepts of cognitive simulation, gamification, deliberate practice, and performance-based competency in the presentation and interpretation of medical images. Specifically, ImageSim presents images as they are experienced in clinical practice and incorporates a normal to abnormal ratio is representative of that seen in emergency medicine. Further, it forces the participant to commit to the case being normal or abnormal and if abnormal, the participant has to visually locate the specific area of pathology on the image. The participant submits a response and gets text and visual feedback with every case. After each case, the participant gets to play again until they reach a desired competency threshold (80% is bronze resident; 90% silver staff emergency medicine physician; 97% gold radiologist). Importantly, the learning experience also emphasizes deliberate practice such that the learning system provides hundreds of case examples and therefore each participants performance has the opportunity to improve along their individual learning curve. Results: Course selection was made based on known medical image interpretation knowledge gaps for practicing emergency physicians. Currently, ImageSim live courses include pediatric musculoskeletal radiographs (2,100 cases, 7 modules) and pediatric chest radiographs (434 cases). In 2018, we will also release a pediatric point-of-care ultrasound course (400 cases, 4 modules) and the pre-pubertal female genital examination (150 cases). For a demo, go to https://imagesim.com/demo/. Using ImageSim, the deliberate practice of about 120 cases (1 hour time commitment) increases accuracy on average by 15%. Currently integrated into 10 emergency medicine training programs and there are about 300 continuing medical education world-wide participants. Conclusion: While acquiring mastery for these images may take years to acquire via clinical practice alone, this learning system can potentially help achieve this in just a few hours.
LO81: Bridging the GAP: A deliberate practice method for learning Genital Abnormalities in Prepubescent girls
- K. Boutis, A. Davis, M. Pecarcic, M. Pusic, M. Shouldice, T. Smith, J. Brown
-
- Journal:
- Canadian Journal of Emergency Medicine / Volume 20 / Issue S1 / May 2018
- Published online by Cambridge University Press:
- 11 May 2018, pp. S35-S36
- Print publication:
- May 2018
-
- Article
-
- You have access Access
- Export citation
-
Introduction: Correctly identifying pathology in pre-pubertal females is a high-stakes physical examination skill. Currently, learning this skill relies heavily on case-by-case exposure, which is variable, limited and often results in suboptimal skill. Thus, there is a need to develop and evaluate learning platforms that simulate the presentation and diagnosis of this important clinical task. We developed an on-line learning and assessment platform that allowed the deliberate practice of 158 pre-pubertal female genital image interpretations . We examined the quantity of skill acquisition by deriving performance metrics and learning curves. Methods: This was a prospective cross-sectional study administered via an on-line learning and assessment platform. Colposcopic images were acquired from a child abuse clinic. Two child abuse experts interpreted images to determine case solutions and 40% of cases had medical or traumatic pathology. Further, to validate image interpretations, a unique set of five child abuse and pediatric gynaecology experts reviewed the cases. Study participants were recruited from the USA and Canada and were required to complete all 158 cases. For each image, learners designated cases as normal or abnormal and if abnormal indicated the abnormal area on the image. The primary outcome was the change in accuracy, sensitivity and specificity. Results: We enrolled 107 participants, 26 medical students, 31 pediatric residents, 24 pediatric emergency fellows, and 26 pediatric emergency attendings. For all participants, the change in accuracy was +9.6% for accuracy (<0.001), +1.4% for sensitivity (p=0.6) and +15.7% (p<0.001) for specificity. The final score for accuracy, sensitivity and specificity was 79.5%, 66.1%, and 87.8%, respectively. There was no difference between learner types with respect to summary performance metrics (accuracy, p=0.15; sensitivity, p=0.44; specificity, p=0.54). Learning curves show maximal learning gains (inflection point) up until 100 cases. Conclusion: Deliberate practice of pre-pubertal female image interpretation was effective for ensuring predictable skill improvement for normal cases but was less effective for abnormal cases. Future research could examine how to refine the education tool to better serve diagnostic skill of abnormal cases.