Introduction
Critically ill patients presenting to emergency departments are assessed urgently through a variety of approaches. Increasingly, emergency physicians are using point-of-care ultrasound (PoCUS) to enhance their assessment of such patients. 1 - Reference Henneberry, Hanson and Healey 3
Performing resuscitative PoCUS (ultrasonography directly related to an acute resuscitation in critical care) 1 is made more complex by the patient’s precarious clinical state and resulting surroundings, creating logistical challenges in terms of timing and access to the patient. Effective resuscitative PoCUS relies on situational awareness, understanding of the rationale for performing a scan at a given time, and the ability to accurately interpret and act on the findings.Reference Socransky and Wiss 4 Generally, individual aspects of critical care should be employed in a “simultaneous, choreographed approach.”Reference Berg, Hemphill and Abella 5 Such choreography strives to minimize interruptions in critical actions while ensuring that the patient undergoes all appropriate assessments and receives required therapies.
As such, emergency medicine (EM) trainees must not only master the skill of generating ultrasound images, but also become efficient at incorporating such scanning into their resuscitation. Competence in resuscitative PoCUS can thus be divided into three broad components: awareness of indications, with associated rationale; the technical skill of image generation, with simultaneous interpretation of findings; and, lastly, the appropriate integration of findings into resuscitation. The role of simulation for ultrasound skill development and competence has only recently gained attention, with the majority of current literature focusing on ultrasound-guided procedures and image generation/interpretation.Reference Mendiratta-Lala, Williams and de Quadros 6 , Reference Sidhu, Olubaniyi and Bhatnagar 7 The incorporation of PoCUS simulation into high-fidelity simulation (HFS), through what has been described as hybrid simulation,Reference Girzadas, Antonis and Zerth 8 may contribute to the development of overall resuscitative PoCUS competence. Such hybrid simulations, or simulated patient encounters, offer an intermediary step whereby the skills learned outside the critical-care context (i.e., at courses, through scanning volunteers and patients who are otherwise well) can be re-integrated into a clinical environment in a way that is developmentally appropriate.Reference Kneebone, Scott and Darzi 9
Many training programs have already incorporated resuscitative PoCUS into HFS to varying degrees. Some centers may use printed images or offer verbal descriptions of findings (representing the simplest method for integrating ultrasound into critical care simulation). In contrast, Kobayashi et al. (2010) published an ultrasound simulation teaching package that incorporated a laptop (placed on an audiovisual cart) as a basic ultrasound simulation device.Reference Kobayashi, Shapiro and Nagdev 10 Expanding upon such integration, Kulyk et al. incorporated a simulated ultrasound probe to the laptop concept (known as the edus2), thus providing trainees with an opportunity to perform scans in real time during HFS.Reference Kulyk and Olszynski 11
The purpose of this study was to evaluate two comparable ultrasound simulation interventions as used during HFS. Comparing two somewhat similar but distinct interventions allowed the study team to assess and isolate for the potential value of basic probe handling and other logistical aspects associated with the use of either intervention.
Methods
Study design
This was a prospective, randomized, crossover trial involving post-graduate specialist trainees in EM (trainees) and EM physicians involved in PoCUS teaching (instructors). Trainees and instructors participated in a day-long simulation-based critical care ultrasound course. This research project was approved by the Behavioural Research Ethics Board of both the University of Saskatchewan Research Ethics Board (BEH #12-292) and by Barts Health Trust, NHS (ReDA #9017).
Study setting and population
This study was conducted at Medical Education Training Suite, Whipps Cross Hospital, Whipps Cross Road, Leytonstone, London, UK. The trainee population was recruited from the London Specialist School of Emergency Medicine. A total of 25 EM trainees participated in the study over four separate course dates. EM instructors were selected by the local study supervisor (T.M.) based on expertise in PoCUS and overall teaching skill-set. Of the eight faculty physicians in the study, five were full-time consultants in EM. The remaining three were senior trainees (chosen by T.H.) deemed sufficiently experienced in EM and PoCUS to serve as instructors for the course.
Study interventions
The edus2 is a PoCUS simulator that is comprised of a modified ultrasound probe and laptop.Reference Kulyk and Olszynski 11 When used within a high-fidelity simulation (HFS) setting, the edus2 plays pre-recorded video clips of areas of interest through the coupling of those videos to specific Radio Frequency Identification Device (RFID) cards placed under the skin of any commercially available HFS mannequin. Passing the edus2 probe over a RFID card located beneath a mannequin’s skin initiates a video clip of the corresponding anatomic area (on the HFS mannequin) to be viewed by the trainee on the simulator screen. Multiple scans are possible during any given scenario (by placing several cards under the skin of the mannequin); however, no image manipulation can take place (clips play to completion once the probe passes over the RFID tag under the skin).
The second ultrasound simulation intervention was a laptop placed on an audiovisual cart. During a given scenario, trainees can bring the laptop to the mannequin’s bedside and choose the desired clip as labeled within the case folder.Reference Kobayashi, Shapiro and Nagdev 10 While this intervention lacks probe manipulation, it is an established method of integrating some cognitive aspects of resuscitative PoCUS into critical care simulation (what to scan, image interpretation, and integration). Both interventions require minimal financial investment (beyond an existing HFS infrastructure), as both can run on used laptops. In addition, both interventions could be used with real standardized patients.
Study protocol
Trainees were randomized based on order of arrival to the simulation centre into two groups (A or B) with two to four trainees per group. Each trainee group was assigned to one of two study arms that involved both the use of the edus2 as well as the simple laptop for a total of four HFS scenarios. Group A trainees completed their first two cases with the edus2 followed by two cases with the use of the laptop. Group B was assigned the same cases but with the PoCUS interventions presented in reverse order. All trainees completed an entrance MCQ (multiple choice question) exam (based on the American College of Emergency Physician’s EMSONO online exam, 12 permission obtained) as a means of establishing level of knowledge as well as the success of group randomization (Figure 1).
Study Design.
Each HFS scenario was followed by a standardized debriefing session led by an instructor and a course facilitator. In order to capture differences in the debriefing experiences that followed each intervention, the EM physician instructors were surveyed. These surveys focused on the performance of the PoCUS interventions as well as the debriefing experiences that followed each case.
Pre- and post-intervention responses (10-point Likert scale) were compared and analyzed in order to determine the strengths of either intervention, as well as for direct comparison.
Scenario development
The four cases were chosen from the ultrasound case simulation package prepared by Kobayashi et al. (2010). They represent the shock or peri-arrest states associated with the following conditions: ruptured aortic aneurysm; blunt abdominal trauma with hemoperitoneum; cardiac tamponade; and cardiac arrest secondary to massive pulmonary embolism.Reference Kobayashi, Shapiro and Nagdev 10
Outcome measures
The primary outcome was trainee and instructor preference for either intervention. Secondary outcomes included identification of strengths and weaknesses of each intervention, as well as overall merits of PoCUS simulation during critical care simulation.
Data analysis
Quantitative data included test and survey scores (Likert scale). The significance (alpha) level for all analysis was set at p<.05. t test analyses were used for comparisons of pre- and post-intervention scores within interventions, as well as comparisons between the two interventions at different time-points.
Qualitative data analysis included thematic and emergent analysis from written responses in self-reports and online survey feedback. The author and a second evaluator (K.T.) independently completed thematic analysis of the qualitative data. The thematic analysis then underwent triangulation, where only themes agreed to by two assessors were included in the final analysis.
Results
Average pre-intervention MCQ score for all trainees was 71.5%, which suggests familiarity with PoCUS, as would be expected for middle-level trainees. Most of the trainees (21/25) had had previous HFS experience, with the majority of trainees having had 3-5 previous HFS experiences. Nearly all had attended a level I PoCUS course or equivalent (21/25). Trainees responded that to date previous HFS experiences had only poorly, if at all, integrated resuscitative PoCUS into HFS, with a mean score of 3.26 out of 10 (SD 2.73).
Primary study question
The primary study outcome was trainee and instructor preference for either intervention. Trainees preferred the edus2 intervention to the simple laptop. Initial survey responses demonstrated that trainees rated the two interventions similarly (see Figure 2). Both interventions were deemed to be an improvement from previous experiences in all of the evaluated aspects of the interventions. No statistically significant difference was identified between the edus2 and laptop intervention. Upon completion of the qualitative survey (a few weeks after the course), all respondents (20/25 study participants) indicated a preference for the edus2. As one trainee put it, “holding the probe makes the simulation closer to reality and real time.” The themes of “real-time” and “hands-on use” dominated the survey responses. This was contrasted with the artificial nature of the videos found on the laptop. “The laptop meant you were trying to find the correct [clip], which detracted from the simulation.”
Trainees were asked how well the simulation scenario (previous, laptop, or edus2) addressed the following aspects of Resuscitative PoCUS. All comparisons to previous experiences were statistically significant (p<0.05). No statistically significant difference was observed between the laptop and edus2.
Instructors rated the edus2 as the superior training intervention, as compared to both previous experiences and the laptop intervention (see Figures 3 and 4). Upon review of qualitative data, features identified with the edus2 included the ability to assess trainee use of PoCUS in real time, as well as basic probe handling: “I was able to assess their use of [ultrasound] in cardiac arrest and the timing of echo with CPR.” This theme relates to the previously described concept of resuscitation choreography.
Instructors were asked how well the simulation scenario (previous, laptop, or edus2) addressed the following aspects of Resuscitative PoCUS. *Laptop intervention scores did not meet statistical significance (p<0.05).
Instructors were asked how well the simulation scenario (previous, laptop, or edus2) addressed the following training aspects of Resuscitative PoCUS. *Laptop intervention scores did not meet statistical significance (p<0.05).
Secondary outcomes
No statistically significant differences in MCQ scores were generated after completion of the first two cases.
In terms of impact of the session on their clinical work, the majority of trainees expressed an increased awareness of indications for use of PoCUS. As one trainee commented, the course increased “the number of situations in which I would consider using ultrasound.” This theme was coupled with what trainees described as an increased “enthusiasm for practicing [ultrasound] scanning [and] an appreciation of the value of [ultrasound] scanning in management of critically ill patients.” The day-long session impacted training by encouraging most trainees “to seek more training in [ultrasound] scanning and to reach Level I sign off and beyond.” More advanced trainees described a lesser impact, as they “already use [ultrasound] scanning as a part of [their] examination” in many patient encounters, but they also noted that the session inspired them to “get involved in [ultrasound] teaching.”
Both interventions were rated favorably in terms of integrating PoCUS into HFS when compared to previous experiences (see Figures 2 and 4). Faculty members commented that both interventions allowed for “a reasonable assessment of [knowledge] of indications” and noted that the interventions clearly “added PoCUS into the decision making process.”
The next series of questions posed to instructors related to the impact the PoCUS interventions had on the debriefing sessions (see Figure 4). One faculty participant explained that the edus2 allowed for “feedback on everything from positioning of equipment, to communication with patient, to documentation and medico-legal issues.” On the other hand, faculty felt the laptop intervention offered them very little insight into trainee skills and as such made it hard to bring up PoCUS skills during debrief. This “lack of ownership of the skill made the feedback less applicable to the trainees.”
Discussion
Our conceptual framework (Figure 5) illustrates key concepts supporting the use of PoCUS simulation in HFS. Learning takes place according to Bloom’s three domains. 12 Development is paired through interplay between the trainee and the instructor/preceptor. As per Cognitive Load Theory (CLT),Reference Bloom, Engelhart and Furst 13 as trainees become more proficient with PoCUS (relying less on short-term memory and more on both long-term and motor memory), they become increasingly capable of focusing on the clinical picture before them. Instructors can identify aspects of trainee PoCUS use that may require further development, and subsequently can create opportunities for deliberate practice. Simultaneously, clinical competence can be assessed using Miller’s framework,Reference van Merriënboer and Sweller 14 while recognizing the challenges inherent to the assessment of critical care skills (namely the infrequency and non-standardizability of such cases). Lastly, Kirkpatrick’s Hierarchy of EvidenceReference Miller 15 allows one to evaluate whether transfer of learning has taken place, and may help with determining if the intervention will have any impact on actual patient care.Reference Kirkpatrick 16
Conceptual Framework PoCUS – Point of Care Ultrasound, ZPD - Zone of Proximal Development.
In terms of PoCUS integration into HFS, trainees rated both interventions favorably. This may be related to the fact that both interventions, though not identical, offered trainees a simplified form of the more complex task of resuscitative PoCUS. As Kneebone et al. stated, one of the strengths of simulation is that it “offers controllable levels of challenge that can be adjusted according to individual need.”Reference Kneebone, Scott and Darzi 9
Thematic analysis of the qualitative data revealed that all trainees preferred the edus2 to the laptop, as it offered “a more hands on experience in real time.” Despite recreating only the basics of image generation, use of the edus2 challenges the trainee to consider the logistical challenges of appropriate use (for example, the proper sequence of PoCUS during cardiac arrest has been questioned).Reference Ripley 17 Furthermore, the established resuscitation sequences of trauma, cardiac arrest, and undifferentiated shock are not identical;Reference Lanctot, Valois and Beaulieu 18 , Reference Weingart, Duque and Nelson 19 neither is the integration of PoCUS into these scenarios. Based on the findings of this study and related simulation literature, the next step should be to teach PoCUS choreography in HFS. Existing evidence of improved team performanceReference Kim 20 , Reference Weller, Nestel and Marshall 21 and resuscitationReference Sawyer, Sierocka-Castaneda and Chan 22 , Reference Norman, Dore and Grierson 23 supports the use of simulation interventions for the integration of resuscitative PoCUS.
Simultaneously, probe driving skills can be taught outside the clinical context. As such, resuscitative PoCUS may be best taught through a combination of training environments. This type of “progressive fidelity”Reference Brydges, Carnahan and Rose 24 may prove to be an efficient way to teach as complex and multidimensional a skill as resuscitative PoCUS.
The edus2 and the laptop represent two ways of integrating PoCUS into HFS. Other options include placing a diagnostic trainer (i.e., CAE Vimedix) beside the HFS mannequin. This trainer can then be turned to and used when indicated during a scenario.Reference Parks, Atkinson and Verheul 25 Each of these approaches represent varying degrees of engineered and psychological fidelity, and as such also represent the spectrum of cost (where increasing engineered fidelity is generally associated with increasing costs).Reference Brydges, Carnahan and Rose 24
Hamstra et al. (2014) recently addressed the challenges associated with fidelity, given that increased engineered fidelity doesn’t always result in increased educational value.Reference Hamstra, Brydges and Hatala 26 They coined the term “functional fidelity,” and moreover, the concept of “functional task alignment,” where a given simulation intervention is defined by the learning objectives and tasks associated with the clinical skill, rather than the intervention's physical resemblance. This approach frames the evaluation of our two PoCUS simulation interventions not by their semblance to the real equipment, but rather by each intervention’s ability to address learning objectives and simulate the desired set of tasks. When introducing resuscitative PoCUS into critical care simulation, it can be argued that it is clinical integration (knowing when to scan, what to scan, and how to interpret findings within the case) that is of greatest interest. Specific image generation, on the other hand, can be taught outside the critical care context. While both interventions met several of the objectives as related to resuscitative PoCUS skills, the edus2 was found to meet more of the desired tasks, and therefore showed better functional task alignment.
Limitations
This study has several limitations. Unlike this study, most simulation intervention studies are designed to be longitudinal and as such harness the fullest potential of key constructs like deliberate practice.Reference Ericsson, Krampe and Tesch-Romer 27 Furthermore, trainees were asked about the perceived impact of the single session on their future clinical work and education, but we did not follow/verify those reports, nor does the study inform us of the impact of repeated exposure to the intervention.
Despite significant efforts to reduce personal bias, it is possible that the lead author’s involvement in the sessions (as the voice of the simulated patients and presenter of the two interventions), while also being co-developer of the edus2, may have biased some of the trainee and faculty responses. Several steps were taken during the design and implementation of the study to mitigate bias. The decision to carry out the study at another institution served to minimize personal biases and conflicts of interest that may exist between the lead author and students/instructors at his institution. Secondly, the use of previously designed cases by Kobayashi et al. ensured that the HFS scenarios were not biased toward one type of intervention or the other. Thirdly, the lead author holds no patents or commercial interests pertaining to the edus2 or related ultrasound simulation products. The edus2 is under a creative commons copyright and as such is to be shared and further developed in an open and non-commercial fashion.
Future Research
Developing a repertoire for use of PoCUS during resuscitation and emergent care will require deliberate practice. Future studies could be directed at determining whether repeated exposures to this training intervention results in better use of resuscitative PoCUS during real resuscitation.
Conclusion
The integration of resuscitative PoCUS into critical care HFS represents an evolution of PoCUS training in residency. This progression is driven by a desire by both EM trainees and faculty to safely integrate PoCUS into the care of critically ill patients. The findings of the study support the integration of PoCUS into HFS. Such integration was found to be of value to both trainees and instructors in terms of allowing trainees to demonstrate knowledge of indications as well as correct image interpretation. The edus2 was identified as being a superior teaching intervention, as it allowed for greater functional integration of PoCUS into critical care (real time and hands-on), significantly better assessment of trainee skills, and subsequently had greater impact on session debrief and formative feedback.
Acknowledgments
A sincere thank you to Ms. Megan Hall-Jackson (MA Clinical Education), whose insights and assistance profoundly improved this study’s design. This submission would not be possible without her enthusiasm for simulation-based medical education. We would also like to thank Ms. Krista Trinder, MSc (College of Medicine, University of Saskatchewan) and Mr. K Shawn Davidson, PhD (Clinical Research Scientist, Department of Medicine, Laval University, Sainte Foy, Quebec) for their assistance with data analysis and technical writing.
Competing Interests: None to declare.
