Introduction
High quality cardiopulmonary resuscitation (CPR) in accordance with the 2010 American Heart Association (AHA) Guidelines has been shown to have a positive impact on survival from cardiac arrest.Reference Berg, Hemphill and Abella 1 , Reference Meaney, Bobrow and Mancini 2 The guidelines recommend chest compressions at a rate of above 100 compressions per minute (cpm) and a depth of >5 cm as a component of adult CPR. A consensus statement released by the AHA in 2013 further elaborated on components of high quality CPR in an effort to facilitate the implementation of these guidelines into practice.Reference Meaney, Bobrow and Mancini 2 They incorporate recent evidence and include an upper compression rate limit, defining adequate as between 100 and 120 cpm, and continue to recommend a minimum depth of 5 cm. They suggest aiming for a chest compression fraction of >80%, based on expert consensus. The chest compression fraction is defined as the proportion of time that chest compressions are performed during a cardiac arrest, with the duration of arrest defined as the time that the arrest is first identified to the time of return of sustained circulation or termination of manoeuvres.Reference Meaney, Bobrow and Mancini 2
Despite the evidence of survival benefit with optimal CPR delivery, CPR quality in both in-hospital and out-of-hospital remains poor.Reference Abella, Alvarado and Myklebust 3 - Reference Wik, Kramer-Johansen and Myklebust 5 Many studies have documented poor retention of CPR skills after standard courses.Reference Chamberlain, Smith and Woollard 6 - Reference Smith, Gilcreast and Pierce 10 Based on the results of their systematic review, Mancini et al. suggest that CPR skill decay occurs as rapidly as 3 months.Reference Mancini, Soar and Bhanji 11 One of the factors contributing to this poor skill retention is suboptimal training procedures during the course itself.Reference Kaye, Rallis and Mancini 12 - Reference Perkins, Boyle and Bridgestock 14 Lynch et al. demonstrate that in a layperson CPR course, the instructors’ ratings of compression depth and hand placement were barely better than chance, and that instructors were more likely to rate poor quality compressions as adequate than the converse.Reference Lynch, Einspruch and Nichol 15 These false positive ratings could lead to students who are unable to perform adequate CPR being certified in basic life support (BLS). The majority of the research, thus far, has been undertaken during formal CPR courses led by certified BLS/advanced cardiovascular life support (ACLS) instructors. The 2013 consensus statement from the AHA recommends further research into clarifying the methods of integration of CPR training into advanced courses and continuing maintenance of competency.Reference Meaney, Bobrow and Mancini 2
Our centre hosts weekly training sessions intended to build on the resuscitation skills learned in formal BLS/ACLS courses through repeated practice of standard ACLS scenarios. Feedback on global performance by the resuscitation team is provided at the end of each scenario on the basis of observations made by the instructor, without the formal use of objective CPR quality measures. The assessment of performance in these simulated resuscitation scenarios is complex with multiple aspects, such as team communication, leadership style, task coordination, algorithm compliance, and CPR quality, all requiring consideration. Cognitive load theoryReference van Merriënboer and Sweller 16 emphasizes the limited capacity of working memory and supports the finite ability of instructors to attend to and acknowledge all aspects of a resuscitation scenario. We sought to determine whether instructors were able to accurately assess the quality of the chest compressions as a component of their global assessment of team performance in a simulated resuscitation setting.
Methods
Study design
A prospective observational study was conducted to compare instructor assessment of chest compression quality to objective data generated by the simulation manikin. Data were collected during established weekly simulated resuscitation sessions from September 2013 to March 2014. This study was approved by the Queen’s University Research Ethics Board, and informed consent was obtained from all instructors who chose to participate.
Setting and participants
The Department of Emergency Medicine at Queen’s University holds weekly “Junior Resuscitation Rounds.” Participants include junior residents in postgraduate year (PGY) 1–2, medical students in their third and fourth years of training, and nursing students in their fourth year. All participants have previously completed BLS training, with junior residents having completed an ACLS course. Blended multidisciplinary groups of learners practice scenarios repeatedly over several two-hour sessions. Participants attend two, three, or four sessions and are randomly assigned to groups each time. The scenarios cover the range of ACLS algorithms, including ventricular fibrillation, pulseless electrical activity, asystole, tachycardia, and bradycardia. The instructors are experienced in simulation and include senior emergency medicine residents (PGY3–5), who have participated as learners in these sessions at an average of 50 times during their first two years of training, and emergency department (ED) staff who teach the sessions once every six weeks on average. Instructor demographics are outlined in Table 1. After each scenario, the instructors debrief the team on their global performance of skills such as airway management, CPR quality, team communication, and the appropriateness of the medical management. The specific content of the debriefing is at the discretion of the instructor, based on their observation of the team’s performance.
Scenario
In order to ensure consistency, data were only collected during a standardized unwitnessed ventricular fibrillation resuscitation scenario (see Appendix 1). The standardized scenario for data collection occurred as the first scenario each week of the academic year. The instructors viewed the scenario from inside the resuscitation room, where they would have been within 2 m of the manikin. They did not have access to the objective data generated by the manikin. After finishing their debrief of the scenario with the learners, the instructors completed a questionnaire assessing the quality of the resuscitation (see Appendix 2). The questionnaire required a dichotomous assessment of the chest compression parameters of rate, depth, and fraction as adequate or inadequate. The questionnaire also included distractor items such as communication, application of ACLS algorithm, and overall performance in an attempt to reduce the Hawthorne effect, because many instructors participated more than one time over the course of the study.
Measurements
We used Laerdal SimMan 3G manikins (Laerdal Medical, Stavanger, Norway). Chest compression parameter measurements were generated by the manikin at 10-second intervals throughout the resuscitations and saved to a debrief file. Measurements used were chest compression rate, depth, and fraction. These metrics were saved and exported to a Microsoft Excel file for analysis. Prior to initiation of the study, an optical tracking device was used to calibrate the manikin depth metrics to ensure that they were accurate. Agreement between the manikin measurements and precise optical tracking measurements was accurate to within 2%.
Data analysis
We defined adequate chest compressions according to the 2013 amendment to the AHA GuidelinesReference Meaney, Bobrow and Mancini 2 that recommends a compression rate of 100–120 cpm, depth >5 cm, and fraction >80%. Chest compression rate and depth values were analysed in two ways: both as an average across the scenario and as percent adequate. For the percent adequate calculation, we analysed the interval units of 10 seconds generated by the manikin (80% of the interval units needed to meet criteria). Adequacy of chest compression rate or depth in a given scenario was defined as either an average within the specified parameters or meeting the specified parameter for >80% of the compressions analysed.Reference Al-Rasheed, Devine and Dunbar-Viveiros 17 The >80% adequate analysis was undertaken to confirm that the analysis using averages was not being adversely affected by outliers.Reference Lynch, Einspruch and Nichol 15 Chest compression fraction was calculated by the Laerdal SimMan 3G as amount of time compressions that were performed divided by the amount of time that compressions were indicated and reported as a percentage.
Sensitivity and specificity were calculated for the subjective assessment versus the objective manikin data. We defined sensitivity as the proportion of scenarios that the instructor assessed as adequate that had a parameter within the guideline range as measured by the manikin. Specificity was defined as the proportion of scenarios that the instructor assessed as inadequate that had a parameter outside of the guideline range, as measured by the manikin.
A sensitivity analysis was performed using a widened adequacy criteria, accepting a rate of 100–125Reference Idris, Guffey and Aufderheide 18 and a depth of >44 mm.Reference Stiell, Brown and Christenson 19 , Reference Stiell, Brown and Calloway 20
Instructor demographics
Sensitivity and specificity of instructor assessment of chest compression rate, depth, and fraction in primary and sensitivity analysis. A sensitivity analysis was performed using a widened adequacy criteria, accepting a rate of 100–125Reference Idris, Guffey and Aufderheide 18 and a depth of >44 mm.Reference Stiell, Brown and Christenson 19 , Reference Stiell, Brown and Calloway 20
SD=standard deviation.
Results
A total of 24 scenarios were analysed, 14 assessed by an ED staff, and 10 by a senior resident. This was a convenience sample that included all sessions run in the 2013–2014 academic year that included junior residents, medical students, and nursing students. There were a total of seven staff assessors and eight resident assessors, with a maximum of three assessments per person. A graphical representation of the objectively measured chest compression rate, depth, and fraction versus instructor assessment is shown as stacked-dot charts in Figure 1. For the purpose of the graph, the data were grouped at intervals of 5 cpm, 5 mm, and 5%. Instructor assessment of the rate between 100 and 120 cpm had a sensitivity of 0.17 (confidence interval [CI] 0.02–0.32) and a specificity of 0.06 (CI −0.04–0.15) compared with manikin objective measures. Instructor assessment of depth >50 mm had a sensitivity of 0 and a specificity of 0.38 (CI 0.18–0.57). Instructor assessment of fraction >80% had a sensitivity of 1 and a specificity of 0.25 (CI 0.07–0.42). The sensitivity analysis using widened adequacy criteria of rate between 100 and 125 cpm had a sensitivity of 0.55 (CI 0.35–0.74) and a specificity of 0.08 (CI −0.03–0.18). The sensitivity analysis using depth >44 mm had a sensitivity of 0.75 (CI 0.58–0.92) and a specificity of 0.40 (CI 0.20–0.60). Sensitivity and specificity calculated in the primary and sensitivity analyses are shown in Table 2. Following the calculation using the session average rate and depth, we repeated the primary and sensitivity analysis using >80% adequate for rate and depth and found similar results.
Objectively measured chest compression A) rate, B) depth, and C) fraction, versus instructor assessment, are shown. Each symbol represents the average of the measurements from one observed session grouped in intervals of 5 cpm, 5 mm, and 5%. Diamonds represent scenarios where the parameter was assessed as adequate by the instructor, and squares represent scenarios where the parameter was assessed as inadequate by the instructor. The shaded boxes indicate the range of values that meet guideline recommendations.
Sensitivity and specificity of instructor assessment
Sensitivity and specificity of instructor assessment of chest compression rate, depth, and fraction in primary and sensitivity analysis. A sensitivity analysis was performed using a widened adequacy criteria, accepting a rate of 100–125Reference Idris, Guffey and Aufderheide 18 and a depth of >44 mm.Reference Stiell, Brown and Christenson 19 , Reference Stiell, Brown and Calloway 20
SD=standard deviation.
Discussion
Our findings suggest that instructors may not assess chest compression quality accurately during simulated resuscitation training sessions. The chest compression parameter with the least accurate assessment was rate, with the majority of sessions having rates within the guidelines being assessed as inadequate and the majority of sessions having rates above the guideline being assessed as adequate. Sensitivity analysis, including rates up to 125 cpm as acceptable, resulted in improved accuracy but continued to be associated with very poor sensitivity and specificity. None of the sessions achieved an average depth within the guidelines, and this limited the accuracy analysis of the depth data. Sensitivity analysis using a minimum depth of >44 mm demonstrated improved, but unacceptable, measures of accuracy. As in previous studies, the trend in rate, depth, and fraction was toward more false positives or inadequate chest compressions being assessed as adequate.Reference Lynch, Einspruch and Nichol 15 This prevalence of false positives is concerning in a training course, because participants who do not meet skill requirements are not given this feedback and are subsequently trusted to employ these skills in patient resuscitation. This study builds on previous work showing poor quality CPR in formal training coursesReference Parnell and Larsen 13 , Reference Perkins, Boyle and Bridgestock 14 and in resuscitation training.Reference Arshid, Lo and Reynolds 21 We have shown that chest compressions often did not occur within the recommended guidelines and, perhaps more importantly, the instructors were unable to accurately assess this. One potential explanation for their inaccuracy is the need to simultaneously assess multiple resuscitation variables. Instructors are expected to give feedback about multiple facets of the resuscitation process, including CPR quality, the application of ACLS algorithms, team dynamics, team communication, and overall scenario management. With this level of cognitive load during a resuscitation training scenario, our finding that assessment of chest compression quality is suboptimal is perhaps not surprising.
Our observations regarding the inaccuracy of chest compression quality assessment during simulated resuscitation scenarios have significant implications for training programs. With the increasing prevalence of simulation-based education interventions, it is crucial that we use valid and reliable assessment strategies.Reference McGaghie, Issenberg and Petrusa 22 From the results of this study, it seems that subjective instructor assessment of chest compression quality is not sufficient for accurate feedback during simulated resuscitations. The use of objective real-time chest compression quality feedback or post-scenario quantitative debriefing of CPR quality using objective measures may ameliorate this deficiency and concurrently reduce the cognitive load of instructors. Studies have shown that real-time CPR feedback improves performance,Reference Kern, Stickney and Gallison 23 - Reference Yeung, Meeks and Edelson 25 and the AHA suggests considering the use of real-time feedbackReference Mancini, Soar and Bhanji 11 and formalized post-resuscitation debriefing.Reference Meaney, Bobrow and Mancini 2 Various devices other than high-fidelity manikins have been used to assess chest compression quality in simulated resuscitation, including metronomes and smartphones.Reference Kern, Stickney and Gallison 23 , Reference Frisch, Das and Reynolds 26 Future research should explore the impact of these feedback techniques on the effectiveness of resuscitation simulation training.
Although not related to our primary objective, we observed an overall trend to increased chest compression rate, with 18/24 scenarios having an average rate of >120 cpm. The current campaign of “Push Hard, Push Fast” advocated by the AHA may in fact be resulting in compressions that are too fast. It has been shown that compressing at a rate beyond 120 cpm can result in a decrease in compression depthReference Stiell, Brown and Christenson 19 , Reference Stiell, Brown and Calloway 20 and worse outcomes.Reference Idris, Guffey and Aufderheide 18 Instructors and resuscitation team leaders should be aware of this potential trend and undertake corrective measures with their teams, whether in the simulation lab or the clinical setting.
Limitations
There are two important limitations to this study. First, we assumed that the instructors were familiar with the guidelines defining adequate CPR. If, in fact, they did not know the appropriate guidelines, we may have been measuring their lack of knowledge as opposed to their accuracy of measurement. This may be particularly evident in the assessment of chest compression fraction, which is a parameter that which some instructors may not be familiar. In all likelihood, our instructors have a good understanding of current guidelines, because they are senior academic emergency medicine physicians and residents practicing in a tertiary care center. They regularly teach resuscitation to junior learners, and some are ACLS instructors in the community. However, we acknowledge that we cannot comment conclusively on the relative contribution of lack of knowledge of current guidelines to the inadequacy of chest compression assessment. Second, despite the use of high-fidelity simulation manikins, the results may not be generalizable to chest compressions in humans.
Conclusions
Assessment of chest compression rate, depth, and fraction by human instructors in a simulated resuscitation training scenario was inaccurate when compared with objective measures from the manikin. These results support the use of objective and technology-supported measures of chest compression quality for feedback during resuscitation education using simulators.
Acknowledgements
We thank Dr. William Pickett for his assistance with the statistical analysis and acknowledge Dr. David Clinkard, the author of the script used to extract data from the Laerdal manikins for analysis. We also thank the staff of the Queen’s University’s Clinical Simulation Centre: Jeremy Babcock, Marcus Brown, and Jenny Xhu.
Competing interests: None declared.
Supplementary Material
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/cem.2015.104
