The transition from medical school to residency can be abrupt and stressful. The start of post-graduate training is associated with substantive changes in professional independence, as trainees take on greater responsibility in managing complex, acutely ill patients.Reference Berridge, Freeth, Sharpe and Roberts 1 - Reference Antonoff, Swanson, Green, Mann, Maddaus and DʼCunha 3 These challenges are exacerbated by the wide variability in clinical knowledge and skills among graduating medical students,Reference Berridge, Freeth, Sharpe and Roberts 1 , Reference Teo, Harleman, OʼSullivan and Maa 2 , Reference Moercke and Eika 4 - Reference Remmen, Derese and Scherpbier 6 some of whom candidly express a lack of self-confidence in their abilities.Reference Berridge, Freeth, Sharpe and Roberts 1 , Reference Antonoff, Swanson, Green, Mann, Maddaus and DʼCunha 3 , Reference Evans, Wood and Roberts 7 As a result, at the onset of residency, trainees may experience increased stress and burnout and may even inadvertently commit errors that compromise patient care.Reference Teo, Harleman, OʼSullivan and Maa 2 , Reference Thomas 8 , Reference Shanafelt, Bradley, Wipf and Back 9 Although this is often mitigated through the experience residents accumulate during training, until this occurs the potential for negative consequences remains high. In fact, recent evidence demonstrates an increase in mortality attributable to medical error in the months when medical graduates begin residency training, suggesting the anecdotal “July Effect” may be a real phenomenon.Reference Phillips and Barker 10 The potential implications of such errors are magnified in high acuity fields like neurosurgery. However, present day post-graduate surgical education is characterized by duty hour restrictions, fiscal constraints, and complexity of surgical care that limit opportunities for skills training in the clinical setting.Reference Reznick and MacRae 11
The undeniable concern for patient safety, coupled with the perceived lack of preparedness among incoming post-graduate trainees, provides a strong rationale for developing programs to ease this transition. In 2012, the Association of Faculties of Medicine in Canada (AFMC), in partnership with the Royal College of Physicians and Surgeons of Canada (Royal College) and other stakeholders, released the Future of Medical Education in Canada Post Graduate (FMEC-PG) project report. One of its recommendations is to ensure effective transitions “along the educational continuum”, 12 including the transition from undergraduate to postgraduate training. As a key transformative action, the report calls for development of pan-Canadian resident orientation programs to standardize entry into residency training. 12
The Canadian Neurosurgery Rookie Camp was launched in July 2012, as a first step in addressing these recommendations. In keeping with similar surgical orientation programs,Reference Selden, Origitano and Burchiel 13 - Reference Fann, Calhoon and Carpenter 15 the goal of the Camp was to foster acquisition of basic technical, cognitive and behavioural neurosurgical skills, so that incoming residents could begin post-graduate training with increased clinical proficiency and self-confidence. Details of the development, delivery and evaluation of the inaugural Rookie Camp are presented in this report.
Methods
Curriculum Development and Course Delivery
The concept of a national neurosurgery training camp was presented (by author DBC) to the Royal College neurosurgery specialty training committee and endorsed in November 2011. Following this, a multidisciplinary organizing committee was convened to develop the curriculum and evaluation plan for the Rookie Camp. The committee included Canadian neurosurgery program directors, faculty, and residents; medical educators; simulation experts; and representatives from the Royal College. The committee developed educational objectives using the Royal College Objectives for Training in Neurosurgery and the CanMEDS competency FrameworkReference Frank and Danoff 16 (Appendix 1) and course content based on a review of existing surgical orientation programs (e.g. the Society of Neurological Surgeons “bootcamp” courseReference Selden, Origitano and Burchiel 13 ) and a national needs-assessment survey (see Brandman et al. in this issue).
The inaugural Rookie Camp was held in July 2012 at the Dalhousie University Skills Centre for Health Sciences (Halifax, Nova Scotia). Post-graduate year one (PGY-1) residents from all Canadian neurosurgical training programs were invited to participate. Funding for registration, materials, and meals was provided through unrestricted educational grants. All materials donated by corporate partners were deemed necessary to meet course objectives.
A pre-course manual was distributed electronically before the Camp and in print on the first day. Residents attended a social event the evening before the course began, while course faculty attended a session on delivering effective feedback. All faculty and participants attended a second social gathering on the evening of the second day. During the two-day course, groups of three to four participants rotated through multiple stations, each of which was facilitated by one or two faculty members. During the seven ‘case-based’ stations, relevant aspects of a patient’s history, physical exam, differential diagnosis and management were reviewed, followed by demonstration of relevant anatomy and surgical approaches using imaging and cadavers, and finally experiential skills training using various simulation modalities (Figure 1). An additional four procedural skills stations featured faculty modeling of appropriate surgical technique for various basic skills, after which participants practiced using part-task training exercises (Figure 2). During emergency management, team training and communication skills stations, residents participated in immersive simulations combining role-play and high-fidelity mannequins, with subsequent debriefing by simulation educators (Figure 3). A detailed summary of these course activities is provided in Table 1.
Case-based stations To contextualize course content, participants rotated in small groups through stations focused on a clinical case. After discussion of case history and physical exam findings, relevant anatomy and surgical approaches were reviewed using imaging and/or pre-dissected cadavers (A, B). Additional time was given to review the appropriate use of surgical instruments (C) and relevant cranial and spinal surface anatomy either during the case or during Case 11 (Craniotomy and Neurosurgical Instruments 101). Finally, time was given for technical skills practice relevant to the scenario using a variety of high and low-fidelity training models, e.g. placement of a real halo ring and vest on a mannequin during the spinal cord injury case (D).
Procedural skills training stations Trainees had the opportunity to engage in deliberate practice of basic neurosurgical skills during the procedural skills stations, which utilized both high and low-fidelity part-task training models. A variety of basic neurosurgical technical skills and procedures were reviewed, including: landmarking and inserting external ventricular drains using a cadaver (A); two-layer scalp closure simulated using pork-belly (B); principles of hemostasis and use of bipolar cautery using grapefruit and/or sand (C); and placing a burrhole using handheld and pneumatic perforators on sheep scapulae (D).
High fidelity simulation Trainees are presented with a case of traumatic head injury (A). After triaging the case over the telephone and discussing transfer of the patient with a simulated physician from a peripheral hospital (B), residents have the opportunity to manage the patient (via a high-fidelity mannequin) in small groups. The residents’ management of the case, team dynamics, and communication skills are debriefed following the scenario (C-D).
Summary of Rookie Camp Curriculum
a Each station was set up with the necessary instructional materials, including: (1) a projector or computer screen and laptop for presentation of case details and imaging; (2) cadavers to facilitate expert demonstration (e.g. craniotomy flaps, patient positioning, etc.); (3) neurosurgical tools and simulation platforms for skills training (e.g. simulated lumbar torso and lumbar puncture/drain kits ); (4) dedicated, high-fidelity simulation space and patient-simulators (e.g. high-fidelity mannequin with cranial trauma) for team, communication and emergency management skills training. CT = computed tomography, SAH = sub-arachnoid hemorrhage, CSF = cerebroplsinal fluid, VP = ventriculoperitoneal, EVD = external ventricular drain, EDH = epidural hematoma, XR = X-Ray
Program Evaluation
The Rookie Camp was evaluated by surveying participants and faculty, who rated their agreement with statements pertaining to each station and the course overall on a 4-point Likert scale (1 = strongly disagree to 4 = strongly agree). In addition, participants and faculty were invited to provide additional free-text comments regarding their experiences. The impact of the course on participants’ knowledge was evaluated using three multiple-choice tests: a pre-test held at the beginning of the first day of the course, post-test held at the end of the second day, and a retention-test conducted three months after the Camp (administered electronically using the Royal College learning management system). The tests contained two to three questions from each station (“taught” questions), as well as four questions on topics not covered in the course (“not taught” questions). The organizing committee reviewed each question to ensure accuracy and an appropriate level of difficulty. Further validity or reliability testing was not conducted.
Data Analysis
Statistical analyses were performed using SPSS 21 (IBM Corp, Armonk, NY). Results from the surveys were analyzed using descriptive summaries (median, range and frequency of each Likert response). Raw scores on the knowledge tests were converted to percentages and analyzed using one-way repeated measures analysis of variance with post-hoc pairwise comparisons using the Bonferroni correction for normally distributed variables, and the Friedman test for non-normal distributions. Separate analyses were conducted for overall test scores, scores on “taught” questions, and scores on “not-taught” questions.
Results
Seventeen of 23 PGY-1 residents (74%) attended the inaugural Rookie Camp, representing 11 of 14 (79%) Canadian neurosurgical training programs. The regional distribution of residents attending the Camp is shown in Figure 4. All participants received the pre-course manual and participated in the entirety of the course.
Distribution of Rookie Camp Participants
Resident and Faculty Course Evaluations
The results of both participant and faculty evaluations are summarized in Table 2, with detailed comments provided in Appendix 2. Between 13 and 17 (76-100%) station-specific and overall course evaluations were completed by participants. All respondents agreed that the stated objectives were met and sufficient time was allotted for skills training in each station. With the exception of the hemostasis module (Station 9), respondents agreed that the information presented was relevant and valuable for the PGY-1 level.
Resident Rating a of Rookie Camp Stations
a All items scored on a 4 point Likert scale, ranging from 1 = strongly disagree to 4 = strongly agree.
b Numeric scores represent median and range of responses to each question.
Of the case-based stations, respondents rated Case 2 (posterior fossa tumour with hydrocephalus, which included external ventricular drain (EVD) insertion) highest with respect to training relevance. Participants found the opportunity to practice EVD insertion using real instruments useful, and one participant stated, “… it was very helpful to learn strategies for breaking bad news.” Similar ratings were observed for cases 6 (pituitary tumour with endonasal navigation), 4 (cauda equina), 1 (subarachnoid hemorrhage and shunt tapping), and 5 (spinal cord injury with halo application). Participants again found the exposure to new topics (e.g. halo application) and the opportunity to learn communication strategies helpful. With respect to procedural skills, station 10 (basic surgical skills) was rated highest, with 93% of respondents strongly agreeing that the skills learned were relevant, practical and valuable for their training. Conversely, the virtual reality bimanual hemostasis training station (station 9) was rated lowest.
The overall course evaluation revealed that for the majority of participants, the content of the course was novel. In addition, all participants agreed that the location of the Camp was appropriate, the knowledge and skills taught were relevant and the content was taught in a comprehensive manner. In particular, many trainees found the pre-course manual and small group-learning format to be beneficial. One participant noted that simulations afforded the opportunity to “work under pressure [and] to see what it is like before it happens on the floor.” Notably, 100% of respondents strongly agreed that they would implement what they learned in clinical practice and that they would recommend the course to future neurosurgery residents. Course faculty echoed these sentiments, commenting: “this course exceeded my expectations; I did not think it would work out this well and be as well received.”
Both participants and faculty also provided suggestions for improvement. In particular, participants requested additional teaching on interpretation of imaging (particularly spinal comuted tomography and magnetic resonance imaging), more anatomical dissection, a larger variety of examples (e.g. during the consent discussion) and more time for individual practice of technical skills (e.g. mounting a halo, inserting an EVD, etc.). In addition, both participants and faculty suggested limiting cases to less than 1.5 hours. Faculty suggested adding various technical and clinical skills, such as high-speed drilling, an introduction to the operating microscope, and observed neurological exam of a comatose and awake patient (using a standardized patient) with faculty feedback. Both residents and faculty found the virtual-reality simulation less useful than other aspects of the course.
Knowledge Test Results
The results of the pre-, post-, and three-month retention tests are presented in Figure 5. A significant increase in the percentage of correct responses was observed for overall test scores (F(2,32) = 8.092, p<0.01) and on questions concerning material taught during the course (F(2,32)=7.572, p<0.002), but not for material not taught during the course (χ2 (2)=1.302, p<0.521). Post-hoc pairwise comparisons revealed that scores improved between the pre-test and immediate post-test (70% to 81%, p < 0.002 for ’taught’ questions), whereas no significant change was observed from post-test to retention test. In addition, a trend of increasing difference in the percentage of correct responses before and after the course was observed between “taught” vs. “non-taught” questions. These results demonstrate that participants’ knowledge improved significantly for content taught in the course, as compared with unrelated content, and that this knowledge was retained up to three months later. In addition, a narrowing of the confidence interval for questions concerning course material, but not for other questions, suggests that the course also narrowed the knowledge gap between participants.
Pre, Post and Retention Knowledge Test Results
Discussion
A number of recent publications describe standardized residency orientation programs for senior medical studentsReference Berridge, Freeth, Sharpe and Roberts 1 , Reference Teo, Harleman, OʼSullivan and Maa 2 , Reference Laack, Newman, Goyal and Torsher 17 and ‘bootcamp’ coursesReference Antonoff, Swanson, Green, Mann, Maddaus and DʼCunha 3 , Reference Antonoff, Shelstad, Schmitz, Chipman and D’Cunha 18 - Reference Parent, Plerhoples and Long 20 for junior residents in in obstetrics and gynecology,Reference Duff 21 cardiac surgery,Reference Fann, Calhoon and Carpenter 15 orthopedics,Reference Sonnadara, Van Vliet and Safir 14 and neurosurgery.Reference Selden, Origitano and Burchiel 13 , Reference Selden, Barbaro, Origitano and Burchiel 22 , Reference Selden, Anderson, McCartney, Origitano, Burchiel and Barbaro 23 Results from these programs are encouraging, with evidence for improved self-confidence and perceived preparedness for clinical practice among participating trainees.Reference Berridge, Freeth, Sharpe and Roberts 1 , Reference Esterl, Henzi and Cohn 19 When augmented with simulation-based training, tangible improvements in clinical knowledgeReference Selden, Origitano and Burchiel 13 , Reference Selden, Anderson, McCartney, Origitano, Burchiel and Barbaro 23 and technical skillsReference Sonnadara, Van Vliet and Safir 14 , Reference Parent, Plerhoples and Long 20 have been observed. Emerging data also suggest these trainees demonstrate improved clinical performance when compared with matched peers within the first month of training, suggesting these educational experiences may also mitigate the “July effect” and improve patient safety.Reference Antonoff, Swanson, Green, Mann, Maddaus and DʼCunha 3 The results from the inaugural Canadian Neurosurgery Rookie Camp add to this growing body of literature. To our knowledge, this Camp is the first national specialty-specific orientation program in Canada, and the second of its kind in the published literature.Reference Selden, Origitano and Burchiel 13
The perceived value of the Camp is evident in participants’ course ratings, which suggest knowledge and skills taught at the Camp were value-added for PGY-1 residents. This is further supported by improved domain-specific knowledge test scores before and after the Camp and the retention of this knowledge after three months. These data suggest the Camp achieved one of its primary objectives: to standardize PGY-1 resident skills on entry into Canadian neurosurgical training. Participants also appreciated the opportunity to network with faculty and peers during the formal curriculum and at social events. This “socialization to the field” has been noted as an added benefit in other courses,Reference Selden, Origitano and Burchiel 13 and may be beneficial in developing residents’ sense of belonging to the Canadian neurosurgical community.
The station-specific comments provide insight into the value of specific teaching methods used during the Camp. For case-based stations, participants commented that incorporating multiple skills was particularly beneficial for learning. This suggests that residents value the opportunity to contextualize and integrate various cognitive, behavioural and technical skills required to manage a neurosurgical patient. Among the procedural skills stations, the highest ratings were awarded to basic skills training, which used simple, low-tech simulation to teach essential techniques such as asepsis, suturing, and bipolar cautery. Conversely, stations using ‘high-tech’ virtual reality simulation and part-task non-cadaveric simulation trainers were not rated as highly, despite their supposed increased fidelity for the clinical setting. This underscores a controversial aspect of simulation pedagogy: that higher fidelity (where fidelity is defined as a closer approximation of visual or surface features of the simulation to real clinical scenarios) does not necessarily translate into better learning.Reference Norman, Dore and Grierson 24 , Reference Cusimano 25 Interestingly, this finding is in opposition to the results from other neurosurgical orientation courses.Reference Selden, Origitano and Burchiel 13 It is clear that additional research is required in this area.
Regardless, simulation is increasingly being used in surgical education to facilitate both deliberate practiceReference Ericsson 26 of fundamental procedural skills and integration of cognitive, behavioural and technical aspects of surgical care delivery, in a setting free from concerns over patient safety.Reference Haji, Hoppe and Morin 27 There is now extensive evidence that simulation training has a large effect on improving knowledge, skills and clinical behaviour outcomes, while having a moderate effect on patient outcomes.Reference Cook, Hatala and Brydges 28 Within the surgical skills domain, multiple studies have demonstrated that skills learned in the simulated setting transfer to clinical practice.Reference Seymour, Gallagher and Roman 29 , Reference Park, MacRae and Musselman 30 However, the majority of these studies do not address neurosurgical skills, and thus further research in this area is warranted.
A number of limitations of the Rookie Camp experience should be acknowledged. First, the content and delivery of the Camp was evaluated using subjective ratings from participants, which may be influenced by the fact that respondents were first year residents who were being taught by senior residents and attending surgeons. Despite emphasizing the anonymous nature of responses, it is possible that participants held back from providing critical comments about the experience, so as not to offend the organizing committee. Second, as these trainees were just beginning their post-graduate training (most had not yet completed a neurosurgery rotation), their ratings are based on a limited knowledge of what neurosurgical residency entails, and should be interpreted with due caution. Third, the validity and reliability of the knowledge tests were not established beyond expert review of the questions. In addition, the same questions were used for each of the tests, introducing the potential for test-retest bias. Fourth, we are unable to comment on the impact of the Camp on practice behaviours of participating residents or subsequent patient outcomes; however, we hope to address this in part through post-course semi-structured interviews. Ongoing monitoring of complication rates for neurosurgical procedures performed by junior residents (e.g. EVD, lumbar puncture, etc.) may also be useful to demonstrate the impact of simulation training on neurosurgical outcomes.
We are similarly not able to comment on the impact of the Rookie Camp on the specific technical and behavioural skills taught, as these were not directly assessed. Although attempts were made by faculty to rate residents’ performance during the various stations, this was found to be extremely difficult given that participants were in small groups and there was insufficient time to both complete the learning objectives and assess each participant in a rigorous manner. This is a major limitation in our evaluation, and in the evaluation of many post-graduate orientation programs.Reference Teo, Harleman, OʼSullivan and Maa 2 In the future, efforts should be made to objectively assess changes in the skill-level of participating trainees (e.g. using objective structured assessment technical skills before and after the course).Reference Parent, Plerhoples and Long 20 , Reference Martin, Regehr and Reznick 31 However, to ensure the appropriateness of these assessments, measures of performance with appropriate validity evidence will need to be developed for the technical and behavioral skills in question.Reference Cook, Bordage and Schmidt 32
Finally, it is important to recognize that the Rookie Camp is a very intensive, but brief learning experience. As such, the educational benefits it affords to participants are necessarily limited, with the greatest impact likely to be affected in the first few months of residency training.Reference Antonoff, Swanson, Green, Mann, Maddaus and DʼCunha 3 Given the ongoing pressures that hinder opportunities for resident education in the clinical setting and recent movements towards competency-based education,Reference Frank, Snell and Cate 33 the Camp should be viewed as an introductory experience to be coupled with longitudinal training opportunities for residents (both clinical and simulation-based) within their respective training programs.Reference Selden, Origitano and Burchiel 13
Based on the successes from this inaugural experience, the Canadian Neurosurgery Rookie Camp has been endorsed as an annual event. The organizing committee is committed to iterative course improvement: by incorporating resident feedback from the 2012 course, a number of improvements were made to the 2013 curriculum, including a revised course manual, reorganization of the hemostasis station and increasing time for fundamental skills training (e.g. EVD insertion). At the same time, the elements of the course that were rated highly (e.g. case-based teaching, small-group sizes, and opportunities for repeated, deliberate practice) were retained. Additional efforts to evaluate changes in residents’ skills, practice behaviours and perhaps even patient outcomes will need to be incorporated in future courses to characterize the impact of the Camp across all outcomes of interest.
Conclusion
The inaugural Canadian Neurosurgery Rookie Camp was successfully delivered, with participation of PGY-1 trainees and faculty from across the country. Participants’ evaluation of the course was positive and knowledge tests revealed the Camp promoted learning, retention, and a narrowing of knowledge gaps between participants on content taught during the course. Our experience has also highlighted areas for future research in neurosurgical education, including issues related to simulation fidelity and its impact on knowledge and skill acquisition, and the transfer of skills learned during courses like the Rookie Camp to clinical practice.
Acknowledgements
The authors thank the following members of the local organizing committee and the Rookie Camp Faculty for their pivotal roles in successfully delivering the Rookie Camp: Dr. Christopher Wallace, Paula Chiasson (Camp Coordinator), Murray Hong, Michelle Murray, and the staff of the Skills Centre for Health Sciences at Dalhousie University. This project was funded through unrestricted education grants from the Canadian Neurosurgical Society, the Royal College of Physicians and Surgeons of Canada, Dalhousie University Faculty of Medicine, the National Research Council of Canada (NeuroTouch), Medtronic Incorporated, Codman & Shurtleff Incorporated, and Karl Storz Endoskope Incorporated.
Disclosures
None to disclose.
Supplementary Material
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/cjn.2014.124.
