In July 2017, the Royal College of Physicians and Surgeons of Canada (RCPSC) implemented a Competence by Design (CBD) system of evaluation throughout all Canadian residency programs in order to mitigate knowledge gaps, enhance preparedness for independent practice, and ascertain when practice demands new abilities or skills. ¹ The CBD system is a hybrid competence-based medical education system and provided a new approach to the country’s postgraduate medical education. ^{Reference McGaghie, Sajid and Miller2} To achieve an intra-hospital context, milestones and entrustable professional activities (EPAs) were created by each specialty competence committee in the RCPSC followed by extensive faculty development efforts. Englander and colleagues (2017) defined EPAs as “essential task[s] of a discipline (profession, specialty, or subspecialty) that a learner can be trusted to perform without direct supervision.” ^{Reference Englander, Frank and Carraccio3} However, it is unclear how many subspecialists were included in the EPA development process, how surgical subspecialists would interpret EPAs in their discipline, and how those EPAs integrate into a CBD framework. To gain insight into this issue, this study examined the differences in the interpretation of a single EPA by subspecialty neurosurgery attendings and identified obstacles to the implementation of the CBD system from the perspective of these faculty.

We recruited vascular neurosurgeons from different neurosurgery residency programs in Canada. The recruitment period happened prior to the launch of CBD in the neurosurgery residency programs (January–March 2019). They were interviewed to capture their understanding of CBD. The interview questions were related to one of five EPAs specific to vascular neurosurgery from the Canadian neurosurgical residency programs CBD system. The vascular EPA was “Performing surgery for patients with an intracranial aneurysm.” Vascular neurosurgeons were asked to reflect on how they would evaluate and provide feedback to residents inquiring about an evaluation on this activity. Interview questions were designed to allow the interviewer to probe participant answers where possible. The interview guide was piloted on two different individuals (one general surgeon and one pediatric neurosurgeon) prior to starting the interviews to assure all the questions were clear and non-biased. Finally, individual interviews occurred remotely through videoconferencing (Zoom Video Communications, San Jose, CA) and were audio-recorded. Data were transcribed verbatim by one of the authors (MC).

Open coding was performed on one sample interview by two coders (MC and SKM), and the development of a codebook occurred as part of data reduction. The same two coders reviewed the codebook and agreed on its terms and on code definitions after various iterations. Two external reviewers (JRC and RP) verified the codebook to assure its precision and clarity. All interview transcripts were then independently coded at the sentence level by the two coders (MC and SKM) using QSR International’s NVivo 12 software. The coding was calibrated with the sample interview. Subsequent categories were created for thematic analysis utilizing inductive approach and performing axial coding with two other independent reviewers (NO and RP). The Conjoint Health Research Ethics Board (CHREB), University of Calgary, reviewed and approved the research protocol REB19-0453.

Vascular neurosurgeons described several elements influencing the level of trust they grant to residents in their program (Table 1). Most of them had a general idea of what an entrustable resident should be during residency training. Participants stated that the fund of knowledge and technical skills of a resident are highly important aspects for entrustability “Surgical anatomy is very important and that’s fundamental. The other aspects in terms of techniques, how are they handling their instruments? How are they retracting the brain?” (Surgeon 5) Similarly, vascular neurosurgeons judged the amount of independence to afford individual residents based on their previous surgical encounters with them and the necessity of the repetition of a task.

Table 1:

Definition of themes derived from qualitative analysis

Entrustability is also influenced by concerns about patient safety and the likelihood of medicolegal involvement in the task to perform such as the clipping of an intracranial aneurysm when compared to a less complex task (i.e. a burrhole). One surgeon commented on the balance between patient safety and EPAs and noted “Independent to me would be the bar that I would set on how I feel they are able to do it on their own, […] but not meaning alone. It’s really about whether it’s translatable to independent practice.” (Surgeon 4) The level of residency training is another factor influencing patient safety. A senior resident as compared to a first-year resident should demonstrate more expertise and have gained more trust from the faculty.

Specifically related to the task of clipping an aneurysm, most interviewees responded that it requires multiple skills and that putting a clip on an aneurysm is simply the final step of a much more complex neurosurgical operation. However, they all had an opinion on how to evaluate a resident on this specific EPA regardless of the details provided to define the EPA. One participant noted “I will evaluate them as I evaluate them now. I don’t think the EPA per se changes the behavior of the evaluation of the procedure.” (Surgeon 2) Similarly, when the neurosurgeons explained what EPAs and milestones meant, either they did not know enough on the subject matter to describe it or provided a definition according to their own knowledge of the concepts. To overcome this issue, a consistent method of constructive feedback to the resident physicians may be helpful to ensure that both the faculty and resident physician have a shared mental model of the EPA and milestone.

This study explored subspecialty neurosurgeons understanding of a specific EPA as well as how they evaluate their trainees in the new CBD era of assessment. When discussing a single EPA, there are significant biases and variabilities when it comes to evaluating residents based on both trainee intrinsic and extrinsic factors. Most neurosurgeons would continue to grade residents the same way they have been doing prior to CBD, independently of EPAs’ assessment form’s information. Helping faculty understand questions such as “why are we changing?” or “how can we do this well?” may not only facilitate faculty knowledge about CBD training but also improve their approach with EPA assessments. ^{Reference Hall, Woods and Frank4} Interestingly, vascular neurosurgeons also expressed their concerns of CBD related to patient safety, which is in contrast with the main idea and rationale behind CBD to support quality and safety concerns in postgraduate medical practice. ^{Reference Holmboe, Sherbino, Englander, Snell and Frank5} These concerns might reflect a wrong interpretation of the “I did not need to be there” level of high entrustability included in the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) scale that is being used to grade residents in CBD. ^{Reference Gofton, Dudek, Wood, Balaa and Hamstra6} Such discrepancies in interpretation of EPA grading should be addressed to perhaps develop simulated surgical scenarios specific to neurosurgery that fosters evaluators’ participation or close observation of the resident. These findings may or may not hold true once neurosurgeons are more familiar with the educational process.

The results of this article describe subspecialty neurosurgeons’ perceptions of EPA and competency-based assessments at a pivotal time in Canadian neurosurgery education when none of them had been involved in CBD before. These findings can translate to other specialties about to shift to competency-based medical education and lead to important lessons for faculty development and engagement and need for consistent constructive feedback mechanisms to resident physician learners.