CLINICIAN'S CAPSULE
What is known about the topic?
A gap exists in our understanding of how well-prepared newly certified emergency physicians feel with supervision and assessment of trainees.
What did this study ask?
How prepared do senior emergency medicine residents and new-to-practice physicians feel with supervision and assessment of trainees?
What did this study find?
Residents are transitioning to practice with little formal training on how to supervise and assess trainees.
Why does this study matter to clinicians?
Clinician educators will need to play a larger role in developing curricula to instruct residents on how to perform these skills.
INTRODUCTION
Multiple organizations have identified transitions from training to practice as unique periods in the medical life-cycle which can be stressful and burdensome.Reference Frank, Snell and Sherbino1–Reference Westerman, Teunissen and van der Vleuten5 The current Canadian paradigm allows residents to progress through their training with graded, progressive responsibility,6 a model whose efficacy has not been demonstrated and whose expectations are at times ambiguous.Reference Kennedy, Regehr, Baker and Lingard7 While many training programs have commenced transitioning to a competency-based educational model, including emergency medicine (EM), this transition is in its infancy. However, regardless of the training paradigm used, inherent in the transition from training to practice is the change from being a supervised trainee to a supervising physician, often for former resident colleagues at senior training levels.
Research involving the medical life-cycle of EM residents has largely focused on the transition from medical student to resident, rather than from resident to independent practice.Reference Cohen-Schotanus, Schonrock-Adema and Bouwkamp-Timmer8–Reference Prince, Van De Wiel and Scherpbier10 Supervision and assessment investigations have focused primarily on junior trainees,Reference Orman and Thornton11–Reference Touchie, De Champlain and Pugh15 leaving a gap in our understanding of how comfortable senior trainees feel when supervising and assessing other senior trainees. While many programs have adopted “residents as teachers” programs, these programs focus primarily on the competencies of bedside teaching, not supervision or assessment.16–Reference Olmos-Vega, Dolmans, Donkers and Stalmeijer18 There are some free open access medical education (FOAMed) materials that are trying to address this gap in training,Reference Chan and Sherbino19–20 as well as curricula in the literature to ease the transition to independent practice.Reference Stodel, Wyand and Crooks21–Reference Dahn, Watts, Best and Bowes23 However, within these materials, there are few, if any, mentions of developing or consolidating skills in supervision and assessment, and outcome measures are scarce.
Given that the Royal College of Physicians and Surgeons of Canada (RCPSC) and the College of Family Physicians of Canada (CFPC) identify supervision and assessment as core competencies,Reference Frank, Snell and Sherbino1–Reference Tannenbaum, Konkin and Parsons2 our study sought to explore senior emergency residents’ and new-to-practice emergency physicians’ sense of preparedness to supervise and assess trainees. We also sought to understand whether there is a perceived need for more formal training in supervision and assessment, and to appreciate what barriers and/or enablers might exist in implementing it.
METHODS
Methodology overview
Our qualitative design was informed by constructivist grounded theory. Grounded theory is a methodology that emphasizes the generation of theory rooted in data.Reference Glaser and Strauss24 Constructivist grounded theory is a modernized version that takes on a relativist approach, recognizing the diverse viewpoints and realities of the investigators and the participants, and uses a reflexive stance towards situations, actions, and people – thereby allowing investigators to develop hypotheses to guide the qualitative interviewing and analysis.Reference Charmaz, Holstein and Gubrium25–Reference Charmaz26 Constructivist grounded theory has been used broadly within healthcare because its guidelines can assist in the examination and subsequent understanding of social, psychological, and behavioural processes.Reference Braun and Clarke27 It allowed us to understand how emergency residents and new attendings developed skills and behaviours through their training history and professional experiences.
The study received ethics approval from the Sunnybrook Health Sciences Centre Research Ethics Board and the Hamilton Integrated Research Ethics Board. Data were collected between September 2016 and November 2017.
Participant criteria and recruitment
Eligible participants were from and/or affiliated with the University of Toronto or McMaster University and were either a fifth year RCPSC EM resident (PGY5), a resident completing an enhanced skills program in EM through the CFPC [CCFP(EM), certification in family medicine with added competency in emergency medicine], or an EM physician within his or her first 3 years of practice. An email invitation and Letter of Information were sent to all PGY5 EM and CCFP(EM) residents and to all EM attendings affiliated with either site by their respective program administrator or director. Potential participates self-identified to a study investigator to schedule an interview.
Semi-structured interviews
A semi-structured interview guide was developed based on pertinent areas of interest from the literature, while favouring open-ended questions designed to elicit experiential data. Interviews were conducted either in person or via Skype with the same co-investigator (SK). Each interview lasted 30–45 minutes, after completion of a verbal consent checklist.
Two phases of interviews were held to allow for refinements to the interview guide.
Phase one took place between September 2016 and July 2017 at the University of Toronto, and phase two between August 2017 and November 2017 at the University of Toronto and McMaster University. Due to scheduling conflicts, we were unable to conduct any interviews with participants from McMaster during phase one. Our final sample size of 18 participants was guided by thematic saturation; interviews were stopped after all investigators agreed that all themes had been fully captured, and that successive interviews did not reveal any further themes or expansions upon relevant themes.5,28–32 See Table 1 for participant distribution.
Distribution of interview participants
CCFP(EM) = certification in family medicine with added competency in emergency medicine; PGY = postgraduate year.
Data analysis
A data analysis examined how participants constructed meanings and actions related to their experiences of supervision and assessment by coding for what was commonly occurring and by comparing responses between participants. Experiences and emotional responses involving supervision and assessment were of specific importance.
The constructivist grounded theory framework involved these steps: 1) simultaneous data collection and analysis, 2) exploration of emergent themes by a constant comparative method, 3) construction of abstract categories that explain these processes, and 4) integration of categories into a theoretical framework.Reference Braun and Clarke27 The analytic team consisted of the principal investigator (KP), a newly practicing RCPSC-certified emergency physician, co-investigator (SK), a senior RCPSC EM resident, and co-investigator (JZ), a psychiatrist and expert in qualitative research methodologies. Interviews were audio-recorded, deidentified, and transcribed verbatim by an external company and were uploaded into NVivo 11 (QRS International, Burlington, Massachusetts). An analysis was performed in parallel with data collection in two separate phases. Interview transcripts were coded by co-investigator SK who consolidated the data into broader themes using open-coding, memo-writing, and categorizing.Reference Plas and Kvale29 Principal investigator KP and co-investigator JZ met regularly with co-investigator SK to discuss the coding and emerging themes. JZ provided methodological support and coded several transcripts independently to ensure consistency and review themes. Responses from each academic institution were also compared to look for any differences that may have emerged.
RESULTS
Five major themes were identified: 1) Supervision and assessment skills were acquired passively through modelling; 2) the training available in these areas is variably used, creating a diversity of comfort levels; 3) competing priorities in the emergency department (ED) represent significant barriers to improving supervision and assessment; 4) providing negative feedback is difficult and often avoided; and 5) competence by design will act as an impetus for formal curriculum development in these areas. For a summary of these five themes and representative quotations, see Table 2.
Summary of overarching themes with representative quotations
CCFP(EM) = certification in family medicine with added competency in emergency medicine; ED = emergency department; PGY = postgraduate year.
Modelling behaviours
Regardless of the academic centre, residents and attendings described learning how to supervise and assess trainees passively through observation of those more senior, modelling those behaviours, and then using a “trial and error” method of identifying which strategies worked best for them personally.
It was a little bit of trial and error in the sense of trying things and seeing what people would respond to, trying something different and seeing if that worked better. (Attending)
When asked whether this was useful or ideal, responses varied; some agreed, with the caveat that they did not see any other way, while others hoped for change. The consensus amongst most participants was that, while non-clinical skills like supervision and assessment are more challenging to teach in a didactic manner, there are opportunities – didactic or otherwise – to improve how we acquire them.
Training and comfort levels
Residents and attendings from both universities identified the scarcity of formal training on supervision and assessment during residency. Residents identified a maximum of two sessions within residency, and some external “residents as teachers” sessions; however, most agreed that the latter were not focused on supervision or assessment.
Participants also identified heterogeneity in exposure to informal learning opportunities where they might practise supervision and assessment such as on dedicated “supervising” shifts. Some residents started supervising trainees in their PGY3 year, whereas others started earlier, including as a senior clerk. There seemed to be greater opportunities to supervise at academic centres and relatively few opportunities at affiliated community sites. Residents specifically noted that they felt uncomfortable supervising fellow senior residents because they had little opportunity to do so. Some participants also mentioned a lack of clear expectations in terms of what the natural progression of these competencies look like.
It's easier to dichotomize an R5 versus an R1. But 2–3, 3–4, 4–5, those little differences in years, I think it's very tough. I don't know what normal is. (Attending)
Most residents who had undertaken extra courses in medical education, such as a master's degree, the ED STAT! course, or a certificate-based teaching program (“Master Teacher” or “University Teaching” program), noted feeling more comfortable with supervision and assessment due to having greater exposure to these topics within these programs.
Competing priorities in the ED
Residents and attendings made mention of the many competing priorities to balance during a shift. When asked where supervision and assessment sat within the priority hierarchy, and how these competencies were considered and managed during a shift, most residents mentioned splitting up responsibilities but felt they lacked knowledge on how to execute it. Some PGY5 residents identified being made responsible for all aspects of ED management without any formal training.
We're given options on shift to work on various roles. You could look at it as flow of the department, care of your own patients and then supervising trainees. We don't really get any formal training on how to balance those priorities. (PGY5)
Residents unanimously stated that responsibility for ensuring a well-balanced experience between teaching, supervising, and developing one's own clinical acumen, was theirs. However, not all residents or attendings felt a strong impetus to push themselves to improve or focus on these skills independently. Some residents identified other priorities in their senior years, such as certification examinations and future practice plans, as barriers to honing their skills, whereas attendings unanimously mentioned departmental flow as being the biggest barrier to being able to be an effective supervisor and teacher.
It can slow you down from a flow perspective…. One of the struggles of being a junior staff is I feel like I'm really expected to see a bunch of patients and work on my flow. (Attending)
Negative feedback and assessment
Residents and attendings unanimously chose providing negative feedback as the most challenging supervisor and assessor task, and all mentioned a lack of formal training in this. Participants brought up a challenge unique to the ED: the expectation of making a judgement regarding a trainee, despite working with them for only one shift. This led to feelings of awkwardness and guilt about not having the authority or the right to provide negative feedback after such short encounters.
Additionally, residents mentioned feeling inadequate in providing negative feedback based on their overall lack of experience in being a formal assessor, and not knowing how a trainee should be performing for their PGY-level. This was also brought up by some attendings who admitted that they were unsure of their ability to accurately identify a failing trainee.
If I had more experience with other learners… and I knew what [their] baseline was, then it [giving negative feedback] would be okay. But I haven't really seen dozens of learners where I can actually say this is the standard that you should be at. (Attending)
Other difficulties with providing negative feedback included potentially offending the trainee, a defensive trainee, and fear of affecting a trainee's progression. Attendings also discussed fear of academic reprisal and a lack of support from their respective departments.
People who have academic positions are working hard to go from being assistant to associate to full professors, and they're worried about teaching evaluations[sic] being negative. I do worry that if I have a very negative teaching evaluation [sic], that will impact my academic progression. (Attending)
Some attendings indicated that some degree of opinion about a trainee comes from hearsay. When assessors have limited interactions with trainees and a lack of clear expectations for where residents ought to be with respect to training level, assessors will establish an impression from informally asking colleagues “by the water cooler.”
The move towards competence by design
As programs transition to competence by design, supervision and assessment will be necessary to ensure that residents demonstrate competence in entrustable professional activities and milestones.33 Participants identified advantages and disadvantages in how competence by design could influence or change supervision and assessment.
The main benefit described by most participants was the hope that trainees would get more frequent and specific feedback.
You can go through residency getting the same evaluation [sic], and a lot of people that are weak don't get honest results. I think there will be a lot of benefit to having feedback that targets a specific behaviour that is observed and actionable. (Attending)
Additional benefits of competence by design included greater continuity of assessments and shifting the feedback atmosphere to one of coaching and entrustment. However, others were concerned that competence by design will not be able to capture all of the skills that an emergency physician needs and will mean an increased amount of work for physicians in an already overburdened system.
At the end of your training, you're going to have a couple of hundred data points, but is that sufficient to say that someone is competent in all of the major things that you need to be in order to perform EM? I'm not sure that it is. (Attending)
DISCUSSION
Our study used thematic analysis informed by the tenets of constructivist grounded theory to explore senior residents’ and new-to-practice emergency physicians’ sense of preparedness to supervise and assess trainees, and understand how current EM curricula support or detract from the development of these competencies.
Our findings suggest that residents are transitioning to practice with little formal training on how to effectively supervise and assess trainees. These results mirror those by Westerman et al., which demonstrated that, while transitioning internists and obstetricians felt prepared to fulfill their role clinically, they did not feel prepared for nonclinical tasks, including supervision and assessment.Reference Westerman, Teunissen and van der Vleuten5 Berendonk et al. also identified three areas of discomfort regarding assessment among supervisors: the importance of being a content expert to be a credible assessor, familiarity with the level of knowledge possessed by the trainee being assessed, and the self-efficacy of the assessor.Reference Berendonk, Stalmeijer and Schuwirth34 This aligns with our work in that participants in our study identified that determining the appropriate performance level of a trainee, particularly those in adjacent years, is difficult and presents a unique challenge.
Our study also reinforces the findings of Dudek et al. who demonstrated that there are multiple barriers to documenting poor performance and providing negative feedback, and that when there is virtually no continuity of feedback between assessors, it can be easy for a struggling resident to fall through the cracks.Reference Dudek, Marks and Regehr35 Our study also confirms that a substantial number of attendings avoid providing negative feedback for fear that a poor teaching evaluation could impact a promotion.Reference Dudek, Marks and Regehr35–37
Competence by design has the potential to address these issues by creating a culture of coaching within medical education, rather than the current culture of staged supervision and assessment. Further, as competence by design assessment tools are developed, it will be critical that residents are afforded the opportunity to practise these skills. It was clear from our interviews that residents in the RCPSC program had significantly more opportunities to supervise and assess junior residents than their CCFP(EM) colleagues. However, both programs suffered from a lack of experience in supervising and assessing senior resident colleagues, a task expected of a new attending physician. It seems that more attention needs to be paid to exploring how educators can better prepare senior residents to supervise other senior resident colleagues.
We also found that role modelling regarding how to supervise and assess was universal. Our study begs the question: “Who is the role model?” Have they demonstrated adequate competence for this behaviour? Would this kind of learning meet the standards set out by a certification college? These are important questions that demand further research if we continue to rely on this as the primary learning method.
Role modelling may also be linked to supervising physicians acting as mentors and coaches. As described by Watling et al., the culture of medical education often involves residents working in parallel with supervisors, with discrete periods of direct observation that progressively decline over the course of training as residents become more independent.Reference Watling, Driessen and van der Vleuten38
Watling et al. also identified that medical trainees, as compared to musicians, seek out clinical excellence in their supervisors as a marker of their credibility, rather than excellence in instruction or coaching.Reference Watling, Driessen and van der Vleuten38 Reconciling these previous findings with the findings in our study that role modelling plays an important role in the development of skills in supervision and assessment will be an important area of further research.
Limitations
This small-scale study captured the opinions of emergency physicians and trainees at only two institutions, limiting its generalizability. Our convenience sampling technique resulted in our interviewing only two CCFP(EM) residents and only four residents from McMaster; this limits our ability to draw conclusions from the CCFP(EM) cohort and from McMaster. However, despite our small sample size at McMaster, we still were able to reach thematic saturation.
Our data sets were primarily coded by a single reviewer (SK), which could introduce bias into the themes. We attempted to mitigate this in the following ways: 1) The principal investigator and co-author JZ coded a minimum number of transcripts, 2) the coding done by SK was reviewed by the principal investigator and co-author JZ for accuracy, and 3) all three authors were involved in discussions on thematic generation. Lastly, whereas the principal investigator is an EM physician and co-author SK is an EM resident, co-investigator JZ is not and therefore assisted with engaging KP and SK in self-reflexivity to limit potential bias.
CONCLUSIONS
The transition from residency to independent practice is a challenging time for any physician. Our study sought to identify how prepared senior EM residents and new-to-practice physicians felt when supervising and assessing trainees. Our findings suggest that both groups feel they do not have adequate training in these areas. Our study also found that managing the competing priorities of an ED and providing negative feedback also lack formal training and need to be addressed in future curricula. We hope that our results will provide support to educators and spur the development of dedicated curricula to address these deficiencies.
Financial support
This work was generously supported by the Canadian Association of Emergency Physicians (CAEP) Junior Investigator Research Grant Program (Control ID: 2661311).
Competing interests
None declared.
