CLINICIAN'S CAPSULE
What is known about the topic?
Emergency medicine faculty attitudes regarding Competence by Design have not been investigated.
What did this study ask?
We surveyed baseline perceptions about Competence by Design, attitudes towards implementation, perceived/prompted/unperceived faculty development needs.
What did this study find?
Emergency medicine education is well-situated to transition to Competence by Design given clinicians’ positive attitudes towards feedback, direct observation, and workplace-based assessments.
Why does this study matter to clinicians?
Education leaders should concentrate on supervision and observation skills but also address concerns regarding how Competence by Design can best complement/augment clinical care.
INTRODUCTION
Competency-based medical education represents a major shift from the traditional educational model in postgraduate medical education towards one that is less time-dependent and hinges more on the achievement of the necessary abilities to practise emergency medicine (EM).Reference Sherbino, Snell and Dath1–Reference Hodges10 Competency-based medical education seeks to remediate some of the traditional training system's perceived flaws by tailoring teaching and assessment to the learner's needs and desired outcomes and ensuring that progression in training is based on achievement of clearly defined competencies required for practice.Reference Carraccio, Englander and Melle11,Reference Cooney, Chan and Gottlieb12 Multiple factors have led to the rapid adoption of competency-based medical education by many national bodies: the need to demonstrate sustained physician competence in an age of increased accountability to the public, variation in the quality of training of physicians in the current system, the prevalence of the “failure to fail” culture in medical education, and the need to ensure smoother transitions from training into clinical practice.Reference Carraccio and Englander4,Reference Cooney, Chan and Gottlieb12–15
Competence by Design was developed by the Royal College of Physicians and Surgeons of Canada (RCPSC) as a change initiative to transform specialty medical education to a competency-based medical education model of teaching and assessment. EM is among six specialties that transitioned to Competence by Design in July 2018. The RCPSC Competence by Design model maintains the current 5-year duration of training but blends in competency-based medical education concepts such as refreshed outcomes required of EM graduates, new stages of training (transition to discipline, foundations, core, transition to practice) with associated milestones, tailored learning approaches, and programmatic assessment.
For the frontline clinical teacher, competency-based medical education implies increased direct observation of trainees’ clinical performance and assessment via entrustable professional activities (EPAs) – stage-specific, key tasks of the discipline.16 Though the EPA construct is new, the EM faculty community is well-versed in direct observation and assessment. Previous research regarding EM faculty perceptions of the 2005 CanMEDS framework (a paradigm that defines high-level competencies of specialist practice) showed that frontline clinicians found the structure helpful in organizing feedback and increasing awareness of the breadth of competencies expected of an independent practitioner.Reference Bandiera and Lendrum17,Reference Frank, Snell and Sherbino18 However, they struggled with the abstract nature of the CanMEDS roles and key competencies.Reference Bandiera and Lendrum17 Daily encounter cards have been used in EM to provide timely feedback to learners around all CanMEDS competencies, although one drawback was a tendency towards leniency bias.Reference Bandiera and Lendrum19,Reference Sherbino, Kulasegaram, Worster and Norman20 Though these initiatives suggest that EM faculty are generally well-prepared for observation and feedback, no Canadian studies to date looked at frontline faculty opinions on how these concepts will change during the transition to Competence by Design.
Given that successful implementation depends on a full understanding of the target audience, we designed a multi-centre needs assessment survey to determine the specific Canadian EM faculty and senior trainee development requirements during the transition to Competence by Design.
METHODS
A multi-site, cross-sectional digital survey was conducted between February and April 2018 with a sample of EM faculty and senior residents across RCPSC EM programs in Canada. Senior residents were included in this survey because they will be potential supervisors for the first Competence by Design cohort.
Survey design
The survey was modelled on a previously described multi-phase needs assessment.Reference Chan, Jo and Shih21 Survey domains included baseline perceptions about Competence by Design, attitudes towards implementation, and perceived/prompted and unperceived faculty development needs. The survey was initially developed by members of our team (AS, JH, TC) and subsequently revised based on expert feedback (JS). The needs assessment (Appendix A) was created on Google Forms (Mountainview, CA, USA).
The first section of the survey explored frontline clinicians’ practice and perceptions around feedback and workplace-based assessments (WBAs). Respondents were asked to reflect on the current state of feedback within the training environment and identify barriers to optimal feedback delivery. We also asked about their pre-existing perceptions of Competence by Design.
Secondly, we asked respondents to select topics that they believed would aid in their delivery of Competence by Design from a provided list and invited them to free-text additional suggestions. We also sought to harness the power of stories to determine what prompted needs might emerge from respondents’ descriptions of difficult scenarios. This technique has previously been used in other needs assessment studies.Reference Chan, Jo and Shih21,Reference Tseng, Jo and Shih22
Finally, the third part sought to use multiple-choice quiz questions (Appendix B) to examine what types of terminology and competency-based medical education-associated language are confusing as a means to determine unperceived educational needs.
This study was reviewed and received exemption by the Hamilton Integrated Research Ethics Board.
Participants
An invitation with the link to the online survey was distributed by email to all of the RCPSC program directors (PDs) via the EM specialty committee electronic mailing list. PDs were invited to nominate local faculty members and senior residents who represented an adequate cross-sectional sampling of EM clinician teachers in the pre-, early-, mid-, and late-career phase. We did not restrict the number of nominees from PDs. Nominees were approached by email; a link to the needs assessment was provided. Nominated individuals were contacted using a modified Dillman technique (three times, approximately 3 weeks apart) to ensure maximal completion compliance.Reference Dilman23
Analysis
We used Microsoft Excel (Redmond, WA, USA) to calculate simple descriptive statistics of our survey responses.
RESULTS
Between February and April 2018, 47 participants (40 faculty, 7 residents) from 11 Canadian schools with EM postgraduate training programs completed the needs assessment survey (58.8% response rate, based on the original 80 nominated individuals). We lacked full participation of francophone and Atlantic sites. All participants (100%) trained in Canada. Most (66.0%) worked full-time at an academic centre. A description of the demographic data for responding participants is displayed in Table 1.
Respondent demographics
Current practice
Half (55.3%) of the respondents felt they were currently providing high-quality feedback. A significant number of participants felt they did not provide adequate feedback (23.4%) or that the feedback provided was not always honest (8.5%). Most (89.4%) respondents reported that meaningful learner feedback should be provided on each shift with the additional 10.6% reporting it should be provided weekly (every three to four shifts). Additionally, a strong majority (76.6%) of respondents indicated that they had received teacher-training on providing feedback.
Barriers to feedback and WBAs
Because Competence by Design emphasizes the use of WBAs (in the form of EPAs), it was important to explore the participants’ perceptions around existing barriers to providing adequate feedback in the workplace. The top five barriers identified by respondents when providing feedback in the clinical setting were time constraints (80.4%), perceived learner disinterest (43.5%), fear of assessment repercussions (41.3%), lack of training (28.3%), and difficulty giving negative feedback (13.0%).
Perceptions of Competence by Design impact
Respondents generally indicated they felt that the transition to Competence by Design would have no impact on patient care (44.7%). A minority (17.0%) perceived it might have a negative impact, whereas another group (12.8%) felt that it might have a positive impact. The reasons cited by those participants who felt patient care would be negatively affected were that direct observation, feedback, and documentation of WBAs will slow down patient flow. Those who felt Competence by Design would improve patient care believed that a more direct observation will result in safer care, because more feedback might result in more competent trainees. This finding warrants consideration and exploration. Prompted needs from free-text reponses tended to concern logistics of Competence by Design and practicality of carrying out increased direct observation in higher acuity/volume settings.
Respondents’ confidence in the implementation of Competence by Design
Respondents (91.5%) were confident that they could incorporate WBAs into their emergency department (ED) environments. Most respondents reported that direct observation is required for adequate and meaningful feedback (19.1% rated it “critical,” and 78.7% rated it “somewhat” or “very important”). A strong majority (78.7%) indicated that the ED environment offered above-average opportunity for direct observation compared with other clinical environments. A majority (55.3%) of respondents indicated that Competence by Design would lead to improved feedback to trainees, whereas 17.0% indicated that feedback would not be improved. Many (44.7%) reported Competence by Design would lead to a better educational experience (38.3% uncertain; 17.0% indicated that it might have negative effects).
Self-reported needs
We polled participants about which faculty development topics they thought are most valuable as we transition to Competence by Design. Table 2 displays these topics in descending order of importance. Despite the overall perception that current feedback practice is of high quality, a majority (63.8%) of respondents were interested in additional professional development around how to deliver high-quality feedback. Completing assessments after a clinical encounter (57.4%) and principles of resident promotion (55.3%) are made were the next two most requested topics for faculty development. Very few respondents (25.5%) endorsed that they would like more information on Competence by Design terminology.
Priority professional development topics as identified by survey respondents
Note: All other items with one or fewer endorsements were excluded. These included documentation of feedback; how to help residents implement personal learning plans; how direct observations can be done (different techniques and approaches); how to integrate many direct observations into one feedback session; how to help the resident in difficulty; and how to recruit staff to be more excited about CBD and the idea of big change.
Unperceived needs
In our unperceived needs (testing) phase of the needs assessment, only 44.6% of respondents could correctly identify key competency-based medical education terms (Appendix B). This suggests that there is a gap between what respondents prefer and what they may require.
DISCUSSION
As we enter into a new era of postgraduate EM education in Canada, the findings of our study will help inform faculty development. We found that surveyed EM faculty members and senior residents valued direct observation, frequent feedback, and increased assessment opportunities – all of which are crucial for implementing Competence by Design.
Despite concerns about time constraints and concurrently balancing patient care and educational needs, respondents in our survey reported perceptions that the ED offered an above-average opportunity for direct observation and regular feedback. This is consistent with prior literature findings.Reference Bandiera and Lendrum19,Reference Bandiera24 Because the competency-based medical education model hinges on extensive direct observation, this is an important advantage and suggests that teachers are culturally ready to transition to Competence by Design.
Implications for EM faculty developers in the age of competency-based medical education
A strong majority (63.8%) of those surveyed stated that they wanted to receive further faculty development in delivering high-quality feedback, although 53.1% of respondents felt that the residents were already receiving adequate feedback at their centre. Our finding is similar to previous literature showing that a majority of academic EM physicians have a strong interest in improving their educational skillset.Reference Brown, Lang and Patel25 Other key areas of faculty development identified by our survey are concepts unique to Competence by Design (e.g., completing new WBAs and understanding how resident performace is linked to promotion decisions). These areas may not have previously been well-covered in faculty development to date, and may be high-yield when training existing educators and new faculty.
Potential pitfalls
Interestingly, it appears that many respondents are unsure whether educational experiences and better feedback will translate into better patient care, given that 44.7% felt that Competence by Design will not impact patient care. We believe this finding may be because faculty are unsure of how their observations and instructions are linked to promotion decisions. This inconsistency is important to explore further in qualitative studies because it may affect buy-in from faculty and residents when transitioning to the competency-based medical education model.
Our survey also identified other potential problems that could affect transition to Competence by Design. Our respondents were also less unified in their opinions regarding whether Competence by Design would lead to a better educational experience or better feedback and the impact that this paradigm shift may have on patient care. Programs must ensure that the impact of this transition on training and clinical care is monitored and that the results are shared with the faculty, in order to address these concerns.
Faculty and senior residents were unfamiliar of some of their terminology/knowledge gaps regarding Competence by Design fundamentals, but are unlikely to be interested in further development in this area. One potential solution would be to bundle information regarding competency-based medical education concepts and terminology with faculty development sessions that target the areas of higher interest, such as completion of WBAs. Though the poor adoption of official Competence by Design language is not likely to cause big problems for front-line teachers,it may be useful to incorporate these terms in faculty development to ensure a common lexicon.
Time constraints in the clinical environment were identified by over 80% of respondents as the most common barrier in delivering high-quality feedback. This is certainly consistent with prior literature on teaching in EM.Reference Thurgur, Bandiera, Lee and Tiberius26–Reference Chan, Dewark and Sherbino28 Fear of repercussions when providing honest feedback and perceived learner disinterest were also major barriers. If we want Competence by Design to succeed, it is paramount to acknowledge these issues in faculty and resident development sessions and begin finding solutions that will address these concerns.
Next steps
This needs assessment marks the beginning of what is likely to be an ongoing and continuous task. We will need to continually assess faculty and trainee engagement over time, especially when the Competence by Design EPAs become the standard. Previous work in competency-based medical education has suggested that these changes can lead to cultural shiftsReference Li, Sherbino and Chan29 or implementation issues.Reference Dayal, O'Connor, Qadri and Arora30 In Canada, we can gain a glimpse into how faculty and resident perceptions and practices may change by watching “early adopter” programs.Reference Li, Sherbino and Chan29,Reference Chan and Sherbino31–Reference Hall, Rich and Dagnone35 Will we create the desired shift in learning culture towards a growth mindsetReference Dweck36 at an early stage in the trainees’ development (which may decrease the fear of feedback and repercussions)? Or will the constant surveillance and monitoring result merely in increased accounting and documentation without educational benefit? Continued flexibility and innovation will also be imperative; we will need to remain vigilant as we implement and refine our specialty's EPAs.
Limitations
For our study, we focused on surveying the needs of frontline faculty. As such, we deliberately excluded questions regarding other important competency-based medical education concepts such as curriculum reform or competence committees, which would be of interest to educational leaders. Our survey response rate was only 58% with representation gaps from the Atlantic provinces and Québec; this may hinder the generalizability of the study findings. Specific to Québec, we did not have a French language version of the survey, although the strong participation from one French-language program suggests that this may not have been the main barrier.
Our sampling technique of PD nomination may also have introduced some level of bias in our data – likely those who were highly engaged were nominated, and so voices who were objectors or resistors of Competence by Design may not be captured. Given this, even with highly engaged teachers, our findings suggest that there is ample room for growth and faculty development. This survey is also limited in its generalizability towards community teaching sites with fewer trainees, but, in this study, our focus was on faculty in academic centres (which are responsible for the majority of postgraduate training, and they may be impacted the most by Competence by Design).
Finally, the types of responses we were able to elicit were limited by the survey methodology (categorical responses to pre-developed closed questions with limited free-text answers). Interviews using open-ended questions might have revealed more nuances about the participants’ questions and concerns about competency-based medical education or Competence by Design, but this methodology was not chosen because of logistical challenges.
CONCLUSIONS
As we move into the era of competency-based medical education, it is critical to understand front-line educators’ perceptions and attitudes towards this new model of education. This cross-sectional needs assessment survey of faculty and senior residents in Canadian RCPSC EM training programs can provide a roadmap for faculty developers going forward. We will need to attend to threats to Competence by Design implementation, including concerns about effects on patient care and learner education.
Acknowledgements
Thank you to our participants for their honesty and answers. We would also like to acknowledge and thank the program directors from all of the participating sites, and, in particular, Dr. Nazanin Meshkat, who encouraged the team to go forward to complete this scholarly project.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/cem.2019.343
Ethics
This project received ethical exemption from the institutional review board (Hamilton Integrated Research Ethics Board).
Financial support
TC holds a PSI Foundation Knowledge Translation Fellowship Award. TC and JS have also previously received funding from the Royal College of Physicians and Surgeons of Canada for various unrelated projects.
Competing interests
None declared.
Previous presentations
This work has not been previously presented in a national forum, but a white paper with the results from our survey was created and distributed to the participating sites for their own faculty development purposes.
