Hostname: page-component-848d4c4894-m9kch Total loading time: 0 Render date: 2024-05-27T22:05:44.233Z Has data issue: false hasContentIssue false

The competency-based medical education evolution of Canadian emergency medicine specialist training

Published online by Cambridge University Press:  30 September 2019

Jonathan Sherbino*
Division of Emergency Medicine, Department of Medicine, McMaster Education Research, Innovation and Theory (MERIT) Program, McMaster University, Hamilton, ON
Glen Bandiera
Division of Emergency Medicine, Department of Medicine, PGME, University of Toronto, Toronto, ON
Ken Doyle
Department of Emergency Medicine, McGill University, Montreal, QC
Jason R. Frank
Department of Emergency Medicine, University of Ottawa, Ottawa, ON
Brian R. Holroyd
Department of Emergency Medicine, University of Alberta, Edmonton, AB
Gord Jones
Department of Emergency Medicine, Queen's University, Kingston, ON
Joanne Norum
Department of Emergency Medicine, University of Alberta, Edmonton, AB
Carolyn Snider
Division of Emergency Medicine, Department of Medicine, University of Toronto, Toronto, ON
Kirk Magee
Department of Emergency Medicine, Dalhousie University, Halifax, NS.
Correspondence to: Dr. Jonathan Sherbino, DBHSC Room 5003, 1280 Main St. West, Hamilton, ONL8P 1H6; Email:


Canadian specialist emergency medicine (EM) residency training is undergoing the most significant transformation in its history. This article describes the rationale, process, and redesign of EM competency-based medical education. The rationale for this evolution in residency education includes 1) improved public trust by increasing transparency of the quality and rigour of residency education, 2) improved fiscal accountability to government and institutions regarding specialist EM training, 3) improved assessment systems to replace poor functioning end-of-rotation assessment reports and overemphasis on high-stakes, end-of-training examinations, and 4) and tailored learning for residents to address individualized needs. A working group with geographic and stakeholder representation convened over a 2-year period. A consensus process for decision-making was used. Four key design features of the new residency education design include 1) specialty EM-specific outcomes to be achieved in residency; 2) designation of four progressive stages of training, linked to required learning experiences and entrustable professional activities to be achieved at each stage; 3) tailored learning that provides residency programs and learner flexibility to adapt to local resources and learner needs; and 4) programmatic assessment that emphasizes systematic, longitudinal assessments from multiple sources, and sampling sentinel abilities. Required future study includes a program evaluation of this complex education intervention to ensure that intended outcomes are achieved and unintended outcomes are identified.



La formation des spécialistes au niveau de la résidence en médecine d'urgence (MU) au Canada est actuellement en voie de connaître la transformation la plus importante de son histoire. Il sera question, dans l'article, de la justification, du processus et de la nouvelle conception de la formation médicale axée sur les compétences en MU. Au nombre des raisons sous-tendant l’évolution de la formation au niveau de la résidence figurent : 1) une plus grande confiance du public à l’égard de cette spécialité grâce à une transparence accrue de la qualité et de la rigueur de la formation; 2) une meilleure reddition de compte fiscale au gouvernement et aux établissements en ce qui concerne la formation des spécialistes en MU; 3) l’établissement de meilleurs systèmes d’évaluation visant à remplacer le fonctionnement inadéquat des rapports d’évaluation en fin de stage et à mettre fin à l'importance exagérée accordée aux examens en fin de formation; 4) un apprentissage adapté aux besoins individuels des résidents. Un groupe de travail, composé de membres provenant de différentes régions et représentant différentes parties, s'est réuni un certain nombre de fois, sur une période de deux ans, et a convenu d'un processus de consensus pour les prises de décision. Le nouveau programme de formation au niveau de la résidence comprend quatre éléments clés : 1) l'atteinte de résultats précis en MU, en tant que spécialité, au cours de la résidence; 2) l’élaboration de quatre stages progressifs de formation, en lien avec les expériences nécessaires d'apprentissage et les activités professionnelles confiables quant aux compétences à acquérir à chaque stage; 3) un apprentissage personnalisé, fondé sur la flexibilité des programmes de résidence et la souplesse des apprenants, afin de s'adapter aux ressources locales et aux besoins des stagiaires; 4) une évaluation « programmatique » qui met l'accent sur les évaluations systématiques et longitudinales provenant de différentes sources, et sur les capacités sentinelles. Enfin, il faudra procéder ultérieurement, dans le cadre d'une étude, à une évaluation du programme afin de s'assurer de l'atteinte des objectifs visés et de relever l’émergence de résultats collatéraux non recherchés.

CAEP Paper
Copyright © Canadian Association of Emergency Physicians 2019


The Royal College of Physicians and Surgeons of Canada certified its first specialist emergency physician (EP) in 1983.Reference Elyas and Stapleton1 Over the past 4 decades, the specialty has matured with 14 residency training programs across Canada offered in French or English. The design of specialist emergency medicine (EM) training programs followed a model organized by time, manifested in a set of standardized clinical rotations that exposed trainees to (assumed) clinical content. All of this changed on July 1, 2018.

On this date, all Canadian EM specialist training programs migrated to a competency-based medical education model. This transition was part of a 7-year, system-wide change to all Canadian specialty training programs.2 EM was in the second cohort of Royal College disciplines, following an initial pilot of two specialties.

A form of outcomes-based education, competency-based medical education focuses on the abilities required of specialist EPs for successful entry into practice.Reference Carraccio and Englander3,Reference Frenk, Chen and Bhutta4 Rather than a simple recitation of relevant knowledge, competency-based medical education requires the demonstration of relevant competencies, where a competency is “an observable ability of a health professional, integrating multiple components such as knowledge, skills, values, and attitudes. Since competencies are observable, they can be assessed to ensure their acquisition. Competencies can be assembled like building blocks to facilitate progressive development.”Reference Frank, Snell and Cate5

The purpose of this study was to describe the rationale, process, and redesign of the most significant change in specialist EM training since the inception of the specialty in Canada.


There are four compelling arguments supporting the transition of specialist EM training to a competency-based medical education model.Reference Holmboe, Sherbino, Englander, Snell and Frank6 Firstly, in exchange for serving the public good, the profession enjoys self-regulation via the determination of the standards for practice certification and accreditation for residency programs. However, the public trust is eroding. The CEO of the Canadian Medical Protective Association indicates that “college matters and hospital matters – based on complaints either attributed to the physician's behaviour or to the competency of the physician – have risen globally…We do track … all articles pertaining to doctors and the vast, vast majority of them are negative…”Reference Colin7

Accountability to the public requires transparency and rigour in training programs. Specific, systematic articulation of all of the specialist EM competencies and the associated assessment processes provides the public reassurance that Canadian specialist EPs are competent to provide emergency care to any patient with any acute health concern on the first day after graduation.8 The transformation of EM programs to a competency-based medical education model is an attempt to provide such transparency and rigour.

Secondly, in an age of fiscal accountability, government seeks evidence of healthcare resource stewardship. Justification for 5 years of specialty training has been supported by a collaborative working group that included representation from the Canadian Association of Emergency Physicians (CAEP), the College of Family Physicians of Canada (CFPC), and the Royal College of Physicians and Surgeons of Canada. “[Specialist EP] graduates, due to their longer dedicated training in EM, have additional abilities, including but not limited to: critical care experience, advanced resuscitation skills, advanced toxicology, pediatric EM, research and a higher level of experience with the management of critically ill patients…. Due to curriculum differences, [specialist EPs] have additional training in some areas, including EM administration, research, and pre-hospital care.”8 The competency-based medical education transition maintains the high specialist EM standards of practice, while developing education system efficiencies. The complete re-examination of all components of EM specialist training mapped to the complexities of contemporary EM practice identified inefficiencies. Moreover, traditional curricula organized around time, where the calendar dictates training progression and allows only a single date of exit from training, is inefficient. Competency-based medical education allows learners to progress when they demonstrate the ability to do so. While most learners will require 5 years to demonstrate competence, competency-based medical education opens up the possibility of shorter training time. Eight years of training data from a University of Toronto pilot orthopedic surgery competency-based medical education residency training program identified a 15 percent decrease in required training time.Reference Nousiainen, Mironova and Hynes9

Thirdly, faculty struggle with ineffective assessment systems that cannot provide useful data on resident performance to assist with educational diagnoses and prescriptions or effective curriculum planning. Traditional curricula over-emphasize end-of-training high stakes knowledge exams with little attention to authentic performance. A competency-based medical education model grounds assessment at the bedside, using multiple, low stakes, biopsies of longitudinal performance mapped to a mastery learning model.Reference Iobst, Sherbino and Ten10 Immediate feedback on performance allows the resident to efficiently acquire skills, while aggregation of hundreds of these data points guide global resident performance decisions, the functioning of rotations and curricula.

Fourthly, adopting a competency-based medical education model serves resident needs. EM is highly competitive in Canada with less than 50 percent capacity to accept medical students who rank specialist EM as their first choice for residency. Successful applicants have a track record of academic success and effective learning processes. Yet, traditional EM curricula assume that residents learn at the same pace, where advancement proceeds as a cohort dependent upon the crude determination of the change of the academic year. EM resident learning trajectories indicate that these assumptions of uniform progress are false.Reference Chan, Sherbino and Mercuri11 By organizing training around the needs of the individual, mastery learning models allow tailored education and variable progress for residents, optimizing their time and limiting calendar watching.

Further, by clearly articulating the progressive stages of training, competency-based medical education addresses the differences in abilities among entering residents. The transition to discipline (i.e., residency) stage ensures that fundamental competencies are acquired before more complex ones. Similarly, the transition to practice stage allows for meaningful acquisition of the tacit knowledge required of practice.


Two years prior to launch, a working group was convened to develop the Canadian specialist EM competency-based medical education model. Membership included the Royal College EM Specialty Committee with democratic representation from the five established regions (Atlantic; Quebec; Ontario and Nunavut; Manitoba and Saskatchewan; Alberta, British Columbia, Yukon, and North West Territories), all residency program directors, representatives from pediatric EM, critical care, clinical pharmacology and toxicology, the national specialty society (CAEP), plus a chair and vice chair. Additional representation included all assistant residency program directors, CFPC representation, two resident representatives, and an educational consultant supplied by the Royal College. Project support was provided by the Royal College, including an administrator and policy writer.

A consensus process was adopted with all final decisions collectively endorsed by every member of the committee. Standard Royal College policies regarding the structure, governance, and accreditation process for residency training were observed. However, the design and content of the competency-based medical education curriculum were constructed de novo. No working group member identified a conflict that influenced the consensus process.

Three in-person retreats (each 3 days in length) were held over an 18-month period. Four subcommittees were struck to address specific issues related to the four stages of residency training. Bimonthly webinar conference calls were held with the whole special working group, and monthly webinar conference calls were held for each subcommittee during periods of intense deadlines. Draft policies and memos were securely distributed for revision and comment electronically. Document version control was maintained by administrative staff.

Queen's University's specialty EM residency program, through a special Royal College approval process, launched their competency-based medical education curriculum on July 1, 2017. A planned iterative program evaluation was conducted at 3, 6, and 12 months to provide pilot data informing the national launch.Reference Hall, Rich and Dagnone12 A change management initiative for residency program directors and EM faculty was developed in the 9 months leading up to the 2018 launch. Educational resources, project Gantt charts, and faculty development resources were supplied. Topics covered included the rationale for change, principles of competency-based medical education, new stages of training, new assessment processes (e.g., entrustable professional activities, competence committees, exam timing), and the new area of concentrated expertise, among others.

Prior to launch, a meeting of all postgraduate medical education deans representing every medical school with a specialty EM residency training program convened to indicate the readiness of their home program to launch. EM was the only discipline with unanimous university-level endorsement among the first three cohorts of transition to competency-based medical education, indicating the EM residency programs had appropriate resources, faculty development, and change management strategies to support the transition.


There are no commonly agreed upon principles of competency-based medical education published in the literature to ensure fidelity of implementation.Reference Van Melle, Frank, Holmboe, Stockley and Sherbino13 The working group adopted the following four principles to ensure common implementation across all 14 programs.

Outcomes defined

The CanMEDS 2015 FrameworkReference Frank, Snell and Sherbino14 is a generic physician competency framework adopted in more than 43 educational jurisdictions around the world.15 While some competencies are common across disciplines, it was essential to tailor the framework to the required competencies of a Canadian specialist EP. Box 1 includes the definition of a specialist EP. Supplemental Appendix 1 contains the key and enabling competencies that describe the practice of a specialist EP.

Box 1. Definition of a Canadian specialist emergency physician

“A Royal College Emergency Medicine specialist is a physician who uses highly developed clinical reasoning skills to care for patients with acute and often undifferentiated health problems, across a broad spectrum of illnesses and injury in all age groups, frequently before complete clinical or diagnostic information is available. An Emergency Medicine specialist is able to determine which conditions require immediate care and which conditions can be investigated and managed in a different setting. They assume a consultant's role in the specialty, providing comprehensive emergency adult and pediatric care in academic/teaching, community, or regional hospital settings.

Royal College EM specialists are resuscitation specialists for patients with undifferentiated presentations, possessing expertise in the anatomy, physiology, pathophysiology, pharmacology, toxicology, and management of all acute presentations. EM specialists use their comprehensive knowledge of related fields at the interface between emergency care provision and the healthcare system, including toxicology, traumatology, prehospital care, environmental medicine, and disaster medicine.

Royal College EM specialists are an academic and community resource, providing advanced clinical patient care; support to other physicians and healthcare professionals in an emergency setting; leadership in the administration of emergency departments, emergency medical systems, healthcare institutions, and related programs; and the conduct of relevant research and education with the goal of advancing knowledge and improving individual and/or community health outcomes.”Reference Dreyfus and Dreyfus16

The working group achieved consensus and endorsed these EM competencies. They serve as the destination to guide trainees through residency. For program directors and faculty members teaching in residency programs, the EM competencies provide a common mental model. Using this framework, a residency training program can be designed to achieve these competencies.

For society and government, this document serves as a specific articulation of the practice of a specialist EP. This is of particular importance in a period where the term emergency physician is often used as a function of geography (i.e., the work environment of the physician).8 Recognizing the tremendous heterogeneity between emergency departments and patient populations across Canada, using competency to define a physician allows for greater clarity with respect to health human resources needs.

Staged progression

With competency-based medical education, time is no longer the determinant of resident progression, where prescribed dwell times (e.g., rotation attendance) serve as the surrogate for the acquisition of experience. Rather, time is used as an independent variable, where a resident actively navigates at an individualized rate based on their demonstration of advancing ability.Reference Iobst, Sherbino and Ten10 While the designation of postgraduate years 1 to 5 has been preserved, this nomenclature is only used by provincial governments with respect to funding positions and resident salaries. The new educational design describes four stages: transition to discipline (i.e., residency), foundations, core, and transition to practice (Figure 1).

Figure 1. Stages of training.

Each of these stages includes specific milestones that describe the knowledge, skills, and attitudes to be acquired at this stage. (See Supplemental Appendix 2.) Specific assessments are tied to each stage. Advancement from one stage to the next requires the demonstration of the abilities specific to each stage.

This developmental design allows the sequencing of foundational to advanced abilities.Reference Ericsson17,Reference Chan, Sebok-Syer and Thoma18 Regular checkpoints requiring a comprehensive review of performance ensure that trainees on failing learning trajectories are identified early with the possibility of less intensive remediation of performance. For trainees on excelling learning trajectories, an earlier increase in responsibility accelerates residency completion.Reference Harris, Bhanji and Topps19

For both trainees and faculty, the stages of learning make transparent the progression of ability required to become a specialist EP. In essence, an explicit educational roadmap is provided. This common mental model relegates expectations of junior versus senior performance that was inconsistent between trainees, faculty, and programs.

Tailored learning

The traditional model of specialist EM training used a common curriculum among all residency programs, where specific rotations, often in the service of other specialties, were required. The new competency-based medical education model returns flexibility to learners and programs. Rather than prescribing a specific rotation and specific specialty service, the new model indicates the type of patient experiences required for the resident to develop competence in managing key EM presentations. The competence by design model does specify training experiences, but these are framed as exposure to specific patient subpopulations, rather than specific discipline rotations. For example, a prescribed 4-week rotation on an orthopedic service has been replaced with a requirement for a resident to develop competence in managing patients with acute musculoskeletal injuries. This allows a program director to effectively design a locally based curriculum.

For residents, an emphasis on tailored learning is reflected in the lack of specific dwell time with various required patient experiences. This introduces some flexibility into individualized training. Of course, institutional (e.g., university, hospital) logistics must be negotiated. (For example, most required training experiences will be structured in increments of 4 weeks to smooth transitions in service requirements.) Residents play a key health human resource in the delivery of healthcare in our teaching institutions. Also, the funding envelope for training is specific to a 5-year period. However, within this envelope of funding, residents, in conjunction with their program directors, have the ability to tailor more experience with one patient population and less with another, depending on their specific needs.

Finally, a specific portion of training is dedicated to an area of concentrated expertise. This training experience allows a resident to pursue deeper expertise in a clinical or academic domain relevant to his or her future practice as a specialist EP. This required learning experience is unique to EM. It ensures that the specialty is continually advancing clinical, administrative, research, and educational domains. In turn, this improves the quality of Canadian emergency health systems.

Supplemental Appendix 3 describes the clinical training experiences, simulated training experiences, and topics requiring formal instruction, mapped to each stage of training.

Programmatic assessment

Programmatic assessment is the systematic, longitudinal acquisition of multiple assessments from multiple assessors addressing multiple competencies to inform a group-based global judgement of a trainee's progression towards a competent practice.Reference Sherbino, Bandiera and Frank20 Programmatic assessment emphasizes authentic performance. For trainees and frontline faculty, programmatic assessment will be the most notable change in the transition to a specialist EM competency-based medical education model.

Traditionally, residency training programs rely heavily on an end-of-training high stakes knowledge exam, ad hoc in-training oral and written exams, and poorly designed and executed end-of-rotation reports (e.g., ITERs).Reference Chan and Sherbino21 EM is well situated to adopt programmatic assessment, because many programs currently employ direct-observation assessment instruments that capture authentic performance.Reference Englander, Frank and Carraccio22

The two most notable additions to assessment processes are entrustable professional activities and competence committees. Entrustable professional activites are key tasks (i.e., work) of the discipline that can be observed and entrusted to a trainee and are assessed via direct observation instruments.Reference Acai, Li, Sherbino and Chan23 A renewed emphasis on direct observation of authentic performance is a key principle of programmatic assessment. Entrustable professional activities describe an element of work of specialist EP practice, integrating competencies from multiple CanMEDS roles. Each activity is supported by 10 to 15 competencies (i.e., milestones). Entrustable professional activities are mapped to a specific stage of training. There are 28 EM entrustable professional activities, representing critical and sentinel, but not exhaustive, specialist EP work. (See Supplemental Appendix 4.)

Unique to entrustable professional activities is an entrustment scale (Table 1) anchored by descriptors of the degree of supervision required for completion. Complementary to the entrustment score is a required narrative included for the completion of each entrustable professional activity. On average, one entrustable professional activity is completed during each emergency department shift. Ideally, the entrustable professional activity to be observed is negotiated between resident and observer at the start of the shift and completed midway through the shift. Once complete, the observer should share his or her observations and entrustment score with the resident. Both the entrustment scale and the narrative feedback are recorded via a web-based platform. The Royal College e-portfolio or university-specific proprietary platforms are used to facilitate the aggregation of data and the provision of personalized assessment dashboards available to residents and residency programs.Reference Chan, Sherbino and Mercuri11

Table 1. Entrustment scaleReference Li, Sherbino and Chan24

While entrustable professional activities are a key part of the resident's overall assessment portfolio, they are complemented by in-training exams and other locally implemented assessments to provide a multi-modality perspective on resident progress. Included in Supplemental Appendix 4 are nine additional special assessment instruments.

The data assembled in each resident portfolio, including entrustable professional activity assessments, special assessments, in-training exams, and other local instruments, are adjudicated on a regular basis by a competence committee.Reference Hauer, Chesluk and Iobst25 This committee, comprising the program director and faculty, and, in some cases, residents or non-physician community members, depending on local practice, systematically reviews the aggregated portfolio data to make a group judgement about a resident's performance towards attaining stage-specific and finally global competence.Reference Hauer, Cate and Boscardin26,Reference Cooney, Chan and Gottlieb27 The competence committee is not bound by accounting practice, where aggregated mean scores drive a decision. Rather, emulating scientific peer review or jurisprudence, competence committees weigh all evidence to determine a summary decision. Competence committees require deliberate structure and systematic practices to avoid group think and mitigate bias.Reference Norman, Norcini and Bordage28 Competence committees indicate when an entrustable professional activity is achieved, and a resident can progress to the next stage. Most notably, the final step in completing specialist EM residency training requires the competence committee to endorse that the resident is ready for practice.

The high-stakes written and oral exam is no longer at the end of training; it is situated between the core and transition to practice stages. Placing the exam at the end of training has led to inappropriate social stressors on trainees, absenteeism from clinical and educational responsibilities, and a de-emphasis on the clinical practice of EM with an overt attention to rare trivia. The final stage of training, transition to practice, requires indirect supervision of the residents to allow them to demonstrate competence in managing a section of an emergency department independently, including teaching junior learners and attending to relevant clinical administrative issues.


The designed versus the delivered versus the experienced curriculum is not a direct correlation. The conceptual frameworks and theories informing this new model of specialist EM training have many assumptions.Reference Van Melle, Gruppen and Holmboe29 Additionally, many unintended consequences of certain design features (including gaming of the system) have yet to be realized. Moreover, while competence by design is informed by education theory, there are a number of untested assumptions. For example, will the cost savings of a more efficient residency design offset the costs required for increased education infrastructure, simulation training, and so forth? Will entrustable professional activity assessments adequately demonstrate global specialist EP competence? Also, will competence by design improve the quality of education and, more significantly, the care of emergency patients? These representative questions are only a portion of the assumptions that require testing.30 Future steps for EM educators and researchers include program evaluation of local and national curricula. Challenging the assumptions that inform this design is critical to refine future iterations and understand areas of educational success and failure.Reference Gofton, Dudek, Wood, Balaa and HAmstra31


The transition of Canadian specialist EM residency training to a competency-based medical education model is not a perfect design. The future of EM education will see refinements and new evolutions in curricula. However, competency-based EM residency education attempts to maintain the social contract, steward government resources, facilitate best education practices for faculty, and improve learning efficiency for residents. The refinement of the competencies required for specialist EM practice, the implementation of stages of residency with associated educational roadmaps, the adoption of tailored learning, and the design of programmatic assessment mark improvements upon traditional residency education. The lessons we learn from this national transition will inform the next generation of specialist EM physicians.

Competing interests

None declared.

Supplementary material

The supplementary material for this article can be found at


The authors thank Brian Au, Isabelle Bertrand, Stephen Choi, Warren Cheung, Brian Chung, Marie-Maud Couture, Jaelyn Caudle, Damon Dagnone, Matt Davis, Pierre Désaulniers, Sandy Dong, Cheryl French, Emerson Genius, David Ha, Carine Haggar, Andrew Hall, April Kam, Janet MacIntyre, Jill McEwen, Sarah McPherson, Nazanin Meshkat, Christine Meyers, Avik Nath, Gina Neto, Wesley Palatnick, Alim Pardhan, Sarah Reid, Helen Ross, Robert Sedran, Samir Sherif, Marco Sivilotti, Brent Thoma, Lisa Thurgur, Caroline Tyson, Kristen Weersink, David White, Rob Woods, and Joel Yaphe for their contributions toward this study.



1.Elyas, R. The birth of a new specialty: The history of emergency medicine in Canada. In: The Proceedings of the 16th Annual History of Medicine Days (ed. Stapleton, M). Calgary: University of Calgary; 2007, 277–86.Google Scholar
2.Royal College of Physicians and Surgeons of Canada. Competence by Design (CBD): moving towards competency-based medical education; 2015. Available at: (accessed March 4, 2019).Google Scholar
3.Carraccio, CL, Englander, R. From Flexner to competencies. Acad Med 2013;88(8):1067–73.10.1097/ACM.0b013e318299396fCrossRefGoogle ScholarPubMed
4.Frenk, J, Chen, L, Bhutta, ZA, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet 2010;376(9756):1923–58.CrossRefGoogle Scholar
5.Frank, JR, Snell, LS, Cate, OT, et al. Competency-based medical education: theory to practice. Med Teach 2010;32(8):638–45.CrossRefGoogle ScholarPubMed
6.Holmboe, ES, Sherbino, J, Englander, R, Snell, L, Frank, JR. A call to action: the controversy of and rationale for competency-based medical education. Med Teach 2017;39(6):574–81.CrossRefGoogle Scholar
7.Colin, L. With college and hospital actions soaring, the CMPA recalibrates. Med Post 2017. Revised April 18, 2017. Available at: (accessed March 12, 2019).Google Scholar
8.Collaborative Working Group on the Future of Emergency Medicine in Canada. Final Report of the Collaborative Working Group on the Future of Emergency Medicine in Canada; 2016. Available at: (accessed March 4, 2019).Google Scholar
9.Nousiainen, MT, Mironova, P, Hynes, M, et al. Eight-year outcomes of a competency-based residency training program in orthopedic surgery. Med Teach 2018;40(10):1042–54.CrossRefGoogle ScholarPubMed
10.Iobst, WF, Sherbino, J, Ten, Cate O, et al. Competency-based medical education in postgraduate medical education. Med Teach 2010;32(8):651–6.CrossRefGoogle ScholarPubMed
11.Chan, TM, Sherbino, J, Mercuri, M. Nuance and noise: lessons learned from longitudinal aggregated assessment data. J Grad Med Educ 2017;9(6):724–9.CrossRefGoogle ScholarPubMed
12.Hall, AK, Rich, J, Dagnone, J, et al. P061: implementing CBME in emergency medicine: lessons learned from the first 6 months of transition at Queen's University. CJEM 2018;20(S1):S78.CrossRefGoogle Scholar
13.Van Melle, E, Frank, JR, Holmboe, ES, Stockley, D, Sherbino, J. A core components framework for evaluating implementation of competency-based medical education programs. Acad Med 2019;epub.CrossRefGoogle ScholarPubMed
14.Frank, JR, Snell, L, Sherbino, J, eds. CanMEDS 2015 Physician Competency Framework. Royal College of Physicians and Surgeons of Canada; 2015. Available at: (accessed March 4, 2019).Google Scholar
15.Personal Correspondence. C. Abbot. Manager of CanMEDS, Office of Specialty Education, Royal College of Physicians and Surgeons of Canada. Contacted February 15, 2019.Google Scholar
16.Dreyfus, HL, Dreyfus, SE. Mind over machine. New York: Simon & Schuster; 1986.Google Scholar
17.Ericsson, KA. Deliberate practice and acquisition of expert performance: a general overview. Acad Emerg Med 2008;15(11):988–94.10.1111/j.1553-2712.2008.00227.xCrossRefGoogle ScholarPubMed
18.Chan, T, Sebok-Syer, S, Thoma, B, et al. Learning analytics in medical education assessment: the past, the present, and the future. AEM Educ Train 2018;2(2):178–87.CrossRefGoogle ScholarPubMed
19.Harris, P, Bhanji, F, Topps, M, et al. Evolving concepts of assessment in a competency-based world. Med Teach 2017;39(6):603–8.CrossRefGoogle Scholar
20.Sherbino, J, Bandiera, G, Frank, JR. Assessing competence in emergency medicine trainees: an overview of effective methodologies. CJEM 2008;10(4):365–71.CrossRefGoogle Scholar
21.Chan, T, Sherbino, J, McMAP Collaborators. The McMaster Modular Assessment Program (McMAP). Acad Med 2015;90(7):900–5.CrossRefGoogle Scholar
22.Englander, R, Frank, JR, Carraccio, C, et al. Toward a shared language for competency-based medical education. Med Teach 2017;39(6):582–7.CrossRefGoogle Scholar
23.Acai, A, Li, S-A, Sherbino, J, Chan, TM. Attending emergency physicians’ perceptions of a programmatic workplace-based assessment system: the McMaster Modular Assessment Program (McMAP). Teach Learn Med 2019;epub:1–11.CrossRefGoogle Scholar
24.Li, S-A, Sherbino, J, Chan, TM. McMaster Modular Assessment Program (McMAP) through the years: residents’ experience with an evolving feedback culture over a 3-year period. AEM Educ Train 2017;1(1):514.CrossRefGoogle Scholar
25.Hauer, KE, Chesluk, B, Iobst, W, et al. Reviewing residents’ competence. Acad Med 2015;90(8):1084–92.CrossRefGoogle ScholarPubMed
26.Hauer, KE, Cate, O Ten, Boscardin, CK, et al. Ensuring resident competence: a narrative review of the literature on group decision making to inform the work of clinical competency committees. J Grad Med Educ 2016;8(2):156–64.10.4300/JGME-D-15-00144.1CrossRefGoogle Scholar
27.Cooney, R, Chan, T, Gottlieb, M, et al. Academic Primer Series: key papers about competency-based medical education. West J Emerg Med 2017;18(4):713–20.10.5811/westjem.2017.3.33409CrossRefGoogle ScholarPubMed
28.Norman, G, Norcini, J, Bordage, G. Competency-based education: milestones or millstones? J Grad Med Educ 2014;6(1):16.CrossRefGoogle ScholarPubMed
29.Van Melle, E, Gruppen, L, Holmboe, ES, et al. Using contribution analysis to evaluate competency-based medical education programs: it's all about rigor in thinking. Acad Med 2017;92(6):752–8.CrossRefGoogle ScholarPubMed
30.Royal College of Physicians and Surgeons of Canada. Emergency medicine competencies; 2018. Available at: (accessed March 4, 2019).Google Scholar
31.Gofton, WT, Dudek, NL, Wood, TJ, Balaa, F, HAmstra, SJ. The Ottawa surgical competency operating room evaluation ()-SCORE): a tool to assess surgical competence. Acad Med 2012;87(10):1401–7.CrossRefGoogle ScholarPubMed
Figure 0

Figure 1. Stages of training.

Figure 1

Table 1. Entrustment scale24

Supplementary material: PDF

Sherbino et al. supplementary material

Sherbino et al. supplementary material 1

Download Sherbino et al. supplementary material(PDF)
PDF 343.9 KB
Supplementary material: PDF

Sherbino et al. supplementary material

Sherbino et al. supplementary material 2

Download Sherbino et al. supplementary material(PDF)
PDF 184.3 KB
Supplementary material: PDF

Sherbino et al. supplementary material

Sherbino et al. supplementary material 3

Download Sherbino et al. supplementary material(PDF)
PDF 278.3 KB
Supplementary material: PDF

Sherbino et al. supplementary material

Sherbino et al. supplementary material 4

Download Sherbino et al. supplementary material(PDF)
PDF 84.6 KB