Interpretation of results

The information gained from this study helps characterize the landscape of the RCEM residency selection process. We identified significant heterogeneity between programs concerning the importance of various elements of the application file where standard deviations approached and often exceeded one point (five-point Likert scale). Although generally being considered of low importance overall, transcripts and dean's letters demonstrated notably high variability, suggesting that they should not be entirely neglected. The understanding of a program director regarding the relationship between the CaRMS ranking of their residents and future performance was also highly variable and possibly relates to an overall lack of regular process/outcome evaluation. Recent calls for regular assessment of process outcomes have been published in Best Practices in Application and Selection (BPAS) for residency programs in Canada.^{Reference Bandiera, Abrahams and Cipolla13,Reference Roger, Epstein and Bandiera14} However, in our study, only 1 of 13 programs had ever performed a formal evaluation of their selection process. As such, outcomes evaluation of Canadian emergency medicine selection processes represents a much needed area of future study. The ultimate goal of selection should be to effectively identify the candidates who most wholly possess the characteristics that a given program desires. These characteristics may vary somewhat between programs but should be made explicit and the processes used to select them internally standardized.^{Reference Bandiera, Abrahams and Cipolla13}

Letters of reference emerged as the most important element of a candidate's application file. Our data would suggest that applicants can improve the perceived quality of their reference letters by performing electives at academic hospitals and by asking for reference letters from emergency medicine preceptors who know them well, rather than targeting a well-known name in the emergency medicine community. The commonly held belief that the “name” of the writer matters above all else has been discredited. Programs can assist candidates with achieving improved familiarity by scheduling students with the same preceptor over multiple shifts. For preceptors, indicating how many students/residents they typically work with and providing a ranking relative to other trainees were helpful for several respondents. Certain respondents also felt that frequently reusing the same letters or ranking multiple applicants in the same high percentile detracted from the quality of a letter.

There is a growing body of evidence suggesting that non-structured or “narrative” letters of reference are highly subjective, inconsistent for the range of qualities described, and demonstrate suboptimal inter-rater agreement among readers.^{Reference Langdorf, Cheaito, Epter and Kazzi17,Reference Fortune18} Including a relative ranking of a candidate, as suggested previously, could help facilitate comparisons between applicants and improve alignment with the BPAS recommendation of striving to incorporate more objective assessment strategies.^{Reference Bandiera, Abrahams and Cipolla13} A potentially more robust solution could be to transition to standardized letters of reference (SLOR). SLORs consist of a set of standardized domains that can be tailored to a given specialty's unique features.^{Reference Langdorf, Cheaito, Epter and Kazzi17} In general, SLORs have been shown to allow for more direct comparisons between candidates and improved inter-rater reliability,^{Reference Langdorf, Cheaito, Epter and Kazzi17–Reference Hartman, Lefebvre and Manthey19} likely improving the overall utility of the letter of reference as a selection instrument.

Consistent with other studies, research experience ranked lower among selection criteria with only 38.5% of respondents identifying it to be important or very important^{Reference Green, Jones and Thomas1,Reference Crane and Ferraro6,Reference Negaard, Assimacopoulos, Harland and Van Heukelom8} ; what is clear from other studies is that perceived commitment to a specialty is a factor often considered by selections committees,^{Reference Crane and Ferraro6,Reference Hartman, Lefebvre and Manthey19} and research is one possible way to demonstrate this. Another is through elective choice. In our study, spending anywhere from 26–75% of CaRMS elective time in emergency medicine demonstrated commitment to the specialty. However, the Association of Faculties of Medicine of Canada has recently enacted a policy, effective beginning with the Class of 2021, that restricts student electives to a maximum of eight weeks in any single entry-level discipline, such as emergency medicine.²⁰ Therefore, we recommend that students spend the maximum allowable CaRMS elective time in emergency medicine to demonstrate commitment to the specialty.

The majority of respondents reported that the interview is weighted more heavily than the application file in determining final rank order, consistent with much of the residency selection literature.^{Reference Crane and Ferraro6,Reference Negaard, Assimacopoulos, Harland and Van Heukelom8} Structured interviews where each candidate is asked the same questions in the same order were used in 76.9% of programs. Less than half (46.2%) use a standardized framework to evaluate responses. This raises some concerns about the reliability of the interview process. Although there is no definitive interview format that is considered to be the gold standard, it is generally well accepted that unstructured interviews are highly subjective and demonstrate low reliability.^{Reference Prideaux, Roberts and Eva4,Reference Stephenson-Famy, Houmard and Manyak21–Reference Gardner, Onofrio and Dunkin23} On the other hand, evidence for the validity and relative reliability of structured interviews has been previously shown.^{Reference Prideaux, Roberts and Eva4,Reference Blouin, Day and Pavlov24} One study demonstrated that implementing a half-day course that teaches the fundamentals of structured interviewing can substantially improve interviewer knowledge and inter-rater agreement, even among experienced interviewers.^{Reference Gardner, Onofrio and Dunkin23} Despite this, the literature continues to demonstrate a poor association between the interview and future resident performance.^{Reference Stephenson-Famy, Houmard and Manyak21,Reference Kenny, Mcinnes and Singh22} Additional research is needed to better understand what interview factors (e.g., format, content, method) are most helpful in identifying candidates who will later become successful residents.

Strengths and limitations

To the best of our knowledge, our study is the first to perform a comprehensive analysis of the Canadian RCEM residency selection process. The 92.9% response rate is a notable strength and helps ensure a balanced national perspective. The retrospective nature of the study is one potential limitation as certain aspects of our data will be subject to recall bias. Another limitation is that data collection using a survey instrument is inherently subject to a degree of self-report and social desirability bias. Having focused exclusively on Canadian emergency medicine residency programs, our results may not be generalizable to other specialties or geographic regions. Finally, as we focused exclusively on program directors and assistant program directors, our data may differ somewhat from the opinions of other selection committee members.

Study implications and future directions

The information gained from this study has characterized the landscape of the Canadian residency selection process in emergency medicine. It immediately adds transparency to the process for prospective candidates allowing for increased confidence when deciding where to focus their attention. For selection committees, the data provide an easy way to identify where their practice differs from other Canadian programs/selection best practices.^{Reference Roger, Epstein and Bandiera14,Reference Paterson, Hartmann, Woods, Martin and Thoma25} Now that we have a firm understanding of what factors are currently driving selection decisions, this study sets the stage for future research that aims to identify selection methods that are more predictive of future resident performance. With the transition to competency-based medical education and the resultant shift to more comprehensive resident assessments, the current climate is timely to conduct such research.