How to use this report

This report is important for educators, program directors, and simulation leaders as they seek to implement simulation-based assessment and collectively navigate this transformative time for postgraduate medical education. Deconstruction of the principles of assessment makes this seemingly large undertaking more manageable and accessible. These recommendations can be used as both a guide to ensure the effective implementation and use of simulation-based assessment, as well as a tool to promote and advocate for high-quality simulation-based assessment. The utility of this report will vary depending on the institution and their existing simulation infrastructure and assessment processes. For some, this may be used as a reference tool to review programs of assessment systematically that already include simulation-based assessment. For others, this may be used to advocate for resources in order to establish a program of assessment that leverages the potential of simulation-based assessment, enabling assessment of performance in domains that are otherwise difficult to capture and illuminating the “blind spots” in current programs of assessment.

Notable findings

As with any assessment method, it is important to question the validity of the assessment process and the decisions that are made based on that assessment (e.g., pass/fail). As our literature review revealed, there is a strong and growing body of validity evidence supporting the use of simulation-based assessment such that we can proceed comfortably using simulation for both lower-stakes assessment and the assessment of specific aspects of competence such as procedural competence and crisis resource management.^{Reference Cook, Brydges, Zendejas, Hamstra and Hatala15} Simulation-based assessment is likely best used for the assessment of observable skills, attitudes, and behaviours and is less suited for the assessment of knowledge. Ideally, educators should seek out tools and processes that have been evaluated, capitalizing on work that has already been done and adapting to local contexts. As we increase the stakes of the assessment, the demand increases for validity evidence for the assessment process.^{Reference Cook, Brydges, Ginsburg and Hatala17} We recommend that EM programs in Canada start using simulation-based assessment initially with tools and processes that have some form of validity evidence when possible. We also challenge the EM community to continue developing validity evidence for more simulation-based assessment tools and programs and, in particular, for competencies that are not otherwise easily assessed.

One of the main advantages of simulation is its inherent potential for standardization; the assessment environment can be controlled to deliver the same “stimulus” to all learners. This gives simulation-based assessment a powerful and unique advantage over traditional workplace-based assessment. However, the challenges in delivering such standardized assessments cannot be overstated. Simulation-based assessment is resource intensive, as compared with most existing methods of assessment. With a need to focus on standardization of scenario delivery, training of confederates, and rater training, the costs and efforts to deliver high-quality assessment can become prohibitive. It is important to remember that high-fidelity simulation equipment is not required to conduct simulation-based assessment. Depending on the assessment objectives, task trainers and low-fidelity equipment may be the optimal choices to achieve functional task alignment.^{Reference Hamstra, Brydges, Hatala, Zendejas and Cook18} Simulation-based assessment resources such as cases or assessment tools should be shared between programs whenever possible to improve access to the highest quality simulation content.

While the benefits of using simulation for medical education are well articulated,^{Reference McGaghie, Issenberg, Barsuk and Wayne9,Reference McGaghie, Issenberg, Cohen, Barsuk and Wayne19} the educational effect (the learning that takes place before the assessment) and catalytic effect (the learning that is stimulated by the assessment) of simulation-based assessment have not been well defined. Effective assessment should encourage both of these and promote the learner's progression to competence.^{Reference Norcini, Anderson and Bollela20} Literature suggests that much of the learning that occurs in simulation is a direct result of timely, objective, and constructive feedback from both peers and faculty involved in the debriefing process.^{Reference Lateef21} Therefore, we must ensure that processes of simulation-based assessment do not become hurdles to overcome but rather opportunities for concrete feedback, reflection, and enhanced performance.^{Reference Harrison, Könings, Schuwirth, Wass and van der Vleuten22}

Simulation-based assessment will not succeed if the training programs, content creators, assessors, and participants are not involved in its planning, implementation, and review. If trainees no longer look forward to learning in the simulation lab because of a poorly designed simulation-based assessment, we will lose the learning environment that we have worked so hard to create. The simulation environment is artificial and requires experience for learners to become comfortable, thus making frequent exposure in low-stakes contexts critical if we are to also use it for assessment. Further defining the stakes of any simulation-based activity can help set the stage for trainee comfort with a shared understanding of expectations and clarity of purpose.^{Reference Watling and Ginsburg23}