Introduction
Objective Structured Clinical Examinations (OSCEs) play a pivotal role in medical education and assist evaluating the “show how” component in Millers Pyramid of assessment (Gormley Reference Gormley2011). This assessment methodology is frequently used to evaluate medical students’ suitability to progress onto medical qualification both at undergraduate and at post-graduate level (Newble Reference Newble2004; Zayyan Reference Zayyan2011). OSCE’s however are associated with significant organisational challenges, especially during the current worldwide COVID-19 pandemic (Wooliscroft, 2000).
COVID-19 was characterised as a pandemic by the World Health Organisation (WHO) on March 11th 2020 (World Health Organisation, 2021). The declaration was followed by the implementation of worldwide restrictions. In Ireland, restrictions included the closure of facilities deemed “non-essential” and resulted in the discontinuation of face-face teaching in universities on March 12th 2020. Thus, all medical students were advised that on-going education was to pivot “on-line” with clinical attachments transferred out of hospital and university settings. These changes had a significant impact on examining students in clinical skills, including in psychiatry and resulted in OSCEs being deferred including the Year 4 (4 MB) OSCE at the National University of Galway (NUI Galway), Ireland
Traditionally 4 MB medical students at NUI Galway rotate through a number of specialities (Psychiatry, General Practice, Obstetrics and Gynaecology and Paediatrics), with a particular focus on clinical skills which are examined as part of the Advanced Clinical Skills (ACS) module. Historically, the assessment of the learning objectives and skills were assessed via a summative OSCE, which in 2019 included 16 stations of 10 minute durations split over 2 days. For students to progress to their final year (5 MB), students are required to pass the ACS module. Consequently, despite the logistical difficulties caused by the COVID-19 pandemic and its’ associated restrictions, an OSCE examination was required for 4 MB students, to assess their attainment of mandatory learning outcomes only attainable in a clinical examination format.
University institutions who have proceeded with OSCEs during the COVID-19 pandemic have utilised either an adapted virtual OSCE assessment (Lara et al. Reference Lara, Foster, Hawks and Montgomery2020; Kakadia et al. Reference Kakadia, Chen and Ohyana2020), or a modified COVID-19 compliant live OSCE (Darling-Pomranz et al., Reference Darling-Pomranz, Gray and Watson2020; Boursicot et al. Reference Boursicot, Kemp, Ong, Wijaya, Goh, Freeman and Curran2020). Positive feedback from stakeholders in particular has been noted with modified live OSCEs, albeit careful planning and circuit modification being required (Darling-Pomranz et al., Reference Darling-Pomranz, Gray and Watson2020; Boursicot et al. Reference Boursicot, Kemp, Ong, Wijaya, Goh, Freeman and Curran2020). On-line examinations have also yielded largely positive feedback; however, technical issues have emerged as a specific challenging feature (Kakadia et al. Reference Kakadia, Chen and Ohyana2020).
On August 26th and 27th 2020, when the first COVID-19 peak had subsided in Ireland, the ACS committee decided to proceed with a modified live OSCE for 4 MB students, based on 1) the mandatory requirement for students to pass learning outcomes pertaining to this module to proceed into final year and 2) due to previous positive, albeit preliminary reports of modified face-to-face OSCEs (Darling-Pomranz et al., Reference Darling-Pomranz, Gray and Watson2020; Boursicot et al. Reference Boursicot, Kemp, Ong, Wijaya, Goh, Freeman and Curran2020). Modifications to the ACS module for the 2020 OSCE are detailed in Table 1. The OSCE consisted of each discipline creating one station of 14 minutes instead of four 10 minute stations per discipline. Of note, otorhinolaryngology is examined as a component of the general practice discipline. Each of the discipline stations assessed distinct learning objectives. The 14 minute time period was chosen to enable all learning outcomes based upon the blueprint of the ACS module to be examined and in line with Medical Council guidelines (Council IM 2012) (see Table 2), with this increased time period allowing additional tasks to be examined. For example, in psychiatry, OSCE stations were chosen that were adapted from previous years, with three tasks examined compared to two previously. This time-period ensured less than 15 minutes of contact occurred between simulated patients and the students being examined. The number of exam circuits increased from three to six circuits, reducing the number of students, examiners and invigilators in any location.
Changes implement in the modified live 2020 ACS OSCE
Examination blueprint with learning outcomes
Communication: h = History taking; Ex = Examination; ep = Effective explanation; GP = General practice; OBGYN = Obstetrics and Gynaecology.
Experienced examiners were recruited to ensure marking standardisation. As less examiners were required for this exam, due to the reduced number of OSCE stations compared to previous OSCEs, we were able to ensure that all examiners, had examined previously in other ACS OSCE examinations in their discipline of interest. In addition, each discipline held examiner briefing sessions prior to the OSCE, and during the OSCE, on-going monitoring of marks across centres was undertaken to ensure minimal discrepancies in marks occurred due to examiner differences, with a lead examiner identified for each station monitoring same. Each of the seven learning outcomes was examined in at least two OSCE stations, with communication examined in each of the four OSCE stations. The psychiatry stations also examined history taking and management (see Table 2).
There were no congregated breaks at any exam location and several measures were incorporated to ensure COVID-19 guidelines were maintained throughout the examination. As previous conducted studies predominantly focused on the feasibility of conducting an OSCE during the COVID-19 pandemic, this study primarily focused on student performance compared to the previous years’ face-to-face examination. Consequently, the aims of this study were: (1) to ascertain if the ACS 2020 OSCE was associated with any significant differences in results compared to the equivalent 2019 OSCE examination for both the entire OSCE and for the discipline sub-components of the OSCE; Psychiatry, General Practice, Obstetrics and Gynaecology and Paediatrics, (2) to assess students’ satisfaction with this modified examination, and (3) to explore if conducting a face-to-face examination during the COVID-19 pandemic was associated with any organisational difficulties.
Methods
Participants involved in this study included all 4 MB medical students sitting the ACS OSCE for the first time, in either 2019 or 2020. All data relating to exam performance were anonymised and securely stored and handled in accordance with the Data Protection Act, 2018. Ethical approval was attained from the Galway University Hospitals Research Ethics Committee (C.A.2351).
Data pertaining to metrics from the 2019 and 2020 OSCE respectively were retrieved from Observe software (Qpercom Ltd, 2020), which is only available to members of the ACS committee.
Students’ anonymised subjective experience of the modified OSCE was measured utilising a study-specific 7-item questionnaire (see Appendix 1). This questionnaire examined data pertaining to students experience and sense of safety on the examination site and communication relating the OSCE. A free-text box was also incorporated allowing students to provide additional data relating to their subjective experience of this assessment.
Statistical analysis was conducted utilising the Statistical Package for Social Sciences 26.0 for Windows (SPSS Inc., IBM, New York, USA). Descriptive analyses (frequencies, means and standard deviations) were attained for all quantitative data (i.e. total OSCE result and scores attained from each OSCE station and each medical discipline) with data checked to ascertain if normally distributed. The student-t test was utilised for parametric data and the Chi square (χ 2) test was utilised for categorical data (comparison of number of students attaining different grades [first class honours (≥70%), second class honours (60–69%), pass (50–59%) and fail (<50%)]. Likert scale data from this anonymized feedback were compared utilising χ 2 or Fisher’s exact data as appropriate. Free-text data from the questionnaire were examined and open-coded based on the framework of the questionnaire.
Results
OSCE metrics
Each OSCE included 200 students. The mean total ACS score in the 2020 OSCE was statistically higher [62.95 (SD = 6.21) vs. 59.35% (SD = 5.54), t = 6.092, p < 0.01] compared to 2019. Three of the disciplines [Psychiatry (t = 5.207, p = 0.001), General Practice (t = 8.549, p < 0.001) and Obstetrics and Gynaecology (t = 3.397, p = 0.001)] demonstrated higher marks statistically in the 2020 OSCE compared to the 2019 OSCE (see Table 2). The Cronbach Alpha result for the 2020 OSCE was 0.796 vs. 0.777 in 2019.
On examining individual grades, the 2020 OSCE demonstrated a statistical difference compared to the 2019 OSCE (χ 2 = 27.257, df = 3, p < 0.001). Post-hoc analysis revealed more students attained H1 grades in 2020 compared to 2019 (11.5% vs. 1.0%, χ 2 = 18.816, Fishers exact p < 0.001) with less students attaining pass grades in 2020 compared to 2019 (29.0% vs. 46.5%, χ2 = 13.032, p < 0.001). There was no statistical difference in the number of students failing the OSCE in 2020 compared to 2019 (3.0% vs. 4.0%, χ2 = 0.296, p = 0.586).
Finally, when analysing the station competencies; the mean score for communication in the 2020 OSCE (whereby both students and simulated patients were mask wearing) was 61.3% compared to 64.8% in 2019.
Questionnaire data
There was a 46.5% (n = 93) response rate to the electronic anonymous feedback questionnaire distributed to 4 MB students who had recently completed the 2020 OSCE (see Table 3). Of students who responded; the majority felt this examination was performed safely (89.2%, n = 82), well-co-ordinated (88.8%, n = 82) and was executed fairly (61.3%, n = 57). However, 47.8% (n = 44) of students reported that wearing a face-mask impacted on their ability to communicate effectively with patients (simulated patients) during the OSCE.
Comparison of ACS marks between 2019 and 2020 OSCEs
Questionnaire: Free-Text data
In total, five themes emerged relating to the 2020 OSCE: (1) timing of the OSCE negatively impacted student well-being (n = 13), (2) deleterious impact of mandated COVID-19 restrictions on OSCE performance (n = 8), (3) student preference for more OSCE stations (n = 6), (4) student preference for live rather than virtual OSCE (n = 5), and (5) the modified OSCE was a positive experience (n = 11) (see Table 4).
2020 OSCE: student feedback
Total number of respondents = 93 (%).
Themes emanating from free-text responses: students comments regarding their experience of the modified OSCE during the COVID-19 pandemic
Discussion
This study demonstrated that despite challenges encountered by the COVID-19 pandemic, conducting modified face-to-face OSCEs are feasible. Students attained a marginally higher overall grade (3%) with the modified 2020 ACS OSCE, with no difference in the number of failing students. Overall student feedback was positive regarding both the co-ordination of the modified-OSCE and their own safety in undertaking the modified-OSCE; however, almost half of the students reported that wearing a face-mask impeded their ability to communicate with the standardised patients during the examination.
The overall mean total assessment result and three of the discipline mean results (Psychiatry, General Practice and Obstetric and Gynaecology) demonstrated statistically higher marks in the modified face-to-face OSCE in 2020 when compared to results obtained in 2019. There are a number of potential reasons which may attribute to this difference. Firstly, students had a longer timeframe to prepare for this assesment (as this examination was postponed from until August), with additional teaching sessions conducted throughout the summer. Secondly, a reduction in the assessment time and stations being assessed by each discipline, was associated with a reduced assessment blueprint. This resulted in all core features being assessed (i.e. risk, communication), but some clinical knowledge being assessed in less detail than with previous examinations (i.e. therapeutics), potentially making some components of the modified OSCE easier for some students (Hijazi & Downing Reference Hijazi and Downing2008; Khan et al. Reference Khan, Gaunt, Ramachandran and Pushkar2013). Our findings are consistent with robust evidence to support the use of reduced assessment in sequential OSCE formats in order to progress competent students (Cookson et al. Reference Cookson, Fagan, Mohsen, McKendree and Mohsen2011; Pell et al. Reference Pell, Fuller, Homer and Roberts2013). It was notable that this modified OSCE had a similar overall pass rate compared to the 2019 OSCE. It is possible that this modified OSCE may have been less sensitive in distinguishing between different grades. However, the metrics of quality of the examination, for example, Cronbach Alpha were noted to be similar between 2019 and 2020.
Quantitative data noted high levels of student satisfaction, however, students expressed concern regarding the possible impairment in communication secondary to wearing a face-mask. Statistical analysis further demonstrated that there was only a very minimal difference in the mean score yielded for communication skills in 2019 vs. 2020. Additionally, the wearing of face-masks replicates current clinical practice for all students; but this was less familiar in August 2020, and would have been the first examination this cohort of students would have undertaken whilst wearing a face-mask. Consequently, we plan to further investigate this concern in the 2021 exam which will again require all students to maintain social distance and wear face-masks. An additional concern notable from qualitative comments related to the timing of the modified OSCE, which was perceived as unfair by some students, occurring just prior to commencement of their final medical year, disrupting students’ ability to have “time-off” studying prior to commencement of a busy academic year. As stated above, this may have paradoxically led to the attainment of higher scores on this examination, given the additional study time available. The timing of the OSCE was however organised in August 2020 for a number of reasons including: (i) COVID-19 rates had reduced significantly, (ii) attainment of the ACS was mandatory prior to commencement of 5 MB (final year of medicine), and (iii) allowed students sufficient time to organise travel from various international jurisdictions and engage in quarantine prior to sitting the OSCE (see Table 1). We plan to compare 5 MB clinical and non-clinical results between 2021 and both 2019 and 2020 to ascertain any potential impact of fatigue on examination performance, relating to this point. It is also possible that student experienced significant anxiety symptoms pertaining to the COVID-19 pandemic and its mandated social restrictions and consequences thereof, which might have attributed to some students’ negative perceptions’ of the examination process (Chandratre Reference Chandratre2020).
It was clear from student feedback and examination scores that conducting this modified OSCE was feasible during the COVID-19 pandemic, albeit significant organisation was required. This is consistent with findings from face-to-face OSCEs conducted in other international institutions, which similarly point to the the importance of re-structuring circuits and amending OSCE venues to create the appropriate environment for this assessment (Darling-Pomranz et al., Reference Darling-Pomranz, Gray and Watson2020; Boursicot et al. Reference Boursicot, Kemp, Ong, Wijaya, Goh, Freeman and Curran2020). Of note, we are unaware of any student, administrator or examiner contracting COVID-19 secondary to undertaking the ACS OSCE.
This study is associated with a number of limitations. Firstly, no formal feedback was attained from the administrative staff and examiners involved in this preparation, organisation and marking of this examination. Secondly, other modified OSCE options (an examination of eight stations for example) might have led to alternate results, potentially more similar to those noted in 2019, and consequently we plan to undertake an eight station ACS OSCE in May 2021, and will compare results with both the 2020 and 2019 examinations. Such an examination will also potentially address the important confounder of the timing of the OSCE in August 2020, which allowed additional time for studying and preparation and may potentially have inflated mean scores.
Conclusion
This study has demonstrated the feasibility of conducting an OSCE examining disciplines including psychiatry during the COVID-19 pandemic and demonstrated only marginal differences in results attained. Additionally, student feedback was predominantly positive in relation to the co-ordination of the modified-OSCE and their own safety in undertaking the exam, albeit some concern was noted in how wearing face-masks might impact communication with simulated patients.
Acknowledgements
We wish to express our gratitude to all staff in the School of Medicine involved in the administration and support of this assessment during this pandemic. We wish also to acknowledgement and commend our students professionalism demonstrated in how they approached and conducted this modified OSCE in August 2020.
Financial support
This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
Conflict of interest
Authors have no conflicts of interest to disclose.
Ethical standards
Ethical approval was attained from the Galway University Hospitals Research Ethics Committee (C.A. 2351; granted December 17th, 2020). The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committee on human experimentation with the Helsinki Declaration of 1975, as revised in 2008.
Appendix 1: Student Feedback Questionnaire
Open access
