Choosing examination formats for the MRCPsych

When setting an examination, some of the key factors^{Reference Van Der Vleuten2} that need to be considered when assessing its utility are shown in Table 1. Each of these factors have to be weighed up against each other, with differing weightings according to the purpose and type of assessment.

Table 1

Key factors to be considered when assessing the utility of an assessment (adapted with permission from reference^{Reference Van Der Vleuten2})

MCQs are a format that lends itself to reliability through standardisation of answers and ease of evaluation of large numbers of candidates via machine marking. Although MCQs have been used since the inception of the MRCPsych in 1972, historically, short answer questions (SAQs) and essays were also utilised; these were phased out as individual marking of SAQs with increasing numbers of candidates was taxing, and there were questions about the reliability of essay marking.^{Reference Tyrer and Oyebode3} The format of MCQs evolved from initial true/false answers to the single best answer or ‘best of five’ in use today, as well as the use of the EMQs, in which there is a theme, several stems and a greater number of options, more easily assessing the application of knowledge.^{Reference Jolly, Swanwick, Forrest and O'Brien4} The MRCPsych is a high-stakes examination, with important consequences for candidates, our patients and society in general. In common with all high-stakes postgraduate medical assessments based in the UK, it is regulated closely by the General Medical Council, and all changes to format and structure must undergo prospective approval by them.

Given these concerns, the reliability of the MRCPsych must be extremely high, so that no trainee passes without the requisite ability. Fortunately, the written papers have excellent reliability (with Cronbach's alpha, a measure of reliability, consistently >0.85), but some have questioned whether this has happened at the expense of validity.^{Reference Shields5,Reference Watkins6} Has the depth of clinical context and its application been lost? Perhaps we fail to reward those trainees who undertake in-depth study of complex issues, such as aetiology, ethics and the history of psychiatry.^{Reference Shields5} The main criticism of MCQs is a ‘cueing’ effect, whereby candidates are cued by the correct answer, rather than actively recalling it.^{Reference Schuwirth, Van der Vleuten and Donkers7} There is evidence that requiring candidates to construct an answer, such as in SAQs, produces better memory than tests that require recognition.^{Reference Wood8} Additional issues with MCQs may include various ‘test-taking’ behaviours, such as eliminating wrong answers to arrive at the correct one, guessing from the options available and seeking clues from the language used to deduce the correct answer independently from the knowledge required.^{Reference Surry, Torre and Durning9}

MCQs end up testing recognition memory, and recall is significantly affected by this cueing effect. Creating a good MCQ with valid and meaningful distractors (incorrect options) can be extremely hard. Poor-quality distractors can make guessing more rewarding. There are a number of areas of the syllabus where it is impossible to write valid distractors, and as a consequence, clinically meaningful knowledge may not be examined and more obscure areas, where MCQs may be easier to write, are more likely to be tested.

As mentioned above, it is intuitive and commonly recognised that the assessment drives learning.^{Reference Epstein1} Areas of the syllabus that are more commonly examined are therefore more likely to be studied by trainees. Assessment factors contribute to strategies used to study,^{Reference Al-Kadri, Al-Moamary, Roberts and van der Vleuten10} which could influence the trainees’ overall learning and the extent of knowledge achieved. Developments in technology have allowed easy access to online MCQ ‘question banks’. Many trainees therefore focus their effort on practicing these questions, rather than focusing on core learning and developing deeper understanding.

The costs of taking the MRCPsych for candidates are high^{Reference Acosta, Ashraph, Bainton, Baird, Banham and Barnes11} because of the high cost of the infrastructure behind the examination, e.g. the professional examinations team, detailed psychometric analysis, and supporting the psychiatrists who volunteer their time freely to create and quality-assure questions, and analyse the results. For several years now, the examination is budgeted not to make excessive surpluses, but if this inadvertently happens, the surplus is directed to the trainees’ fund, which has previously funded the creation of the Trainees Online learning resource, among other projects. Moving to digital platforms may reduce costs to trainees as they no longer need travel or accommodation, and potentially could reduce overall costs as no physical venues are required; however, this is uncertain, and the costs of commercial contracts for software, training and ongoing IT support may counteract this.

Digitisation of examinations

The COVID-19 pandemic led to a rapid and unpredicted introduction of online examinations for the MRCPsych, although the College had plans to begin moving toward digitisation before the pandemic. Although there is a relative paucity of literature on online examinations,^{Reference Butler-Henderson and Crawford12} one small study, in which a direct comparison of online examinations versus paper examinations was made, showed equivalent reliability and validity.^{Reference Hüseyin and Özturan13} In terms of candidate performance in online versus paper examinations, the few studies directly testing this have shown no significant differences.^{Reference Hüseyin and Özturan13,Reference Stowell and Bennett14} Candidates’ perception of online examinations are often favourable, and one study found reduced anxiety when taking online compared with traditional paper-based examinations.^{Reference Stowell and Bennett14} Possibly, the fact that candidates are not able to see their peers might account for this. However, it is clear that the rapid introduction of digitisation for the MRCPsych caused considerable anxiety in trainees; the same study^{Reference Stowell and Bennett14} recognised that the first sitting of online examinations can cause anxiety, which later subsides with familiarity upon repeated testing.

Very short answer questions

Very short answer questions (VSAQs) are a novel format of written questions.^{Reference Sam, Hameed, Harris and Meeran15–Reference Sam, Peleva, Fung, Cohen, Benbow and Meeran19} A VSAQ consists of a short question for which an answer is required to be manually entered on computer screen from free recall, as open text. There are no options provided to choose from as in MCQs/EMQs. Generally, the answer would be only a few words. Box 1 shows some examples of how VSAQs may look. Any correct response will attract one mark and any incorrect response will attract zero marks. Examination software would be programmed to recognise multiple versions of correct answers, using smart algorithms. These would allow different versions of a correct response to be recognised. For example, the first question in Box 1 provides an example of several possible correct answers for that question; all of these answers would attract a full mark, and centre around the idea of a reduction or suppression of the default mode network. The software would additionally be programmed to highlight any answer that is a non-exact match (approximate) to any possible correct answers, and these will be manually reviewed by a designated and trained examiner to ascertain whether that represents a correct response. This will ensure that any unforeseen versions of correct responses will not go unrecognised and unrewarded. That response will then be saved in the list of correct answers for that question for any future examinations. Examiners will also review all other marking done by the computer, to ensure accuracy. Minor spelling errors or typos (e.g. ‘inihbited’ rather than ‘inhibited’) will not be penalised and will be picked up during the review process. VSAQs also allow for two entirely different but correct answers, as illustrated in the second example in Box 1. In this example, again, either of the responses will attract a full mark.

The free recall tested by the VSAQs can be more easily focused on clinically relevant topics, and allow freedom to assess a wider spectrum of the syllabus where MCQs may be impossible to write. This should encourage trainees to refocus on core learning through textbooks and primary papers, and make their knowledge base more clinically relevant in the long term.

In the studies to date, VSAQs have been shown to have higher reliability than MCQs, and reduce the cueing effect.^{Reference Sam, Hameed, Harris and Meeran15–Reference Sam, Westacott, Gurnell, Wilson, Meeran and Brown17} They may improve validity by testing nascent knowledge and clinical skills, rather than the ability to pass examinations.^{Reference Sam, Hameed, Harris and Meeran15} In one study of 300 medical students,^{Reference Sam, Hameed, Harris and Meeran15} 69% of students undertaking VSAQs felt that they were more representative of how they would be expected to answer questions in actual clinical practice, and about half felt that they would change their learning strategies in response. However, these studies were conducted on undergraduate medical students and may not be generalisable to postgraduate psychiatry trainees. Additionally, as far as we are aware, there has not been any published data that uses VSAQs from a high-stakes examination such as the MRCPsych, although at least one other College are considering their introduction for UK medical trainees.^{Reference Phillips, Jones and Dagg20} Finally, as VSAQs require recall rather than recognition, candidates appear to universally score lower in them when compared with MCQs;^{Reference Sam, Hameed, Harris and Meeran15–Reference Sam, Peleva, Fung, Cohen, Benbow and Meeran19} this must be carefully accounted for in the standard setting process that sets the pass mark, so that standard setting judges are aware of likely lower scores in comparison with MCQs, particularly in first iterations of the test when they are lacking comparative past data. To account for this, there would be pilot questions tested and a full analysis undertaken to inform future standard setting.

Trainees’ views on digitisation and VSAQs

The opinion of psychiatry trainees was obtained via a presentation by the Chief Examiner, Dr Ian Hall, to the Psychiatric Trainees’ Committee. The Examinations Sub-Committee's Trainee Representative also sought feedback on the Psychiatric Trainees’ Committee collaborative platform, ‘Workplace’. The questions submitted to the College's webinar, ‘MRCPsych Exam – Changes to exam delivery this Autumn’, attended by over a thousand psychiatry trainees and supervisors, were also reviewed in summarising concerns with regards to the digitisation of the theory examinations.

Psychiatry trainees raised several concerns with regards to the digitisation of the theory examinations (Table 2). In the context of sitting the examinations from home, a common theme was how technical issues, such as insufficient internet connectivity, would be resolved, what support would be available to assist with this, and how the College would ensure candidates were not disadvantaged as a result of technical issues. Trainees also expressed concerns as to how cheating would be identified, particularly the potential to ‘trick’ proctoring technology, to prevent inflated examination marks disadvantaging other trainees. Similarly, they expressed concerns that trainees may be falsely accused of cheating if they write notes or look away from the screen. The concerns regarding cheating are in keeping with the published literature of both candidates’ and examination setters’ perceptions of online examinations.^{Reference Butler-Henderson and Crawford12} Trainees also noted that some trainees’ home environments may be unsuitable for sitting examinations, because of caring commitments or house-sharing arrangements. Trainees were also keen to understand how candidates with dyslexia and other specific learning needs would be accommodated. Furthermore, trainees expressed an expectation that examination fees would be reduced in the context of digital examinations.

Table 2

Common themes of trainees’ concerns and responses

Despite the concerns raised, trainees generally appeared to agree with the prospect of the digitisation of the theory examinations, even outside the current context of COVID-19. However, many expressed a strong preference for these to be conducted in test centres to prevent technical issues or cheating, and to ensure candidates with home settings unsuitable for sitting examinations were not disadvantaged.

With regards to the introduction of VSAQs, the trainee response was generally positive. Trainees felt it addressed their request for a greater emphasis on the testing of core knowledge and that VSAQs were better at testing the application of knowledge than the current format. However strong concerns were raised with regards to the examinations not becoming a ‘spelling test’, and particularly that this may disadvantage candidates with dyslexia, other specific learning needs and international medical graduates. They noted that not all spelling errors are of equal clinical significance and where it is clear that a candidate's intended meaning is correct, that this should be accepted as a correct answer.