MCQ technique

et al (1976) have advocated the bolder strategy of attempting all questions, reasoning that a ran dom guess has a 50% chance of being correct and so any attempt based on some knowledge should increase this probability and so improve the score. Holden (1987) has suggested that can didates might usefully obtain a specimen paper, answer alternate questions using the different recommended techniques (i.e. 'cautious' v. 'bold')

In this study, candidates who had taken a MCQ paper were asked to go back and guess the questions initially marked 'don't know'. The resultant 'guess scores' ranged from -0.6% to 6%. It was concluded that many candidates could substantially increase their scores by adopting a bolder approach. It is proposed that the guess score is a useful measure of the effectiveness of the candidate's MCQ technique.
Candidates taking multiple choice question (MCQ) examinations are often uncertain about how many questions they should answer in order to do full justice to the knowledge that they have without risking marks by reckless guessing. The Royal College of Psychiatrists' Handbook for Inceptors recommended a relatively cautious approach, advising candidates to attempt questions when they are reasonably sure of the answer and to mark the rest 'don't know'. Harden et al (1976) have advocated the bolder strategy of attempting all questions, reasoning that a ran dom guess has a 50% chance of being correct and so any attempt based on some knowledge should increase this probability and so improve the score. Holden (1987) has suggested that can didates might usefully obtain a specimen paper, answer alternate questions using the different recommended techniques (i.e. 'cautious' v. 'bold') and compare the marks obtained to see which technique is more effective.
Fleming (1988) advised a group of medical students that, although wild guessing (guessing on the basis of total ignorance) is as likely to lose marks as gain them, educated guesses (based on some knowledge of the subject) are more likely to be right than wrong. In a subsequent MCQ ex amination the students answered an increased proportion of the questions and this increase was most marked for those who had previously been most cautious. It was also shown that when this cautious group answered more boldly they increased their scores and their performance relative to the other students.

The study
A sample MCQ paper was sent to 23 trainees on the Nottingham SHO/registrar rotational train ing scheme in psychiatry who had completed at least one year and not more than three years of psychiatric training. The paper was obtained from a course for candidates taking the MRCPsych Part I examination and comprised 50 MCQs, each with five parts, to be answered in 90 minutes. Ten of the trainees were due to take the MRCPsych Part I examination the following month, six were due to take the MRCPsych Part II and seven were between exams. Immediately af ter completing the paper, each trainee was asked to go back and attempt all the questions that had initially been marked 'don't know', using a red pen. All the trainees were invited to take the test under examination conditions, but only six were able to do so. The other test papers were sent to trainees by post, along with instructions to take the paper under conditions that approximated as closely as possible to examination conditions. Replies were received from all ten of those due to take MRCPsych Part I, three of those about to take MRCPsych Part II and four of those between exams (overall response rate 17/23=74%). The papers were initially marked with a mark gained for each correct answer, a mark lost for each incorrect answer and no marks for 'don't know' responses to obtain the score. Next, the trainees' attempts to guess the questions initially marked AUDIT 'don't know' were marked, using the same method, to obtain the guess score. It is proposed that when candidates obtain substantially more correct guesses than incorrect guesses, resulting in a high guess score, this indicates poor MCQ technique as they have considerable hidden knowledge that is not expressed due to excessive use of the 'don't know' option. When, however, the number of correct and incorrect guesses is approximately equal, resulting in a low guess score, this suggests that candidates have done full justice to the knowledge that they have and so their MCQ technique is good. Table 1 shows that the mean proportion of ques tions answered was 79% and the mean score for the test paper was 44.2%. There was a strong positive correlation between the proportion of questions answered and the score (r=0.73, P<0.001).

Findings
The mean guess score was 2.4%. There was a significantly higher number of correct than in correct guesses (Wilcoxon. P=0.0013). When the nine trainees with below-average guess scores (range -0.8%-1.6%) were examined separately, the difference between the number of correct and incorrect guesses did not reach statistical signifi cance (Wilcoxon, P=0.0929). Non-parametric testing was used as the differences were not normally distributed.
Six of the eight trainees who answered rela tively cautiously (attempting less than 79% of the questions) obtained above-average guess scores (suggesting poor MCQ technique). In contrast. seven of the nine trainees who answered more boldly (attempting more than 79% of the questions) obtained below-average guess scores (suggesting good MCQ technique).

Comment
Before drawing conclusions from this study, it is necessary to point out a number of caveats that may limit generalising from the findings. The test paper was not a genuine MRCPsych paper, it was not taken (with a few exceptions) under examination conditions and the consequences of success or failure were not as great as those associated with the MRCPsych examination. The number of trainees taking the test paper was relatively small so the results are best regarded as exploratory rather than conclusive. Despite these limitations, the test has demonstrable predictive validity as the two trainees who sub sequently failed the MCQ section of the MRCPsych Part I examination scored lower in the test paper (26.8% and 38.4%) than the eight who passed (42%-52%).
Candidates with good MCQ technique would be expected to obtain a guess score approximat ing to 0%. It was shown that the performance of trainees obtaining guess scores within the range -0.8% to 1.6% was not significantly different from this optimal performance, suggesting that these trainees have good MCQ technique. Although it is acknowledged that the failure to find a significant difference may be due to a type II error it is suggested, as a guideline, that can didates obtaining guess scores within the range -2% to 2% can consider their MCQ technique to be satisfactory. Those obtaining guess scores over 2% should consider their MCQ technique to be poor, this argument becoming increasingly persuasive as the magnitude of the guess score increases. None of the trainees obtained guess scores less than -2%. This indicates that, al though there is a theoretical risk of losing marks by guessing, in practice the risks are minimal.
The results of this study can be interpreted in different ways. Those advocating a bold 'guess everything' strategy could point to the fact that half of the trainees (8/17) substantially improved their score by guessing the questions initially marked 'don't know' and none of the trainees lost more than 0.8% by doing so. On the other hand, those advocating a more cautious 'never guess' approach could argue that the 'guess everything' strategy provokes high levels of anxiety in train ees without substantially improving the scores in half of the cases.
The potential importance of MCQ technique is best understood by considering the effect of add ing the guess score to the score for each candi date. If the mean score (44.2%) is taken as the pass mark for the examination, eight would pass and nine would fail. Adding the guess score to the scores would not effect the outcome for 14 of the 17 candidates, emphasising that it is the candidate's level of knowledge that is the main determinant of examination success or failure. On the other hand, three of the candidates who scored less than 44.2% raised their scores above the pass mark when their guess scores were added, indicating that good MCQ technique can potentially make the difference between passing and failing for candidates close to the pass/fail borderline.
The best general advice for candidates taking MCQ examinations is that they should attempt at least 80% of questions otherwise it is likely that they will fail to do justice to the knowledge that they have due to overcaution. It is further suggested that candidates can usefully make an individualised assessment of their MCQ tech nique by taking a number of sample papers of a standard similar to the examination to be taken and calculating the guess score for each paper. Those consistently obtaining low guess scores can be confident that their MCQ technique is good. Those consistently obtaining high guess scores are encouraged to adopt a bolder strategy in subsequent MCQ examinations.