MULTIPLE CHOICE EXAMINATIONS: THEIR PREDICTIVE VALUE IN THE PRELIMINARY TEST AND THE MEMBERSHIP EXAMINATION By CHRISTINEHASSALLand W. H. TRETHOWAN The Queen Elizabeth Hospital, Birmingham Earlier analyses (Hassall and Trethowan, 1974, 1976) have examined the pass rates in the Preliminary Test and Me

Earlier analyses (Hassall and Trethowan, 1974, 1976) have examined the pass rates in the Preliminary Test and Membership Examinations in relation to the characteristics of the candidates and after separating those sitting for the first time and those re-sitting. This paper explores the relationship between per formance in the Multiple Choice Questionnaire (MCQ) and performance in the examination as a whole, though without distinguishing between candi dates making first and those making later attempts. Although the use of MCQ examinations is now widespread, many candidates and examiners still regard them with suspicion. It has been suggested that they are particularly unsuited to testing psy chiatric knowledge, as they are thought only to test knowledge of facts and some believe that in psychiatry facts are of relatively minor importance. However, it is clear that sound clinical judgement can only be based on a knowledge of the factual basis of the discipline. The fact that MCQ examinations do not test the candidates' skills in assembling facts and setting them


MULTIPLE CHOICE EXAMINATIONS: THEIR PREDICTIVE VALUE IN THE PRELIMINARY TEST AND THE MEMBERSHIP EXAMINATION
By CHRISTINEHASSALLand W. H. TRETHOWAN The Queen Elizabeth Hospital, Birmingham Earlier analyses (Hassall andTrethowan, 1974, 1976) have examined the pass rates in the Preliminary Test and Membership Examinations in relation to the characteristics of the candidates and after separating those sitting for the first time and those re-sitting.This paper explores the relationship between per formance in the Multiple Choice Questionnaire (MCQ) and performance in the examination as a whole, though without distinguishing between candi dates making first and those making later attempts.
Although the use of MCQ examinations is now widespread, many candidates and examiners still regard them with suspicion.It has been suggested that they are particularly unsuited to testing psy chiatric knowledge, as they are thought only to test knowledge of facts and some believe that in psychiatry facts are of relatively minor importance.However, it is clear that sound clinical judgement can only be based on a knowledge of the factual basis of the discipline.
The fact that MCQ examinations do not test the candidates' skills in assembling facts and setting them down clearly and legibly in essay form is also put forward as a deficiency.Unfortunately, where candidates are required to write essays the results demonstrate only too often that the vast majority do not have these skills, or under examination pressure fail to display them.
Furthermore, there is a considerable degree of subjectivity involved in the marking of essay questions and a substantial variability between marks awarded by the examiners.In contrast, the marking of an MCQ, examination, by computer, is free from all such contamination.This being the case, it was thought useful to look at the relationship between the perfor mance of candidates in the MGQ, their performance in the essay question paper, and the results overall.
Table I gives the proportion of candidates in each of the last six Membership Examinations for whom performance in MCQ.correctly predicted the overall result of the examination.It may be seen that the proportion is fairly constant throughout all the examinations and is 73 per cent for the total of 1,164 candidates.
It has been argued that candidates from some countries where the MCQ type of examination is less common, for instance those from the Indian sub continent and Arab states, are at a disadvantage.If this were true, one might expect the predictive value of the MCQ to be lower for these groups than for those from the UK.This would be particularly true of the Membership Examination, where MCQ supplies only a quarter of the marks, so that a can didate who fails MCQ because of the unfamiliar technique required rather than from lack of knowl edge has a good chance of making up the marks if he does well in other sections.
The percentages of overall results correctly predicted by the MCQ are shown in Table II for each of the three groups used in these analyses.While there is little difference between any of the groups, if anything, the MCQ is a marginally less successful predictor for the UK group.
In the Preliminary Test, where the MCQ.accounts for half the marks, it might be expected to be a more successful predictor.This is indeed the case.Table III shows that for the last three Preliminary Tests, giving a total of just over 1,000 candidates, the predictive level of the MCQ, is go per cent.gain looking at the candidates by origin, Table IV, there is little difference between the three groups.If we examine the performance of the essay paper as a predictor of success or failure in the whole preliminary test, it turns out to be much less satis factory than the MCQ.â€"the predictive level of the essay paper over the total of the three examinations being only 73 per cent compared with 90 per cent for the MCQ,.Furthermore, it can be demonstrated that there is less agreement between examiners marking the essay paper in the Preliminary Test than in the Membership Examination.
From these results it can be argued that for the Preliminary Test the MCQ is such a good predictor that the essay paper could be abandoned.If this were c'one, on the present showing, there would be about io per cent of candidates whose result under the present system would have been reversed.This group comprises 34 candidates who failed the examination but passed the MCQ and would therefore have passed on the MCQ alone, and 63 candidates who passed the examination but failed the MCQ.
In the case of the Preliminary Test, though not of the Membership, it is possible to examine the per formance of these two groups in subsequent exam inations.Unfortunately, only a small proportion of each group have taken another examination, but the results of their performance seem worth taking into account.
Candidates from the three examinations under review, that is Preliminary Tests io, 11 and 12, who passed and have subsequently taken the Membership, number 126.If we divide this group according to their performance in the Preliminary Test, we get the categories in

X1 = 6-78 P < o-oi (Yates correction)
the MCQ to pull up a failure on the essay paper.Category 2 is those candidates who failed the MCQ, but managed to make up the necessary marks with their essay paper.Although the number in the second group is very small, there is a marked difference between the performance of the two groups when they make their first attempt at the Membership Exam, and the x2 test demonstrates a significant difference between the two groups with a probability level of P = < o-oi.
The first category in Table Vb comprises those candidates who failed the Preliminary Test, including the MCQ.while the second comprises candidates who failed the Preliminary Test overall although they passed the MCQ.If the results of the next attempt at the Preliminary Test are examined, it is clear that those who passed the MCQ on the previous attempt had a much higher pass rate than those who failed it.There is, then, a significant difference between the candidates shown in the two groups.Thus, although if the essay paper were abandoned as part of the Preliminary Test some candidates who now pass would fail and vice versa, it would be fair to say from these early results that those who would be failed under the MCQ.alone appear to be the less able candidates, while the reverse is true of those who would pass.
We tested the notion of abandoning the essay in the Preliminary Test on two senior academic psy chiatrists.In each case the reaction was the same, an acceptance that, based on the evidence presented, the possibility was worth exploring, but there was a strong feelingâ€"perhaps what the Americans call a 'gut reaction"â€"that the essay should and probably would be retained.There are a number of reasons why this reaction might be general.There is the suspicion of MCQ, type examinations mentioned earlier, and also of computers.Stories of 'computer error' in the press always fail to mention that it is the 'human error' in the programming that is responsible for the mistake.The chances of this happening in the marking of an MCQ examination are negligible.Or perhaps it is that examinees get a great deal of satisfaction out of covering sheets of paper with their essays rather than marking little boxes.In the former they can range freely if not always accurately and legibly, in the latter they are forced into what many see as the unsatisfactory confines of prepared alter native answers.
Our suggestion is, then, that the evidence currently available indicates that the possiblity of abandoning the essay paper in the Preliminary Test examination should be further explored.It would not be a precedent, as the Royal College of Physicians has for several years been content to use MCQ_ type of examination alone in Part I of its Membership Examination.

TABLEIV
Percentage of candidates in each Preliminary Test Examinationwhere the overall result was correctly predicted by Multiple Choice Questionnaire by Origin

Table Va .
The first category is those who passed the Preliminary Test, either by passing both parts or by passing sufficiently well on

TABLE Va
IOO26

TABLEVb
Comparison of subsequent performance of candidates passing orfailing MCQ_â€"nextattempt at Preliminary Test