Assessment of psychiatric teaching: examining examinations

Aims and method The study set out to consider whether the ‘end-of-firm’ examination in psychiatry fulfils its goals by examining the internal consistency of the end-of-firm examination and final medical board examination and the correlation between these examinations9 scores and sub-scores. Students who failed their end-of-firm examination were followed to monitor their progress at final medical board examinations. Results One hundred and eighty students sat end-of-firm examinations and final medical board examinations. All the components of the end-of-firm examinations and the final correlated significantly with the total mark. The psychiatric essay was less well correlated with the final mark but correlated well with total essay marks. Students who failed their end-of-firm psychiatry examination at first attempt received extra tuition and only two failed finals. Clinical implications Essay writing ability may not carry over into high performance in other areas tested and may be a test of other skills. Extra tuition benefited failing students whose results fell outside the pattern of prediction. Thus the end-of-firm examination may be a useful tool if acted on to prevent student failure.

examination and the correlation between these examinations' scores and sub-scores. Students who failed their end-of-firm examination were followed to monitor their progress at final medical board examinations. Results One hundred and eighty students sat end-offirm examinations and final medical board examinations. All the components of the end-of-firm examinations and the final correlated significantly with the total mark. The psychiatric essay was less well correlated with the final mark but correlated well with total essay marks. Students who failed their end-of-firm psychiatry examination at first attempt received extra tuition and only two failed finals.

Clinical implications
Essay writing ability may not carry over into high performance in other areas tested and may be a test of other skills. Extra tuition benefited failing students whose results fell outside the pattern of prediction. Thus the end-of-firm examination may be a useful tool if acted on to prevent student failure.
The work of academic departments is tradition ally split between two areas, teaching and research. For some years research has been rigorously assessed by peer review because academic promotion has depended upon re search productivity; teaching productivity, in contrast, has received less attention.
A traditional method of assessing medical teaching is the examination, which tests not only the student, but also, directly, evaluates the teaching. Important functions of the inter mediate ('end-of-firm') student examination are to help students pace their learning; to identify students who are having difficulties, so that they can access additional help: to practise for the final medical board examination and to establish that they have achieved a necessary level of knowledge. In contrast, medical board finals serve to establish minimum standards for safe practice, while also identifying excellence.
Psychiatrists and other disciplines have shown considerable interest in evaluating components of both intermediate and final examinations. The importance of both written and oral examinations in evaluating both factual knowledge and clinical skills has been stressed (Love et al 1993). Criteria-based rating scales for assessing stu dent's clinical performance has been advocated (Carline et al, 1992, Parenti, 1993 although Metheny (1991) has shown that clinicians tend to make global ratings of student performance, failing to distinguish between the different evalu ation categories. However, these may be an improvement on most examinations which are based upon simple recall, despite the complex thinking required in psychiatry (Miller et al, 1991 ).
The present study is the result of an exercise designed to improve teaching quality. It sets out to consider whether one particular format of intermediate examination fulfils its goals. It does this by examining the internal consistency of both an end-of-firm examination and final medical board examinations, and the correlation between these examinations' scores and subscores. Students who failed their psychiatry end-of-firm examination were followed to monitor their progress at final medical board examinations.

The study
The study examined a whole year cohort of students who passed through an inner London teaching hospital. The components of their psychiatry end-of-firm examination were com pared with each other, as were those of the final medical board examinations.
The examinations consist of the following components.
(a) Psychiatry end-of-firm examinations: (i) a multiple choice question paper (MCQ) (li) a video-recording of a patient inter view. Students were asked to record the mental state findings, differential and preferred diagnosis. Their papers were marked using a structured scheme. (iii) a clinical viva with two independent members of the department, using a 'case vignette' format. (iv) an overall mark, calculated by adding totals of 1-3 given above. Each com ponent was given equal weighting, (b) The final medical board examinations: (i) a compulsory psychiatric essay (il) a medicine/paediatric/psychiatry es say paper as a whole (iii) a final medical board MCQ paper (iv) a final medical board clinical (long and short cases) (v) a final medical board clinical viva (vi) a final medical board total mark. Comparisons were made with Spearman's technique for non-parametric data using SPSS-PC.
Lastly, we examined the cohort to see if students who failed their intermediate examin ations also tended to fail final medical board examinations.

The end-qf-firm examination
The viva, MCQ and mental state were all highly correlated with the total mark (P<0.0001). Internal comparison of the components of the end-of-firm examination were all significant (at least P<0.05). In particular, marks on the MCQ and the mental state were highly significant (P< 0.0001).

Final medical board examinations
Internal comparison of the marks for the components of medical board finals (essay total, clinical examination and the viva) were all significant (at least P<0.05).
However, the psychiatric essay only correlated well with the total essay mark (P< 0.0001) but not with the clinical examination or the viva.
Seventeen students failed their end-of-firm psychiatry examination, but only two of these later failed final medical board examinations. Ten students who-passed the end-of-firm psy chiatry examination at the first attempt, with a mean score of 62% (range 54-72%) nevertheless failed medical board finals.

Findings
The cohort was 180 students who sat under graduate psychiatry examinations in the academic year 1991-1992 and the final medical board examination in 1993. Correlation coeffi cients for the comparison of intermediate end-offirm examinations with the grades obtained at final medical board examinations are given in Table 1. All the components of the end-of-firm examination correlated significantly with the total score that candidates achieved at final medical board examinations.
Correlations between the individual compo nents of the two examinations were also gen erally strong, although the psychiatric essay was an exception.

Comment
This study was designed to examine the correla tion between the end-of-firm results and the later performance at medical board finals. There is a very strong relationship between the total mark attained in the end-of-firm examination and the components of medical board finals. The one exception is the relatively low correlation between the end-of-firm examinations and the psychiatric essay at finals. It is interesting to note that in the final examination itself, the psychiatric essay has the lowest correlation with the other components. Thus, it appears that some factor, perhaps organisational skills, good handwriting, or being a native English speaker, links essay writing ability, regardless of subject, but that this may not carry over into a high performance in other areas being tested.
While the total score obtained at the end-offirm examination correlates well with the results at final medical board examinations, so too does the score from the end-of-flrm MCQ alone. This raises the question of whether the other parts are valuable. In particular, the mental state compo nent correlated least well with both the other marks and with the total obtained in the end-offirm examinations. The mental state element of the end-of-firm examination also correlated least well with components and the total score at final medical board examinations.
This study formed part of an assessment of departmental teaching. The findings have been used to effect changes in the structure of the end-of-firm examination. The mental state from a pre-recorded videotape and the case vignette were abandoned. Instead, a new viva based upon presentation of a case has been instituted. Strict rules of case selection are applied, and the individual case chosen for each student is by the consultant of their clinical 'firm'. The new viva format thus approaches more closely the examination conditions of final medical board examinations, and was retained in order to give students practice of case presentation under examination conditions.
We also looked at students who failed their end-of-firm examination at the first attempt. Seventeen students failed; they received extra tuition and were then able to pass the psychiatry examination at a later date, only two of these subsequently failed medical board finals. Thus it seems that this technique benefited students whose results stood outside the overall pattern of prediction of final medical board results by the intermediate examination. The intermediate ex amination may be a useful indicator of students who are in need of remedial tuition. Extension of this technique by other departments might identify further students who would benefit from increased teaching input. There was, however, a small cohort of students (10/180) who show an average ability in psychiatry, but who fail final medical board examinations.
This study shows the potential for analysing examination data as part of the programme for improving teaching and examinations. Correlat ing the marks from intermediate and final medical board examinations allows academic departments to ensure that the effort invested in intermediate examinations is not being squan dered.
The present study is of interest to a wider audience than academic departments of psy chiatry. It may be surprising that intermediate examinations in psychiatry seem to identify those students who perform satisfactorily in all disciplines when they later sit final medical board examinations. By the same token, stu dents who do poorly in their intermediate psychiatry examination also do poorly at final medical board examinations, with the exception of those who failed their intermediate examin ations and received extra tuition.