JAME S WARN ER Psychiatrists ’ appreciation of statistical v . clinical significance : a quick test

AIMS AND METHOD Pharmaceutical advertising material can confuse clinical and statistical significance.We used a brief questionnaire (five questions) to evaluate psychiatrists’ appreciation of this difference. This approximated to the level of critical appraisal competence of the MRCPsych part 3 examination. RESULTS Of the 113 questionnaires distributed 93 were returned complete (response rate 82%). Senior trainees were significantly better than junior trainees at correctly interpreting data (mean score (maximum 5) 2.61v. 2.08; P = 0.04). Consultants did less well than senior trainees, although our sample of consultant respondents was too small for significance testing. CLINICAL IMPLICATIONS Learning critical appraisal for the MRCPsych examination may provide psychiatrists with valuable transferable skills and prevent gaps in our knowledge being exploited by misleading study data. Psychiatrists of all grades need to maintain their research appraisal skills and should not regard the MRCPsych examination as the end of their learning.

Pharmaceutical advertising material can confuse clinical and statistical significance.We used a brief questionnaire (five questions) to evaluate psychiatrists'appreciation of this difference.This approximated to the level of critical appraisal competence of the MRCPsych part 3 examination.

RESULTS
Of the 113 questionnaires distributed 93 were returned complete (response rate 82%).Senior trainees were significantly better than junior trainees at correctly interpreting data (mean score (maximum 5) 2.61 v. 2.08; P = 0.04).Consultants did less well than senior trainees, although our sample of consultant respondents was too small for significance testing.

CLINICAL IMPLICATIONS
Learning critical appraisal for the MRCPsych examination may provide psychiatrists with valuable transferable skills and prevent gaps in our knowledge being exploited by misleading study data.Psychiatrists of all grades need to maintain their research appraisal skills and should not regard the MRCPsych examination as the end of their learning.
As psychiatrists, we are presented with research data from various sources, and we need to be able to apply the findings to our clinical practice.We are often given presentations by pharmaceutical representatives, both on a one-to-one basis and in larger groups.Our observations suggest that the mode of data presentation in these settings can be misleading.Data are presented in ways that can appear to show large benefits for a particular medication over existing treatments.This is particularly the case when pharmaceutical companies are trying to differentiate their medication from others in the same class.However, scrutiny of the often colourful charts and tables can reveal less positive results.We decided to assess to what extent learning critical appraisal skills equipped psychiatrists to identify 'statistical spin' and measure how confident and competent psychiatrists are in interpreting the results of study data.A questionnaire was devised to test critical appraisal skills and, it was hoped, to stimulate psychiatrists of all grades to maintain these skills.We aimed to make the evaluation educational, and issued doctors with worked answer sheets once they had completed the questionnaire.

Method
We designed a questionnaire to assess psychiatrists' ability to critically analyse clinical trial data.This was designed to reflect the skills required for the MRCPsych examination (J.W. is Chair of the MRCPsych critical appraisal question panel).After piloting to ten doctors, the questionnaire was distributed at three educational meetings within St Mary's and Charing Cross psychiatry training rotations.At each meeting, a short presentation was given to introduce the study aims, and time was scheduled for doctors to complete the questionnaire and hand it in at the end of the afternoon.Most doctors completed the questionnaire on the day; those who took the questionnaire away were asked to return it the following week.The questionnaire was anonymous, but respondent psychiatrists were asked their grade and year of qualification, whether they qualified and trained in the UK, and to rate how confident they are (on a scale of 0 to 10) in interpreting results of trial data presented by pharmaceutical companies.The questionnaire consisted of five questions and a 'best fit' answer from two or three statements (see Appendix 1 in the online supplement to this paper).Those doctors wishing to see the answers were asked to give their email address.We waited until we had collected around 100 completed questionnaires before distributing the worked answers (see Appendix 2 in the online supplement to this paper).Analysis was undertaken using SPSS version 15 for Windows with chi-squared, Mann-Whitney and Kruskal-Wallis tests as appropriate.

Results
In total, 93 questionnaires were returned from 113 distributed (82% response rate); 87 respondents returned the completed questionnaire on the day of distribution and the remaining 6 returned the completed questionnaire within 1 week.All grades of psychiatrist were represented in the study.The year of qualification ranged from 1977 to 2005, with 2000 as the most common year (20 of the 93 doctors qualified that year).Table 1 lists the number of correct answers (out of 5) and the average score per grade.Table 2 lists the number of correct replies (out of 93) and the percentage giving a correct answer for each question.
The modal score was 3 out of 5, with an average of 2.29.In the senior house officer/specialty training (ST) 1-3 year group, 2 of the 51 doctors got all five questions wrong, but 2 in this group got all five correct.One specialist registrar/ST4-6 got all the questions correct.No staff-grade respondent or consultant achieved full marks, but these groups were small compared with the other grades.The average score by grade increased with experience from senior house officer/ST1-3, through staff and specialist registrar/ST4-6 grades.Junior trainees (senior house officers and ST1-3 trainees) were Smith & Warner Psychiatrists' appraisal of significance original papers significantly less confident than specialist registrars in their critical appraisal skills (z = 72.45,P = 0.01) and also significantly less competent (z = 72.05,P = 0.04).The average score appeared to fall again at consultant level, although in our small consultant sample this was not significant (w 2 = 4.9, d.f.= 3, P = 0.18).We split the year of qualification results into two groups around the median (pre-2001, and 2001 and beyond) but found no significant difference in scores (w 2 = 5.7, d.f.= 5, P = 0.34).No significant difference was found whether the respondents were UK-qualified or not (z = 70.78,P = 0.44), or whether they had the bulk of their postgraduate psychiatric training in the UK or not (z = 71.81,P = 0.07).There did appear to be a slight positive correlation between higher ratings of 'confidence' and score achieved (r = 0.24, P = 0.02).However, all of the 13 doctors who achieved four or five correct answers rated themselves only 'quite confident' rather than 'very confident'.Most respondents did best on questions 1 and 4. Question 3 appeared to be the most difficult, with only 23% of doctors achieving the correct answer.

Discussion
Our good response rate was largely due to our approaching psychiatrists at local training scheme educational meetings and achieving a 'captive' audience.Of the 93 respondents, 83 completed and returned the questionnaire on the day of distribution.Five of the remaining six respondents were consultants who wanted more time to answer the questions, and returned the answer sheet within 1 week.(We do not feel that this altered the overall study results, and assume no conferring took place.) The results show an increase in critical appraisal confidence as psychiatrists progress from junior to senior posts, which is matched by an increase in competence up to consultant level.It is noteworthy that skills in the consultant group do appear to tail off from those achieved by the specialist registrar/staff grades.This probably reflects the need to develop good appraisal skills for the MRCPsych examination, which can attenuate following membership attainment.Another explanation is that a few consultants had qualified before the critical appraisal examination was introduced, so never acquired the skills.One concern was that the questionnaire was too difficult.However, there were 13 doctors who scored four or more questions correctly.Question 3 was answered least well, and on reflection could be somewhat of a 'trick' requiring respondents to spot that the data are non-parametric and t-tests are not applicable.In case this question skewed the findings we looked at the effect of removing it from the results.No significant difference was seen in the results by grade, year qualified, UK-qualified/trained or confidence.
Psychiatrists need to be confident and able to critically appraise the wide variety of research evidence presented to them.An obvious example of this need is our evaluation of pharmaceutical research, although critical appraisal skills are required for many aspects of clinical practice.Many doctors regard their time with pharmaceutical company representatives as an important source of clinical and prescribing information.However, data can be presented in misleading ways and we need to maintain our critical appraisal skills.Recent studies suggest that the colourful charts and tables presented to us can be biased, or even false.Casares et al examined promotional material given by pharmaceutical representatives to family doctors in Spain. 1 The promotional material was cross-referenced and scrutinised alongside the original study on which the information was supposedly based.In 45% of cases the advertising material was not supported by the research, highlighting the need to view such material with the original study data.Another study investigated whether research funding by pharmaceutical companies had any bearing on the outcomes: Lexchin et al concluded that studies sponsored by pharmaceutical companies were more likely (by an odds ratio greater than 4) to have outcomes favouring the sponsor. 2

Implications of the study
In order to draw valid and clinically relevant conclusions from research data, we must retain our critical 'eye'.Psychiatrists need to be able to critically appraise various types of evidence, including that presented by drug companies.This study suggests that the MRCPsych critical appraisal paper may help trainees develop these skills; but we need to maintain them beyond the MRCPsych examination.

P E T E R L A W R E N C E Z A K I L A B I B A N D L I S A B R O W N E L L
Factors affecting patient satisfaction with the psychiatric ward round: retrospective cross-sectional study

AIMS AND METHOD
A questionnaire was distributed to patients in a psychiatric hospital in Birmingham, UK, to identify the factors that affect their satisfaction with the ward round.

RESULTS
The questionnaire was completed by 42 patients (53% response rate).Waiting time was the only variable to be significantly correlated with total score of patient satisfaction.Regression analysis also identified diagnosis and patients meeting their consultant before the first ward round as significant predictors of patient satisfaction.

CLINICAL IMPLICATIONS
Reducing waiting time and ensuring that the consultant meets the patient before the first ward round would make a significant improvement to the in-patient experience, without causing much disruption to standard clinical practice.
The ward round plays an essential role in the review and management of patients in psychiatric hospitals.It has been shown that up to a half of patients report negative experiences relating to the ward round. 1 Patients have found it uninformative and stated that it can provoke anxiety and distress.2^4 Several studies have looked at the effects that demographic and ward round variables have on patient satisfaction with the ward round.4^6 This study was conducted to investigate some variables that had not been previously researched, as well as the aforementioned variables.Our main focus was to identify factors that affect patient satisfaction with the ward round.By identifying these factors, changes to standard practice could be implemented to improve the in-patient experience.

Question 2
Answer: C The 95% confidence interval for the relative risk includes 1 so is not statistically significant.With such small percentages no conclusion can be drawn about the sideeffect profile comparison given.As both drugs are equally efficacious, they should be prescribed equally on the basis of this study.

Question 3
Answer: B Data for length of stay are flawed because t-tests cannot be used for non-parametric data such as length of stay in hospital (continuous variable and not normally distributed).A simple check for standard deviation normal distribution is to calculate if 2 standard deviations (includes 95.4% of data) taken away from the mean are still within the possible range for the variable.
So the sample is not normally distributed and t-tests cannot be applied.Non-parametric statistical tests are required.

Question 4
Answer: B The relative risk reduction of suicide appears convincing (26%), but the actual risk reduction (ARR) of 1.1% is less startling for a rare event like suicide.If there is apparent disparity between these indices it is useful to calculate the number needed to treat (NNT).The NNT is 1/ARR.So in this study NNT = 1/1.1% = 1/0.011= 91 patients.Therefore, 91 patients are needlessly given the group II combination (carbamazepine + new mood stabiliser) to prevent just one suicide in 5 years.As the hepatic sideeffect profile of treatment group II is twice as high as group I, the group II patients are at considerably greater risk of serious liver problems.Therefore the combination is not recommended and the clinician should continue to prescribe carbamazepine alone.

Question 5
Answer: B This study highlights the importance of 'intention to treat' and how data can be manipulated if it is ignored.Looking at the 'treatment received' data (Fig. 1), the TAU + drug A group appears to do much better than the TAU only group.The graph suggests a positive outcome for drug A in 28% more patients, at the highly significant P-level of 0.001.However, as patients unable to tolerate the side-effects of drug A revert to TAU, the differences presented are actually statistical artefacts.The 'treatment received' graph shows the outcome only in those who ultimately receive TAU + A (260 of original 350 participants) and loses all those who had to revert to TAU owing to troublesome side-effects of drug A (90 of original 350 participants, expanding the TAU group to 440).If 'intention to treat' is applied (looking at outcomes of all the original participants as randomised, the effect of offering treatment rather than receiving it), the actual result is very different.The 'as randomised' graph (Fig. 2) looks at all the 350 original TAU + A patients compared with the 350 original TAU only controls.It shows a non-significant P-value of 0.5 and no actual benefit of drug A. Drug A is therefore not helpful, and has a wide side-effect profile so could actually be harmful.It should therefore not be prescribed on the basis of this study.
Smith & Warner Psychiatrists' appraisal of significance original papers

Patients 2 Worked answers for questionnaire Question 1
Patients were purposively sampled from five wards (four general adult wards and one mother and baby unit) of a psychiatric hospital in Birmingham over 1 month.Patients were excluded if they lacked capacity to consent to the study, their consultant practised novel ward round Smith & Warner Psychiatrists' appraisal of significance original papers Appendix Answer: B Both are statistically significant at the 95% confidence interval, but treatment B shows a greater improvement in scores on the HRSD so should be favoured in clinical practice.

Table 2 .
Respondents' answers analysed by question (n = 93) literature, the skills examined here (and in the MRCPsych critical appraisal paper) are fundamental to many aspects of clinical practice.Despite there being many sources of 'readymade' appraisal such as the Cochrane Collaboration and BMJ Clinical Evidence, clinicians may still need to appraise individual studies prior to making treatment decisions, undertaking audits, writing reports and business cases.We are pleased such skills appear to develop with training but are concerned that the skills may attenuate if not used frequently.