Journal clubs have traditionally been important means by which clinicians, academics and trainees appraise research that relates to their field. In recent years the evidence-based style has become more prevalent, allowing knowledge gleaned from research to be applied to clinical situations. Currently, the main approach of evidence-based journal clubs is quantitative. This paper describes the evaluation of a modified journal club format, developed in an academic department of a medical school and combining the appraisal of both qualitative and quantitative papers.

The place of evidence-based journal clubs is now established. Since the first series of articles from McMaster University in Canada, when a set of guidelines for evidence-based journal clubs was suggested, there has been a growing literature on the subject (McMaster University, 1981). Subsequently, numerous established international journals have all published their own versions of these guidelines. Gilbody (Reference Gilbody1996) suggested the format which has been adopted in many psychiatric journal clubs. Warner & King (Reference Warner and King1997) reported that once implemented in this format, 88% of participants improved their critical appraisal skills and 100% felt it was an appropriate use of academic programme time. Geddes (Reference Geddes1998) suggested that the critical appraisal skills of clinicians were, for many, at best rusty. He also highlighted the importance placed on this skill by the College, which introduced the critical review paper as part of the core skills tested in the MRCPsych examination. Dhar & O’Brien (Reference Dhar and O'Brien2001) highlight the usefulness of this approach for trainees in preparation for postgraduate examinations. Owen et al (Reference Owen, House and Worrall1995) point to the usefulness of this approach in improving research practice and Geddes & Harrison (Reference Geddes and Harrison1997) suggest that adopting this approach improves clinical practice.

Although the argument that qualitative research has an important role is becoming more widely accepted, there is little evidence of an increasing ability to evaluate the quality of qualitative research. There is an ongoing debate about how this can be done and what kind of criteria should be used. In 1998, the National Health Service commissioned a review of the literature and their report highlighted criteria that could be used to differentiate between the quality of qualitative papers (Reference Murphy, Dingwall and GreatbatchMurphy et al, 1998). Pope & Mays (Reference Pope and Mays1999) summarise some of these points and offer further guidelines.

The idea to change the journal club format within the Department of Mental Health (Learning Disability) from an older style to a more evidence-based approach had been suggested for some time. However, the expectation that this would be based on quantitative research was not acceptable because of the unique mix of medical and social science skills within the group. Each theoretical perspective was felt to have equal validity and therefore required equal consideration in the journal club. It was proposed to establish a journal club, which would present and assess both qualitative and quantitative research papers on the same subject at the same session. It was envisaged that critical appraisal skills in both quantitative and qualitative research among all members, regardless of their discipline, would develop. As the critical appraisal of qualitative papers has not been previously described, this component of the process was evaluated.

Method

Journal clubs are held regularly as part of the regular postgraduate academic meetings of the Department of Mental Health, Learning Disability. They are attended by doctors, psychologists, clinical and social science researchers, and multidisciplinary community team members, with a variable attendance averaging about ten individuals. It was decided to pilot the joint (i.e. qualitative and quantitative) process of assessment with a view to extending it to a wider audience if successful.

The structure of our new journal club was modelled loosely on that described by Gilbody (Reference Gilbody1996). A different presenter was chosen for each session, a psychiatric specialist registrar for the quantitative section of the session and social science researcher for the qualitative section. The structure was modified based on comments received throughout the period of study.

Lists of questions used in quantitative appraisal are available from numerous sources (Reference GreenhalghGreenhalgh, 1997; Reference Sackett, Strauss and RichardsonSackett et al, 1999). These were summarised and provided to the facilitators. For the qualitative papers a series of questions based upon the framework suggested by Pope & Mays (Reference Pope and Mays1999) were developed (see Appendix). A glossary of terms, collected from available literature, was also provided to both groups (available from the authors).

Two short questionnaires composed of a mixture of open answer boxes as well as some 5-point Likert scales were developed. The first questionnaire was used to obtain baseline opinions. The second, distributed after four sessions over a 6-month period, was used to assess any change in confidence when appraising qualitative papers, and participants’ enjoyment and the perceived usefulness of the new format.

The collected data were analysed using Stata, version 7, for Windows. Non-parametric statistics were used to assess the change in confidence in those completing the journal clubs based on the null hypothesis that there would be no change.

Results

Table 1 provides a breakdown of those attending both the initial and final journal club for the pilot study. Table 2 shows how confidence levels changed during the course of the study and gives a breakdown of the perceived educational value, enjoyment and usefulness of this new format. In all areas, seven or more of those attending reported positive statements.

Table 1.

Breakdown of those attending the pilot study

Speciality/role	Number attending initial session	Number attending final session
Consultant psychiatrist	6	3
Specailist registrar	3	3
Speech and language therapist	1	1
Social scientist	2	2
Training advisor	1	0
Nurse	1	0
Associate specialist	1	0
Total	15	9

Table 2.

Confidence levels of nine participants in assessing qualitative papers before and after the pilot study based on a 5-point Likert scale

	n	%
Confidence in evaluating qualitative papers at start of study1,2	2	22
Confidence in evaluating qualitative papers at end of study1,3	7	78
Considered this format of educational value	8	89
Enjoyed the new format	8	89
Considered this was a useful process	9	100

The majority of people at the start of the pilot study showed little confidence in appraising qualitative papers. Those who were more confident tended to be researchers who were already familiar with qualitative research methods. Owing to the small number in the study, the non-parametric Wilcoxon matched-pairs sign-rank test was used. A significant level of change in confidence was noted (z=2.535, P=0.012). The largest changes were seen in those with the least initial experience of reading and appraising qualitative papers.

Discussion

Despite increasing recognition of the value of the evidence-based journal club, there has been little, if any, appraisal of qualitative journal papers in a medical institution. However, there is an increasing body of evidence to show that qualitative research is considered to be as equally important as quantitative research. Rosser (Reference Rosser1999) argues that only the quality and the relevance of evidence blended with the context and values of the patient will achieve the benefit of medical evidence for patients. Greenhalgh & Hurwitz (Reference Greenhalgh and Hurwitz1999) highlight the importance of narrative, particularly its role in an evidence-based world. They emphasise the importance of listening and understanding the patients’ views and suggest that the process of taking a history can be comparable to methods of qualitative research. They further argue that it is important not to ignore the relevance of qualitative research, as it often seeks a deeper truth and aims to understand the significance of phenomena (Reference GreenhalghGreenhalgh, 1997; Reference Greenhalgh and HurwitzGreenhalgh & Hurwitz, 1999). Malterud (Reference Malterud2001) argues that qualitative enquiry could contribute to a broader understanding of medical science and that methods of patient care are based on more than just the results of clinical experiments.

The results of this study suggest it is possible to apply the principles of evidence-based journal clubs to qualitative papers. As the new-format club progressed, it became more difficult to find both qualitative and quantitative articles on the same topic. Also, many apparently qualitative papers appear to contain both quantitative and qualitative elements. Furthermore, some of the qualitative papers were very long, and owing to time constraints more guidance as to the areas within the paper to read was needed. Following the conclusion of the study, it was decided to separate the journal clubs into two separate sessions. Box 1 summarises practical tips to help run a qualitative journal club.

The multidisciplinary nature of the Department of Mental Health, Learning Disability meant that initially senior colleagues familiar with qualitative research were able to facilitate the journal clubs. As sessions progressed other people were increasingly able to facilitate the sessions using the guidelines (see Appendix) and the glossary of terms (available from the authors) as a resource. The criteria of the guidelines tended to be strictly adhered to during the early sessions, but as people became more confident, a less rigid adherence developed. This suggests that, with the help of the guidelines, it would be possible to extend this format to other settings, even if experience of reading and appraising qualitative research was minimal.

Further benefits are that members of the department are now more aware of qualitative methods and may be more comfortable in using such methods in research projects - an evaluation supported by Owen et al (Reference Owen, House and Worrall1995).

The pilot study has shown that the critical appraisal of both qualitative and quantitative papers can easily be introduced to an existing journal club. It suggests that there are advantages achieved by further developing critical appraisal skills to include qualitative research papers.

Appendix

Declaration of interest

None.