Skip to main content Accessibility help

Methodology and reporting of systematic reviews and meta-analyses of observational studies in psychiatric epidemiology: Systematic review

  • Traolach S. Brugha (a1), Ruth Matthews (a1), Zoe Morgan (a1), Trevor Hill (a1), Jordi Alonso (a2) and David R. Jones (a3)...



Relatively little is known of the use of systematic review and synthesis methods of non-randomised psychiatric epidemiological studies, which play a vital role in aetiological research, planning and policy-making.


To evaluate reviews of psychiatric epidemiological studies of functional mental disorders that employed synthesis methods such as systematic review or meta-analysis, or other forms of quantitative review.


We searched the literature to identify appropriate reviews published during the period 1996 to April 2009. Selected reviews were evaluated using published review guidelines.


We found 106 reviews in total, of which 38 (36%) did not mention method of data abstraction from primary studies at all. Many failed to mention study quality, publication bias, bias and confounding. In 73 studies that performed a meta-analysis, 58 (79%) tested for heterogeneity and of these, 47 found significant heterogeneity. Studies that detected heterogeneity made some allowance for this. A major obstacle facing reviewers is the wide variation between primary studies in the use of instruments to measure outcomes and in sampling methods used.


Many deficiencies found in systematic reviews are potentially remediable, although synthesis of primary study findings in a field characterised by so many sources of heterogeneity will remain challenging.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Methodology and reporting of systematic reviews and meta-analyses of observational studies in psychiatric epidemiology: Systematic review
      Available formats

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Methodology and reporting of systematic reviews and meta-analyses of observational studies in psychiatric epidemiology: Systematic review
      Available formats

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Methodology and reporting of systematic reviews and meta-analyses of observational studies in psychiatric epidemiology: Systematic review
      Available formats


Corresponding author

Professor T. Brugha, Department of Health Sciences, Division of Clinical Psychiatry, Leicester General Hospital, Leicester LE5 4PW, UK. Email:


Hide All

Declaration of interest




Hide All
1 Derry, CJ, Derry, S, McQuay, HJ, Moore, RA. Systematic review of systematic reviews of acupuncture published 1996–2005. Clin Med 2006; 6: 381–6.
2 Peters, JL, Sutton, AJ, Jones, DR, Rushton, L, Abrams, KR. A systematic review of systematic reviews and meta-analyses of animal experiments with guidelines for reporting. J Environ Sci Health B 2006; 41: 1245–58.
3 Stroup, DF, Berlin, JA, Morton, SC, Olkin, I, Williamson, GD, Rennie, D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis of Observational Studies in Epidemiology (MOOSE) group. JAMA 2000; 283: 2008–12.
4 Sutton, AJ, Jones, DR, Abrams, KR, Sheldon, TA, Song, F. Systematic reviews and meta-analysis: a structured review of the methodological literature. J Health Serv Res Policy 1999; 4: 4955.
5 Brugha, TS, Bebbington, PE, Jenkins, R. A difference that matters: comparisons of structured and semi-structured diagnostic interviews of adults in the general population. Psychol Med 1999; 29: 1013–20.
6 Kessler, RC, Abelson, J, Demler, O, Escobar, JI, Gibbon, M, Guyer, ME, et al. Clinical calibration of DSM-IV diagnoses in the World Mental Health (WMH) version of the World Health Organization (WHO) Composite International Diagnostic Interview (WMHCIDI). Int J Methods Psychiatr Res 2004; 13: 122–39.
7 Thompson, C, Thompson, C. The Instruments of Psychiatric Research. Wiley, 1988.
8 EPREMED Consortium. European Policy Information Research for Mental Disorders: Final Report of the Project SANCO 2004123. EU Commission, 2008.
9 White, VJ, Glanville, JM, Lefebvre, C, Sheldon, TA. A statistical approach to designing search filters to find systematic reviews: objectivity enhances accuracy. J Information Sci 2001; 27: 357–70.
10 Sutton, AJ, Abrams, KR, Jones, DR, Sheldon, TA, Song, F. Methods for Meta-analysis in Medical Research. Wiley, 2000.
11 Egger, M, Davey Smith, G, Schneider, M. Systematic reviews of observational studies. In Systematic Reviews in Health Care: Meta-analysis in Context (eds Egger, M, Davey Smith, G, Altman, D): 211–27. BMJ Books, 2001.
12 Saha, S, Chant, DC, Welham, JL, McGrath, JJ. The incidence and prevalence of schizophrenia varies with latitude. Acta Psychiatr Scand 2006; 114: 36–9.
13 Saha, S, Welham, J, Chant, D, McGrath, J. Incidence of schizophrenia does not vary with economic status of the country: evidence from a systematic review. Soc Psychiatry Psychiatr Epidemiol 2006; 41: 338–40.
14 Singer, GHS. Meta-analysis of comparative studies of depression in mothers of children with and without developmental disabilities. Am J Ment Retard 2006; 111: 155–69.
15 Saha, S, Chant, D, McGrath, J. Meta-analyses of the incidence and prevalence of schizophrenia: conceptual and methodological issues. Int J Methods Psychiatr Res 2008; 17: 5561.
16 McGrath, J, Saha, S, Welham, J, El Saadi, O, MacCauley, C, Chant, D. A systematic review of the incidence of schizophrenia: the distribution of rates and the influence of sex, urbanicity, migrant status and methodology. BMC Med 2004; 2: 13.
17 Saha, S, Chant, D, Welham, J, McGrath, J. A systematic review of the prevalence of schizophrenia [see comment]. PLoS Med 2005; 2: e141.
18 McGrath, J, Saha, S, Chant, D, Welham, J. Schizophrenia: a concise overview of incidence, prevalence, and mortality. Epidemiol Rev 2008; 30: 6776.
19 Waraich, P, Goldner, EM, Somers, JM, Hsu, L. Prevalence and incidence studies of mood disorders: a systematic review of the literature [see comment]. Can J Psychiatry 2004; 49: 124–38.
20 Goldner, EM, Hsu, L, Waraich, P, Somers, JM. Prevalence and incidence studies of schizophrenic disorders: a systematic review of the literature [see comment]. Can J Psychiatry 2002; 47: 833–43.
21 Somers, JM, Goldner, EM, Waraich, P, Hsu, L. Prevalence and incidence studies of anxiety disorders: a systematic review of the literature. Can J Psychiatry 2006; 51: 100–13.
22 Roy, MA, Maziade, M, Labbe, A, Merette, C. Male gender is associated with deficit schizophrenia: a meta-analysis. Schizophr Res 2001; 47: 141–7.
23 Scott, J, McNeill, Y, Cavanagh, J, Cannon, M, Murray, R. Exposure to obstetric complications and subsequent development of bipolar disorder. Systematic review. Br J Psychiatry 2006; 189: 311.
24 Cantor-Graae, E, Selten, JP. Schizophrenia and migration: a meta-analysis and review [see comment]. Am J Psychiatry 2005; 162: 1224.
25 Geddes, JR, Verdoux, H, Takei, N, Lawrie, SM, Bovet, P, Eagles, JM, et al. Schizophrenia and complications of pregnancy and labor: an individual patient data meta-analysis. Schizophr Bull 1999; 25: 413–23.
26 Mojtabai, R. Duration of illness and structure of symptoms in schizophrenia [see comment]. Psychol Med 1999; 29: 915–24.
27 McLeod, BD, Weisz, JR, Wood, JJ. Examining the association between parenting and childhood depression: a meta-analysis. Clin Psychol Rev 2007; 27: 9861003.
28 Cole, MG, Dendukuri, N. Risk factors for depression among elderly community subjects: a systematic review and meta-analysis. Am J Psychiatry 2003; 160: 1147–56.
29 Robertson, E, Grace, S, Wallington, T, Stewart, DE. Antenatal risk factors for postpartum depression: a synthesis of recent literature [see comment]. Gen Hosp Psychiatry 2004; 26: 289–95.
30 Wohl, M, Gorwood, P. Paternal ages below or above 35 years old are associated with a different risk of schizophrenia in the offspring. Eur Psychiatry 2007; 22: 22–6.
31 Cole, MG, Bellavance, F, Mansour, A. Prognosis of depression in elderly community and primary care populations: a systematic review and meta-analysis. Am J Psychiatry 1999; 156: 1182–9.
32 Dickens, C, McGowan, L, Clark, CD, Creed, F. Depression in rheumatoid arthritis: a systematic review of the literature with meta-analysis. Psychosom Med 2002; 64: 5260.
33 King, M, Semlyen, J, Tai, SS, Killaspy, H, Osborn, D, Popelyuk, D, et al. A systematic review of mental disorder, suicide, and deliberate self harm in lesbian, gay and bisexual people. BMC Psychiatry 2008; 8: 70.
34 Ali, S, Stone, MA, Peters, JL, Davies, MJ, Khunti, K. The prevalence of co-morbid depression in adults with Type 2 diabetes: a systematic review and meta-analysis. Diabet Med 2006; 23: 1165–73.
35 Gilbody, S, Lightfoot, T, Sheldon, T. Is low folate a risk factor for depression? A meta-analysis and exploration of heterogeneity. J Epidemiol Commun Health 2007; 61: 631–7.
36 Sutton, A, Duval, S, Tweedie, R, Abrams, K, Jones, D. Empirical assessment of effect of publication bias on meta-analyses. BMJ 2000; 320: 1574–7.
37 Costello, EJ, Erkanli, A, Angold, A. Is there an epidemic of child or adolescent depression? J Child Psychol Psychiatry 2006; 47: 1263–71.
38 Gaynes, BN, Gavin, N, Meltzer-Brody, S, Lohr, KN, Swinson, T, Gartlehner, G, et al. Perinatal depression: prevalence, screening accuracy, and screening outcomes. Evid Rep Technol Assess (Summ) 2005; 119: 18.
39 Fazel, S, Danesh, J. Serious mental disorder in 23000 prisoners: a systematic review of 62 surveys [see comment]. Lancet 2002; 359: 545–50.
40 Gavin, NI, Gaynes, BN, Lohr, KN, Meltzer-Brody, S, Gartlehner, G, Swinson, T. Perinatal depression: a systematic review of prevalence and incidence. Obstet Gynec 2005; 106: 1071–83.
41 Abrams, RC, Horowitz, SV, Zweig, RA. Personality disorders after age 50: a meta-analytic review of the literature. In Personality Disorders in Older Adults: Emerging Issues in Diagnosis and Treatment (eds Rosowsky, E, Abrams, RC, Zweig, RA): 5568. Erlbaum, 1999.
42 DiMaggio, C, Galea, S. The behavioral consequences of terrorism: a meta-analysis. Acad Emerg Med 2006; 13: 559–66.
43 Skeem, JL, Edens, JF, Camp, J, Colwell, LH. Are there ethnic differences in levels of psychopathy? A meta-analysis. Law Human Behav 2004; 28: 505–27.
44 Light, RJ, Pillemar, DB. Summing Up: The Science of Reviewing Research. Harvard University Press, 1984.
45 Mendelson, T, Rehkopf, DH, Kubzansky, LD. Depression among Latinos in the United States: a meta-analytic review. J Consult Clin Psychol 2008; 76: 355–66.
46 Rosenthal, R, Rubin, DB. Comparing significance levels of independent studies. Psychol Bull 1979; 86: 1165–8.
47 Maag, JW, Reid, R. Depression among students with learning disabilities: assessing the risk. J Learn Disabil 2006; 39: 310.
48 Bennett, HA, Einarson, A, Taddio, A, Koren, G, Einarson, TR. Prevalence of depression during pregnancy: systematic review. Obstet Gynec 2004; 103: 698709 [erratum 1344].
49 Chida, Y, Hamer, M, Steptoe, A. A bidirectional relationship between psychosocial factors and atopic disorders: a systematic review and meta-analysis. Psychosom Med 2008; 70: 102–16.
50 Dragovic, M, Hammond, G. Handedness in schizophrenia: a quantitative review of evidence. Acta Psychiatr Scand 2005; 111: 410–9.
51 Lorant, V, Deliege, D, Eaton, W, Robert, A, Philippot, P, Ansseau, M. Socioeconomic inequalities in depression: a meta-analysis [see comment]. Am J Epidemiol 2003; 157: 98112.
52 Aleman, A, Kahn, RS, Selten, JP. Sex differences in the risk of schizophrenia: evidence from meta-analysis. Arch Gen Psychiatry 2003; 60: 565–71.
53 Pinquart, M, Sorensen, S. Differences between caregivers and noncaregivers in psychological health and physical health: a meta-analysis [see comment]. Psychol Aging 2003; 18: 250–67.
54 Kim, D. Blues from the neighborhood? Neighborhood characteristics and depression. Epidemiol Rev 2008; 30: 101–17.
55 Fazel, S, Doll, H, Langstrom, N. Mental disorders among adolescents in juvenile detention and correctional facilities: a systematic review and metaregression analysis of 25 surveys. J Am Acad Child Adolesc Psychiatry 2008; 47: 1010–9.
56 Breh, DC, Seidler, GH. Is peritraumatic dissociation a risk factor for PTSD? J Trauma Dissoc 2007; 8: 5369.
57 Starr, LR, Davila, J. Excessive reassurance seeking, depression, and interpersonal rejection: a meta-analytic review. J Abnorm Psychol 2008; 117: 762–75.
58 Swinnen, SGHA, Selten, J-P. Mood disorders and migration: meta-analysis. Br J Psychiatry 2007; 190: 610.
59 Orth, U, Wieland, E. Anger, hostility, and posttraumatic stress disorder in trauma-exposed adults: a meta-analysis. J Consult Clin Psychol 2006; 74: 698706.
60 Woodberry, KA, Giuliano, AJ, Seidman, LJ. Premorbid IQ in schizophrenia: a meta-analytic review. Am J Psychiatry 2008; 165: 579–87.
61 Porter, M, Haslam, N. Predisplacement and postdisplacement factors associated with mental health of refugees and internally displaced persons: a meta-analysis. JAMA 2005; 294: 602–12.
62 Folsom, D, Jeste, DV. Schizophrenia in homeless persons: a systematic review of the literature. Acta Psychiatr Scand 2002; 105: 404–13.
63 Kleintjes, S, Flisher, AJ, Fick, M, Railoun, A, Lund, C, Molteno, C, et al. The prevalence of mental disorders among children, adolescents and adults in the Western Cape, South Africa. S Afr Psychiatry Rev 2006; 9: 157–60.
64 Friedl, MC, Draijer, N, de Jonge, P. Prevalence of dissociative disorders in psychiatric in-patients: the impact of study characteristics. Acta Psychiatr Scand 2000; 102: 423–8.
65 Myers, JE, Thompson, ML. Meta-analysis and occupational epidemiology. Occup Med (Lond) 1998; 48: 99101.
66 Fryers, T, Brugha, T, Morgan, Z, Smith, J, Hill, T, Carta, M, et al. Prevalence of psychiatric disorder in Europe: the potential and reality of meta-analysis. Soc Psychiatry Psychiatr Epidemiol 2004; 39: 899905 (erratum Soc Psychiatry Psychiatr Epidemiol 2005; 40: 85).
67 Sutton, AJ, Abrams, KR, Jones, DR, Sheldon, TA, Song, F. Systematic reviews of trials and other studies. Health Technol Assess 1998; 2: 1276.
68 Egger, M, Smith, GD, Altman, DG. Systematic Reviews in Health Care: Meta-analysis in Context. BMJ Books, 2001.
69 Alonso, J, Angermeyer, MC, Bernert, S, Bruffaerts, R, Brugha, TS, Bryson, H, et al. Sampling and methods of the European Study of the Epidemiology of Mental Disorders (ESEMeD) project. Acta Psychiatr Scand Suppl 2004; 420: 820.
70 Demyttenaere, K, Bruffaerts, R, Posada-Villa, J, Gasquet, I, Kovess, V, Lepine, JP, et al. Prevalence, severity, and unmet need for treatment of mental disorders in the World Health Organization World Mental Health Surveys. JAMA 2004; 291: 2581–90.
71 Centre for Reviews and Dissemination. Strategies for Searching for Observational Studies. University of York CRD, 2006.
72 Cochrane Collaboration. The Cochrane Policy Manual, issue 1, 2007 (
73 Moher, CD, Eastwood, S, Olkin, I, Rennie, D, Stroup, DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 1999; 354: 1896–900.
74 Boaz, A, Ashby, D, Denyer, D, Egan, M, Harden, A, Jones, DR, et al. A multitude of syntheses: a comparison of five approaches from diverse policy fields. Evid Policy 2006; 2: 479–52.
Type Description Title
Supplementary materials

Brugha et al. supplementary material
Supplementary Material

 PDF (510 KB)
510 KB

Methodology and reporting of systematic reviews and meta-analyses of observational studies in psychiatric epidemiology: Systematic review

  • Traolach S. Brugha (a1), Ruth Matthews (a1), Zoe Morgan (a1), Trevor Hill (a1), Jordi Alonso (a2) and David R. Jones (a3)...


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed

Methodology and reporting of systematic reviews and meta-analyses of observational studies in psychiatric epidemiology: Systematic review

  • Traolach S. Brugha (a1), Ruth Matthews (a1), Zoe Morgan (a1), Trevor Hill (a1), Jordi Alonso (a2) and David R. Jones (a3)...
Submit a response


Methodology and reporting of systematic reviews and meta-analyses

Fred Lerner, Information Scientist
14 November 2012

In "Methodology and reporting of systematic reviews and meta-analysesof observational studies in psychiatric epidemiology: a systematic review"Brugha et al discussed the search strategies employed by the compilers of the systematic reviews and meta-analyses that they analysed. We wish that they had pursued this issue in more detail.

Brugha et al wrote that "Authors generally gave comprehensive detailsof search strategies employed, including details of electronic databases searched, exact search terms, dates covered by search and other methods used." In examining many systematic reviews and meta analyses of psychiatric literature in the course of our work with the PILOTS Database,an online index to the worldwide literature of posttraumatic stress disorder that we produce at the National Center for PTSD, we have often observed the inadequacy of the search strategies described by their authors. It is evident that few of these studies have made proper use of the controlled indexing vocabularies used by databases such as MEDLINE andPsycINFO or displayed evidence that the thesauri in which these controlledvocabularies are published have been consulted. The reader familiar with these tools will often have reason to question the reliance that can be based on systematic reviews and meta analyses whose authors have not consulted them.

In Lerner and Hamblen (2010) we explain in detail the importance of properly using controlled vocabularies in the compilation of systematic reviews and meta-analyses, demonstrate problems that may arise from not doing so, and offer suggestions for improving the literature searches underlying these compilations.

REFERENCELerner, F; Hamblen, J.L. (2010) . Surveying the traumatic stress literature: the effective use of bibliographic databases in preparing literature reviews and meta-analyses. Journal of Traumatic Stress, 23, 819-822.

... More

Conflict of interest: (The authors have no financial interest in the work reported. The PILOTS Database is produced by the National Center for PTSD, a program of the U.S. Department of Veterans Affairs.)

Write a reply

Methodology and reporting of systematic reviews and meta-analyses of observational studies in psychiatric epidemiology

Golam M. Khandaker, Wellcome Trust Clinical Research Fellow
11 July 2012

We agree with Brugha and colleagues that the field of psychiatric epidemiology poses particular challenges to systematic reviewers [1]. Heterogeneity between studies may arise from differences in outcomes and it is certainly true that psychiatry still lacks 'biologically based gold standards' regarding their definition. However, we disagree that these arenecessarily linked. For the purpose of systematic reviews and meta-analyses the issue is not to what extent an outcome is definable, with biological tests or otherwise. Rather, the key issue is how comparable individual studies are in their measurement of whatever outcome they use. For example, studies of schizophrenia defined by standard diagnostic toolssuch as the ICD-10, and applied using common operationalised-criteria, should be looking at the same construct to a large extent. Definitions of physical health conditions also vary, even when specific tests are available for diagnosis. For example, definition of hypertension is not the same across national guidelines used in the USA and Europe [2, 3]. We acknowledge that differences exist in psychiatry between diagnostic tools which attempt to define the same or similar conditions, such as schizophrenia in ICD-10 vs. DSM-IV. Often studies include outcomes, such as psychotic, depressive or other symptoms instead of a diagnostic category which can make comparison harder. Therefore, we recommend systematic reviews pay close attention to how outcome is defined in individual studies so that they are comparable. This should be considered as part of mandatory reporting of individual study quality in systematic reviews, as we have recently done [4], and as Bhugra et al. rightly encourage. Biologically-based outcomes may help in due course but, currently, attention needs to be focused on the principle of comparabilityof outcomes we have now.

Another important contributor to heterogeneity is variation in exposure measurement which we think needs to be emphasised. In our systematic review and meta-analysis of premorbid IQ in schizophrenia we found that the effect size varied as a result of differences in methods and age of IQ testing [5]. Therefore, as well as ensuring that measurement of exposure is similar across included studies, differences should be explored further by sub-group and sensitivity analysis.

With regards to meta-analysis, combining methodologically incomparable studies will have serious implications for the validity and generalizability of findings. For example, a pooled odds ratio of 1.34 wasreported for schizophrenia for exposure to Herpes Simples Virus Type 2 (HSV-2) in a recent meta-analysis [6]. Unfortunately, this tells us very little because the reviewers conflated studies which considered HSV-2 infection in early life and subsequent schizophrenia (i.e. prospective designs) with those which considered the prevalence of infection in peoplewith established schizophrenia (i.e. a cross-sectional design). Such differences may not be picked up by tests for heterogeneity. The responsibility that individual studies are sufficiently comparable in design and other aspects in order to justify combining their results in a meta-analysis lies with researchers conducting systematic reviews, as wellas with the reader.

It was not clear from the meta-review how many original reviews followed some kind of guidelines. Guidelines for reporting of systematic reviews including those of observational studies already exist, such as PRISMA and MOOSE. They include comprehensive checklists for the assessmentof outcome, exposure, effects of bias and confounding among others in individual studies. We believe more widespread use of these guidelines, something that can be mandated by journal editors and peer reviewers should greatly increase comparability of individual studies, and overall, lead to an improvement in the quality of systematic reviews and meta-analyses.

Reference:1.Brugha, T.S., et al., Methodology and reporting of systematic reviews and meta-analyses of observational studies in psychiatric epidemiology: systematic review. Br J Psychiatry, 2012. 200: p. 446-53.2.Chobanian, A.V., et al., The Seventh Report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure: the JNC 7 report. Jama, 2003. 289(19): p. 2560-72.3.Mancia, G., et al., 2007 Guidelines for the management of arterial hypertension: The Task Force for the Management of Arterial Hypertension of the European Society of Hypertension (ESH) and of the European Society of Cardiology (ESC). Eur Heart J, 2007. 28(12): p. 1462-536.4.Kirkbride, J.B., et al., Incidence of Schizophrenia and Other Psychosesin England, 1950-2009: A Systematic Review and Meta-Analyses. PLoS One, 2012. 7(3): p. e31660.5.Khandaker, G.M., et al., A quantitative meta-analysis of population-based studies of premorbid intelligence and schizophrenia. Schizophr Res, 2011. 132(2-3): p. 220-7.6.Arias, I., et al., Infectious agents associated with schizophrenia: A meta-analysis. Schizophr Res, 2011.

... More

Conflict of interest: None declared

Write a reply

Improving reporting quality in systematic reviews of reviews

Jay P. Singh, Postdoctoral Research Fellow
06 June 2012

As the number of primary studies in a literature grows, reviews of the field will be published. Such reviews are helpful in that they allow large quantities of information to be quickly assimilated by readers, be they researchers, clinicians, policymakers, or non-professionals. With some fields having had dozens or even hundreds of reviews published, a novel reviewing methodology has recently emerged, that of a metareview, ora systematic review of reviews. Brugha et al. present a metareview of observational studies in psychiatric epidemiology. The four main strengthsof using metareview methodology are exemplified in this work: First, metareviews allow researchers to investigate the general quality of the review literature on a given topic. This may be accomplished using a standardized reporting checklist. Second, metareviews can advance researchers' understanding of heterogeneity. If consistent evidence of between-study variability is found, a metareview can suggest that sources of heterogeneity be investigated using methods such as subgroup analysis or metaregression regardless of the statistical significance of any meta-analytic findings. Third, by using the review as the unit of analysis, theconsistency of publication bias findings can be explored. Fourth, which effect sizes are commonly used to summarize study findings can be identified and their strengths and weaknesses described. Finally, metareviews can highlight areas that could benefit from further research at the secondary level.

According to guidelines published by the Cochrane Collaboration,1 literature reviews should assess, at a minimum, whether included studies stated their objectives a priori, report a reproducible search strategy, include unpublished reports, provide a list of included studies, summarizethe sample and design characteristics of included investigations, conduct an inter-rater reliability check to assess the consistency of the data extraction process, explore sources of heterogeneity, assess evidence of publication bias, and disclose conflicts of interest. To encourage a transparent and consistent reporting of such characteristics in future metareviews, a 20-item checklist - the Metareview Assessment of Reporting Quality (MARQ) - was developed in consultation with expert systematic reviewers (available from the author on request). The use of similar reporting checklists is recommended by over 200 peer-reviewed journals.

A systematic search to February 11, 2012 of PsycINFO, MEDLINE, EMBASE, and CINAHL identified four previously published metareviews of themedical sciences literature. Topics covered included interventions in key areas of liaison psychiatry,2 the utility of violence risk assessment in forensic psychiatry,3 the short-term effectiveness and safety of antidepressants for treating depression,4 and the epidemiology and reporting characteristics of systematic reviews in the field of medicine.5When applied to these four metareviews, an average of 15 (SD = 3) MARQ criteria were met, indicating a moderate level of reporting quality. In order to test inter-rater reliability, a research assistant working independently of the author (Christie Leung) coded each of the four metareviews on the 20 MARQ criteria. A high level of inter-rater agreementwas established (kappa = 0.93), supporting use of the checklist in future research to ensure transparent and consistent reporting of metareview methodology.

References1. Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. West Sussex: John Wiley & Sons; 2008.2. Ruddy R, House A. Meta-review of high-quality systematic reviews of interventions in key areas of liaison psychiatry. Br J Psychiatry 2005, 187: 109-20.3. Singh JP, Fazel S. Forensic risk assessment: a metareview. Crim JusticeBehav 2010, 37: 965-88.4. Cipriani A, Geddes JR, Furukawa TA, Barbui C. Metareview on short-term effectiveness and safety of antidepressants for depression: an evidence-based approach to inform clinical practice. Can J Psychiatry 2007, 52: 553-62. 5. Moher D, Tetzlaff J, Triccol AC, Sampson M, Altman DG. Epidemiology andreporting characteristics of systematic reviews. PLoS Med 2007, 4: 447-55.

CorrespondenceJay P. Singh, Department of Mental Health Law and Policy, University of South Florida, 13301 Bruce B. Downs Blvd., Tampa, FL, 33612. Tel: 813-974-9032; Fax: 813-974-6411

Declaration of InterestThe author declares no competing interests.

... More

Conflict of interest: None declared

Write a reply


Reply to: Submit a response

Your details

Conflicting interests

Do you have any conflicting interests? *