Skip to main content
    • Aa
    • Aa

Inadequate reporting of trials compromises the applicability of systematic reviews

  • Gerald Gartlehner (a1), Patricia Thieda (a2), Richard A. Hansen (a2), Laura C. Morgan (a2), Janelle A. Shumate (a3) and Daniel B. Nissman (a4)...

Background: Uncertainty about the applicability of controlled trial findings is an increasing concern for clinicians and policy decision makers. This study aimed to determine whether information reported in studies included in systematic reviews was adequate enough to assess their applicability.

Methods: We used the databases of four recently conducted systematic reviews on the comparative efficacy and safety of second-generation antidepressants, inhaled corticosteroids, Alzheimer's drugs, and targeted immune modulators. We developed and pilot-tested a questionnaire to assess the adequacy of reporting with respect to seven previously validated criteria of study design that distinguish explanatory from pragmatic studies. For each of the 137 included studies, two reviewers independently assessed the adequacy of reporting.

Results: Overall, only 12 percent of the included studies provided sufficient information to reliably distinguish explanatory from pragmatic studies. The areas with the greatest lack of reporting were the setting of the study, methods of adverse event assessment, and sample size considerations to determine a minimally important difference from a patient perspective.

Conclusions: Substantial shortcomings in reporting exist in aspects of study design important to determine whether a study is applicable to specific populations of interest.

Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

2. DG Altman , KF Schulz , D Moher , The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001;134:663694.

3. D. Atkins Creating and synthesizing evidence with decision makers in mind: integrating evidence from clinical trials and other study designs. Med Care. 2007;45 (Suppl 2):S16S22.

4. AW Chan , A Hrobjartsson , MT Haahr , Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291:24572465.

5. JC Dumville , DJ Torgerson , CE Hewitt . Reporting attrition in randomized controlled trials. BMJ. 2006;332:969971.

6. M Ethgen , I Boutron , G Baron , Reporting of harm in randomized, controlled trials of nonpharmacologic treatment for rheumatic disease. Ann Intern Med. 2005;143:2025.

7. M Fortin , J Dionne , G Pinho , Randomized controlled trials: do they have external validity for patients with multiple comorbidities? Ann Fam Med. 2006;4:104108.

9. G Gartlehner , RA Hansen , D Nissman , A simple and valid tool distinguished efficacy from effectiveness studies. J Clin Epidemiol. 2006;59:10401048.

11. M Godwin , L Ruhland , I Casson , Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity. BMC Med Res Methodol. 2003;3:28.

12. CP Gross , R Mallory , A Heiat , Reporting the recruitment process in clinical trials: who are these patients and how did they get there? Ann Intern Med. 2002;137:1016.

15. B Haynes . Can it work? Does it work? Is it worth it? The testing of healthcare interventions is evolving. BMJ. 1999;319:652653.

16. L Hazell , SA Shakir . Under-reporting of adverse drug reactions: a systematic review. Drug Saf. 2006;29:385396.

17. C Hewitt , S Hahn , DJ Torgerson , Adequacy and reporting of allocation concealment: review of recent trials published in four general medical journals. BMJ. 2005;330:10571058.

18. K Huwiler-Muntener , P Juni , C Junker , Quality of reporting of randomized trials as a measure of methodological quality. JAMA. 2002;287:28012804.

19. JP Ioannidis , SJ Evans , PC Gotzsche , Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med. 2004;141:781788.

21. A Malmivaara , BW Koes , LM Bouter , Applicability and clinical relevance of results in randomized controlled trials: the Cochrane review on exercise therapy for low back pain as an example. Spine. 2006;31:14051409.

23. D Moher , KF Schulz , D Altman . The CONSORT Statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials 2001. Explore (NY). 2005;1:4045.

24. M Mullner , H Matthews , DG Altman . Reporting on statistical methods to adjust for confounding: a cross-sectional survey. Ann Intern Med. 2002;136:122126.

25. CD Mulrow . Rationale for systematic reviews. BMJ. 1994;309:597599.

28. D Schwartz , J Lellouch . Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967;20:637648.

30. T Sokka , T Pincus . Eligibility of patients in routine care for major clinical trials of anti-tumor necrosis factor alpha agents in rheumatoid arthritis. Arthritis Rheum. 2003;48:313318.

31. M Zwarenstein , S Treweek , JJ Gagnier , Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337:a2390.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

International Journal of Technology Assessment in Health Care
  • ISSN: 0266-4623
  • EISSN: 1471-6348
  • URL: /core/journals/international-journal-of-technology-assessment-in-health-care
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Full text views

Total number of HTML views: 2
Total number of PDF views: 7 *
Loading metrics...

Abstract views

Total abstract views: 38 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 23rd May 2017. This data will be updated every 24 hours.