Introduction
In order to provide high-quality healthcare, clinicians are expected to have knowledge of the best available research evidence conducted in their field. They should be able to combine this knowledge with their experience and their patient's wishes when making clinical decisions, which is an approach known as evidence-based practice.Reference Dawes, Summerskill, Glasziou, Cartabellotta, Martin and Hopayian 1 This means that clinicians should be able to find and select publications that are relevant for clinical practice and which have sufficient methodological quality.Reference Grol and Grimshaw 2 , Reference Burton 3
There is a large quantity of publications in otolaryngology, of which a major proportion is thought to lack clinical relevance or be of insufficient quality.Reference Ioannidis, Greenland, Hlatky, Khoury, MacLeod and Moher 4 High-quality research is important because it can change clinical practice, foster patient experiences and improve healthcare benefits.
This study aimed to assess the quality of methods in otolaryngological studies. As most studies published concern the evaluation of treatment effects, and the assessment tools for such intervention studies are well established and accepted, we focused on the quality of such study publications.Reference Shin, Rauch, Wasserman, Coblens and Randolph 5 The quality of a study depends largely on the quality of reporting and the risk of bias of the applied study design. Without adequate reporting, risk of bias cannot be assessed.Reference Turner, Shamseer, Altman, Schulz and Moher 6 General standards and conditions for reporting intervention studies and avoiding risk of bias are widely known and available. 7 , 8
Clinicians rely, among other things, on impact factor to select the best available evidence. The impact factor is widely assumed to be an indicator of the quality of research journals.Reference Brody 9 It is calculated at the end of every year, based on the number of citations in the previous two-year period, relative to the number of publications in this period. 10 Researchers strive to publish in journals with the highest impact factor. Consequently, these journals are seen as the leading and most prestigious journals.Reference Seglen 11 , 12
For our study, we selected the leading field-specific and general medical journals. We extracted publications on treatment outcomes in otolaryngology, and we investigated their quality of reporting and risk of bias. We compared the findings based on the publication source, either selected otolaryngology journals or selected general medical journals. We expected to find a major portion of high-quality otolaryngology publications in the selected general medical journals.
Materials and methods
Publication search and classification
We defined our cohort of publications in the year 2012. Using the most recent impact factors, from 2011, we selected 10 otolaryngology journals with the highest impact factor and searched for citable articles (i.e. peer-reviewed publications) from 2010. 13 We ranked all medical journals in the 2011 Journal Citation Report 13 by their impact factor and selected 20 with the highest impact factor that were likely to publish otolaryngology-related research. We selected journals with a subspecialty related to otolaryngology (e.g. oncology, allergy, respiratory) and journals that transcend multiple disciplines (e.g. The Lancet, The BMJ, PLOS Medicine). Two authors (NK and KS) independently selected journals and searched for otolaryngology-related publications; initial disagreement was resolved by discussion until agreement was reached. We selected journals from both 2010 and 2011, because we expected to find few articles and aimed to include a sufficient number of studies to compare between journals.
Two authors (NK and KS) independently retrieved and reviewed all publications. We selected studies reporting first-hand and original data, conducted on living patients. We selected only clinical research, with a determinant and outcome relevant for patient care. We searched for therapeutic studies; that is, studies estimating the effect of an intervention on the course and outcome of a disease. We included all therapeutic intervention studies, both randomised and non-randomised studies, multiple and single group comparisons, and prospective and retrospective trials. Studies reporting on less than 10 patients were excluded. 14 Initial disagreement on the selection and categorisation of articles was discussed with a third author (GH) until agreement was reached; the selection is therefore based on a full consensus.
Publication quality assessment
Based on pre-defined criteria, we evaluated the quality of selected articles by their study design. Assessment involved the evaluation of: selection bias, notably the study design characteristic of treatment assignment by (1) random and (2) concealed allocation; information bias, notably standardisation of (3) treatments and (4) outcome assessments; performance bias, (5) blinding of outcome assessment, and attrition bias; and (6) completeness of reported data (Table I). 14 , Reference Grobbee and Hoes 15 The Cochrane Handbook includes criteria 1, 2, 5 and 6 in their assessment of bias.Reference Higgins and Green 16
Checklist items for assessment of therapeutic publications
When item information was not provided or not clearly reported, we rated it as insufficient. When item information was clearly reported, it was rated as sufficient. Studies were assigned a high reporting quality for reporting five or six items, a moderate reporting quality for reporting three or four items, and a low reporting quality for reporting zero, one or two items. For the classification, it did not matter which item was fulfilled: all items were assigned an equal weight.
After determining reporting quality, we assessed risk of bias per item. When the reporting allowed assessment (i.e. when reporting was rated sufficient), we rated the item as either satisfied or not satisfied. When the reporting quality was insufficient, the item was rated as not satisfied. Two authors (NK and KS) independently assessed articles and resolved initial disagreements by discussion.
Because selection bias is most important in therapeutic studies, these items were assigned most weight in terms of risk of bias.Reference Grobbee and Hoes 15 Studies that did not satisfy criteria 1 and 2 (random and concealed allocation) were considered to have a high risk of bias, even if they fulfilled all other items. Studies were classified as having a low risk of bias if they satisfied criteria 1 and 2 plus all other study design features. If studies satisfied criteria 1 and 2 but failed on one or two of the other four features, they were rated as having a moderate risk of bias. Studies that failed on more items were classified as having a high risk of bias.
According to current quality standards, studies should have good reporting quality, and moderate or low risk of bias.Reference Higgins, Altman, Sterne, Higgins and Green 17
Data analysis
We entered and analysed data using SPSS statistical software, version 23.0 (Chicago, Illinois, USA). We compared ordinal outcomes using the Mann–Whitney U test. Binary outcomes were tested using the chi-square test. We used a Bonferroni post-hoc correction to determine inflated risks of a type 1 error in cases of multiple testing. For prediction of outcomes, ordinal logistic regression was performed.
Results
The 10 selected otolaryngology journals had impact factors ranging from 1.8 to 2.8. The 20 selected medical journals had impact factors ranging from 6.0 to 101.8. For details on the selected journals and their impact factors, see Appendix 1.
Of 1500 citable articles in the otolaryngology journals, we identified 262 (17 per cent) therapeutic articles. Of these, 94 (36 per cent) had a high reporting quality (Table II); 7 (3 per cent) had a moderate risk of bias and 5 (2 per cent) had a low risk of bias (Table III).
Overall reporting quality
* Overall significance calculated using a Mann–Whitney U test. Post-hoc tests with a Bonferroni correction showed a significant difference between the moderate and high categories.
Overall risk of bias
* Overall significance calculated using a Mann–Whitney U test. Post-hoc tests with a Bonferroni correction showed significant differences between the low and high and between the moderate and high categories.
Of 10 967 citable articles in the selected medical journals, 183 (2 per cent) were original clinical studies related to otolaryngology. We identified 76 (42 per cent) therapeutic studies, 36 from 2010 and 40 from 2011. There were no statistical differences between the two years. Overall, 57 (75 per cent) had a high reporting quality (Table II); 10 (13 per cent) a moderate risk of bias and 8 (11 per cent) had a low risk of bias (Table III).
The results per checklist item are reported in Table IV. For the selected medical journals, only blinding was remarkably poorly reported. For the otolaryngology journals, both blinding and completeness of data were remarkably poorly reported. Other items were moderately to well reported. Risk of bias items were more often rated sufficient for publications in the selected medical journals. Randomisation, concealed allocation and blinding often had a low score, which explains the overall high risk of bias.
Reporting quality and risk of bias per checklist item
* Number of articles that adequately reported the item.
† Calculated using the chi-square test.
‡ Number of articles that were rated sufficient.
OR = odds ratio; CI = confidence interval
Table V shows the results of reporting quality against risk of bias. For a low or moderate reporting quality, the probability of low or moderate risk of bias was 0. High reporting quality was therefore a pre-requisite in order to even qualify for a low or moderate risk of bias.
Reporting quality against risk of bias
Data represent numbers and percentages of articles, unless indicated otherwise. *Probabilities calculated by an ordinal regression model, with p < 0.05 for fitting of the model (Nagelkerke's R2 = 0.23, goodness of fit test p > 0.05 and test of parallel lines p = 0.61).
The results per journal are shown in Figure 1 (reporting quality) and Figure 2 (risk of bias). The details can be found in Appendix 2. From the 10 selected otolaryngology journals, 1 journal did not publish original research. The top three otolaryngology journals published little clinical research (their focus is on fundamental research); the number one otolaryngology journal did not publish any therapeutic studies (Appendix 2).
Reporting quality per journal according to impact factor.
Risk of bias per journal according to impact factor.
Discussion
Otolaryngology journals have a relatively low impact factor compared to the selected medical journals. Overall, the reporting standards were met for 45 per cent of therapeutic otolaryngology publications, and 9 per cent met our quality standards (having a low or moderate risk of bias). This is the first time such a study has been carried out in otolaryngology.
As a result of our selection approach, we missed publications, but there is unlikely to be high-quality otolaryngology research published outside of our selected journals. For efficiency, we selected studies from 2010, and assumed that our sample from 2010 would be representative of the studies published in the previous and subsequent year. We therefore presume that including more studies from adjacent years would lead to a reduction in random error, but would not change the outcome of our study. We included studies from the year 2010 because we expected to find more articles available in full text. Publications from 2010 (and even from before that) are still widely used in daily practice and guidelines. This is because it usually takes several years before publications are used in daily practice.Reference Glasziou and Haynes 18
We could have introduced bias in our study because the authors and journals that published the studies we assessed were not blinded. Therefore, the researchers who evaluated the quality of the articles could have been influenced by its authors or by the impact factor of the journal it was published in. However, by performing a systematic evaluation, which was executed independently by two authors, we reduced this risk of bias.
In evaluating the quality of publications, we limited the methodological aspects assessed, to provide an overview of the results. Adding items to the current classification would not change the risk of bias, but it could have an effect on the reporting quality. We believe we chose valid items for our rating, as all are derived from the Cochrane Handbook. We did not include the selective reporting item because we did not compare outcomes between different studies.Reference Higgins and Green 16
There are important differences between otolaryngology journals and the selected medical journals. Our results show that the selected medical journals, which have a high impact factor, have better reporting quality and published considerably higher quality research. However, 76 per cent of the publications in these journals still had a high risk of bias, so a high impact factor is definitely not a guarantee of high quality. It was not possible to compare impact factors between journals, given the small differences in impact factor and the limited number of publications; nevertheless, Figures 1 and 2 show that there is variation between journals.
There are several explanations for the higher quality of publications in the selected medical journals. Medical journals can select from different research fields and are more attractive for authors to submit their research to, given their high impact factor. 12 As a result, general medical journals have the first and most comprehensive choice when selecting publications. Additionally, high impact factor journals have stricter rules about reporting, and often authors are requested to fulfil checklists (such as the Consolidated Standards of Reporting Trials (‘CONSORT’)). 7
Our study provides important information for both research and clinical practice. The results show that the reporting of study designs needs to improve, especially for the lower impact factor otolaryngology journals. Even if not all items for risk of bias can be met, the reporting quality can and should always be high. In order to achieve improvement and help reduce research waste, there is a significant role for authors, reviewers and journals to play.Reference Ioannidis, Greenland, Hlatky, Khoury, MacLeod and Moher 4 Better reporting of studies and improved methodological quality is warranted. Furthermore, researchers, reviewers and editors can play an important role.
In the context of clinical practice, it is important for medical doctors to realise that they should always critically assess an article, even if it is published in a high impact factor journal. It is also important to be aware that relevant otolaryngology articles are not only published in otolaryngology journals, but also in general medical journals.
-
• For good medical practice, it is important that studies of sufficient quality are published
-
• The majority of otolaryngology papers do not meet current quality standards
-
• General medical journals have a considerably higher impact factor and publish more research of higher quality
-
• Overall, the goal should be better reported and designed otolaryngological studies
Publications should meet other conditions, besides those concerning reporting and methodological quality. Studies should preferably report original data, and should not simply review or summarise existing data, or express opinions. The study should be relevant and innovative; that is, the outcome should preferably have an impact on current policy. The journal or publication should be open access; furthermore, publishing studies driven by publication rates (‘bean counting’), rather than by the above factors, must be avoided.Reference Ioannidis, Greenland, Hlatky, Khoury, MacLeod and Moher 4 , 12
Conclusion
The majority of therapeutic research in leading otolaryngology journals does not meet current standards for methodological quality. Selected medical journals, with a considerably higher journal impact factor, tend to publish research of higher quality. Nevertheless, almost half of their publications are of low quality. Both researchers and medical publishers have a responsibility to improve the quality of methods and reporting. Medical doctors should critically assess publications before applying their findings in daily practice, and should look outside their specialised literature when searching for high-quality evidence in their field.
Appendix 1 Impact factors (2011) for selected journals
Appendix 2 Reporting quality and risk of bias per journal
