Hostname: page-component-89b8bd64d-x2lbr Total loading time: 0 Render date: 2026-05-08T02:26:58.599Z Has data issue: false hasContentIssue false
  • English
  • Français

On the potential distortions of highly cited papers in emerging research fields: A critical appraisal

Published online by Cambridge University Press:  15 July 2019

Edoardo G. Ostinelli Open the ORCID record for Edoardo G. Ostinelli [Opens in a new window] ,
Orsola Gambini  and
Armando D'Agostino
Edoardo G. Ostinelli
Affiliation:
Department of Health Sciences, University of Milan, 20142 Milan, Italy. edoardo.ostinelli@unimi.itorsola.gambini@unimi.itarmando.dagostino@unimi.ithttp://www.diss.unimi.it
Orsola Gambini
Affiliation:
Department of Health Sciences, University of Milan, 20142 Milan, Italy. edoardo.ostinelli@unimi.itorsola.gambini@unimi.itarmando.dagostino@unimi.ithttp://www.diss.unimi.it
Armando D'Agostino
Affiliation:
Department of Health Sciences, University of Milan, 20142 Milan, Italy. edoardo.ostinelli@unimi.itorsola.gambini@unimi.itarmando.dagostino@unimi.ithttp://www.diss.unimi.it
Get access Rights & Permissions [Opens in a new window]

Abstract

Citation-based metrics are increasingly used as a proxy to define representative, considerable, or significant papers. We challenge this belief by taking into account factors that may play a role in providing citations to a manuscript and whether/how those highly cited studies could shape a scientific field. A different approach to summarisation of relevant core publications within a topic is proposed.

Information

Type
Open Peer Commentary
Information
Behavioral and Brain Sciences , Volume 42 , 2019 , e77
Check for updates
Copyright
Copyright © Cambridge University Press 2019 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Berer, K., Mues, M., Koutrolos, M., Rasbi, Z. Al, Boziki, M., Johner, C., Wekerle, H. & Krishnamoorthy, G. (2011) Commensal microbiota and myelin autoantigen cooperate to trigger autoimmune demyelination. Nature 479(7374):538–41. Available at: https://doi.org/10.1038/nature10554.Google Scholar
Bramer, W. M., Giustini, D. & Kramer, B. M. (2016) Comparing the coverage, recall and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: A prospective study. Systematic Reviews 5:39. Available at: https://doi.org/10.1186/s13643-016-0215-7.Google Scholar
Catalogue of Bias Collaboration, Aronson, J. K., Bankhead, C. & Nunan, D. (2017a) Hot stuff bias. In: Catalogue of bias. University of Oxford. Available at: https://catalogofbias.org/biases/hot-stuff-bias/.Google Scholar
Catalogue of Bias Collaboration, Plüddemann, A., Banerjee, A. & O'Sullivan, J. (2017b) Positive results bias. In: Catalogue of bias. University of Oxford. Available at: https://www.catalogueofbiases.org/biases/positive-results-bias.Google Scholar
Catalogue of Bias Collaboration, Spencer, E. A., Brassey, J. & Heneghan, C. (2017c) One-sided reference bias. In: Catalogue of bias. University of Oxford. Available at: https://www.catalogofbias.org/biases/one-sided-reference-bias.Google Scholar
Cheek, J., Garnham, B. & Quan, J. (2006) What's in a number? Issues in providing evidence of impact and quality of research(ers). Qualitative Health Research 16(3):423–35. Available at: https://doi.org/10.1177/1049732305285701.Google Scholar
Dickersin, K., Scherer, R. & Lefebvre, C. (1994) Identifying relevant studies for systematic reviews. British Medical Journal 309(6964):1286–91. Available at: https://doi.org/10.1136/bmj.309.6964.1286.Google Scholar
Greenberg, S. A. (2009) How citation distortions create unfounded authority: Analysis of a citation network. British Journal of Medicine 339:b2680. Available at: https://doi.org/10.1136/bmj.b2680.Google Scholar
Hafeez, D. M., Jalal, S. & Khosa, F. (2019) Bibliometric analysis of manuscript characteristics that influence citations: A comparison of six major psychiatry journals. Journal of Psychiatric Research 108:90–94. Available at: https://doi.org/10.1016/j.jpsychires.2018.07.010.Google Scholar
Ioannidis, J. P. A. (2005a) Contradicted and initially stronger effects in highly cited clinical research. JAMA 294(2):218–28.Google Scholar
Ioannidis, J. P. A. (2005b) Why most published research findings are false. PLoS Medicine 2(8):e124. Available at: https://doi.org/10.1371/journal.pmed.0020124.Google Scholar
Ioannidis, J. P. A., Boyack, K. W., Small, H., Sorensen, A. A. & Klavans, R. (2014) Bibliometrics: Is your most cited work your best? Nature 514:561–62. Available at: https://doi.org/10.1038/514561a.Google Scholar
Ioannidis, J. P. A. & Khoury, M. J. (2014) Assessing value in biomedical research: The PQRST of appraisal and reward. JAMA 312(5):483–84. Available at: https://doi.org/10.1001/jama.2014.6932.Google Scholar
Ioannidis, J. P. A. & Panagiotou, O. A. (2011) Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses. JAMA 305(21):2200–10. Available at: https://doi.org/10.1001/jama.2011.713.Google Scholar
Kulkarni, A. V., Aziz, B., Shams, I. & Busse, J. W. (2009) Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. JAMA 302(10):1092–96. Available at: https://doi.org/10.1001/jama.2009.1307.Google Scholar
Lefebvre, C., Manheimer, E. & Glanville, J. (2011) Searching for studies. In: Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011), ed. Higgins, J. P. T. & Green, S.. The Cochrane Collaboration. Available at: http://www.handbook.cochrane.org.Google Scholar
Mackinnon, S., Drozdowska, B. A., Hamilton, M., Noel-Storr, A. H., McShane, R. & Quinn, T. (2018) Are methodological quality and completeness of reporting associated with citation-based measures of publication impact? A secondary analysis of a systematic review of dementia biomarker studies. BMJ Open 8:e020331. Available at: https://doi.org/10.1136/bmjopen-2017-020331.Google Scholar
Rathbone, J., Carter, M., Hoffmann, T. & Glasziou, P. (2016) A comparison of the performance of seven key bibliographic databases in identifying all relevant systematic reviews of interventions for hypertension. Systematic Reviews 5:27. Available at: https://doi.org/10.1186/s13643-016-0197-5.Google Scholar
Seglen, P. O. (1998) Citation rates and journal impact factors are not suitable for evaluation of research. Acta Orthopaedica Scandinavica 69(3):224–29. Available at: https://doi.org/10.3109/17453679809000920.Google Scholar
Tajika, A., Ogawa, Y., Takeshima, N., Hayasaka, Y. & Furukawa, T. A. (2015) Replication and contradiction of highly cited research papers in psychiatry: 10-Year follow-up. British Journal of Psychiatry 207:357–62. Available at: https://doi.org/10.1192/bjp.bp.113.143701.Google Scholar
Tatsioni, A., Bonitsis, N. G. & Ioannidis, J. P. A. (2007) Persistence of contradicted claims in the literature. JAMA 298(21):2517–26. Available at: https://doi.org/10.1001/jama.298.21.2517.Google Scholar
Wieland, S. & Dickersin, K. (2005) Selective exposure reporting and Medline indexing limited the search sensitivity for observational studies of the adverse effects of oral contraceptives. Journal of Clinical Epidemiology 58(6):560–67. Available at: https://doi.org/10.1016/j.jclinepi.2004.11.018.Google Scholar