Skip to main content
×
×
Home

Does Peer Review Identify the Best Papers? A Simulation Study of Editors, Reviewers, and the Scientific Publication Process

  • Justin Esarey (a1)
Abstract
ABSTRACT

How does the structure of the peer review process, which can vary among journals, influence the quality of papers published in a journal? This article studies multiple systems of peer review using computational simulation. I find that, under any of the systems I study, a majority of accepted papers are evaluated by an average reader as not meeting the standards of the journal. Moreover, all systems allow random chance to play a strong role in the acceptance decision. Heterogeneous reviewer and reader standards for scientific quality drive both results. A peer review system with an active editor—that is, one who uses desk rejection before review and does not rely strictly on reviewer votes to make decisions—can mitigate some of these effects.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Does Peer Review Identify the Best Papers? A Simulation Study of Editors, Reviewers, and the Scientific Publication Process
      Available formats
      ×
      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about sending content to Dropbox.

      Does Peer Review Identify the Best Papers? A Simulation Study of Editors, Reviewers, and the Scientific Publication Process
      Available formats
      ×
      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about sending content to Google Drive.

      Does Peer Review Identify the Best Papers? A Simulation Study of Editors, Reviewers, and the Scientific Publication Process
      Available formats
      ×
Copyright
References
Hide All
Baxt William G., Waeckerle Joseph F., Berlin Jesse A., and Callaham Michael L.. 1998. “Who Reviews the Reviewers? Feasibility of Using a Fictitious Manuscript to Evaluate Reviewer Performance.” Annals of Emergency Medicine 32 (3): 310–17.
Bornmann Lutz, Mutz Rudiger, and Daniel Hans-Dieter. 2010. “A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants.” PLoS One 5 (12): e14331. doi:10.1371/journal.pone.0014331.
Breuning Marijke, Backstrom Jeremy, Brannon Jeremy, Isaak Gross Benjamin, and Widmeier Michael. 2015. “Reviewer Fatigue? Why Scholars Decline to Review Their Peers’ Work.” PS: Political Science & Politics 48 (4): 595600. doi:10.1017/S1049096515000827.
Breuning Marijke, and Sanders Kathryn. 2007. “Gender and Journal Authorship in Eight Prestigious Political Science Journals.” Political Science & Politics 40 (2): 347–51. doi:10.1017/S1049096507070564.
Cole Stephen, Cole Jonathan R., and Simon Gary A.. 1981. “Chance and Consensus in Peer Review.” Science 214 (4523): 881–6.
Coronel Ruben, and Opthof Tobias. 1999. “The Role of the Reviewer in Editorial Decison-Making.” Cardiovascular Research 43: 261–4.
Djupe Paul. 2015. “Peer Reviewing in Political Science: New Survey Results.” PS: Political Science & Politics 48 (2): 346–51.
Garand James C., and Giles Micheal W.. 2003. “Journals in the Discipline: A Report on a New Survey of American Political Scientists.” PS: Political Science & Politics 36 (2): 293308.
Giles Micheal W., and Garand James C.. 2007. “Ranking Political Science Journals: Reputational and Citational Approaches.” PS: Political Science & Politics 40 (4): 741–51.
Goodman Steven N., Berlin Jesse, Fletcher Suzanne W., and Fletcher Robert H.. 1994. “Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine.” Annals of Internal Medicine 121: 1121.
Jacoby William G., Lupton Robert N., Armaly Miles T., and Carabellese Marina. 2015. “American Journal of Political Science Report to the Editorial Board and the Midwest Political Science Association Executive Council.” April. Available at https://ajpsblogging.files.wordpress.com/2015/04/ajps-editors-report-on-2014.pdf.
Kojadinovic Ivan, and Yan Jun. 2010. “Modeling Multivariate Distributions with Continuous Margins Using the Copula R Package.” Journal of Statistical Software 34 (9): 120.
Lee Kirby P., Schotland Marieka, Bacchetti Peter, and Bero Lisa A.. 2002. “Association of Journal Quality Indicators with Methodological Quality of Clinical Research Articles.” Journal of the American Medical Association 287 (21): 2805–8.
Li Danielle, and Agha Leila. 2015. “Big Names or Big Ideas: Do Peer-Review Panels Select the Best Science Proposals?” Science 348 (6233): 434–8.
Mahoney Michael J. 1977. “Publication Prejudices: An Experimental Study of Confirmatory Bias in the Peer Review System.” Cognitive Therapy and Research 1 (2): 161–75.
Mayo Nancy E., Brophy James, Goldberg Mark S., Klein Marina B., Miller Sydney, Platt Robert W., and Ritchie Judith. 2006. “Peering at Peer Review Revealed High Degree of Chance Associated with Funding of Grant Applications.” Journal of Clinical Epidemiology 59: 842–8.
Miller Beth, Pevehouse Jon, Rogowski Ron, Tingley Dustin, and Wilson Rick. 2013. “How to Be a Peer Reviewer: A Guide for Recent and Soon-to-Be PhDs.” PS: Political Science & Politics 46 (1): 120–3.
Mulligan Adrian, Hall Louise, and Raphael Ellen. 2013. “Peer Review in a Changing World: An International Study Measuring the Attitudes of Researchers.” Journal of the American Society for Information Science and Technology 64 (1): 132–61.
Nexon Daniel H. 2014a. “Ask the Editors: Desk Rejections (Part I), International Studies Association.” May 5. Available at www.isanet.org/Publications/ISQ/Posts/ID/1377/Ask-the-Editors-Desk-Rejections-Part-I.
Nexon Daniel H. 2014b. “Ask the Editors: Desk Rejections (Part II), International Studies Association.” July 7. Available at www.isanet.org/Publications/ISQ/Posts/ID/1427/Ask-the-Editors-Desk-Rejections-Part-II.
Nexon Daniel H. 2014c. “ISQ Annual Report, 2014.” December 5. Available at www.isanet.org/Portals/0/Documents/ISQ/ISQ%202014%20Annual%20Report.pdf.
Nylenna Magne, Riis Povi, and Karlsson Yngve. 1994. “Multiple Blinded Reviews of the Same Two Manuscripts: Effects of Referee Characteristics and Publication Language.” Journal of the American Medical Association 272 (2): 149–51.
Park In-Uck, Peacey Mike W., and Munafo Marcus R.. 2014. “Modelling the Effects of Subjective and Objective Decision Making in Scientific Peer Review.” Nature 506 (February): 93–6.
R Core Team. 2015. R: A Language and Environment for Statistical Computing (Version 3.2.5). Vienna, Austria. Available at www.R-project.org.
Schroter Sara, Black Nick, Evans Stephen, Carpenter James, Godlee Fiona, and Smith Richard. 2004. “Effects of Training on Quality of Peer Review: Randomised Controlled Trial.” British Medical Journal 328: 673–78.
Schroter Sara, Black Nick, Evans Stephen, Godlee Fiona, Osorio Lyda, and Smith Richard. 2008. “What Errors Do Peer Reviewers Detect, and Does Training Improve Their Ability to Detect Them?” Journal of the Royal Society of Medicine 101: 507–14.
Smith Richard. 2006. “Peer Review: A Flawed Process at the Heart of Science and Journals.” Journal of the Royal Society of Medicine 99: 178–82.
Somerville Andrew. 2016. “A Bayesian Analysis of Peer Reviewing.” Significance 13 (1): 32–7. doi:10.1111/j.1740-9713.2016.00881.x.
Sweitzer Bobbie Jean, and Cullen David J.. 1994. “How Well Does a Journal’s Peer Review Process Function? A Survey of Authors’ Opinions.” Journal of the American Medical Association 272 (2): 152–3.
Weber Ellen J., Katz Patricia P., Waeckerle Joseph F., and Callaham Michael L.. 2002. “Author Perception of Peer Review.” Journal of the American Medical Association 287 (21): 2790–3.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

PS: Political Science & Politics
  • ISSN: 1049-0965
  • EISSN: 1537-5935
  • URL: /core/journals/ps-political-science-and-politics
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 333
Total number of PDF views: 803 *
Loading metrics...

Abstract views

Total abstract views: 4078 *
Loading metrics...

* Views captured on Cambridge Core between 10th October 2017 - 16th December 2017. This data will be updated every 24 hours.