Problems with increasing author and editor discretion in peer review

To address the problems with reviewer bias and error, Allen et al. offer several recommendations aimed at redistributing power from reviewers to authors and editors. Examples include allowing authors to evaluate reviewers, increasing author utilization of appeals, reducing the involvement of reviewers in revisions, and empowering editors to discern the importance and weight of reviewer comments. These discretionary actions could potentially be used to screen out low-quality reviewers, mitigate the impact of reviewer error on manuscript decision-making, or protect authors from engaging in inappropriate or unnecessary revisions suggested by reviewers.

Despite these potentially useful applications, we argue that increasing author and editor discretion is likely to exacerbate, rather than mitigate, problems with peer review. This is because authors and editors—like reviewers—are subject to human motivations, biases, and errors that shape their perceptions and decision-making. As a result, adopting procedures that rely on their subjective evaluations is likely to exacerbate, rather than mitigate, bias and error in the peer review process. For example, authors whose papers are rejected are motivated to protect their positive views of themselves and their work, cope with the pain of rejection, and ultimately get their work accepted. As a result, author perceptions of reviewer incompetence or bias or the fairness of the review process are likely to be contaminated by the outcome of the process and the extent of criticism received. These sources of contamination undermine the validity and utility of authors’ subjective perceptions (e.g., ratings of reviewers) and decisions based on these perceptions (e.g., decisions to appeal) while also raising the risk of incentivizing problematic reviewer behavior (e.g., providing a positive review while privately recommending rejection, or overlooking significant problems to receive a positive rating). Similarly, editors are subject to error and biases that affect their perceptions and decision-making. For example, editors likely have developed strong heuristics regarding what scholars produce “good work” or are “good reviewers”, in addition to professional relationships with many authors and reviewers. Because authors and reviewers are typically not anonymous to the editor, these subjective perceptions and relational motivations may color editors’ evaluations, feedback, and decisions. For example, editors may implicitly judge manuscripts written by well-known scholars as better quality or more capable of improvement than those written by scholars they do not know (i.e., familiarity bias). Likewise, editors may implicitly weigh the comments of well-known reviewers more heavily in their decision-making, regardless of the quality of their comments.

We are also concerned about the impact of recommendations that increase reliance on a single individual’s subjective opinion. For example, Allen et al. suggest that only editors should be able to read the revised manuscript after initial review, and that reviewers should focus their comments on aspects of the manuscript for which they have expertise. Although we agree that these changes may increase the efficiency of peer review, they may also increase error and bias. As noted by Allen et al., authors and editors are unlikely to have depth of expertise regarding all aspects of a manuscript. The involvement of multiple reviewers is intended to address potential gaps in editors’ and authors’ knowledge, and make them aware of problems, solutions, assumptions, and contributions that they may not have otherwise considered. When decisions rely on a single decision maker, there is a risk that problems and assumptions may go unseen (resulting in acceptance of work that should be revised or at least challenged) or that important solutions and contributions may go unarticulated (resulting in rejection of work that has high potential for meaningful contribution).

We are also concerned that increasing reliance on editors shifts decision-making power toward individuals who are most well-versed in the implicit norms of the field and the rules of “the publication game”—in other words, those with the highest levels of mastery over the “hidden curriculum” of academic life. This depth of implicit knowledge may make even minor norm violations highly salient, triggering implicit evaluations that a submission is low quality. For example, citation style errors or “overexplanations” of widely held assumptions (e.g., defining a p-value or justifying the use of a 7-point response scale) are minor norm violations that may be disproportionately engaged in by less experienced authors or by those who have been socialized into different academic norms. Yet, although these norm violations are highly correctable, they may implicitly signal competence or quality to those most familiar with the conventions of the field, “tipping the scales” toward rejection. In addition, well-established scholars often share taken-for-granted assumptions and terminology, which can undermine the introduction of new approaches or ways of thinking and the accessibility of the research to a broader audience. For these reasons, we are concerned that increasing reliance on the editor in the review process may further undermine the diversity, novelty, reach, and accessibility of our science, raising concerns that organizational research will only be understood by a small pool of peers with similar assumptions, training, and worldviews.

We agree with Allen et al., that reviewer bias and error are significant problems plaguing our science. Yet, the voluminous literature on questionable research practices, inaccurate citation, and methodological errors suggests that authors are also subject to bias and error, and that these errors often go uncorrected by editors and reviewers alike (Banks et al., Reference Banks, Rogelberg, Woznyj, Landis and Rupp2016; Kepes et al., Reference Kepes, Keener, McDaniel and Hartman2022). Likewise, there is strong multidisciplinary evidence that human decision-makers—regardless of expertise—are subject to a variety of biases that distort their judgment and undermine the fairness and validity of decisions (Hilbert, Reference Hilbert2012; Kahneman, Reference Kahneman2013). Addressing these issues requires less, not more, reliance on subjective evaluations in peer review.

Reducing subjectivity in the peer review process using AI

Addressing the problems associated with subjectivity in the review process requires increasing objectivity. To accomplish this, we suggest the same approach used in other performance appraisal contexts: The review process can become more objective and developmental by clarifying the criterion, developing evaluation measures that align with the criterion, and providing developmental feedback on the extent to which performance aligns with criterion dimensions (DeNisi & Murphy, Reference DeNisi and Murphy2017). Indeed, the importance of objective review criteria is already partially recognized through the use of quantitative dimension-based ratings in the review process and in journal instructions to reviewers. However, quantitative dimensions are often imprecisely defined in review portals, reviewers’ qualitative comments hold considerably more weight in the review process relative to their quantitative ratings, and there are no formal systems in place to ensure that reviewers are following journal instructions. These practices have the effect of increasing the potential for error and bias in the review process; mitigating them requires careful attention to performance criteria for both manuscripts and reviewers. Thankfully, comprehensive criteria for high-quality manuscripts (e.g., Aguinis et al., Reference Aguinis, Ramani and Alabduljader2018; Appelbaum et al., Reference Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu and Rao2018) and peer reviews (Köhler et al., Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha, Woo and Gulick2020) already exist. However, without tools to reinforce these standards, we run the risk that they will be ignored. Given constraints on author, reviewer, and editor time and attention, and the breadth of criteria proposed for manuscript and reviewer evaluation, we suggest the use of artificial intelligence (AI) as a tool to aid in the process of structured, criterion-based appraisal and feedback during peer review. Using ChatGPT, we created two custom GPTs to aid authors, reviewers, and editors in the review process: Friendly Reviewer and Review Checker. Links to the models, example use cases, and checklists are available on the Open Science Framework: https://osf.io/nbcqy/.

Friendly Reviewer is a custom GPT that evaluates quantitative, deductive manuscripts against a checklist of best practices for reviewers in the organizational sciences (see OSF). We developed the checklist by incorporating best practices from a range of sources, including American Psychological Association (APA) journal reporting standards (Appelbaum et al., Reference Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu and Rao2018), journal instructions (Journal of Applied Psychology, n.d.), published papers on rigor and contribution in the organizational sciences (Aguinis et al., Reference Aguinis, Ramani and Alabduljader2018; Lange & Pfarrer, Reference Lange and Pfarrer2017), and our own disciplinary expertise. Users can upload PDF manuscript drafts and copy and paste the review checklist into the chat, and Friendly Reviewer will engage in a structured review based on the checklist, providing a categorical evaluation of whether a checklist item is present, partially present, missing, or not applicable, with the evaluation grounded in specific evidence from the manuscript. In addition, the GPT provides guidance on how to address deficiencies based on Aguinis et al. (Reference Aguinis, Ramani and Alabduljader2018) and other sources. Users can also interact with the GPT for additional targeted feedback; for example, users can ask the GPT to be more critical of particular sections (e.g., theorizing and hypothesis development) or to provide examples or steps for addressing deficiencies. The aim of Friendly Reviewer is to improve manuscript quality and address reporting deficiencies before submitting a paper for peer review. Friendly Reviewer increases objectivity in the review process by evaluating manuscripts against a standard set of best practices rather than using the idiosyncratic and undefined standards adopted by human authors and reviewers. In doing so, it promotes research quality across a breadth of important criteria while reducing reliance on human knowledge and subjectivity.

Review Checker is a custom GPT that evaluates peer reviews against a checklist of best practices and provides reviewers with developmental feedback. We developed the reviewer checklist based on best practices from Kohler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha, Woo and Gulick2020) and our own disciplinary expertise (see OSF). Users can upload their review comments as PDF documents or text into the chat, and Review Checker will engage in a critical structured review of the content based on the checklist. As with Friendly Reviewer, users can interact with Review Checker to revise their reviews (e.g., asking for suggestions for how to rephrase a harshly worded criticism). The aim of Review Checker is to improve the quality, comprehensiveness, collegiality, and objectivity of reviews prior to submission and decision recommendations.