What is the purpose of peer review?

This is one of the questions that we all think we know the answer to, but over which we might not all agree. Peer review as we know it is often traced back to eighteenth-century incarnations such as the Royal Society of Edinburgh’s Medical Essays and Observations of 1731, and the taking over of Philosophical Transactions by the Royal Society in 1752 (Kronick, Reference Kronick1990). However, the pre-publication peer review of that era did not emphasize the validity of studies, so much as novelty and interest. The Literary and Philosophical Society of Manchester (1785) makes this explicit in the preface to its Memoirs, 1785 Volume 1.

What do we expect from peer review?

The context in which a researcher in applied science consults a journal now differs considerably from that of an eighteenth-century aristocrat-scientist, so what is expected from peer review today? This probably depends on who you are. In the next couple of paragraphs, I will set out some views that I have formed over the last 35 years, progressively moving from (1) practitioner to (2) practitioner and author to (3) author and reviewer to (4) author and editor (my status now). My perceptions of the process have changed dramatically as I have moved through these stages, and it is likely that the processes have also changed over that time.

For most researchers and scientist-practitioners, as readers and users of journal articles, the main concerns are probably veracity – honesty and truthfulness – and validity – soundness and rigour. Is it possible for journals to ensure complete veracity and validity? Probably not – not without a very substantial increase in resources and unacceptably extended times to acceptance. At the Journal, as with most journals, several levels of checks are applied: an initial examination of the manuscript by one or two editors, together with plagiarism checks and checks for duplication. Some obvious problems are picked up at this point, and manuscripts might be rejected. At peer review, it cannot be guaranteed that all important problems of veracity and validity are found. As often parodied in the ‘Reviewer 2’ trope, two reviewers rarely identify exactly the same issues in their reviews of the same manuscript. Lack of agreement between reviewers on the first round is not necessarily a critical problem – editorial input, together with subsequent rounds of revision and review usually, but not always, either improves a manuscript or sees it classified as likely unreliable for one reason or another, and therefore rejected. Despite all this work, we rarely have certainty that a published paper is true and correct in all respects, as evident from multiple reports of failure of academic publishing, the replication crisis and frequent retractions.

What research scientists as writers want most from peer review is fairness, impartiality, transparency and efficient process. It is not fair for someone to use the privileged cloak of anonymity to deliver a quick and incorrect judgement on work that you and potentially a large team have slaved over, possibly for entire PhD programmes, possibly with huge amounts of funding support that might have taken years to obtain. It is not fair to have a valid paper rejected because a reviewer had a personal objection to some aspect of the approach taken by the researchers or their writing style or their country of origin. It is not fair for a manuscript to be stuck ‘awaiting review’ for a year. It is not fair for editors or reviewers to discriminate for or against specific researchers or research groups because of perceptions they might have about the country of origin, gender, race and ethnicity, previous work or reputation of the researcher(s) or their institution(s). It is also not fair for reviewers to recommend against the publication of novel work but to take the ideas and carry out similar studies themselves. We know that there is huge potential for inappropriate gatekeeper action in the publication process. The effects of persistent rejection and some possible responses to the current system are described by Nguyen and Vuong (Reference Nguyen and Vuong2025).

What publishers and the owners of journals want is something different again. There is no question that, despite the widespread, usually nominal, acceptance of the sensible principles of the San Francisco Declaration on Research Assessment (DORA; https://sfdora.org/), high citation rates remain important for the survival of scholarly journals. Publishers and editors care about impact factors because they can be important for academic reward and recognition structures (even in institutions that have signed up to DORA), so authors tend to actively choose journals with higher impact factors. For more applied journals, other metrics can be important: readership, downloads, shares, social media mentions and policy uptake, but as yet, these are less formalized and less recognized. Hence, editors of journals are on the lookout for manuscripts that seem likely to generate interest and frequent citation, whereas they might tend to discriminate against manuscripts that seem to have limited potential readership. In any case, editors of journals generally remain motivated to some extent to publish articles that have maximal readership and potential for citation. Hence, from the peer-review process, they expect clear guidance on the novelty, innovation and potential impact (socially or scientifically) of a manuscript.

What can we do to make it work better?

Allen et al. (Reference Allen, Cury, Gaston, Graf, Wakley and Willis2019) conducted a literature review on the topic and suggested five principles of good peer review: content integrity, content ethics, fairness, usefulness and timeliness. Their ‘content integrity’ broadly corresponds with my ‘validity’, their ‘content ethics’ with my ‘veracity’, ‘fairness’ with ‘fairness’ and ‘timeliness’ with ‘efficiency’. Their ‘usefulness’ is an important element that I have sometimes overlooked – how helpful is the peer-review process to the author for subsequent submissions? Overall, their paper is a useful guide to how we can improve the process of selecting manuscripts for publication.

Firstly, in relation to veracity and validity (content integrity and ethics), we are not ready to do away with peer review just yet. Despite its flaws, it is useful for identifying academic malpractice and errors in experimental design, implementation, analysis and interpretation. It should be directed more carefully to those characteristics of a manuscript that affect its veracity and validity: aims and objectives, experimental design, implementation, analysis and interpretation. Reviewers should not feel that they must review every aspect of a manuscript and correct the grammar and expression. If a reviewer doesn’t have the expertise to comment on some aspects of a multidisciplinary study, that is fine. They can comment on the bits that they know and understand and clearly state where the boundaries are on their competence. Detailed comments limited to a component of the manuscript can be much more useful than vague statements about the manuscript overall. Lack of agreement between reviewers should be addressed to some extent by the handling editor and editor-in-chief. It should be made explicit when there are apparent contradictions in views, and the authors should be invited to address these contradictions with evidence-based arguments. As suggested by Allen et al. (Reference Allen, Cury, Gaston, Graf, Wakley and Willis2019), many issues of content ethics can be detected at the initial editorial office level, but experience shows that reviewers commonly raise alarms about ethical issues that were missed at the initial editorial check.

Secondly, in relation to fairness, impartiality, transparency and efficiency, much of the responsibility falls to the editorial team. Before even going out to a handling editor, the editor-in-chief is expected to perform an initial check to ensure that every manuscript is within scope for the journal, likely of some potential interest to at least some of the potential readers of the journal (an old and possibly anachronistic concept given the fact that few truly read a journal issue these days but usually search for content by key words and topics), has no indications of plagiarism, image manipulation or inappropriate use of artificial intelligence (AI), has been conducted appropriately, and is written in intelligible English. A surprising number of manuscripts can be identified at this early stage as not suitable for putting out to review – we don’t want to waste reviewers’ time if even a non-specialist can identify fundamental problems. Desk-reject rates for science journals can be very high, around 80–90% for some particularly prestigious journals. At this stage, the potential for the editor’s biases to affect selection is great, and transparency traditionally has been low. Feedback is usually limited to something along the lines of ‘out of scope for the journal’ or ‘not suitable for publication in the journal’. This is regrettable and non-transparent; however, it reflects the widespread view that providing more detailed reasons provides more potential points of argument and a greater chance that the author of a rejected manuscript will initiate contact with the editor and argue their case. My own experience suggests that this is the case; however, the very limited empirical data don’t support this view. A meta-analysis of whether by increasing explicitness in general argument (in our case, the reasons for rejection), the apparent ‘disagreement space’ is enlarged, and the effectiveness of the argument is reduced (O’Keefe, Reference O’Keefe1998), did not find any adverse effects on the effectiveness of the argument. Allen et al. (Reference Allen, Cury, Gaston, Graf, Wakley and Willis2019) stated, ‘We have not isolated transparency as a principle in its own right – although it is pertinent to many of recommendations – because transparency is not an unqualified good. Some aspects of the peer-review process require confidentiality and discretion’. This comment was not made specifically in relation to rejection messages. Taking into consideration concerns such as those of Nguyen and Vuong (Reference Nguyen and Vuong2025), who particularly criticize the lack of information provided in desk-reject messages and the lack of humility on the part of editors and journals generally, some measures are required to improve the transparency of assessment of manuscripts prior to peer review. This is a very challenging area, and other than trying to be as aware as possible of my own potential biases as the editor-in-chief, from a policy perspective, I’m going to let it lie for the time being and come back to it in the future.

Regarding what the Literary and Philosophical Society of Manchester refers to as ‘novelty, ingenuity or importance’, external reviewers are widely considered to be important. For the Journal of Dairy Research, as a predominantly applied science journal, many of whose outputs are intended to be put to immediate use in the field, I argue that, of these three criteria, novelty should not be the highest ranked. There is abundant evidence and (mostly Bayesian) statistical theory to show that the more novel and unexpected a reported finding is, the more likely it is to be false, for example, Forstmeier et al. (Reference Forstmeier, Wagenmakers and Parker2017). The challenge is to decide whether a manuscript makes a meaningful contribution to our understanding of the world, and in this, objective, reliable opinions can be difficult to obtain. Here, peer reviewers with strong expertise in a field and deep knowledge of the literature might also have strong positions on a topic and can be quite harsh on manuscripts for lack of novelty. My preference is for reviewers to form a view on novelty and communicate it clearly, but not to recommend rejection of the manuscript based on lack of novelty without first commenting on its veracity and validity.

Finally, as a component in the system and being a frequent perpetrator of inefficiency, I spend a lot of time trying to identify points of blockage in our process. Probably, the greatest source of variation in processing time for manuscripts is the time taken for the recruitment of qualified peer reviewers and obtaining useful reports from them. The accompanying instructions for reviewers will be followed by detailed instructions for authors, and both are intended to improve efficiency in the publication process. Increasing the use of automated reminders can help in some respects, but can also reduce responsiveness in handling editors and reviewers. By providing more explicit instructions for every stage of the process, by enforcing them, and by establishing more human relationships with handling editors, reviewers and authors, it should be possible to increase the proportion of participants in the process who know what they are doing and can do it efficiently.

Peer review is being challenged as a paradigm, but we are stuck with it for the foreseeable future, and we can improve the system so that it works better for all participants. The process should move away from a gatekeeping mentality in which the journal editorial team and reviewers see themselves as exalted custodians of true knowledge to one in which there is a genuine collaboration to develop and distribute knowledge. The gatekeeping remains, but is primarily required to deal with the increasing challenge of academic fraud, at all levels. We need to (1) engage more with potential reviewers from low- and middle-income countries, (2) ensure that reviewers understand what we are asking from them and (3) always carefully consider whether reviewers are expressing unbiased, reasonable opinions. Most importantly, we need to remember that whatever approach we take at the Journal, we will never eliminate all unreliable, invalid, fraudulent papers. Readers have an obligation to apply their discriminating abilities – we need to do away with the concept that if something is peer-reviewed, it is true.

Now, the big question: Should we be paying peer reviewers? Anyone who suggests this idea has obviously never heard of the concept of transaction cost economics. In floating or debunking this idea, most of the online discussions I have seen focus on the expected small cost of likely payments to peer reviewers relative to article processing fees. However, most pundits completely ignore the real issues: the administrative armies that publishers would need to recruit to manage tens of thousands of relatively tiny international payments, each requiring a separate contract, taxation documentation, identification verification and international transaction fees; the absolute barriers caused by economic sanctions on some countries; the disadvantaged position in international banking facilities of low- to middle-income countries; the potential for evolution of new and potentially fraudulent business models to exploit the new economic opportunity. The lack of feasibility of remunerating reviewers probably makes redundant the question of whether it is right. Authors and reviewers who take the view that the system should pay them for the application of their discriminating abilities should reconsider with whom they publish and for whom they review – better to publish with and review for the journals of learned bodies, where you are confident that you are working to the benefit of a legitimate scientific community.

In conclusion, with the revised instructions to reviewers, we are hoping to improve the experience for authors and reviewers. I look forward to feedback.