What is peer review?

Peer review is a ‘formal system whereby a piece of academic work is scrutinised by people [the peer reviewers] who are not involved in its creation but are considered knowledgeable about the subject’ (Reference Wager, Godlee and JeffersonWager 2002).

The peer-review process is important and all reviewers have a duty to ensure that the highest possible standards are met. The quality of this process is rightly cited as one of the main influences on a journal's reputation and standing (Reference Scott-LichterScott-Lichter 2009). It serves as a ‘quality control’ mark for published articles.

Peer reviewers may find it helpful to think of themselves as ‘consultant advisors’. The reviewer has two broad aims, which are directed at two main parties:

1. • the author(s) – to help improve quality and therefore the chances of publication (in any journal)

2. • the editor – to aid decision-making.

These should be at the forefront of any reviewer's mind. Ultimately, the editor makes the decision. It would be wise to be mindful of not conveying your decision to the author as this could be construed as going over the editor's head. More information about the reviewers’ responsibilities towards authors, editors and readers is available from the Council of Science Editors’ White Paper on promoting integrity in scientific journal publications (Reference Scott-LichterScott-Lichter 2009).

Types of review

Broadly speaking, there are two types of review – open and masked reviews.

Advantages and disadvantages of a peer-reviewed journal

A peer-reviewed journal is one in which submitted articles are independently examined by a panel of external experts prior to publication. The Psychiatrist and the British Journal of Psychiatry have at least two independent peer reviews before approving an article for publication, and Advances in Psychiatric Treatment usually uses one reviewer.

The role of an editor

The editor is the person who has ultimate responsibility for the direction and content of the journal. When a scientific paper is submitted, the editor has to read the paper to ascertain that it is deserving of peer review. Sometimes the initial in-house review is carried out by a member of the editorial board with expertise in that particular area. Many journals have a committee to decide on this – in the British Journal of Psychiatry it is known as the Janitor Committee.

After review, it is the editor who makes the final decision on whether a paper should be published or not. The role of the reviewer is to act as advisor to the editor and their recommendations may be overridden in certain circumstances, such as when the editor believes the reviewer did not adequately consider the paper, made injudicious comments or made a recommendation to publish without appreciating the pressures on space in the journal. In circumstances where the editor is dissatisfied with the standard of the review, a further review may be sought. Some journals, such as the BMJ, have a ‘hanging committee’ whose role is to aid the editor in reaching a decision on publication in the event that there is uncertainty.

When two reviewers disagree on one paper, the response of the editor will differ depending on the stage of the review process. If it occurs early in the process, for example after the initial submission, it is important to read both reviewers’ comments carefully since the score given may not be commensurate with their review or it may be apparent that the reviewer has not been diligent enough or has been excessively harsh. The editor can then make a decision based on their own evaluation of the merits of the paper, taking into account the reviewers’ comments. Alternatively, another reviewer may be selected to replace the outlier. If the disagreement occurs after the paper has been resubmitted, the editor may make the decision with an explanatory letter to the author or ask for a third review. In this way the likelihood of an appeal is lessened and may be seen as fairer by the author.

The editor of a journal has many different roles and is extremely busy. Good reviewers help enormously in aiding the decision-making process.

How an editor chooses reviewers

Submitting papers to journals is now mainly done electronically. The process is usually explained on the journal's front or back cover, or on its website. Softwares may differ between journals but essentially, when authors submit papers, their names are added to the e-submission system as potential reviewers in their field, whether their papers are published or not. Those who have expressed an interest in reviewing will also be added, thus building up a list of reviewers. All are asked to provide information on their areas of expertise.

Performance as a reviewer is monitored automatically to include the number of reviews carried out over a set period, the turnaround time and dates of the reviews. There is a facility in some databases for the editor to rate the quality of the person's reviews. However, these ratings are not shared with the reviewers. A drawback is that the database needs regular updating so that it is amended when a reviewer moves or develops another area of expertise, and it is the reviewer who is responsible for this. The lists on many databases are out of date.

If a reviewer is contacted but does not reply within a specified deadline, the reviewing software alerts the editorial office. The reviewer invitation process then begins again. Thus, failure to update the database has the effect of increasing the time to publication for the individual paper. P.C. (Editor of The Psychiatrist) once had to contact 14 possible reviewers before finding one who would agree.

What to do on receipt of invitation to review

It is a privilege to peer review a paper. The reviewer plays an important part in the advancement of scientific knowledge and ensuring that the quality and standards of the journal are met. Being a peer reviewer is a great addition to a CV: for trainees it covers several of the competencies required by curricula for the portfolio; for specialty doctors and consultants it will help with revalidation. This does not mean one should merely jump at the chance when asked to review, tempting as it may be to agree with H.G. Wells that ‘No passion in the world is equal to the passion to alter someone else's draft’. The role is one that should be taken responsibly and seriously. Reference Wager, Godlee and JeffersonWager et al(2002) suggest that one should ask oneself five preliminary questions before accepting a manuscript for review (Box 1). In the rest of this section we expand on their discussion of these questions from the point of view of an invited reviewer.

What to do on receipt of the paper

We recommend the following first steps on receipt of a manuscript that you have agreed to review.

1. • Right from the start, ask yourself whether the paper suits the journal. If you are not familiar with the journal, the editorial office can send a copy of the instructions to authors. This will act as a guide as to whether the authors have successfully adhered to what the journal is looking for. You may wish to read other articles published in the journal to get a feel for the style and level of what gets published. It may help to see whether the reference list in the submitted paper contains papers published in the same journal.

2. • Read through the paper once fully, resisting the temptation to start critiquing it at this stage.

3. • Make a note of first impressions. Unlike reviewers, most readers will only read it once. Therefore your first impressions are likely to be similar to those of other like-minded colleagues. If it does not make much sense to you, it probably would not make sense to other readers.

4. • Read the paper again slowly. This time make notes. Guidance to what is required is shown in the template in Fig. 1.

   FIG 1

   Template for peer review.

   

5. • You may find it helpful to then sleep on it and come back to it the next day. This may give new insights on previously missed points.

6. • Bear in mind that two reports need to be written (one to the editor and one to the author), but often the same report can go to both author and editor.

7. • In the case of research papers, you may find it helpful to use the PICO rule (Box 2) as guidance to make sure that the basic elements are covered.

8. • Some reviewers prefer an even simpler checklist – the four ‘W's (Box 3) (Reference AlbertAlbert 2009). You should be able to adequately describe to a colleague the points covered by the four items.

9. • The underlying factor driving your critique should be the scientific merit of the paper.

How to write the review

Templates can be useful for making sure nothing major is omitted. This is rather akin to many trainees who learn mental state examinations by initially using templates to remember all the sections. However, even for more experienced reviewers, templates such as that shown in Fig. 1 can be useful.

The summary of the review ideally should contain no more than three sentences. The first should describe the topic and method. The second should comment on major shortcomings and/or changes that need to be made. The third should make helpful concluding comments to the author/editor about your overall opinion. Following this, the review can be broadly split into major and minor points (Fig. 1).

An alternative is to go through the paper systematically in the order the authors have written it, writing a paragraph under each subheading. These paragraphs should be numbered to make it easier for authors to respond. For example, starting with the abstract, this is often the first and only thing many psychiatrists read when searching the literature. Bearing this in mind, you need to make sure that it accurately reflects the body of the manuscript. Note that before accepting or declining the review, you will have been able to read the author's abstract.

You then need to check that there is a stated aim or hypothesis. Following this, it is important to make sure the method is clearly described, valid and appropriate to the question posed. The results should not be repeated in the discussion and the conclusions should be based on the results. The conclusions should be true, accurate and justifiable from the data available. The discussion should be appropriately focused. You might suggest additional references, which can be helpful to authors. It is important to scrutinise the references – if they are too old they may be inappropriate. It is pertinent at least to read the abstracts of the references listed since authors sometimes cite inappropriate references without having read them.

Editors rightly place a lot of emphasis on grammar, syntax and language. For example, Advances in Psychiatric Treatment aims for concise, lucid and unambiguous language (Reference BouchBouch 2009). Although journals may have copy-editors who check such detail, it is good practice for reviewers to pick up any errors and comment in the specific section of the review.

You may need to do your own literature search and add it to the review. This is important when discussing originality, if only old papers are referenced or if one is dealing with a paper concerned with peripheral or esoteric areas (e.g. the reviewer has an interest in suicide and is asked to review a paper on suicide by motorcycle accident).

Remember, no paper is perfect and most papers require revision.

The length of review will depend on the complexity of the individual manuscript and how much you have to say. One- or two-lined reviews are not acceptable; nor is raising new methodological flaws on a redrafted paper that were not spotted on a previous review. The words of Einstein ring true here: ‘Everything should be made as simple as possible, but not simpler’ (Reference CalapriceCalaprice 2005).

A summary of the do's and don'ts of writing a review is presented in Box 4.

What to do after the review has been completed

For the British Journal of Psychiatry and The Psychiatrist reviewers are asked to score the submission on a 5-point rating scale (Box 5), but other journals will have a different scoring system.

Very few manuscripts are accepted first time (P.T., Editor of the British Journal of Psychiatry, estimates less than 1%). Joe Reference BouchBouch (2009) stated that the record during his first 2 years as Editor of Advances in Psychiatric Treatment was six resubmissions. For the Psychiatric Bulletin, out of 214 new submissions between October 2007 to September 2008, 21 (10%) were rejected without review, 32 (15%) were rejected but invited to submit the paper as a letter, and 40 (19%) were rejected after review. The total number rejected was 93 (44%) and the total number accepted following review was 108 (50%). The remaining 13 (6%) were withdrawn by the authors or deleted by the editors.

Once you have submitted your review, you have to wait for the editor's decision, which is usually communicated via email several weeks later.

What makes a good reviewer?

Some authors believe a good reviewer is one that combines a critical eye with a positive creative attitude aimed at improving manuscripts and educating fellow researchers. The best reviewers concentrate on offering useful advice to authors rather than giving summary judgements to editors. Other core attributes include courtesy, punctuality, thoroughness, keenness and being knowledgeable in the discipline (Golbeck-Wood 1998).

Many younger reviewers of junior academic status are reluctant to review, feeling inadequately skilled, especially when the author is a well-known senior figure. However, the evidence suggests that younger reviewers write better-quality reviews (Reference StosselStossel 1985; Reference Evans, McNutt and FletcherEvans 1993; Reference Kliewer, Freed and DeLongKliewer 2005). Research training and postgraduate qualifications are not associated with review quality. Reference Black, Rooyen and GodleeBlack et al(1998) attempted to determine the characteristics of reviewers who produce high-quality reviews for the BMJ as determined by both editors and authors. In a logistic regression analysis, the only significant characteristics were younger age and those with training in epidemiology or statistics. Other authors have found no significant correlation between quality of reviews and gender, years of reviewing and academic rank (Reference Kliewer, Freed and DeLongKliewer 2005).

Obtaining feedback is always a good thing in order to support reviewer development. Many journals send reviewers the comments of other reviewers of the same paper. You can also ask the editor for feedback, perhaps using a tool developed by Reference Van Rooyen, Black and GodleeVan Rooyen and colleagues (1999) called the Review Quality Instrument (RQI). This instrument considers seven aspects of a review (importance of research question, originality, methodology, presentation, constructiveness of comments, substantiation of comments, interpretation of results), each scored on a 5-point Likert scale (1, not at all or poor; 5, discussed extensively or excellent). The total score is the mean of the 7-item scores. There is an additional eighth item for the overall assessment of the quality of the review. One may assume that reviewers who work through questions from the RQI (Box 6) might write better reviews.

The bottom line is that editors love to hear from enthusiastic potential reviewers of any age and background who feel that they can do a good job. After reading this article, we hope that you are in a better position to do just that.

Reviewing books

Writing a book review is a good way of getting your own name in print, perhaps for your CV. You will be sent a book or books to read and asked to write a review. It is tempting simply to list the contents and contributors and to make a comment on the potential audience, length or cost of the book. More interesting for the reader, however, and a challenge for the reviewer is to write a short article using the book as a starting point.

Writing to potential reviewers, the book review editors of the British Journal of Psychiatry and The Psychiatrist comment that a review is an opportunity to inform the reader of a book's merits, but should also be interesting in its own right. Pulling out one or two themes and exploring how the book has been approached requires careful reading and thinking about what the book is presenting.

You can develop your skills as a book reviewer by taking on different types of book to review, from single-author titles focused on a specific issue to books covering more varied material and with sometimes many contributors. To quote Simon Wessely, ‘a review should inform, entertain and occasionally provoke’ and the best reviews ‘are often jewels of argument and exposition, essays in their own right’ (Reference WesselyWessely 2000). You should respect the amount of time that goes into writing and editing a book; this is not to say that a review cannot be critical – but any critical comments should not be offensive (Reference Crown, Lee and RamsayCrown 2000).

To find out more

The following books and articles may be useful to peer reviewers of today and tomorrow.

1. • The Doctor's Guide to Critical Appraisal (Reference Gosall and GosallGosall 2009)

2. • How to Read a Paper (Reference GreenhalghGreenhalgh 2000)

3. • How to Write a Paper (Reference HallHall 2008)

4. • Critical Appraisal for Psychiatrists (Reference Lawrie, McIntosh and RaoLawrie 2000)

5. • How to get Published. All you Need to Know to Publish in Psychiatry (Reference BayléBaylé 2006)

6. • How to Survive Peer Review (Reference Wager, Godlee and JeffersonWager 2002)

7. • ‘A systematic guide to reviewing a manuscript’ (Reference Provenzale and StanleyProvenzale 2005).

A great deal of emphasis is currently placed on psychiatrists’ ability to systematically appraise published literature, and it now forms a key part in the MRCPsych exams and journal clubs. Interestingly, there is little such emphasis for adopting a robust and systematic approach to peer reviewing submitted manuscripts.

Next steps: how do I become a reviewer?

If you wish to become a reviewer, you should contact the editor of the journal and explain why you would like to do this work. You should also indicate your areas of scientific interest. It is important to provide a reference attesting to your ability to critically consider papers and also to meet deadlines. You will be required to demonstrate that you have a true expertise in a particular area, not simply a more-than-average interest, since you are likely to be asked to consider papers by authors who have published extensively in a particular area.

Good reviewers are highly sought after and desperately needed. In the Peer Review Survey (Sense about Science 2009) one in five researchers questioned thought that peer review is unsustainable owing to few willing volunteers. But 86% of volunteers enjoyed peer reviewing, although the majority said that there is a lack of guidance and that formal training would help. This article goes some way to addressing the issue.

MCQs

Select the single best option for each question stem

1. 1 Among the advantages of masked review over open review is that:

   1. a it is cheaper and easier

   2. b it reduces personal bias

   3. c it offers more transparency

   4. d it makes the reviewer more accountable

   5. e authors can communicate with reviewers.

2. 2 An advantages of a peer-reviewed journal is that:

   1. a reviewers can guess the identity of the author

   2. b there is no chance of personal bias operating

   3. c peer review helps to weed out unsatisfactory papers

   4. d positive findings are more likely to be accepted

   5. e there is less delay from submission to publication.

3. 3 The role of a peer reviewer is to:

   1. a decide whether a paper should be published in a journal

   2. b rewrite the paper to correct flaws

   3. c help the editor in the decision-making process

   4. d write only to the editor regarding a decision

   5. e criticise the author.

4. 4 A peer review should include:

   1. a personal comments to the author

   2. b a score out of 5 to the author

   3. c comments on the generalisability of the study

   4. d comments on how you would rewrite the methods

   5. e comments on your own research in the field.

5. 5 Factors that have been shown to positively influence the quality of a reviewer's work include:

   1. a being of a particular ethnic origin

   2. b the seniority of the reviewer's position

   3. c having a research-based degree

   4. d punctuality in writing reviews

   5. e being female.

MCQ answers

1

b

2

c

3

c

4

c

5

d