What is peer review?

Peer review in an integral part of scientific publishing by helping maintain the credibility and integrity of scientific literature. Peer review is the process by which scientists who are knowledgeable in a particular field of study give their constructive feedback on scientific papers for research in that field, generally prior to publication in a scientific journal. Journal editors (and likely also your PI if you are a student) will often invite you to participate in this peer review process.

Why be a peer reviewer?

There are many reasons why you may be interested in participating in peer review, even as a student. For example, participating in peer review is a great way to keep up to date with the latest work in your field of study and to contribute to the advancement of science. Peer reviewing is also a critical skill that will help improve your CV and help you think critically about your own research and writing. The peer review process also allows you to build relationships with journal editors, which may become useful in the future when you are publishing.

Your role as a peer reviewer

As a peer reviewer, it is your responsibility to:

1. Complete your review in a timely fashion (as defined by the editor) and, if you are unable to do so, turn down the invitation to peer review or notify the editor of any delays
2. Be fair and constructive in your peer review
3. Commit to dealing with multiple iterations. Peer reviewing a paper is often not a onetime commitment; the paper will likely be revised, and you will be called on again to review the revision.

Ethics in peer review

Peer review requires a number of ethical considerations. Peer reviewers are generally given privileged access to works-in-progress and thus should exercise care to avoid abusing this privilege. The Committee on Publication Ethics (COPE) provides detailed guidelines to reviewers on the ethics of peer review. A few summarized highlights are provided below:

• You should turn down the invitation if you have any conflicts of interest that may impede your ability to provide an unbiased evaluation of the research.
• You should treat any paper you are reviewing as confidential.
• You should not use information you have gained from reviewing the paper to your own advantage.
• You should not use your role as a peer reviewer to the detriment of others, including the authors.

The peer review report

Once you have accepted a peer review assignment, you will need to write up a peer review report that will help the authors improve their final paper as well as help the editor to decide whether or not the paper should be considered further for publication.

Report structure

Some journals provide their own report structure (such as via online questionnaires and forms) that they expect peer reviewers to follow. Generally, your report should be structured as the following:

1. Executive summary
2. Major comments
3. Minor comments

Executive summaries should be 1 to 2 paragraphs while major and minor comments should be numbered lists. Major and minor comments referencing specific texts should be referenced with page and line numbers where possible.

Executive summary

The executive summary is primarily for the editor. To provide an example, I will use text from my own review, including my publicly available review in F1000, an open-peer review journal.

The executive summary should address the following questions:

1. Why did the authors carry out the study?

   Analysis of single-cell RNA-seq data is often complicated by large amounts of zeros, of some which represent true lack of expression, while others are reflective of poor capture efficiency or other technical limitations. Several methods have been developed to impute the zeros and recover the true gene expression values.

2. What did the authors do? What did the authors find out?

   Here, the authors compare the performance of 5 of these single-cell imputation methods using both simulated data and artificially permuted single-cell RNA-seq data. They evaluate the extent to which these methods introduce false differential expression

3. What is your main opinion of the paper?

   A number of clarifications are needed to improve the understandability of the manuscript. Performance benchmarks using additional datasets are also needed to ensure that observed performance differences between methods are not biased by how well the datasets conform to underlying distributions assumed by each method.

4. If requested, what is your suggestion to the editor?

   This paper is not suitable for publication at this time and will require major revisions.

Major comments

Major comments are significant and substantial flaws in methodology, logic, missing controls, and such that if not addressed will preclude the paper from publication. Examples of major comments may include but are not limited to:

• Additional experiments or analyses to address conclusions presented in the paper that are currently not well supported by the data
• Additional text and clarifications, especially in methods, to ensure help reproducibility
• Text corrections to avoid inaccurate statements
• Noting improper or incorrect citations (this one can be quite challenging)

Minor comments

Minor comments, in my opinion, are things that should be address if possible, but may not preclude the paper from publication if unaddressed. Examples of minor comments may include but are not limited to:

• Improvements to language for clarity (scientific word choice)
• Improvements to figures (labels, legends, etc)

General tips

In my personal experiences as a peer reviewer, my intuitive reaction when I read a really “bad” paper has always been anger. It’s as if I paid to watch a movie that had a great trailer and then the movie sucked so I’ve just wasted my time. I’m not alone in feeling this way and perhaps you feel this way too. A number of reviews that I’ve received have been quite angry, condescending, or just mean. These are a few approaches I now use to help ensure that my peer review reports are constructive and beneficial to the authors, editor, and the scientific community at large. When good work is made better by helpful reviews, we all win.

1. Reframe your perspective: This paper is a work-in-progress. You have been requested as a reviewer for your expertise and ability to provide a critical and independent perspective. The editor and the authors have asked for your help so be helpful.

2. Be pedagogical: Treat the authors as if they are your students. Teach them about where their analysis went wrong and why their conclusions may be inaccurate by justifying your recommendations with concrete examples.

3. Write your review as if you are writing a paper: This means your review should be something you are proud of and are willing to stand by if it is made public (even if it is not).

4. Come back to it: It’s ok to take a breather, walk away, and come back.

Things to avoid

Beyond just being rude and mean, there are other things you should avoid in your peer reviewer report:

• Do not suggest experiments or analyses that are not beyond the scope of the paper. Yes, everyone can always do more. But you should not suggest something just because you are curious if it is not relevant to the paper’s research questions.
• Do not correct spelling, grammar, or typos. In my opinion, this is the role of the copy editor. You have higher level scientific and intellectual contributions. You should only correct these things if it interferes with your understanding of the science.
• The worst thing you can do as a reviewer is ask for a bunch of experiments or analyses when you know the underlying premise of the paper is so flawed that you would never accept even if all your comments are addressed. It’s ok to reject! If a paper is not of sufficient quality and won’t be without effectively becoming a new paper, just reject it! Note: just because you recommend a paper for rejection doesn’t mean that it will be rejected! There are other reviewers who may recommend differently, and ultimately an editor may choose to accept a publication in spite of your criticisms. Even if you are recommending rejection, it is often still helpful to provide a set of major and minor comments. At the very least, it will provide the authors with your feedback.

Limitations of peer review

Peer review is not a foolproof process and comes with many limitations.

Peer review relies on reviewers. So inherently, the effectiveness of the peer review process will depend on the thoroughness, dedication, and availability of these reviewers. But reviewers are people too. So inevitably, peer reviews represent the subjective opinions of these reviewers. These opinions may be subject to implicit biases regarding the author’s institution, gender, level of fame, or other characteristics. It is up to all of us, reviewers and editors alike, to do our best to recognize and minimize these biases.

Peer review may mitigate but will not always prevent fraud. It is up to the scientific community to remain vigilant and to always read scientific papers critically regardless of the paper’s peer reviewed status. Adoption of automated plagiarism checks, image manipulation checks, and other artificial intelligence algorithms by journals will hopefully one day help further mitigate fraud in scientific publishing.

Ultimately, peer review is an ongoing process. Even if a paper does though a few rounds of peer review, is accepted and published, it is still subject to peer review by the greater scientific community who will continue to critically evaluate, replicate, and eventually build upon this research.

Additional resources

• Focus on Peer Review - an online class from Nature Masterclasses
• How to write a peer review according to PLOS
• Jeff Leek’s lab guide to reviewing papers