Guide

April 22, 2020

How to Review a Paper

During the COVID-induced three-month lab shutdown in 2020, I had the opportunity to review multiple manuscripts with my PI. To help me and other lab members navigate how to review a paper, I collated advice based on others’ articles and summarized that here in a guide. You can find the papers that I referenced at the bottom of this post. Many of the sentences or phrases may be copy-pasted directly from these articles, and I do not claim this guide to be my original work but instead hope that It can serve as a useful reference for those reviewing a paper for the first time.

I try to limit my reviews to one page with a summary paragraph followed by enumerated major comments.

Last edited: September 8, 2023.

Summary of the guide

  1. Check for journal specific guidelines
  2. Skim the paper once taking notes as you go.
  3. Draft a quick summary
  4. Take a (quick or long) break
  5. Sit down to write enumerated major comments
  6. Write actual suggestion

General Criteria

  • Does the paper provide insight into an important issue?
  • Does the paper tell a good story?
  • Is the paper interesting for an international audience?
  • Does the insight from the paper stimulate new, important questions?
  • Is there a high probability that the paper will be read and cited by others?

Stage 1: Check for journal specific guidelines

Check what format the journal prefers the review to be in. Familiarize yourself with any journal-specific guidelines (these will be noted in the journal’s guide for authors available on each journal’s homepage).

Stage 2: First read, skim

The first reading is to get an overall impression of the paper. Take particular note of the parts of the paper that your expertise speaks best to. Be upfront and honest with the authors and the editor about which scientific aspects you will focus on in your review.

Take notes as you go:

  • Is it in the scope of the journal?
  • What are its aims?  What is the paper about? What question or problem does the article address? What is the thesis/main argument. Highlight or underline where it is stated in intro/conclusion.
    • Is it relevant, interesting, or original?
    • How does the question being addressed fit into the current status of our knowledge? What does it add to the subject area compared with other published material?
    • Does the hypothesis make sense or is it interesting?
  • Do they address the main question posed?
    • Do the findings support the claims made in the paper? How convincing are the results and how careful is the description?
    • If the author is disagreeing significantly with the current academic consensus, do they have a substantial case? If not, what would be required to make their case credible?
  • What are the key data? Does it suffer from any issues with insufficient data/statistically non-significant variations? Do they have:
    • The sufficient use of control experiments?
    • Comprehensive and technically correct analysis and interpretaiton of the results?
    • Any other fatal flaws in the study or experimental designs?
  • Is the paper structured properly (abstract, keywords, material and methods, discussion, conclusions, references, etc.) and logically?
    • Do the authors explain every table, figure, and scheme?
    • Does the theoretical argument make sense?
  • Does it have a fatal flaw?
    • Is the method fatally flawed?
      • Unsound methodology
      • Discredited method
      • Missing processes known to be influential on the area of reported research
    • A conclusion drawn in contradiction to the statistical or qualitative evidence reported in the manuscript
    • Is an entire section missing?
    • Is the paper utterly unreadable?
    • Do they ignore a process that is known to have a strong influence on the area under study
    • Insufficient data points, statistically non-significant variations and unclear data tables.
    • Do the authors make claims that overreach the data?

Stage 3: Summarize

Draft the summary paragraph(s) of your review:

  1. Summarize the research question addressed
  2. Summarize the contribution of the work.

This tells the authors how you — the reader — have interpreted the aims, results and novelty of their research. 

Stage 4: Take a Break

After reading the manuscript, taking a break will help you decide which aspects really matter and to distinguish between major and minor issues. 

Stage 5: Drafting the Review

General

  • Use short, clearly-defined bullet-points and make it easy for the editor and author to see what section you’re referring to. Give specific comments and suggestions.
  • Specify exactly the point of weakness and where in the paper? When you ask for a revision, you have to be specific about what you want exactly from the author. If you feel that what you want is not possible in less than three months, it is better to reject the paper at the first stage.
  • Only mention flaws if they matter. Substantiate concerns/problems enough so that the authors can’t say, “Well, that’s not correct” or “That’s not fair.”

Summary

  • Introduction: state your expertise and whether the paper is publishable, or whether there are fatal flaws;
  • You can also discuss the possible readership for the manuscript. Is it interesting for researchers, or can it be of value to practitioners?
  • Brief summary of what the paper is about: understanding of the paper and what it claims. This should state the main question addressed by the research and summarize the goals, approaches, and conclusions of the paper. Help the editor properly contextualize the research and add weight to your judgement. Show the author what key messages are conveyed to the reader, so they can be sure they are achieving what they set out to do. Focus on successful aspects of the paper so the author gets a sense of what they’ve done well.
  • Overall assessment. Highlight strengths before briefly listing weaknesses. Give your main impressions of the article, including whether it is novel and interesting, whether it has a sufficient impact and adds to the knowledge base.
    • Indicate the work’s strengths, its quality and completeness.
    • This should provide a conceptual overview of the contribution of the research. So consider:
      • Is the paper’s premise interesting and important?
      • Are the methods used appropriate?
      • Do the data support the conclusions?
      • Indicate the significance of the work and if it is novel or mainly confirmatory
    • State any major flaws or weaknesses and note any special considerations. For example, if previously held theories are being overlooked

Major Comments — These should be the only comments that matter

  • Major comments may include suggesting a missing control that could make or break the authors’ conclusions or an important experiment that would help the story
  • Try to identify the authors’ claims in the paper that were not convincing and guide them to ways that these points can be strengthened (or, perhaps, dropped as beyond the scope of what this study can support). 
  • List the most critical aspects that the authors must address to better demonstrate the quality and novelty of the paper
  • State major flaws and what the severity of their impact is on the paper
  • Raise any similar work that has already been published or any relevant work that would contradict the authors’ work that are not addressed by the authors.
  • Identify if there are any major presentational problems that prevent you from accurately assessing the work.

Minor Comments — These can be ignored for the first review

  • Highlight obvious typos, grammatical errors, or any ambiguous language.
  • Flag mislabeling of figures or tables.

Stage 6: Suggestion

Some journals will not want the recommendation included in any comments to author. If recommending rejection or major revision, state this clearly in your review.

Acceptance

If you’re recommending acceptance, give details outlining why, and if there are any areas that could be improved. 

Revise & Resubmit

If I can see originality and novelty in a manuscript and the study was carried out in a solid way, then I give a recommendation for “revise and resubmit,” highlighting the need for the analysis strategy, for example, to be further developed. 

If recommending revision, state specific changes you feel need to be made. The author can then reply to each point in turn.

Rejection

If the research presented in the paper has serious flaws, I am inclined to recommend rejection, unless the shortcoming can be remedied with a reasonable amount of revising.

References & Sources

Tags:

Leave a Reply

Your email address will not be published. Required fields are marked *

css.php