How to write a reviewer report?

Every semester, I have graduate students coming to me asking how to write a reviewer report for journals. I have been a reviewer for IEEE Transactions and conferences for a long time. Recently, I also have the privilege to serve as an editor / conference area chair. I do not claim to be a super expert, but I think the following tips would be useful to first-time reviewers. If you disagree with anything I say below, feel free to ignore them.

Why should you accept review invitations?

• Learn how to criticize a paper: If you know how to criticize a paper, you know how to write your own paper.
• Build connections with editors: This will be very useful for your future career.
• Stay updated: It’s a great opportunity to read the latest (and unpublished) articles in your field.

What an editor cares about:

Should the paper be accepted or rejected?

Most beginning reviewers do not understand this single, most important objective of the review business. Editors are here to recommend acceptance or rejection of a paper. They do not “edit” papers. Your job is to provide your judgment so that the editor can make an appropriate decision.

A lot of reviewers provide the editor with a list of things they think the paper should change. But this is NOT what an editor is looking for! Tell the editor whether the paper should be published. If you want to accept the paper, make a strong case that it deserves publication. If you want to reject it, make a strong case that the paper is problematic. In any case, give evidence and support to back up your claims.

Things a reviewer should comment on:

Many editors (including myself) are more concerned about big pictures rather than technical details. Therefore, don’t write a large number of scattered comments. Focus on answering the following questions:

• Is there a problem? Sometimes people just create a problem and make complicated solutions. In some cases, the problem is just too ill-posed that you know there is no realistic solution. There are also problems that are too narrowly focused. Your job is to point these out.
• Are the contributions real contributions? Usually the authors will write their contributions in the introduction. Your job is to examine whether their claimed contributions are justified. Sometimes a contribution is just a combination of many existing methods.
• Does the theory support their claims? Every method is based on certain intuition and theory. If the intuition is fundamentally problematic, the whole method will collapse.
• Does the paper have sufficient experimental support to justify their claimed contributions? Do they have enough testing images? Do they have enough testing configurations? Do they provide standard performance metrics like MSE or PSNR?

Examples

Here are some examples I made up. I assume that the papers are submitted to IEEE Transactions on Image Processing. If you are reviewing papers for other journals, they may want different things.

• Example 1. (Lack of motivation) Suppose that you receive a paper about image denoising. The goal of the paper is to tackle a noise level greater than $200/255$. Without reading any further, the number one question I would ask is under what situation will there be noise greater than $200/255$. (It could happen, but the authors have to justify it.)
• Example 2. (Unrealistic assumption) Suppose that you receive a paper about image deblurring. The paper claims it has a powerful deblurring method, but it requires the knowledge of spatially varying PSFs at every pixel. Without reading any further, the comment I have would be that such an assumption is too strong to hold in practice.
• Example 3. (Known techniques in literature) Suppose that you receive a paper about depth estimation. The authors claim that they make new contributions by combining super-pixels, loopy belief propagation, graph cut, and hole-filling. But after reading the paper, you find that all methods are standard stuff in the literature. In this case, I would point out the references that contain these techniques.
• Example 4. (Incremental contribution) Suppose that you receive a paper about compressive sensing. The authors claim that they can tackle a Poisson model rather than the standard Gaussian model. However, the only thing they change is to replace the Gaussian likelihood by a Poisson likelihood. The body of the paper is a 5-page ADMM algorithm. In this case, I would say that the contribution is incremental.
• Example 5. (Flawed arguments) Suppose that you receive a paper about video denoising under heavy noise. The authors claim that they have a two-stage approach: Step 1: Use motion estimation to align the images; Step 2: Apply temporal filtering. I would be very skeptical about the paper because when noise is heavy, no motion estimation can provide accurate motion maps. So the temporal filtering will be very problematic.
• Example 6. (Insufficient experimental support) Suppose that you receive a paper about image super-resolution. The authors show a 2x super-resolution result of Cameraman.tif with no noise. If I were the reviewer, I would ask how the method will perform on other images. I would also ask about other testing configurations, e.g., noise level, 4x and 8x super-resolution.

Please do NOT do the following:

• Do not write a long summary of the paper (sometimes even longer than the paper’s abstract). Bear in mind that editors are usually experts in the field. You do not need to educate them on what the paper is about.
• Do not give a laundry list of typos, symbol issues, equations, etc. If the paper has too many typos, reject it and tell the authors to resubmit. If symbols are not clear, tell the authors to explain the symbols. Giving one or two examples is sufficient.
• Do not reject a paper by simply saying “the paper is not novel.” Well, there is nothing really novel in academia. Convince the editor and the authors with evidence. If something has been done before, give a reference.
• Do not reject a paper if it is actually a good paper.