How to write a reviewer report?

Every semester, I have graudate students coming to me asking how to write reviewer report for journals. I have been a reviewer for IEEE Transactions and conferences for quite a long time. Recently I also have the previledge to serve as editor / conference area chair. I do not claim that I am a super expert but I think the following tips would be useful to first time reviewers. If you disagree with anything I say below, feel free to ignore them.

Why should you accept review invitation?

  • Learn how to criticize a paper. If you know how to criticize a paper, you know how to write your own paper.

  • Build connection with editors. Will be useful for your future career.

  • Opportunity to read the latest (and unpublished) articles in your field.

What editor cares about:

  • Should the paper be accepted or rejected?
    Most beginning reviewers do not understand this sinlge and the most important objective of the review business. Editors are here to recommend acceptance or rejection of a paper. They do not “edit” papers. Your job is to provide your judgement so that the editor can make appropriate decisions. A lot of reviewers provide the editor a list of things they think the paper should change. But this is NOT what an editor is looking for! Tell the editor whether the paper should be published. If you want to accept the paper, make a strong case that it deserves publication. If you want to reject, make a strong case that the paper is problematic. In any case, give evidence and support to backup your claims.

Things a reviewer should comment on:

Many editors (including myself) are more concerned about big pictures rather than technical details. Therefore, don't write a large number of scattered comments. Focus on answering the following questions.

  • Is there a problem? Sometimes people just create a problem and make complicated solutions. In some cases, the problem is just too ill-posed that you know there is no realistic solution. There are also problems that are too narrowly focused. Your job is to point these out.

  • Are the contributions real contributions? Usually the authors will write their contributions in the introduction. Your job is to examine whether their claimed contributions are justified. Sometimes a contribution is just a combination of many existing methods.

  • Does the theory support their claims? Every method is based on certain intuition and theory. If the intuition is fundamentally problematic, the whole method will collapse.

  • Does the paper have sufficient experimental support to justify their claimed contributions? Do they have enough testing images? Do they have enough testing configurations? Do they provide standard performance metrics like MSE or PSNR (if you are reviewing for IEEE TIP or associated conferences).

Examples

Here are some examples I made up. Pardon me if they offend anyone. I assume that the papers are submitted to IEEE Trans Image Processing. If you are reviewing papers for other journals they may want different things.

Example 1. (Lack of motivation)
Suppose that you receive a paper about image denoising. The goal of the paper is to tackle noise level greater than 200/255. Without reading any further, the number one question I would ask is under what situation will there be noise greater than 200/255. (It could happen, but the authors have to justify.)

Example 2. (Unrealistic assumption)
Suppose that you receive a paper about image deblurring. The paper claims it has a powerful deblurring method but it requires the knowledge of spatially varying PSFs at every pixel. Without reading any further, the comment I have would be that such assumption is too strong to hold in practice.

Example 3. (Known techniques in literature)
Suppose that you receive a paper about depth estimation. The authors claim that they make new contributions by combining super-pixels, loopy belief propagation, graph cut and hole-filling. But after reading the paper you find that all methods are standard stuff in the literature. In this case, I would point out the references that contain these techniques.

Example 4. (Incremental contribution)
Suppose that you receive a paper about compressive sensing. The authors claim that they can tackle a Poisson model rather than the standard Gaussian model. However, the only thing they change is to replace the Gaussian likelihood by a Poisson likelihood. The body of the paper is a 5-page ADMM algorithm. In this case, I would say that the contribution is incremental.

Example 5. (Flawed arguments)
Suppose that you receive a paper about video denoising under heavy noise. The authors claim that they have a two stage approach. Step 1: Use motion estimation to align the images; Step 2: Apply temporal filtering. I would be very skeptical about the paper because when noise is heavy, no motion estimation can provide accurate motion maps. So the temporal filtering will be very problematic.

Example 6. (Insufficient experimetal support)
Suppose that you receive a paper about image super-resolution. The authors show a 2x super-resolution result of Cameraman.tif with no noise. If I were the reviewer I would ask how the method will perform on other images. I would also ask about other testing configurations, e.g., noise level, 4x and 8x super-resolution.

Please do NOT do the followings:

  • Please do not write a long summary of the paper, sometimes even longer than the paper's abstract. Bare in mind that editors are usually experts in the field. You do not need to educate them what the paper is about.

  • Please do not give a laundary list of typos, symbols issues, equations, etc. If the paper has too many typos, reject it and tell the authors to resubmit. If symbols are not clear, tell the authors to explain the symbols. Giving one or two examples is sufficient.

  • Please do not reject a paper by saying “the paper is not novel.” Well, there is nothing really novel in academia. Convince the editor and the authors with evidence. If something has been done before, give reference.

  • Please do not reject a paper if it is actually a good paper.