How to write a reviewer report?Every semester, I have graudate students coming to me asking how to write reviewer report for journals. I have been a reviewer for IEEE Transactions and conferences for quite a long time. Recently I also have the previledge to serve as editor / conference area chair. I do not claim that I am a super expert but I think the following tips would be useful to first time reviewers. If you disagree with anything I say below, feel free to ignore them. Why should you accept review invitation?
What editor cares about:
Most beginning reviewers do not understand this sinlge and the most important objective of the review business. Editors are here to recommend acceptance or rejection of a paper. They do not “edit” papers. Your job is to provide your judgement so that the editor can make appropriate decisions. A lot of reviewers provide the editor a list of things they think the paper should change. But this is NOT what an editor is looking for! Tell the editor whether the paper should be published. If you want to accept the paper, make a strong case that it deserves publication. If you want to reject, make a strong case that the paper is problematic. In any case, give evidence and support to backup your claims.
Things a reviewer should comment on:Many editors (including myself) are more concerned about big pictures rather than technical details. Therefore, don't write a large number of scattered comments. Focus on answering the following questions.
ExamplesHere are some examples I made up. Pardon me if they offend anyone. I assume that the papers are submitted to IEEE Trans Image Processing. If you are reviewing papers for other journals they may want different things. Example 1. (Lack of motivation) Suppose that you receive a paper about image denoising. The goal of the paper is to tackle noise level greater than 200/255. Without reading any further, the number one question I would ask is under what situation will there be noise greater than 200/255. (It could happen, but the authors have to justify.) Example 2. (Unrealistic assumption) Suppose that you receive a paper about image deblurring. The paper claims it has a powerful deblurring method but it requires the knowledge of spatially varying PSFs at every pixel. Without reading any further, the comment I have would be that such assumption is too strong to hold in practice. Example 3. (Known techniques in literature) Suppose that you receive a paper about depth estimation. The authors claim that they make new contributions by combining super-pixels, loopy belief propagation, graph cut and hole-filling. But after reading the paper you find that all methods are standard stuff in the literature. In this case, I would point out the references that contain these techniques. Example 4. (Incremental contribution) Suppose that you receive a paper about compressive sensing. The authors claim that they can tackle a Poisson model rather than the standard Gaussian model. However, the only thing they change is to replace the Gaussian likelihood by a Poisson likelihood. The body of the paper is a 5-page ADMM algorithm. In this case, I would say that the contribution is incremental. Example 5. (Flawed arguments) Suppose that you receive a paper about video denoising under heavy noise. The authors claim that they have a two stage approach. Step 1: Use motion estimation to align the images; Step 2: Apply temporal filtering. I would be very skeptical about the paper because when noise is heavy, no motion estimation can provide accurate motion maps. So the temporal filtering will be very problematic. Example 6. (Insufficient experimetal support) Suppose that you receive a paper about image super-resolution. The authors show a 2x super-resolution result of Cameraman.tif with no noise. If I were the reviewer I would ask how the method will perform on other images. I would also ask about other testing configurations, e.g., noise level, 4x and 8x super-resolution. Please do NOT do the followings:
|