1 Thoughts on how to improve reviews Paul Francis (Presented by Robbert Van Renesse) Cornell

2 Main problem with current system Reviews are often of poor quality I suspect main reason is that PC members are overworked Both ideas in this presentation are focused on this one problem alone

3 Idea One: Allow PC members to ask authors simple “where in the paper can I find this?” questions Authors can answer ONLY with page, column, and line numbers Why?  Often when I reject a paper, it is based on one or a few specific flaws. It would be good to verify that I’m not overlooking something.

4 A reviewer comment “But you can do this today...just ask the PC chairs to forward an ” My rebuttal: 1. Bothering the PC chairs is a significant deterrent 2. Should really limit author reply to text already in paper...otherwise author is effectively submitting more than what is in the paper

5 Today

6 Proposal One: Pass reviews on

7 Observations Reviewer gets the benefit of previous reviewers Various ways this can be exploited:  To improve review: Reviewer first does complete unbiased review, then uses previous reviews as sanity check  To reduce work: Reviewer scans paper, previous reviews, and rebuttal, and then does only enough work to decide if author has overcome previous criticisms This can be done in good faith

8 Observations Good paper should be able to overcome earlier criticisms  Without having to torque paper to satisfy future reviewers....  Instead author can use rebuttal Truly bad paper should not be able to overcome earlier criticisms  Prevents bad paper from retrying until it gets lucky  Might discourage “hail-mary” submissions

9 A reviewer comment “This will promote reviewer laziness” My rebuttal:  A good-faith reviewer will benefit from earlier reviews  A lazy reviewer already has many ways of being lazy (form instant opinion, do minimal work to support opinion...)