Rob Holte University of Alberta holte@cs.ualberta.ca ICML’2003 Minitutorial on Research, ‘Riting, and Reviews Reviews Rob Holte University of Alberta holte@cs.ualberta.ca January 11, 2019

Purpose Evaluate the paper’s scientific merit Check the validity of the paper’s claims and evidence Judge the paper’s relevance and significance Provide constructive feedback to the author January 11, 2019

Example: IJCAI’99 review form 1. How RELEVANT is this paper to AI researchers? 2. How SIGNIFICANT is this paper? 3. How ORIGINAL is this paper? 4. Is this paper technically SOUND? 5. How well is this paper PRESENTED? Further comments, advice or explanations (Please be specific and constructive, especially with respect to any negative judgements above. Point to the section(s) where an error occurs, cite omitted references, etc.) January 11, 2019

Excerpts from ICML’03 form GOALS/MOTIVATION. Does the submission state the goals of the research, including the criteria by which readers should evaluate the results? Is the learning problem well-motivated? CLAIMS/EVIDENCE. Do the authors make explicit claims or draw clear conclusions, and do they present reasonable evidence to support their position? If the claims are theoretical, are the proofs correct? If the claims are empirical, are the experiments appropriate, and do the results reveal the underlying reasons or causes for phenomena? How might the authors strengthen their claims, evidence, or reasoning? January 11, 2019

Example: COLT’97 instructions FOUNDATIONAL/CONCEPTUAL CONTRIBUTION: Note things like a new model, new notion, new definition, new approach. Note the significance and reasons for this novelty (and note the absence of such a novelty). TECHNICAL DEVELOPMENT: - Introduces a new technique - Novel use of known technique - Talented use of known technique - Traditional use of known technique - Trivial technically January 11, 2019

COLT’97 continued RELATION TO OPEN PROBLEMS: Does the paper solve completely/partially/ or does it address an open question? How important is this question? (Central/ important/ interesting/ legitimate/ stupid). How much effort has been invested before in solving it and by whom? January 11, 2019

COLT’97 continued SOCIAL INTEREST IN PAPER: Is it potentially interesting to the whole COLT community, to a major subarea, to everyone in a restricted area, interesting only to the authors. HOW WILL IT CONTRIBUTE: Fertilization, satisfy curiosity, who knows? PAPER TYPE: Is it a - First step (opens a new area) - Last step (closes an important area) - Giant step (makes essential progress) - None of the above. January 11, 2019

Good Reviews Polite Fair Concise Clear Constructive Specific Well-documented Represent the scientific community January 11, 2019

Anonymity Reviewers are supposed to be anonymous BUT, sometimes their identity becomes known or is at least guessed by the authors Write your reviews so that you would not be embarrassed if your identity was revealed January 11, 2019

“In my opinion…” Reviews are necessarily subjective A paper must convince its readers that its claims are valid and significant A reviewer is more careful, thorough and patient than an average reader January 11, 2019

Reviews that “miss the point” Not uncommon, very frustrating Don’t blame the reviewer. Fix your paper. January 11, 2019

Conference reviews short time frame each reviewer has several papers to review Expect reviews to be terse, less thorough, less satisfying January 11, 2019