1. Scholarship Skills Tim Sheard & Todd Leen 1 Lecture 19 Scholarship Skills Tim Sheard, PSU Todd Leen, OGI-OHSU All material © 1996, 1997, 1999, 2000 David Maier, © 2006 Todd Leen Lecture 19: Reviewing

2. Scholarship Skills Tim Sheard & Todd Leen 2 Lecture 19 Reviewing Basically an altruistic activity Why do it? –Altruism is a good thing –Intelligent, thoughtful reviews enhance the quality of the journal or conference –Enhance your reputation (with editor or program chair) –Step to becoming an editor or program committee member –Shows professional participation You may be called upon to review a journal or conference paper as a grad student. (Either directly or by a professor.)

3. Scholarship Skills Tim Sheard & Todd Leen 3 Lecture 19 Basic Job You must determine –Are the results worthwhile? –Are they probably right? (How much time/effort is expected?) –Are they presented well? You might – suggest ways to improve investigation – suggest ways to improve exposition – point to work authors may have overlooked – break anonymity if it’s helpful

4. Scholarship Skills Tim Sheard & Todd Leen 4 Lecture 19 Read the Paper Start by reading the paper carefully –take notes as you go –mark typos on the MS –if it’s dreadful writing, can say that the expositional deficiencies prevented you from giving it a careful reading –In extreme cases can abandon paper -- but TELL THE EDITOR WHY! –checking every example and proof? not always, but SAY SO

5. Scholarship Skills Tim Sheard & Todd Leen 5 Lecture 19 Evaluation Evaluate results, defense, presentation Is it technically sound, valid methodology? Is there a solid defense? Is it significant? If not, need to say why it is obvious, simple or of low utility Archival quality -- interesting for 5+ years? Is length appropriate to contribution

6. Scholarship Skills Tim Sheard & Todd Leen 6 Lecture 19 Evaluation (cont’d) Is it novel? –either results, or methods, or conceptual synthesis Is it interesting? Motivation, context, interpretation Helps understand nature of computation Can be used to solve problems Will spawn other research Timely to the field

7. Scholarship Skills Tim Sheard & Todd Leen 7 Lecture 19 Evaluation (cont’d) Is it deep? Might you get it as a homework problem in a graduate class? Is it concise? Irrelevant parts, needless detail But not too concise: shouldn’t be hard to follow Can it be understood by people in the field? Mostly self-contained Something for non-experts in its particular area

8. Scholarship Skills Tim Sheard & Todd Leen 8 Lecture 19 Evaluation (cont’d) Is the paper complete? Methodology adequately described No gaps in the development Is the research complete? Style and polish Has it been prepared with thought and care? Easy to read? Does it give proper credit to others?

9. Scholarship Skills Tim Sheard & Todd Leen 9 Lecture 19 Decide Conferences sometimes have a numeric scale, journals may ask for numerical judgment in several areas (novelty, completeness, importance, clarity). – Reject -- no hope (rare) – Revise – for exposition, may also ask for additional experiments or theoretical elements – Accept as is (unlikely )

10. Scholarship Skills Tim Sheard & Todd Leen 10 Lecture 19 Professional Responsibility Be objective and fair Put aside personalities and favored approaches Alert editor or program chair to conflicts of interest Speed Try for deadline in cover letter – editorial staff will send reminders Make an acceptable paper as good as possible -- that is, tell the author how to improve the paper Confidential Courteous and honest report How certain and qualified are you in your judgements?

11. Scholarship Skills Tim Sheard & Todd Leen 11 Lecture 19 Typical Reports Out of scope Wrong journal or conference Already done Give citation Too simple You could solve it if posed as an exercise Not very significant Maybe okay if shortened Major errors Essential parts are wrong Not obligated to determine if fixable, but give an opinion if worth trying to fix, or fixable.

12. Scholarship Skills Tim Sheard & Todd Leen 12 Lecture 19 Typical Reports ( cont’d) Boring Grind through the details, tedious experiments High overhead for results Lots of work for little output Poorly presented Not a good job of writing the paper Is there a basis for a good paper? (Telling people how to salvage their paper is time consuming. These tend to be the longest and most tedious reviews to write.) Acceptable with minor (or no) revisions

13. Scholarship Skills Tim Sheard & Todd Leen 13 Lecture 19 Typical Reports ( cont’d) There is sometimes a review form provided (e.g. IEEE). Don’t feel constrained by it, especially for a journal (TkL almost never uses them.)

14. Scholarship Skills Tim Sheard & Todd Leen 14 Lecture 19 Typical Reports ( cont’d) For editor (PC chair) and authors Shorter for a conference paper Statement of significance If you are recommending rejection, say why. General comments, suggestions for improvement Detailed notes Usually by page or section

15. Scholarship Skills Tim Sheard & Todd Leen 15 Lecture 19 Kinds of Comments Typos, grammar, confusing statements References to add Requests to compare to other work Additional points to address in paper Organization Sketchy sections, parts to omit, pieces to move Presentation Better formulation of definitions, need for examples Technical errors If you think some major point is wrong, try to include a counterexample.

16. Scholarship Skills Tim Sheard & Todd Leen 16 Lecture 19 Cover Letter Private Remarks to Editor Not seen by the authors Identify paper –title –author –for which journal or conference –number Your expertise and effort (time spent), caveats (“this isn’t an area of my greatest expertise...”) Summary of paper and your opinion One or two paragraphs

17. Scholarship Skills Tim Sheard & Todd Leen 17 Lecture 19 Cover Letter Private Remarks to Editor Your recommendations and justification Parts of report you want to emphasize -- specific comments Anything else not for the authors