Research Methods and Techniques Lecture 1 Introduction & Paper Review 1 © 2004, J S Sventek, University of Glasgow

30 September 2004RMaT/Paper Review 12 Module Logistics Web site: Course director: Prof J S Sventek, Meetings: Thursdays, 16:00-17:00, F171 Textbooks: none, all readings and lecture slides available from the web site one week prior to the lecture at which the topic is covered Assessment: 30% assessed coursework, 70% examination

30 September 2004RMaT/Paper Review 13 Assessed Coursework Breakdown for assessed coursework: 10% for 1-page summaries for three assigned papers 10% for in-depth review of one assigned paper 10% for annotated bibliography on an assigned topic Completed assignments must be handed in at the beginning of the lecture noted on the schedule. No late assignments accepted.

30 September 2004RMaT/Paper Review 14 Course Schedule Lecture DateTopic Assignment Due 30 SeptemberPaper Review 1 – Summaries 7 OctoberPaper Review 1 – In-depth Paper 1 Summary 14 OctoberLiterature Survey 1 Paper 2 Summary 21 OctoberLiterature Survey 2 Paper 3 Summary 28 OctoberDesign of Experiments 4 NovemberTools for Experimental Use Paper 4 In-depth 11 NovemberResearch Career Skills 18 NovemberPresentation Skills 25 NovemberTechnical Writing 1 2 DecemberTechnical Writing 2 Annotated bibliography

30 September 2004RMaT/Paper Review 15 Why are reviews needed? Researchers produce papers that document knowledge resulting from their research Each paper is refereed by the authors’ peers (peer review) Referee decides whether a paper makes a sufficient contribution to the field Researchers produce proposals for funding to carry out future research Reviewer must determine: is the proposed topic significant? is the methodology reasonable? do the proposers have sufficient expertise to carry out the research? is the budget reasonable? are necessary facilities available?

30 September 2004RMaT/Paper Review 16 Why are we discussing this in ATiCS? Many of the skills of a good referee are required to review published literature as part of your research Reviewing submitted papers and research proposals is one of the professional obligations of a computing science professional Sound knowledge of how reviews are done will assist you in producing papers and proposals that are more likely to be accepted Your academic supervisor may ask you to review one or more papers or proposals that he/she receives while you are here as raw input to the final reviews that he/she returns to the editor/program chair/funder

30 September 2004RMaT/Paper Review 17 A Taxonomy of Research Papers Breakthrough: solves a long-standing problem, or introduces a totally new way of looking at the world Ground-breaking: opens up a field that has not been well explored or understood, and lays a firm foundation Progress: raises and solves important new problems in a well- established field Reprise: reworks an existing result (by the authors or others), yielding new insight Tinkering: extends a known result by a more careful, but non- obvious, methodology Debugging – elucidates and repairs a flawed previously-published result Survey: surveys and unifies a body of knowledge All categories are important if a genuine contribution is made.

30 September 2004RMaT/Paper Review 18 Evaluating a Research Paper What is the purpose of the paper? Is the paper appropriate for the venue? Is the goal of the research significant? Is the method of approach valid? Is the actual execution of the research correct? Are the correct conclusions drawn from the results? Is the presentation satisfactory? Will readers learn anything new from this paper?

30 September 2004RMaT/Paper Review 19 Ethical considerations For the referee Objectivity Fairness Speed Professionalism Confidentiality Honesty Courtesy For the author Appropriate attributions for the work of others The same work is not submitted simultaneously to two or more venues The same work has not been previously published One accepted form of republication: the ACM and IEEE permit republication of conference papers in journals if the revised paper meets a higher standard than for the conference.

30 September 2004RMaT/Paper Review 110 Desirable attributes for a paper Correctness – there are no flaws in the logic used by the authors to reach their conclusions Significance – the results should not be obvious, trivial, or simple – i.e. they make a sufficiently new contribution to the field Innovation – results are original and innovative, not trivial extensions or rework of old results that do not introduce new understanding Interest – will anyone outside of the immediate area care? Timeliness – results should be motivated by recent interest from the science community in related subject areas

30 September 2004RMaT/Paper Review 111 Desirable attributes for a paper (continued) Succinctness – the entire paper should be crisp and to the point Accessibility – should be largely self-contained, and should include careful explanations of basic concepts Elegance – proofs should be elegant/intellectually satisfying Readability – the information in the paper should be available to the reader with a minimum of effort Style – should be well-structured, with flowing technical prose and scholarly vocabulary and grammar – no poetry Polish – should reflect care taken by the author[s] in all aspects of its preparation

30 September 2004RMaT/Paper Review 112 Review Template Briefly summarize the paper in 3-5 sentences. What problem does this contribution solve? Is it an important problem? What are the claimed contributions of the paper? Upon what (and whose) previous work has this research been based? What methodology has been used? Is it appropriate? What conclusions are drawn from the results? Are they correct? Has the research been performed correctly? Is the presentation satisfactory? Is the paper appropriate for the target venue?