1. CISC 879 : Machine Learning for Solving Systems Problems John Cavazos Dept of Computer & Information Sciences University of Delaware www.cis.udel.edu/~cavazos/cisc879 Lecture 7 How to Read, Critique, Write, and Present a Research Paper

2. CISC 879 : Machine Learning for Solving Systems Problems Lecture 7: Overview Effective and Efficient Reading Good and Bad Critical Reviewing How to Write a Paper How to Present a Paper

3. CISC 879 : Machine Learning for Solving Systems Problems Disclaimer Not easy way to read, write, critique and present! Practice, practice, practice Study good examples Ask advisor for examples of reviews Read award papers Example: http://www.usenix.org/publications/library/proceedings/best_papers.html Attend/watch talks by good speakers Videos available on internet Example: http://www.researchchannel.org/prog/

4. CISC 879 : Machine Learning for Solving Systems Problems What do I mean by “Read?” Effective and efficient reading Do not waste time on irrelevant papers Maximize your time on relevant papers The Whys, Whats, and Hows of Reading Critical Reading Essential to writing good critical reviews

5. CISC 879 : Machine Learning for Solving Systems Problems Problems with Some Papers Solving problems not applicable to an actual need CS has not adopted Scientific Method Least Publishable Unit (LPU) Never clearly contrasting to related work Hard to Read Scientific Method

6. CISC 879 : Machine Learning for Solving Systems Problems Questions to Ask What are the motivations? What is solution and how is it evaluated? What is your analysis of problem, solution, and evaluation? What are the major results? Correct, new, clearly presented, worth publishing What are contributions? Are there any questions not answered? Reference: http://www-cse.ucsd.edu/users/wgg/CSE210/howtoread.html

7. CISC 879 : Machine Learning for Solving Systems Problems Efficient Reading Preparation Quiet place and note-taking material Read for Breadth Read title and abstract Skim intro, headers, tables, graphs, and conclusions Look for primary contributions Read in Depth Challenge arguments, assumptions, evaluations, and conclusions Fallacy: Must read from beginning to end Reference: www.cs.columbia.edu/~hgs/netbib/efficientReading.pdf

8. CISC 879 : Machine Learning for Solving Systems Problems Lecture 7: Overview Effective and Efficient Reading Good and Bad Critical Reviewing How to Write a Paper How to Present a Paper

9. CISC 879 : Machine Learning for Solving Systems Problems Anatomy of Good Review Summary of main points of paper (3 to 5 sentences) Evaluation with regard to validity and significance Evaluation quality of work Overall recommendation and thorough justification Reference: http://www.cs.utexas.edu/users/mckinley/notes/reviewing-smith.pdf Evaluation of novelty, significance, correctness, readability.

10. CISC 879 : Machine Learning for Solving Systems Problems Bad Critical Reviewing Seek to find all flaws Criticize paper in a negative way Use words like “Idiot” and “Trash” Criticize paper with little comment After all, they know it’s bad Criticize/praise all papers Be known as person that is overly harsh/praising everything! Reference: http://www.cs.utexas.edu/users/mckinley/notes/reviewing.html

11. CISC 879 : Machine Learning for Solving Systems Problems Good Critical Reviewing Can Always be Positive Think face-to-face constructive criticism Consider Recently-Published New Directions Explain your Decision No explanation will carry little weight Decide which paper is best Not which paper least worthy of rejection Reference: http://www.cs.utexas.edu/users/mckinley/notes/reviewing.html

12. CISC 879 : Machine Learning for Solving Systems Problems Bad Review Example This paper is concerned with blah blah blah. The authors develop a method to automatically generate blah blah blah. The solution is based on genetic algorithms and evolution-based methodology. This is a solid paper with respect to methodology and the experiments performed. My score is relatively low because I am not particularly excited by genetic, evolutionary algorithms. Not a Reasonable Justification!

13. CISC 879 : Machine Learning for Solving Systems Problems Good Review Example I like this work a lot. There are two things that would substantially improve it: 1) Investigation of the reasons why blah blah blah. Is there room for more improvement here, perhaps combining the two in some way? If not, what does this say about what technique we should apply, if forced to choose? 2) Quantitative evaluation of the predictive power of blah blah blah. You seem to have a lot of the necessary data (ie, you can compare the results of blah blah blah from Fig 1). Starts positive, then suggests how to “substantially improve paper.” Reviewer justifies remark!

14. CISC 879 : Machine Learning for Solving Systems Problems Lecture 7: Overview Effective and Efficient Reading Good and Bad Critical Reviewing How to Write a Paper How to Present a Paper

15. CISC 879 : Machine Learning for Solving Systems Problems Getting a Paper Accepted Follow the Rules! paper format, length, on time, etc. Spell and Grammar Check Write, Read, Edit, Repeat (many times!) Have colleagues review After-the-fact outline (check flow of paper) Read good papers Ask advisor and your teachers for good papers Seminal papers

16. CISC 879 : Machine Learning for Solving Systems Problems Getting a Paper Accepted Important: Write to the reader What does the reader know and what do they expect next? Quality must be recognized quickly What are contributions? Is paper stimulating? Is paper relevant? Have a planned organization of paper Enough details to reproduce experiments

17. CISC 879 : Machine Learning for Solving Systems Problems Anatomy of a Good Paper Abstract (4-8 sentences) State the problem and why its important Briefly describe solution Implication of solution Introduction (10% of paper) Describe problem State your contributions (bulleted list) Describe the problem (10%) Motivate the problem

18. CISC 879 : Machine Learning for Solving Systems Problems Anatomy of a Good Paper Describe your idea (20%) Convince reader idea can solve problem Implementation details The details (45%) Convince reader you solved problem State reasonable counterexamples Related work (10%) Convince of novelty Conclusion (5%) Summary of results and significance of contributions

19. CISC 879 : Machine Learning for Solving Systems Problems Icing on the Cake Attention Grabber Sentence Great Introduction! More time editing = less time reading Convey intuition One reader has intuition, can follow details Good looking graphs Can understand (paper) from reading graph captions Use Examples!! Present the general case

20. CISC 879 : Machine Learning for Solving Systems Problems Additional Resources The Elements of Style By Strunk and White Style, Toward Clarity and Grace By Joseph M. Williams Talks of Research Writing http://research.microsoft.com/~simonpj/papers/giving-a-talk/writing-a-paper-slides.pdf http://www.cs.utexas.edu/users/mckinley/pl-summer-2007/presentations/session1/william- cook.pdf

21. CISC 879 : Machine Learning for Solving Systems Problems Lecture 7: Overview Effective and Efficient Reading Good and Bad Critical Reviewing How to Write a Paper How to Present a Paper

22. CISC 879 : Machine Learning for Solving Systems Problems What is Your Talk is about The Paper = The Beef The Talk = Beef Advertisement Do not confuse the two! Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz

23. CISC 879 : Machine Learning for Solving Systems Problems Your Talk is NOT About: To impress your audience with your brainpower To tell them all you know on the topic To present all the technical details Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz

24. CISC 879 : Machine Learning for Solving Systems Problems Your audience Have never heard of the topic Just had lunch and ready to nap Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz Your mission is WAKE THEM UP and make them glad you did.

25. CISC 879 : Machine Learning for Solving Systems Problems What’s in your Talk Motivation (20%) 2 minutes to engage audience Why should I listen? What is the problem? Why is it an interesting solution? Key Idea (80%) Be specific Organize talk around specific goal Ruthlessly prune everything else! Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz

26. CISC 879 : Machine Learning for Solving Systems Problems Tips for a Good Talk Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz Narrow, deep rather than wide, shallow Avoid shallow overview talks Cut to the chase: technical “meat” Examples are your main weapon! To motivate talk To convey intuition To illustrate key idea

27. CISC 879 : Machine Learning for Solving Systems Problems What to leave out Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz Lots of animation and color Slides with lots of text You do not want to read everything that is on your slides, otherwise your audience might wonder why they just don’t just simply read your slides. You do not want a lot of text on your slides as people will want to read everything you have written and will be distracted from your presentation. Use your slides as a guide. Technical Detail Extensive formulas and code Dense slides will put audience to sleep! Present specific aspects; refer to paper for details Related Work Know it, but don’t elaborate on it

28. CISC 879 : Machine Learning for Solving Systems Problems What NOT to Do Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz Use small fonts Reveal one point at a time Do not apologize “I didn’t have time to prepare the talk” “I don’t feel properly prepared to give this talk” Go over your time limit Stand in front of projected material Stand with back to audience Point to laptop

29. CISC 879 : Machine Learning for Solving Systems Problems Final Tips Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz Be Enthusiastic Practice Memorize first few sentences Move around, use arms Speak to someone at back of room Identify nodders and speak to them Watch for questions