Research Seminar – CPIT 695 Reading Skills 1

What is required from you? how research papers are written, how to read such papers critically and efficiently, how to summarise and review them. how to gain an understanding of a new field, in the absence of a textbook how to judge the value of different contributions how to identify promising new directions

What is a Paper? A paper: the prime medium for reporting scientific results In computer science: the conjunction of something conceptual (and preferably new), something sound, and something practical that is relevant today (read: IT).

Reading Skills In principle, a paper provides enough information for its reader to reproduce its contents: proof, experiment. The three stages of reading (as one grows up): 1. the books say blah and thus it is true; 2. this book says blah; 3. the author wrote blah. On reading critically Don’t swallow the author’s propaganda, accepting the paper as the author shaped it: disassemble it to identify its real thrust; appreciate; probe / question / stress; (if needed) reassemble: minimize / expand.

Reading Skills What if you don’t understand something? What if you think you found a bug? Recommendation: 1. Consult people locally. 2. Send a very polite to the author (keeping in mind that you may well be wrong yourself). Prudence: You don’t just represent yourself. You represent your adviser and your institution.

Some elements for a review 1. Convey your understanding of the paper with a summary. 2. Double up with an analysis. 3. Sum up with an assessment and a recommendation. 4. Add a list of remarks, if any. Context of the review: Be objective. Be fair. Don’t delay. Be courteous. Remain confidential. To summarize Reviews should be as comprehensive as possible. Reviews should be as courteous as possible. Reviews should be as selfless as possible.

Reading Skills Don't let other people's frame of mind limit your creativity. Proofreading each other What: an investment. How: with consideration. Be prepared to be misunderstood. Why read papers So you know what’s happening Avoid reinventing the wheel does happen commonly, too many wheels already Find interesting research topics

Why not to read papers Cannot read everything Should not read everything Can suppress innovation: once you see solutions using a particular theme, often hard to think differently Pitfall It is of no use to spend almost all of your time in literature review and seminars. It is impossible to ``finish a literature review and then start research." New literature is always appearing, and as your depth and breadth increases, you will continually see new connections and related areas that must be studied. Active listening and reading must be viewed as ``continuing education'' that will involve you for the rest of your career. Don't fool yourself into thinking it must be finished before you can begin research.

Read or not to read, that is the question Read, of course Know what’s important Know what can be ignored without significant loss of information

What’s in a paper Abstract Introduction Motivation Problem description Solution... Performance Analysis Conclusions Future Work

How to read a paper? Know why you want to read the paper To know what’s going on (e.g., scanning proceedings) title, authors, abstract Papers in your broad research area introduction, motivation, solution description, summary, conclusions sometimes reading more details useful, but not always Papers you may want to improve on read entire paper carefully

What to note Authors and research group Need to know where to look for a paper on particular topic Theme of the solution Should be able to go back to the paper if you need more info Approach to performance evaluation Note any shortcomings

Reading Skills To develop an effective reading style for research papers, it can help to know what you should get out of the paper, and where that information is located in the paper. Typically, the introduction will state not only the motivations behind the work, but also outline the solution. Often this may be all the expert requires from the paper. The body of the paper states the authors' solution to the problem in detail, and should also describe a detailed evaluation of the solution in terms of arguments or an empirical evaluation (case study, experiment, etc.). Finally, the paper will conclude with a recap, including a discussion of the primary contributions. A paper will also discuss related work to some degree. Papers are often repetitive because they present information at different levels of detail and from different perspectives. As a result, it may be desirable, to read the paper ``out of order'' or to skip certain sections.

Questions to be answered while reading a paper What are motivations for this work? For a research paper, there is an expectation that a solution to a problem has been found that no one else has published in the literature. The paper should describe why the problem is important and why it does not have a trivial solution; that is, why a new solution may be required. Implicitly there is implication that previous solutions to the problem are inadequate. Occasionally an author will fail to state either point, making your job much more difficult. What is the proposed solution? This is also called the hypothesis or idea. There should also be an argument about why the solution solves the problem better than previous solutions. There should also be a discussion about how the solution is achieved (designed and implemented) or is at least achievable. Are all concepts and notations introduced before their first usage?

Questions to be answered while reading a paper What is the evaluation of the proposed solution? An idea alone is usually not adequate for publication of a research paper. What argument and/or experiment is made to make a case for the value of the ideas? What benefits or problems are identified? Are they convincing? What alternative solutions exist? Read a paper critically. What is your analysis of the identified problem, idea and evaluation? Is this a good idea? What flaws do you perceive in the work? What are the most interesting points made? What are the most controversial ideas or points made? For work that has practical implications, you also want to ask: Is this really going to work, who would want it, what it will take to give it to them, and when might it become a reality?

Questions to be answered while reading a paper What are the contributions? How does this contribution relate to work previously encountered? The contributions in a paper may be many and varied. Ideas, software, experimental techniques, and area survey are a few key possibilities. What are future directions for this research? Not only what future directions do the authors identify, but what ideas did you come up with while reading the paper? Sometimes these may be identified as shortcomings or other critiques in the current work. What questions are you left with? What questions would you like to raise in an open discussion of the work? What do you find confusing or difficult to understand? By taking the time to list several, you will be forced to think more deeply about the work. What is your take-away message from this paper? Sum up the main implication of the paper from your perspective. This is useful for very quick review and refreshing your memory. It also forces you to try to identify the essence of the work. What are the important references cited in this paper by the author?

Reading Skills Presumably, the introduction should provide motivation. The introduction and conclusion may discuss the solutions and evaluation at a high level. Future work is likely in the concluding part of the paper. The details of the solution and the evaluation should be in the body of the paper. You may find it productive to try to answer each question in turn, writing your answer down. In practice, you are not done reading a paper until you can answer all the questions.  Use the form (PaperReviewForm.pdf) given to fill OR write an abstract using the answers from the questions.

Getting the bigger picture What problem is the paper trying to solve? Highlight three to four lines What are the limitations of prior work? Highlight maximum two to three lines How is the problem solved by this paper? What is the paper s’ contribution? Highlight three to five lines How is the proposed solution evaluated? What kind of data/experiments were conducted? No need to highlight anything; you can highlight two to three words here

Getting the Details What problem is the paper trying to solve? Read the System Model carefully The whole paper is going to be based on this model Understand the gist of the contribution/proposal Does it make sense? Do you think it will work? How will the proposal be evaluated? Don’t try to read all the math in one go Read the assumptions and system model Try to work out a solution to the problem Is the evaluation fair and comprehensive? Try to find the next paper you want to read from this section Understand the results Do not miss a single figure and table Find the corresponding discussions in the paper and read them thoroughly

Reading Skills – What to retain? The problem The basic idea of the proposed solution Your personal notes on the paper’s mathematics Shortcomings of the proposed approach

Reading Skills – Ask yourself What is the message you take away from this paper? Are you convinced that the paper attempted an important problem? If your answer is NO, Justify it! If your answer is NO, Justify it! ‰Are you convinced that the paper proposed a viable solution? If your answer is NO, Justify it! ‰What questions still remain unanswered?

If you find a well-written paper After reading a well written paper, consider the presentation. It may be nicely organized, or it may underscore concepts with thoughtful examples. What makes this paper easy to read? What level of detail is provided? What examples are used to demonstrate important concepts? What questions are left unanswered? Can the results be generalized?

Getting most out of what you read Be organized Keep an electronic bibliography with notes & pointers to the paper files Keep and file all the papers you have read or skimmed Be efficient - only read what you need to Start by reading only the conclusion, scanning figures & tables, and looking at their references Read the other sections only if the paper seems relevant or you think it may help you get a different perspective Skip the sections that you already understand (often the background and motivation sections) Take notes on every paper you find worth reading What problem are they trying to solve? What is their approach? How is it different from other approaches?

Getting most out of what you read Summarize what you have read on each topic - after you have read several papers covering some topic, note the: key problems various formulations of the problem they are addressing relationship among the various approaches alternative approaches Read PhD theses - even though they are long they can be very helpful in quickly learning about what has been done is some field. Especially focus on: Background sections Method sections Your advisor's thesis - this will give you an idea for what he/she expects from you.

Summary in matrix or table form

References CPIT 695 Notes of Dr. Ahmad Barnawi Materials to refer: How to read a CS research paper? – Philip Paper Review Form ading.pdf ading.pdf “How to have Your Abstract rejected” Mary-Claire van Leunen and Richard Lipton “Maxims for Malfeasant Speakers” Norman Ramsey 26