James Harland 1 COSC2148/2149/2150 Research Methods Reading and Assessing Research Literature James Harland

James Harland 2 Generating PDF from Word Despite the lab exercise, I believe the easiest way is: Download and install Open Office Open.doc file with Open Office Save as PDF Close.doc file

James Harland 3 Overview Reading is a fundamental research activity Assessing what is read can be difficult Assessment is required of active researchers 1 paper published = 3 papers reviewed, on average Need to know where and when published May need to investigate publication details Need to know publishing culture Be rational, skeptical and humble

James Harland 4 Publication Types Book Journal Conference Workshop Technical Report Web page Thesis Manuscript...

James Harland 5 Book Published because someone believes many copies will be sold. Author/s detailed development of ideas Gathering of scattered research results into one volume Collection of papers by various authors from a meeting `Handbook on X' series by invited experts Commemorative volume dedicated to an individual PhD thesis published as a book Not necessarily just for the academic market

James Harland 6 Book Often become standard references. Publication can be slow. Reviews are often written by other experts. Good sources of basic concepts and background material. Rarely have up-to-the-minute results. Refereeing process varies from rigourous to virtually none.

James Harland 7 Journal Published by major companies exclusively for academia. Viewed generally as the most important kind of publication Often seen as an archive of research Published at regular intervals (monthly, quarterly) Available in most (good) libraries No bounds on number of papers No real-time refereeing constraints Increasingly publishing special issues on topic X Main customers are university libraries

James Harland 8 Journal Rigourous refereeing by (at least) two experts Only accepted for publication when editor is satisfied Can take many iterations to succeed Generally no bounds on pages or time to (re-)submit Anything that is appropriate will eventually be published Can sometimes have good but unimportant papers Any good paper should eventually appear in a journal

James Harland 9 Conference Proceedings published for distribution to attendees, and sometimes more widely. Probably the most common form of publication Not always refereed (but generally are in CS) Proceedings sometimes published well after conference Tend to have up to date material Quality more variable than journals Worldwide, regional, national, local,... Main papers, poster abstracts, short papers, system demonstrations, invited papers, panel sessions,...

James Harland 10 Conference Generally 3-5 referees, not all from the programme committee Often use numeric scores to simplify the process Real-time refereeing constraints exist ``One-shot'' chance at publication Programme committee selects best-scoring papers No chance to re-evaluate papers later Reports generally not as detailed as for journals Common for acceptance rate to be under 50%

James Harland 11 Workshop More informal than conferences Deadlines closer to actual meeting More specialised audiences Refereeing generally more lightweight Often self-selecting Usually has ``cutting edge'' material Quality again more variable Proceedings often not generally available

James Harland 12 Technical Report Earliest of all publications Not refereed in any sense Used to disseminate information quickly Often same paper submitted to a conference or journal Trend away from these per se to links on personal Web pages A typical evolution is TR to Workshop to Conference to Journal. Any of these steps may be missing.

James Harland 13 Manuscript Sometimes you may get a personal copy of some material not widely available. This is generally from a trusted source only. Examples including incomplete material from colleagues or working notes from a supervisor. Generally does not include justification, background or discussion.

James Harland 14 Theses Paper (or book) submitted by a student towards a degree PhD theses can be hundreds of pages MSc theses around one hundred pages Honours/minor thesis around fifty pages Read in detail by at least two examiners Designed to show that a student is ``qualified'' Often read and criticised in greater depth than other publications Possibly more arcane material than journal or conference publications Don't have to be important, just good

James Harland 15 Reading Papers Read the abstract first If still interested, read the introduction and conclusion If still interested, read the rest of the paper If still interested, look for follow-up papers It is not necessary to understand everything in detail to get some benefit

James Harland 16 Evaluation What? When? Where? How did I hear about it? What's new? What is/is not obvious? What is/is not justified? What confidence do I have in my judgement?

James Harland 17 Evaluation Approaches you could use: Discuss the paper with colleagues Ask your supervisor about it Work through techniques from paper Reproduce results from the paper Ask the author/s about the paper Look up references mentioned in the paper