Presentation on theme: "Publication process Andy King Abstract: In the UK, under-graduate students have little exposure of the."— Presentation transcript:
Publication process Andy King Abstract: In the UK, under-graduate students have little exposure of the scientific literature and are largely oblivious to the process of publishing. Therefore, as part of the induction of our first year PhD students, we run a short course explaining the publishing landscape. Acknowledgments: these slides have been adapted from the slides of Howard Bowman and CACM 46(10)
What is a theory? zWhat is the hypothesis that you are testing? yIs algorithm X faster than algorithm Y? yDoes this engineering tactic lead to better code? zHow do state your theory? yFor example, does it require formal language? zHow do you test your theory? yAre 3 case studies enough? yDo you require proof and where do you stop? yHow do you know that your benchmarking is adequate and unbiased? yAre you comparisons fair and against state-of-the-art?
Why bother to publish? zEnables physically distributed researchers to communicate zProvides a firm platform of existing knowledge zEnables ideas to tracked and ownership claimed zEstablishes a virtual research network zPeer-reviewing provides a feedback mechanism (even if the work is not published, for instance, new anti-unification that was proposed in 2002) zIncremental approach to thesis construction
What is the primary/ secondary literature? PrimarySecondary ResearchTeaching Post-graduate levelUnder-graduate level Peer reviewed with quality guarantee Lightly refereed with variable quality Focused on researchSynthesis of research Journals, research monographs, conference proceedings (CS) Conference proceedings (Astronomy)
What is the grey literature? GL’99 defined grey literature as: "That which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers" zGrey literature is non-conventional and ephemeral. zIt includes technical reports, workshop pre- proceedings, web pages, newsgroup articles, memoranda, manuals. zDue its nature, librarians have had difficulty acquiring and making accessible grey literature.
Two classic examples of grey literature zVaughan Pratt, “Two Easy Theories Whose Combination is Hard”, MIT, 1997 – a counter- example on theory composition for Greg Nelson zNow from
Final-year report versus a thesis/academic paper zReport describes the unfolding software process zUsually not very reflective zSize matters – an exercise in documentation zPaper focuses on a novel algorithmic, a correctness argument, a new result – science zUsually reflective zSize matters – an exercise in clarity and conciseness zPhD by Boyer, “Locking: a restriction of resolution”, University of Texas, 1971 is 40 pages long.
How do you construct a MSc thesis? zTypically structured as: yIntroduction and problem statement yReview of related work yDescription of the technique or model yOutline of a prototype implementation yExperimental work and conclusions yDirections for future work zMSc theses are often reworked a workshop paper z30 pages of insight, carefully structured with thoughtful reflection is better than 50 pages of waffle
How do you construct a PhD thesis? z3 papers in good conferences or 2 journal papers significantly simplifies the job of the examiners zNot clear that 6 weak papers are sufficient zTwo models for constructing a thesis: yBig-bang model -- compose PhD around mini-thesis material; pass viva; then carve up into papers yIncremental model -- publish results as they arise; compose thesis around papers; pass viva. With creativity the thesis can be presented as a “unified whole”. ySee the thesis by Ulf Nilsson on “Abstract Interpretation and Abstract Machines”
Plagiarism (material from the Part II handbook) zPlagiarism is the act of claiming the ideas or discoveries of another as one’s own – academic thief zCopying examples, code, sentences or even striking expressions without acknowledgement is plagiarism zMerging bulleted points into sentences is plagiarism zSevere penalties will be imposed if plagiarism is found (and it is easy to find) zIt is galling to read a paper that uses one of your sentences, or even some notation, without proper reference
How do you read a research paper? zRead the introduction and related work zCarefully study any worked examples, checking the working by hand if possible zRead a paper looking for weaknesses and technical holes zAsk sceptical technical questions: yDoes the method scale-up to real problems? yDoes is the technique engineered around the data-set? yWhat is the main weakness of the method and how can it be patched?
Tracking the literature zFor biblio and co-author searches use zFor topic searches use zFor citation and sources use zFor awareness scan ACM Computing Surveys zFor pre-history use supervisor and
Targeting a conference zStudy the topics list: “Contributions are sought in all areas of blah including but not restricted to …” zMany theory conferences solicit application papers: “Specific attention will be given to new applications of thingamajig” zSearch the literature for related papers published in the conference series (work by case law) zConsider the quality of the symposia: “A total of 128 submissions were received of which 28 were selected for publication” zDeadlines are surprising flexible (ESOP)
What the sub-referee sends to the PC member REVIEWER-NAME: Andy King PAPER-NUMBER: 14 B: I can accept this paper, but I will not champion it (accept, but could reject). CLASSIFICATION: Y: I am knowledgeable in the area, though not an expert; REVIEWER-EXPERTISE: 1: Research PAPER-TYPE: 4 - Summary of the paper: [32 lines] 5 - Comments for the committee only (Not shown to the author(s)): [10 lines] 6 - Comments for the author(s) - will be sent to the submitters: [44 lines] 7 - Points in favour or against 8 - Co-reviewer(s) for this paper: None
What the author receives from CyberChair From: ESOP CyberChair Subject: ESOP 2003 Notification Dear Andy King, We are pleased to inform you that your paper, titled "Goal-Independent Suspension Analysis of Logic Programs with Dynnamic Scheduling" has been accepted for presentation at the conference ESOP Actually, the committee selected only 25 papers out of 99 submissions. Please carefully take into account the enclosed comments by the reviewers when preparing the camera-ready version. It is incumbent upon you to do so. Your camera-ready paper, NOT exceeding 15 pages, is due on January 17, 2003, in the format specified by Springer-Vela at the URL Sincerely, Pier Paolo Delano (ESOP 2003 PC Chair)
What the author receives from CyberChair (part II) First reviewer's review: >>> Summary of the paper <<< [59 lines] >>> Comments <<< [47 lines] >>> Points in favour or against <<< [69 lines] Second reviewer's review: Third reviewer's review: [Note that the scores are not seen by the author with CyberChair. Better conferences, such as POPL and ICLP, tend to use 4 or more referees]
What happens in a PC meeting? zThe chair presents a ranking all the papers AAAB, …, DDDD (updated during meeting) zThe list is top and tailed zThe discussion focuses on the middle-ground and the number of conference streams. zSometimes borderline papers are refereed again over night or over lunch, sometimes remotely. zThe referees reports are tweaked to remove bias and offensive language zPC member papers are considered at the end
Is your work ready for publication? zCan you write a convincing introduction? zCan you fill 8 LNCS pages with a suitably novel and original core? The rest goes on the mandatory abstract, intro, preliminaries, related work, conclusions, references. zEven if you decide after writing, that you work is too premature, then you have still made an advance. Deadlines drive research.
Tips on how to improve a conference paper zCraft the introduction around the themes in the call for paper – be creative if necessary zItemize your 3 contributions in the introduction for the lazy referee using say, “this paper is not an exercise in aesthetics but has a number of important practical implications:” zWrite the paper early and sent it to an expert zAdd a proof or implementation appendix that will be removed in the final LNCS version zUse a CGI script to demonstrate your system zFollow the style of heroes who define the genre
Finding a co-author zPoster sessions zSocial events at conferences zSeminars at other institutions zRemember that a network of good PhD buddies age into a network of senior academics
Routes to a journal paper zExtend a workshop or conference paper with new results, for example, implementation details, proof of correctness (regular paper) zSubmit to a conference with a journal route, for instance, ESOP in SCP (special issue) zWrite a pearl, letter or technical note (fast track) zMake accessible a version of your thesis work, ideal for TOPLAS (generalist paper with meat in an electronic appendix)
Referee/author interaction (stage 1, 22/11/02) To: Bart Demoen Subject: Submission to TPLP Dear Bart, Please find enclosed as an attachment a copy of our paper “Computing Convex Hulls with a Linear Solver” for consideration as a pearl in TPLP. The paper is a revised and extended version of a paper that appeared in Logic-based Program Synthesis and Transformation (LOPSTR) in Best regards, Florence Benoy, Andy King and Fred Mesnard [Paper is 6 pages long but it takes a month to journalise a section of the original 20-page LOPSTR paper since portable code has to be written, exposition added and add a new correctness result proven in computational geometry. Bart is the area editor.]
Referee/author interaction (stage 2, 12/3/03) From: Bart Demoen Subject: TPLP/LN/5 Andy, Your paper TITLE: Computing Convex Hulls with a Linear Solver AUTHORS: Florence Benoy, Andy King and Fred Mesnard has been accepted conditionally for publication in the Pearls Section of TPLP. I will get back to you with one more referee report and shortly afterwards with the conditions for acceptance - they are related to some rewriting and polishing of the text. Regards, Bart
Referee/author interaction (stage 2 cont’d, 30/3/03) From: Bart Demoen Subject: TPLP/LN/5 Dear Andy and other authors of Computing Convex Hulls with a Linear Solver. Here are additional comments of mine... my apologies that it took so long. Personally I tend to agree with referee 3 that there is no LP pearl in your paper. This is not related to what referee 4 refers to: # Whether it could be a logic programming pearl depends # on your view of the relationship between logic and constraint # programming, but I'm inclined to say yes. The actual CLP code must be improved and explained better. One referee complains very justifiably about the variable naming convention. That is one point that can be improved. Another point is… [262 lines on instructions on 4 referees reports of 98, 65, 92 and 59 lines]
Referee/author interaction (stage 3, 30/7/03) To: Bart Demoen Subject: TPLP/LN/5 Bart, Thanks for your meta-report on the referees' reports. We have revised the paper to address these comments -- see the enclosed attachment. In fact we provide responses to each of your comments so that you can check things off. Sorry that our response has been so slow. The new complexity counter-example in (which is now very simple) represents several weeks of thinking. Best regards, Andy, Florence and Fred > I would appreciate if you do not take the minimalist approach: some > comments of the referees should be addressed even if I do not mention them. We have expanded the proof of correctness and have now added intuition to make it more readable. [letter of response is 367 lines and paper is now 11 pages]
Referee/author interaction (stage 5, 26/9/03) From: Bart Demoen Subject: TPLP/LN/5 Hi Andy, Here are the comments of my harshest referee :-) (it is also the only referee at this point, because we are in the end phase) I have added my own comments prefixed by > I have to ask you for another iteration. If it is of any consolation: of all papers submitted/accepted by TPLP, pearls take most time and most iterations (…); I think this iteration is not very demanding on you and you are very close. If you need more explanation on the comments, please let me know. Cheers, Bart [Bart’s new meta-report on the harshest referee’s new report is 124 lines]
Referee/author interaction (stage 6, 14/10/03) To: Bart Demoen Subject: TPLP/LN/5 Dear Bart, Please find enclosed a revised copy of our paper as an attachment. As before, a response to each of the issues you raised is given. Best regards, Florence, Andy and Fred. > > ignore the part where the referee complains about "this is not a pearl" > > the remark that a naive implementation (without copy_term and > > prepare_dump) would at first be easier to understand, is correct and > > would make the presentation better from the didactic point of view; > > for a pearl, that would be indeed very nice - can you do it ? We have followed this suggestion, restructuring the development of project in 3 stages. [New revision of paper is 12 pages and the new letter of response is 151 lines]
Referee/author interaction (stage 7, 17/10/03) From: Bart Demoen Subject: TPLPLN5 Dear Andy, Florence, Fred, I am pleased to tell you that your nightmare is coming to an end: your paper is really in the final stage of acceptance, meaning that I will now bug you only with syntactical, grammatical, typographical and similar matters. Allow me to comment on your most recent changes: # We have followed this suggestion, restructuring the development of # project in 3 stages. I would suggest that you change the sentence "This leads to the following revision" on page 6 into "This leads to the following (SICStus Prolog specific) revision" or something in the same spirit. [142 line report on rewording, grammar and “political” comment]
Referee/author interaction (stage 8, 17/10/03) From: Bart Demoen Subject: TPLPLN5 Dear authors of the TPLPLN5 submission, I am very happy that your submission Computing Convex Hulls with a Linear Solver as a TPLP Programming Pearl is in the final stage of acceptance: I herewith recommend it to Maurice Bruynooghe - the editor-in-chief - for publication in that particular section of TPLP. Maurice will send you additional requirements you should comply with for the final version: they relate to form and what exactly needs to be send to him as final copy. From now on, you deal with him. Thanks for submitting to the programming pearls section of TPLP: I can only wish we receive more submissions like yours. Regards, Bart Demoen
Things that you shouldn't do (but I’ve done) zSend a paper to the wrong journal, for instance, JoPaDC submission rejected after 2 years zNot respond to the referee’s comments, for example, JLP paper with inadequate discussion of related work zForget to update related work in the final copy, for instance, embarrassing erratum zDon’t confuse your versions, for instance, JLP technical note
Are journal papers longer than conference papers? zMany good journals take technical notes, short papers or “letters”: zSome of the best algorithms are incredibly simple (but very clever):
Problematic papers zExperimental but useful papers that quantifiably compare algorithms U, V, W, X, Y and Z zCounter-example papers that reveal errors can inflame the referee (run the letter past the author so that you “do not misrepresent their work”) zLong papers that build on existing work can be tedious to referee (adopt technical note style) zAlgorithm papers that rely on tricky theory can be hard to penetrate for referee (provide worked example and implementation section)
Refereeing papers zIt is inevitably and it inevitably takes 2 day per conference paper, longer for a long article. zPC may have to initiate new refereeing if your report is late. Can jeopardise the PC meeting zIndicate to the editor/chair any doubts in your understanding (usually it concurs with others) zThe rejection of a paper is never pleasant especially to a new researcher. Thus be tactful and supply corrections and do’able suggestions. zOne flaw is enough to kill a paper; never use bluster.
Why bother to quantify quality (biblio-metrics)? zPublication in prestigious forums influences peer recognition, RAE, promotion decisions, etc. zQuality research provides a foundation for further scientific achievements zTherefore citation analysis is considered to be an objective way of assessing quality zTo ensure a reasonable citation history, it is normal to allow a 2-year lag between the publication and the citation analysis
Citation analysis techniques zCPA – average number of citations per article. This measures influence whilst factoring out the size effect, for example, TCS. zUCR – the un-cited ratio, that is, the percentage of journal articles that were not cited at all. z20+ – percentage of articles with 20+ citations. zC2C – cited-to-citing ratio, that is, the total number of citations against the total number of references. Forums that emphasize basic research are likely to be cited more often than they cite others. “Source” versus “store”.
Digression – how to write a paper with 20+ citations zWrite an early paper on an emerging topic that grows into a sub-field of 20+ papers zWrite a survey paper that helps 20+ researchers in their work zWrite a simple paper on an algorithm or technique and then distribute the code so that 20+ researchers use it zWrite a problem paper that poses lots of issues but does not provide many answers that keeps 20+ researchers entertained for a few years
Sample of journals with abbreviations JournalAbbrevJournalAbbrev MIS QuarterlyMISQInformation and ManagementI&M Information Systems ResearchISRIEEE ComputerIEEECom Communications of the ACMCACMInformation Systems JournalISJ J. of the ACM (JACM)JACMInformation SystemsIS IEEE T. on Software EngineeringIEEESEInformation Systems ManagementISM Artificial IntelligenceAIDecision Support SystemsDSS Human-Computer InteractionHCIKnowledge Based SystemsKBS IBM Systems JournalIBMJ. of Strategic ISJISC AI MagazineAIMagJ. of Information TechnologyJoIT ACM T. on Database SystemsACMDBExpert Systems with ApplicationsESA Int’l J. of Human-Computer StudiesHCSComputer JournalCJ ACM Computing SurveysACMCSJ. of Computer ISJCSI J. of Computer and System SciencesJCSSJ. of Systems and SoftwareJSS European J. of ISEJIS
Citation-based indices of journal quality AbbrevCPAUCR20+C2CAbbrevCPAUCR20+C2C MISQ I&M ISR IEEECom CACM ISJ JACM IS IEEESE ISM AI DSS HCI KBS IBM JISC AIMag JoIT ACMDB ESA HCS CJ ACMCS JCSI JCSS JSS EJIS
Conclusions from quality analysis zCPA, UCR and 20+ almost concur and their ranking reflects our expectations for quality zOn average, journals with a technical or a specialty focus attain higher rankings zJournals that not receive much recognition by general audiences should not be shunned zCaveat – journals are not necessarily more prestigious than conferences in CS.
Ranking of journals, conferences, workshops OSDI, USENIX Symposium on Internet Technologies and Systems, PLDI, SIGCOMM, MOBICOM, ASPLOS, USENIX Annual Technical Conference, TOCS, SIGGRAPH, JAIR, SOSP, MICRO, POPL, PPOPP, Machine Learning, Computer Networks, Computational Linguistics, JSSPP, VVS, FPCA, LISP and Functional Programming, ICML, Data Mining and Knowledge Discovery, SID, ICSE, ACM Transactions on Networking, OOPSLA, Workshop on Workstation Operating Systems, Journal of Computer Security, TOSEM, Workshop on Parallel and Distributed Debugging, Workshop on Hot Topics in Operating Systems, Journal of Cryptology, CSFW, ECOOP, Evolutionary Computation, TOPLAS, SIGSOFT FSE, CAV, PODS, Artificial Intelligence, NOSSDAV, ACM Conference on Computer and Communications Security, IJCAI, VLDB Journal, TODS, USENIX Winter, HPCA, LICS, JLP, Computer Networks, ICCV, IEEE Real-Time Systems Symposium, AES Candidate Conference, KR, TISSEC, ACM Conference on Electronic Commerce, TOIS, PEPM, SIGMOD Conference, Formal Methods in System Design, Mobile Agents, REX Workshop, NMR, LOPLAS, STOC, Distributed Computing, KDD, Symposium on Testing, Analysis, and Verification, Software Development Environments, SIAM J Comput, CRYPTO, Multimedia Systems, ICFP, Lisp and Symbolic Computation, ECP, CHI, ISLP, ACM Symposium on User Interface Software and Technology, ESOP, ECCV, ACM Transactions on Graphics, CSCW, AOSE, ICCL, Journal of Functional Programming, RTSS, ECSCW, TOCHI, ISCA, SIGMETRICS/Performance, IWMM, JICSLP, Automatic Verification Methods for Finite State Systems, IEEE Transactions on Pattern Analysis and Machine Intelligence, AIPS, IEEE Transactions on Visualization and Computer Graphics, VLDB, Symposium on Computational Geometry, FOCS, ATAL, SODA, PPCP, AAAI, COLT, USENIX Summer, Information and Computation, Java Grande, ISMM, ICLP, SLP, Structure in Complexity Theory Conference, IEEE Transactions on Multimedia, Rules in Database Systems, ACL, CONCUR, SPAA, J Algorithms, DOOD, SIGSOFT FSE, ICDT, Advances in Petri Nets, ICNP, SSD, INFOCOM, IEEE Symposium on Security and Privacy, Cognitive Science, TSE, Storage and Retrieval for Image and Video Databases (SPIE), NACLP, SIGMETRICS, JACM, PODC, International Conference on Supercomputing, Fast Software Encryption, IEEE Visualization, SAS, …
Ranking of journals, conferences, workshops TACS, International Journal of Computer Vision, JCSS, Algorithmica, ToCL, Information Hiding, Journal of Automated Reasoning, ECCV, PCRCW, Journal of Logic and Computation, KDD Workshop, ML, ISSTA, EUROCRYPT, PDIS, Hypertext, IWDOM, PARLE, Hybrid Systems, American Journal of Computational Linguistics, SPIN, ICDE, FMCAD, SC, EDBT, Computational Complexity, International Journal of Computational Geometry and Applications, ESORICS, IJCAI, TACAS, Ubicomp, MPC, AWOC, TLCA, Emergent Neural Computational Architectures Based on Neuroscience, CADE, PROCOMET, ACM Multimedia, IEEE Journal on Selected Areas in Communications, Science of Computer Programming, LCPC, CT-RSA, ICLP, Financial Cryptography, DBPL, AAAI/IAAI, Artificial Life, Higher-Order and Symbolic Computation, TKDE, ACM Computing Surveys, Computational Geometry, Autonomous Agents and Multi-Agent Systems, EWSL, Learning for Natural Language Processing, TAPOS, TAPSOFT, International Journal of Computational Geometry and Applications, IEEE Transactions on Parallel and Distributed Systems, Heterogeneous Computing Workshop, Distributed and Parallel Databases, DAC, ICTL, IEEE Computer, IEEE Real Time Technology and Applications Symposium, ACM Workshop on Role-Based Access Control, WCRE, Applications and Theory of Petri Nets, ACM SIGOPS European Workshop, ICDCS, Mathematical Structures in Computer Science, Workshop on the Management of Replicated Data, ECCV, PPSN, Middleware, OODBS, ECCC, UML, Real-Time Systems, FME, Evolutionary Computing, AISB Workshop, IEEE Conference on Computational Complexity, IOPADS, IJCAI, ISWC, SIGIR, Symposium on LISP and Functional Programming, PASTE, HPDC, Application and Theory of Petri Nets, ICCAD, Category Theory and Computer Science, Recent Advances in Intrusion Detection, JIIS, TODAES, Neural Computation, CCL, DPDS, ACM Multimedia, MAAMAW, Computer Graphics Forum, HUG, Hybrid Neural Systems, SRDS, TPCD, ILP, ARTDB, NIPS, Formal Aspects of Computing, ECHT, ICMCS, Wireless Networks, Advances in Data Base Theory, WDAG, ALP, TARK, PATAT, ISTCS, Concurrency - Practice and Experience, CP, Computer Vision, Graphics, and Image Processing, FTCS, RTA, COORDINATION, CHDL, Theory of Computing Systems, CTRS, COMPASS/ADT, TOMACS, IEEE Micro, IEEE PACT, ASIACRYPT, MONET, Computer Networks, HUC, Expert Database Conference, Agents, CPM, Symposium on Compiler Construction, International Conference on Evolutionary Computation, TAGT,Workshop on Parallel and Distributed Simulation,FTRTFT, TPHOLs, Intelligent User Interfaces,Journal of Functional and Logic Programming, Cluster Computing,ESA,PLILP,COLING-ACL, META,IEEE MultiMedia
Citation-based measures of academic quality AcademicCitesAcademicCitesAcademicCites D. Johnson12119J. Smith6052R. Agrawal5143 J. Ullman11041D. Knuth5793D. Goldberg5050 A. Gupta8407S. Shenker5716L. Zhang4952 R. Milner7900E. Clarke5674R. Karp4951 R. Rivest6930S. Floyd5598G. Hinton4945 M. Garey6732J. Hennessy5512J. Quinlan4812 R. Tarjan6525A. Aho5411R. Jain4768 J. Dongarra6522R. Johnson5259C. Leiserson4764 V. Jacobson6494A. Pnueli5212C. Hoare4758 L. Lamport6410C. Papadimitriou5210J. Pearl4737 See for the top cited authors of the cited authors in computer science
Finishing well zTactful: referees are often selected from the related work so say, “Recently,  have shown how Def formulae can be cleverly encoded as ACI-terms. The initial experimental results reported in  are promising though widening is required for the larger programs” zBroad: “In fact, surprisingly, the same Galois connection is required for the demand-driven analysis of imperative programs” zInsightful: “ overlooks the boundedness requirement of sub-goals, no doubt because the technique is goal-independent”
Making the most of what you have got zKeep your web page up-to-date. If you add links to ps, then you are more likely to be cited. zCheck what you have signed: y“The Author may publish his/her contribution on his/her personal Web page provided that he/she creates a link to the above mentioned volume of LNCS at the Springer-Verlag server or to the LNCS series homepage at yIf copyright is a problem, then create an “almost identical to the journal” technical report version. yMany insist on the Computing Research Repository (CoRR)