1. Alternative Peer Review: Quality Management for 21 st Century Scholarship Gerry McKiernan Science and Technology Librarian and Bibliographer Iowa State University Library Ames IA USA gerrymck@iastate.edu http://www.public.iastate.edu/~gerrymck/APR.ppt

2. Workshop on Peer Review in the Age of Open Archives International School for Advanced Studies Interdisciplinary Laboratory Trieste, Italy May 23-24, 2003

3. THANK YOU! Workshop Advisory Board [(Marco Fabbrichesi (INFN/SISSA Italy), Stevan Harnad (University of Southampton, UK), Stefano Mizzaro (University of Udine, Italy) and Corrado Pettenati (CERN Library, Geneva, Switzerland)] Iowa State University, Faculty Senate, Committee on Recognition and Development European Commission Iowa State University Library Heike Kross, Ph.D.

4. DISCLAIMER (1) The screen prints selected for this presentation are for educational purposes and their inclusion does not constitute an endorsement of an associated product, service, place, or institution.

5. DISCLAIMER (2) The views and opinions expressed in this presentation are those of the presenter and do not constitute an endorsement by Iowa State University or its Library.

6. NOTICE No editors, authors, or referees were harmed in the preparation of this presentation.

7. http://www.sciencecartoonsplus.com/

8. Giuseppe De Nittis (1846-1884) The Macchiaioli / Italian Impressionists

9. Campo di Biche (1875)

10. PEER REVIEW: DEFINITION “Peer review is the assessment by an expert of material submitted for publication.” Carin M. Olson, “Peer Review of the Biomedical Literature,” American Journal of Emergency Medicine 8 no.4 (July 1990): 356-358.

11. PEER REVIEW: PURPOSES Carin M. Olson, “Peer Review of the Biomedical Literature,” American Journal of Emergency Medicine 8 no.4 (July 1990): 356-358. Peer review helps to ensure that published research is: ImportantOriginal TimelyTechnically-reliable Internally-consistentWell-presented Benefited from guidance by experts

12. PEER REVIEW: STRENGTHS Anne C. Weller, Editorial Peer Review: Its Strengths and Weaknesses. (Medford, NJ: Information Today, 2001). The underlying strength of peer review is“…the concerted effort by large numbers of researchers and scholars who work to assure that valid and valuable works are published and conversely to assure that invalid or non- valuable works are not published ….”

13. “Houston, We Have a Problem!”

14. PEER REVIEW: PROBLEMS Subjectivity Bias Abuse Detecting defects Fraud and Misconduct Delay Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15 no. 4 (October 2002): 247-258. Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdfhttp://www.jisc.ac.uk/uploaded_documents/rowland.pdf

15. SUBJECTIVITY Summary rejections by editor without sending the paper to referees Choice of referees by the editor (choosing for example, a known harsh referee for a paper the editor wishes to see rejected)

16. BIAS Discrimination against authors because of their nationality, native language, gender or host institution Situations where author and referee are competitors in some sense, or belong to warring schools of thought

17. ABUSE Too many articles out of one piece of research, or duplicate publication Intellectual theft: omission or downgrading of junior staff by senior authors Plagiarism (stealing others yet unpublished work that has been sent for review) Delaying publication of potentially competing research

18. DETECTING DEFECTS Identification of factual errors within submission

19. FRAUD and MISCONDUCT Fabrication of results Falsification of data False claim of authorship for results

20. DELAY “There is much muttering about publication delay, a real enough problem, especially in paper publication, but peer review itself is often responsible for as much of the delay as the paper publication and distribution process itself.” Stevan Harnad Stevan Harnad, “Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals, in Scholarly Publication: The Electronic Frontier, edited by Robin P. Peek and Gregory B. Newby (Cambridge MA: MIT Press, 1996). http://www.ecs.soton.ac.uk/~harnad/Papers/Harnad/harnad96.peer.review.html

21. http://www-marketing.wharton.upenn.edu/ideas/pdf/peerpaper.pdf

22. “ Peer review is slow, expensive, profligate of academic time, highly subjective, prone to bias, easily abused, poor at detecting gross defects, and almost useless in detecting fraud.” Richard Smith, “Opening Up BMJ Peer Review,” BMJ 318 (7175) (January 2 1999): 4-5 Richard Smith Editor, BMJ

23. Stephen Lock, A Difficult Balance: Editorial Peer Review in Medicine (Philadelphia, PA: ISI Press, 1986).

25. EXAMPLE Jan Hendrik Schön

26. http://archive.salon.com/tech/feature/2002/09/16/physics/ -

27. Science Nature http://physicsweb.org/article/news/6/9/15/

28. http://agenda.cern.ch/fullAgenda.php?ida=a01193

29. RECOMMENDATIONS Workshop on the Open Archives Initiative (OAI) and Peer Review Journals in Europe, CERN, Geneva Switzerland, March 22-24, 2001 “The participants were unanimous in their belief that the certification of scholarly work remains a fundamental part of a system for scholarly communication.” “It was [also] generally believed that the electronic environment allows for novel approaches to accord a stamp of quality to scholarly works.” Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful Owners,” European Science Foundation Policy Briefing 21 (April 2003): 1-8. http://www.arl.org/sparc/SPB21_OAI.pdf

30. “Let us be imaginative in exploring the remarkable possibilities of this brave new medium.” Stevan Harnad, “Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals, in Scholarly Publication: The Electronic Frontier, edited by Robin P. Peek and Gregory B. Newby (Cambridge MA: MIT Press, 1996). http://www.ecs.soton.ac.uk/~harnad/Papers/Harnad/harnad96.peer.review.html

35. “Let us be more imaginative in exploring the remarkable possibilities of this brave new medium.” With Apologies to Stevan Harnad

37. http://lockss.stanford.edu/ TM

38. LOCKSS For centuries libraries and publishers have had stable roles: publishers produced information; libraries kept it safe for reader access. There is no fundamental reason for the online environment to force institutions to abandon these roles. The LOCKSS model capitalizes on the traditional roles of libraries and publishers. LOCKSS creates low-cost, persistent digital "caches" of authoritative versions of http-delivered content.

39. LOCKSS The LOCKSS software enables institutions to locally collect, store, preserve, and archive authorized content thus safeguarding their community's access to that content. The LOCKSS model enforces the publisher's access control systems and, for many publishers, does no harm to their business models.

40. LAMPSS Lots of Alternative Models Provide Sensible Solutions

42. DISCLAIMER The alternative peer review models profiled are for informational and educational purposes only and do not necessarily constitute an endorsement. ON

43. ALTERNATIVE PEER REVIEW Neo-Classical Certification- based Open Peer Review Commentary- based Collaborately- filtered Institution-based Citation-based Index-based Metadata-based Computer-assisted NO Peer Review Moderator-based Tier-based

44. NEO-CLASSICAL PEER REVIEW

45. Neo-Classical Peer Review

48. CERTIFICATION-BASED

49. Certification-Based “The process of pre-publication peer review could be improved and become a more reliable indicator of manuscript quality if reviewers were trained in, and subsequently applied systematically, critical skills and use of a hierarchy of evidence to classify submitted articles being reviewed.” Stephen Pritchard, “Peer Review - a Proposal for Change,” Paper presented at Thinking Globally - Acting Locally: Medical Libraries at the Turn of an Era, 8 th European Conference of Health and Medical Libraries, September 16-21, 2002, Deutsche Zentralbibliothek für Medizin, Köln, Germany. http://www.zbmed.de/eahil2002/abstracts/pritchard.pdf

50. OPEN PEER REVIEW

51. Open Peer Review IDENTIFICATION OF REVIEWERS / SIGNED REVIEWS BMJ bmj.com BioMed Central biomedcentral.com eMJA (Medical Journal of Australia) www.mja.com.au/public/information/project.html

52. COMMENTARY-BASED

53. Commentary-based Readers can comment before and/or after classic peer review, or instead of classic peer review Electronic Transactions on Artificial Intelligence (www.etaij.org)www.etaij.org OPEN REVIEW / REFEERING Journal of Interactive Media in Education (www-jime.open.ac.uk)www-jime.open.ac.uk PRE- AND POST- COMMENTARY Psycoloquy ( psycprints.ecs.soton.ac.uk) psycprints.ecs.soton.ac.uk POST PEER REVIEW COMMENTARY)

54. Electronic Transactions on Artificial Intelligence

55. Journal of Interactive Media in Education

56. http://psycprints.ecs.soton.ac.uk/

59. COLLABORATIVELY-FILTERED

60. Collaboratively-Filtered “DEFINITION: “Guiding people's choices of what to read, what to look at, what to watch, what to listen to (the filtering part); and doing that guidance based on information gathered from some other people (the collaborative part)." Paul Resnick http://www.cni.org/Hforums/cni-announce/1996/0031.html

61. ResearchIndex / CiteSeer http://www.researchindex.com

62. INSTITUTION-BASED

63. Institution-Based INSTITUTIONAL INITIATIVES DSpace™ (MIT) www.dspace.org eScholarship (University of California) escholarship.cdlib.org Glasgow ePrint Service (University of Glasgow) eprints.lib.gla.ac.uk

64. CITATION-BASED

65. Citation-Based Citations to Open Access / OAI-compliant documents are indicators of document importance http://citebase.eprints.org/

70. INDEX-BASED

71. Index-Based INDEXING OF EPRINTS BY COMMERICAL ABSTRACTING AND INDEXING SERVICE Chemical Abstracts (American Chemical Society) (CAS) indexes select appropriate e-prints from the arXiv.org eprint server as well from the Chemical Preprint Service (Elsevier)arXiv.orgChemical Preprint Service Its “selection criteria for this kind of electronic document are essentially the same as for the traditional printed documents: they must report new information of chemical or chemistry-related interest and must be original publications. Also, the electronic publication must be publicly available and have some relative permanence ….” Eric Shively / Chemical Abstracts Service

72. Index-Based “CAB Abstracts doesn’t currently include Eprints or Preprints, but we are looking at the implications and possible mechanisms for accessing and indexing Eprints and/or Preprints related to the applied life sciences.” Tracy Shaw / CAB International

73. METADATA-BASED

74. Metadata-Based internal peer review CERN http://www.cern.ch/policies/review.html William Y. Arms, “Quality Control in Scholarly Publishing On The Web. What Are the Alternatives to Peer Review?” PowerPoint presentation given at the Workshop on the Open Archives Initiatives (OAI) and Peer Review Journals in Europe, March 22-24, 2003, CERN, Geneva, Switzerland. http://www.cs.cornell.edu/wya/papers/CERN-2001.ppt

75. COMPUTER-ASSISTED

76. Computer-Assisted (1) SOFTWARE THAT ASSISTS IN THE EVALUATION OF A SUBMITTED MANUSCRIPT A Software Program to Aid in Peer Review Alvar Loria and Gladys Faba Objective: To characterize a personal computer-based software program developed as an aid to peer review of medical papers. Design: The software is a Windows-based application that records automatically a numeric score to a series of questions related to 8 sections of scientific papers (introduction, methods, results, and discussion, plus 4 other sections). The questions and sections vary according to type of paper (original reports, case reports, or reviews), and the final output is a score with a maximum of 100 for a "perfect" paper. The software was tested using a single reviewer to judge 289 papers (169 original reports, 50 case reports, and 70 reviews) from 44 Mexican medical journals. All statistical analysis of scores were done with nonparametric tests.

77. Computer-Assisted (2) Results: The paper scores ranged from 29 to 97 with slightly higher median and less dispersion of scores for reviews as compared with original reports and case reports, but these differences did not reach significance. Two observations suggest that the software operated reasonably well: a) there were some differences in the section scores by type of paper that agreed well with differences in their complexity; b) the journal scores showed an association with their number of original papers and their percentage of original papers (Kruskal-Wallis test, P=.06 and 0.07, respectively). Conclusions: The software operated reasonably well when used to compare the relative quality of 289 papers. The validity of the program is restricted in this study to the experience of 1 reviewer. An analysis of the raw scores helped in detecting some ambiguous and redundant questions that have been modified in an improved version. The program has a potential as a training tool for inexperienced reviewers or as a scorekeeper for experienced peer reviewers. Alvar Loria and Gladys Faba, “A Software Program to Aid in Peer Review,” Abstract of paper presented at the Third International Congress on Biomedical Peer Review and Global Communications, September 18-20, 1997, Prague, Czech Republic. http://www.ama-assn.org/public/peer/arev.htm

78. NO PEER REVIEW

79. NO Peer Review http://xxx.arXiv.cornell.edu

80. MODERATOR-BASED

81. Moderator-based (1) The intent of this model is to allow the widest range of scientific manuscripts to be archived, searched, and distributed electronically at the lowest possible cost. This would be accomplished through very minimal filtering and subsequent placement of eprints on a non- commercial archival server by a subject-specific Moderator appointed by a society (or consortia of societies). A society-appointed Editorial Board (with double-blind peer review approved by the non-profit Peer Review Inc. organization) would then the identify the most important materials from among these archived items, and the stamp of approval for these items would be included in a secondary Virtual Collection.

82. Moderator-based (2) There are no direct submissions to the Editorial Board; manuscripts would be directed to the Editorial Board in one of three ways: 1. nominated by the eprint Moderator upon receipt for the archival server, 2. notification sent to the Editorial Board when a threshold number of hits are generated by any one manuscript on the archive server, and 3. nominated by readers of material from the archive; this process requires a letter of support outlining the importance of the work to the Editorial Board.

83. Moderator-based (3) The Virtual Collection could be produced as a variety of products: enhanced abstracts email threads (with comments) virtual reviews of sub-disciplines SDIs (selective dissemination of information) current awareness tools This process: reduces the load on the Editorial Boards, which results in a faster review process; differentiates those items worthy of higher recognition from those worthy of archiving, making it easier for a reader to filter material, based upon a society and discipline authority (rather than commercial reasoning); provides for search/browse/sdi from the Virtual Collection for filtered info, reducing this more expensive option for only those items recognized as of the highest quality.

84. David Stern, “The eprint Moderator Model,” Newsletter on Serials Pricing Issues no. 214 (February 8, 1999). http://www.lib.unc.edu/prices/1999/PRIC214.HTML#214.5 1 2 3

85. TIER-BASED

86. Tier-based Two separate domains Standard Tier Any and all submissions would be accepted after a cursory examination of or other pro forma certification. The review process could be “minimally labor- intensive, perhaps relying primarily on an automated check of author institutional affiliation, prior publication record, research grant status, or other related background; and involve human labor primarily to adjudicate incomplete or ambiguous results of an automated pass.”

87. Tier-based Upper Tier “At some later point (which could vary from article to article, perhaps with no time limit), a much smaller set of articles would be selected for the full peer review process. The initial selection criteria for this smaller set could be any of a variety of impact measures, to be determined, and based explicitly on their prior widespread and systematic availability and citability: e.g., reader nomination or rating, citation impact, usage statistics, editorial selection,....” Paul Ginsparg, “Can Peer Review be Better Focused?,” Science & Technology Libraries 22 No. 3/4 (In press). http://arxiv.org/blurb/pg02pr.html

88. DISCLAIMER Presented for Your Consideration OFF

89. FREEDOM OF IDEAS http://www.nrm.org/exhibits/current/four-freedoms.html

90. ‘FUTURE OF IDEAS’ The explosion of innovation we have seen in the environment of the Internet was not conjured from some new, previously unimagined technological magic; instead, it came from an ideal as old as the nation. Creativity flourished there because the Internet protected an innovation commons.

91. ‘FUTURE OF IDEAS’ The Internet’s very design built a neutral platform upon which the widest range of creators could experiment. The legal architecture surrounding it protected this free space so that culture and information – the ideas of our era–could flow freely and inspire an unprecedented breadth of expression. Lawrence Lessig, The Future of Ideas: The Fate of the Commons in a Connected World. (New York: Random House, c2001).Lawrence Lessig, The Future of Ideas: The Fate of the Commons in a Connected World. (New York: Random House, c2001).

92. “IT’S NOT ABOUT PUBLICATION; IT’S ABOUT IDEAS.”

93. The Association of Learned and Professional Society Publishers (ALPSP) Survey Authors and Electronic Publishing Scholarly research communication has seen far- reaching developments in recent years. Most journals are now available online as well as in print, and numerous electronic-only journals have been launched; the Internet opens up new ways for journals to operate. Authors have also become conscious of alternative ways to communicate their findings, and much has been written about what they ought to think.

94. ALPSP felt that it would be timely to discover what they actually thought and what they actually did. This survey aimed to discover the views of academics, both as authors and as readers. Some 14,000 scholars were contacted across all disciplines and all parts of the world, and nearly 9% responded; their detailed comments make thought-provoking reading. Alma Swan and Sheridan Brown. Authors and Electronic Publishing: The ALPSP Research Study on Authors' and Readers’ Views of Electronic Research Communication. (West Sussex, UK: The Association of Learned and Professional Society Publishers, 2002 ). http://www.alpsp.org/pub5.htm

95. “When asked to predict what would be the most common form of quality control in five years time, only a bare majority answered ‘traditional peer review’.” Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15 no. 4 (October 2002): 247-258. Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdfhttp://www.jisc.ac.uk/uploaded_documents/rowland.pdf

96. FURTHERMORE … 16% said that the referees would no longer be anonymous 27% said that traditional peer review would be supplemented by post-publication commentary 45% expected to see some changes in the peer-review system within the next five years Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15 no. 4 (October 2002): 247-258. Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdfhttp://www.jisc.ac.uk/uploaded_documents/rowland.pdf

97. Importance of the Peer Review Process http://www.alpsp.org/pub5.ppt

98. What is Gray/Grey Literature ? Papers are often written to inform funding bodies about the results of research projects, to support grant applications, to inform rapidly a specific scientific community, to present preliminary results at conferences or as dissertations. Such material is disseminated quickly, often in limited numbers, before or without the formal publication process. Such documents are called non-conventional or grey literature. http://www.kb.nl/infolev/eagle/what_is_gl.htm

99. The Value of Grey Literature Grey literature is really a type of informal communication, which on a scale of formality, fits in somewhere between conversation and normal publication. A formal publication may follow later but in many cases - contrary to the common assumption - these papers may not been made publicly available at all. http://www.kb.nl/infolev/eagle/what_is_gl.htm

100. Nevertheless, grey publications may contain comprehensive, concrete and up-to-date information on research findings, and investigations have shown, that even when grey documents are published officially at a later stage, detailed information on techniques, methods, measured values and details of experiments are frequently omitted. For these details of importance for further research, the non-conventional literature is then the first and only source of information. http://www.kb.nl/infolev/eagle/what_is_gl.htm

101. Veterinary Medicine 12 Major Veterinary Medicine Journals Overall, 6.38% of cited literature was Gray/Grey Literature The figures for individual journals ranged from about 2.5 % to 10% gray/grey literature Research journals cited a higher percentage of Gray/Grey Literature than did Clinical titles William H. Weise and Nancy Pelzer, “Bibliometric Study of Grey Literature in Core Veterinary Medicine Journals,” Journal of the Medical Library Association 91 no. 4 (October 2003): In press.

102. Indexing and Abstracting Services SIGLE: System for Information on Grey Literature National Technical Information Service (NTIS) PsychINFO (Psychological Abstracts)

103. SIGLE System for Information on Grey Literature Grey literature documents covered by SIGLE are technical or research reports, preprints, committee reports, working papers, dissertations, conference papers, discussion and policy papers, government reports, market surveys, etc. http://www.kb.nl/infolev/eagle/frames.htm

104. SIGLE System for Information on Grey Literature No. of Records | Category 4,158 | Aeronautics 17,044 | Agriculture, plant & veterinary sciences 17,668 | Environmental pollution, protection & control 256,657 | Humanities, psychology & social sciences 81,269 | Biological & medical sciences 25,089 | Chemistry http://www.kb.nl/infolev/eagle/frames.htm

105. NTIS (National Technical Information Service) NTIS Database provides bibliographic data and abstracts of unclassified and publicly available information from research reports, journal articles, data files, computer programs and audio visual products, from U.S. and non- U.S. governmental, organizational, and commercial sources The NTIS Database produced by the National Technical Information Service, is the preeminent resource for accessing the latest U.S. government-sponsored research and worldwide scientific, technical, engineering, and business-related information. http://www.csa2.com/csa/factsheets/ntis.shtml

106. Subject Coverage Administration and Management Aeronautics & Aerodynamics AgricultureBehavior & Society BusinessChemistryCommunicationsComputer Science EducationEnergyEngineeringEnvironmental Sciences Health CareInternational Trade Library & Information Science Materials Sciences Mathematical Sciences Natural Resources & Earth Sciences Nuclear SciencePhysics RegulationsTechnologyTele- communications Transportation

107. PsycINFO PsycINFO provides access to international literature in psychology and related disciplines. Unrivaled in its depth of psychological coverage and respected worldwide for its high quality, the database is enriched with literature from an array of disciplines related to psychology such as psychiatry, education, business, medicine, nursing, pharmacology, law, linguistics, and social work. http://www.csa2.com/csa/factsheets/psycinfo.shtml PsycINFO includes psychological research and its applications; the database is of prime relevance to many industries and research establishments worldwide. The sources include over 1,400 professional journals, chapters, books, reports, theses and dissertations, published internationally.

108. Subject Coverage Applied Psychology Communicatio n Systems Developmental Psychology Educational Psychology Experimental Psychology PersonalityPsychological and physical disorders Professional personnel issues Physiological Psychology and Neuroscience Psychometrics And Statistics Social Psychology Treatment and Prevention

109. DatabaseCoverageSize SIGLE1976 – Present 781,410 records (November 2002) NTIS1964 - Present2,168,400 records (October 2001) PsycINFO1872- Present 1,870,180 records (September 2002) DATABASE COVERAGE AND SIZE

110. http://www.neci.nec.com/~lawrence/papers/online-nature01/

111. Conference papers are typical gray/grey literature!

112. EPrints are Gray/Grey Literature Daniela Luzi (1998) “E-Print Archives: a New Communication Pattern for Grey Literature,” Interlending & Document Supply 26 no. 3 (1998): 130-139.

113. Gray/Grey Literature “It’s good enough, it’s smart enough, and doggone it, people use it!” With apologies to Stuart Smalley

114. http://www.openarchives.org/Register/BrowseSites.pl

118. http://software.eprints.org/ June 2003

119. http://eprints.anu.edu.au/

120. http://caltechcstr.library.caltech.edu/

121. http://eprints.lub.lu.se/

122. http://ndltdpapers.dlib.vt.edu:9090/ NETWORKED DIGITAL LIBRARY OF THESES AND DISSERTATIONS

123. http://dspace.org/index.html

124. http://rocky.dlib.vt.edu/~etdunion/cgi-bin/browse.pl

125. http://www.ncstrl.org/

127. Quality Control in Scholarly Publishing on the Web "Most of the high quality materials on the Web are not peer-reviewed and much of the peer-reviewed literature is of dubious quality.” William Y. Arms, "What Are the Alternatives to Peer Review? Quality Control in Scholarly Publishing On The Web." Journal of Electronic Publishing, 8 no. 1 (August 2002). http://www.press.umich.edu/jep/08-01/arms.html

128. http://www.update-software.com/Cochrane/MR000016.pdf

129. Cochrane Methodology Review Despite its widespread use and costs, little hard evidence exists that peer review improves the quality of published biomedical research. There had never even been any consensus on its aims and that it would be more appropriate to refer to it as ‘competitive review’. Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,” BMJ 326 (February 1, 2003): 241 http://bmj.com/cgi/reprint/326/7383/241/a.pdf

130. Cochrane Methodology Review On the basis of the current evidence, ‘the practice of peer review is based on faith in its effects, rather than on facts,' state the authors, who call for large, government funded research programmes to test the effectiveness of the [classic peer review] system and investigate possible alternatives. Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,” BMJ 326 (February 1, 2003): 241 http://bmj.com/cgi/reprint/326/7383/241/a.pdf

131. Cochrane Methodology Review The use of peer-review is usually assumed to raise the quality of the end-product (i.e. the journal or scientific meeting) and to provide a mechanism for rational, fair and objective decision-making. However, these assumptions have rarely been tested. Tom O. Jefferson, Phil Alderson, Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of Biomedical Studies. (Middle Way, Oxford: Update Software Ltd, 2003). http://www.update-software.com/Cochrane/MR000016.pdf

132. Cochrane Methodology Review The available research has not clearly identified or assessed the impact of peer-review on the more important outcomes (importance, usefulness, relevance, and quality of published reports) … [G]iven the widespread use of peer-review and its importance, it is surprising that so little is known of its effects Tom O. Jefferson, Phil Alderson,Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of Biomedical Studies. (Middle Way, Oxford: Update Software Ltd, 2003). http://www.update-software.com/Cochrane/MR000016.pdf

133. http://www.biomedcentral.com/news/20030203/04/ Royal Society

134. UNCITEDNESS David P. Hamilton, "Publishing by and for? -- the numbers,” Science (New Series) 250 (4986) (December 7 1990): 1331-1332. http://www.garfield.library.upenn.edu/papers/hamilton1.html

135. David P. Hamilton,“Research Papers: Who’s Uncited Now?,” Science (New Series) 251 (4989) (January 4, 1991): 25 http://garfield.library.upenn.edu/papers/hamilton2.html

136. David P. Hamilton,“Research Papers: Who’s Uncited Now?,” Science (New Series) 251 (4989) (January 4, 1991): 25 http://garfield.library.upenn.edu/papers/hamilton2.html

137. REJECTED CITATION CLASSICS NOBEL PRIZE RESEARCH Severo Ochoa Polynucleotide phosphorylase Hans Krebs Citric acid cycle Rosalind Yalow Radioimmunoassay Harmut Michel Photosynthetic processes Juan Miguel Campanario, “Commentary: On Influential Books and Journal Articles Initially Rejected Because of Negative Referees’ Evaluations, Science Communication 16 no. 3 (March 1995): 306-325

138. PEER REVIEW: PURPOSES Peer review helps to ensure that published research is: Important ?Original ? Timely ?Technically-reliable ? Internally consistent ?Well-presented ? Benefited from guidance by experts ? Carin M. Olson, “Peer Review of the Biomedical Literature,” American Journal of Emergency Medicine 8 no.4 (July 1990): 356-368.

139. FILTERING (1) UpStream / DownStream “Researchers look at … [certain types] of electronic publications because, despite being tentative, may be relevant to their work. Researchers are expected to do their own ‘downstream-filtering’ of relevant information, which in the electronic world can be facilitated by providing meta- information.”

140. FILTERING (2) UpStream / DownStream Some have expressed the concern that having non-peer reviewed documents with peer- reviewed documents on the same server would ‘contaminate’ the latter and compromise its quality: Readers could have trouble in distinguishing the different sections ‘Making non-peer-reviewed as well as peer- reviewed material will confuse both scientists and the public ….’

141. FILTERING (3) UpStream / DownStream “‘However, this perhaps belittles the ability of scientists to recognize different levels of evidence and to be able to interpret [quality] labels that could make clear that certain materials is non-peer-reviewed content’ after all this is the age of transparency rather than paternalism ….” Gunther Essenbach, “The Impact of Preprint Servers and Electronic Publishing on Biomedical Research,” Current Opinion in Immunology 12 no. 5 (October 2000): 499-503. http://yi.com/home/EysenbachGunther/scans/Eysenbach2000e_CurrOpImmunol_preprint_servers.pdf

142. INVISIBLE HAND OF CLASSICAL PEER REVIEW “The refereed journal literature needs to be freed from both paper and its costs, but not from peer review, whose ‘invisible hand’ is what maintains its quality.” Stevan Harnad http://www.presidentmoron.com

143. INVISIBLE HAND OF CLASSICAL PEER REVIEW

144. “Human nature being what it is, it cannot be altogether relied upon to police itself. Individual exceptions there may be, but to treat them as the rule would be to underestimate the degree to which our potential unruliness is vetted by collective constraints, implemented formally.” Stevan Harnad, “The Invisible Hand of Peer Review,” Exploit Interactive no. 5 (April 2000). http://www.exploit-lib.org/issue5/peer-review/

145. INVISIBLE HAND OF CLASSICAL PEER REVIEW “The system is not perfect, but it is what has vouchsafed us our refereed journal literature to date, such as it is, and so far no one has demonstrated any viable alternative to having experts judge the work of their peers, let alone one that is at least as effective in maintaining the quality of the literature as the present imperfect one is.”

146. INVISIBLE HAND OF CLASSICAL PEER REVIEW “Remove that invisible constraint -- let the authors be answerable to no one but the general users of the Archive [arXiv.org] (or even its self-appointed "commentators") -- and watch human nature take its natural course, standards eroding as the Archive devolves toward the canonical state of unconstrained postings: the free-for-all chat-groups of Usenet …, that Global Graffiti Board for Trivial Pursuit -- until someone re-invents peer review and quality control.” Stevan Harnad, “The Invisible Hand of Peer Review,” Exploit Interactive no. 5 (April 2000). http://www.exploit-lib.org/issue5/peer-review/

147. INVISIBLE HANDS

148. Personal reputation Institutional reputation Pride Self-respect Professional respect Peer pressure ‘ Critical Peer Response’ Invisible College Self-Archiving-Process- Itself Open access Common Sense Self-correcting dynamics

149. RECOMMENDATIONS Workshop on the Open Archives Initiative (OAI) and Peer Review Journals in Europe, CERN, Geneva Switzerland,March 22-24, 2001 “It was [also] generally believed that the electronic environment allows for novel approaches to accord a stamp of quality to scholarly works.” Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful Owners,” European Science Foundation Policy Briefing 21 (April 2003): 1-8. http://www.arl.org/sparc/SPB21_OAI.pdf

150. Examples of new metrics that can be extracted from a fully electronic communication system are: Usage counts of a work Automatically extracted citation information with a scope beyond the ISI- core journals Amount of discussion generated by a paper submitted in a system with open peer review and peer comment Etc. Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful Owners,” European Science Foundation Policy Briefing 21 (April 2003): 1-8. http://www.arl.org/sparc/SPB21_OAI.pdf

151. Scientific Publishing as Rhetoric The problems with peer review become evident once the fact that science has a rhetorical element is accepted. On the one hand, the traditional mode of peer review obscures the problems of reference and the rhetorical dimension of science. The rhetorical process which is at the heart of science and peer review conveniently disappears with the final publication of the manuscript. In its place is an ideal typical representation (the scientific paper) of the realist assumptions about empirical reference. All the academic world sees is a polished manuscript where the personal involvement of the researcher and reviewers has been systematically eliminated. Mike Sosteric, “Interactive Peer Review: A Research Note,” Electronic Journal of Sociology 2 no. 1 (1996). http://socserv.socsci.mcmaster.ca/EJS/vol002.001/SostericNote.vol002.001.html http://socserv.socsci.mcmaster.ca/EJS/vol002.001/SostericNote.vol002.001.html

152. ‘IDEAL SPEECH SITUATION’ A theoretical construct that describes the ideal type of interpersonal interaction that should exist in a rhetorical situation. Jürgen Habermas

153. IDEAL SPEECH SITUATION the ideal speech situation permits each interlocutor an equal opportunity to initiate speech; there is mutual understanding between interlocutors; there is space for clarification; all interlocutors are equally free to use of any speech act; there is equal power over the exchange. Applied in the context of peer, the Ideal Speech Situation ‘would permit unimpeded authorial initiative, endless rounds of give and take, [and] unchecked openness among authors, editors, and referees.’ Mike Sosteric, “ Interactive Peer Review: A Research Note,” Electronic Journal of Sociology 2 no. 1 (1996). http://socserv.socsci.mcmaster.ca/EJS/vol002.001/SostericNote.vol002.001.html

154. CONTINUA Continuum of PUBLICATION (‘Scholarly Skywriting’) WEAK MEDIUM Continuum of REVIEW (‘Scholarly Skyreading’) STRONG

155. OPEN ACCESS and OPEN RETRIEVAL without OPEN USE Incongruent Contradictory Ironic Paradoxical Cognitively Dissonant

156. Three-Legged Stool ACCESS -- RETRIEVAL -- USE

158. ACCESS OPEN ACCESS INITIATIVES

159. RETRIEVAL OPEN ARCHIVES INITIATIVE FOR METADATA HARVESTING

160. USE OPEN SCHOLARSHIP

161. ACCESS -- RETRIEVAL -- USE

162. Lots of Alternative Models Provide Sensible Solutions

163. Four-Legged Stool ACCESS -- RETRIEVAL -- USE -- NAVIGATION

164. INFORMATION OVERLOAD

165. OAIster A search engine for freely available, difficult-to-access, academically- oriented digital resources that are OAI -compliant http://oaister.umdl.umich.edu/

167. University of Michigan Digital Library Production Service institutional repositories departmental repositories e-Journal collections technical reports dissertations and theses discipline eprint collections working papers Internet resources audio video images

169. Cognitive Psychology

171. Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) METADATA ELEMENT for QUALITY ADD

172. “Within the framework of OAI, there is a need for a new protocol for certification. There was strong support for the extension of the usage of the OAI protocol beyond discovery-related metadata. Given the focus of the [1 st OAI] workshop on peer review, concrete actions were suggested to address the exchange of certification-related metadata using the OAI protocol in a trusted environment.” http://www.arl.org/sparc/SPB21_OAI.pdf Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful Owners,” European Science Foundation Policy Briefing 21 (April 2003): 1- 8.

173. QUALITY METADATA (1) internal peer review CERN http://www.cern.ch/policies/review.html William Y. Arms, “Quality Control in Scholarly Publishing On The Web. What Are the Alternatives to Peer Review?” PowerPoint presentation given at Workshop on the Open Archives Initiatives (OAI) and Peer Review Journals in Europe, March 22-24, 2003, CERN, Geneva, Switzerland http://www.cs.cornell.edu/wya/papers/CERN-2001.ppt

174. QUALITY METADATA (2) internal ***** 42 http://www.cern.ch/policies/review.html

176. CERTIFICATION SERVICES Faculty of 1,000,000

177. CERTICATION SERVICES New roles for Indexing and Abstracting Services Expanded Role for Learned and Professional Societies Establishment of Formal/Commercial Reviewing Services

178. PEER REVIEW: STRENGTHS The underlying strength of peer review is“…the concerted effort by large numbers of researchers and scholars who work to assure that valid and valuable works are published and conversely to assure that invalid or non-valuable works are not published ….” Anne C. Weller, Editorial Peer Review: Its Strengths and Weaknesses. (Medford, NJ: Information Today, 2001).

179. Faculty of 1000 / BioMed Central BioMed Central (biomedcentral.com) publishes Faculty of 1000 (F1000), the leading literature evaluation service and “new online research tool that highlights the most interesting papers in biology, based on the recommendations of over 1000 leading scientists.” F1000 is managed “by scientists for scientists” …. [and] provide[s] a rapidly updated consensus map of the important papers and trends across biology.” www.facultyof1000.com

180. Faculty of 1000 / BioMed Central Among its many benefits, F1000: systematically organizes and evaluates the mass of information within scientific literature; provides scientists with a continuously updated insider's guide to the most important papers within any given field of research; highlights papers on the basis of their scientific merit rather than the journal in which they appear; offers the researcher a consensus of recommendations from well over 1000 leading scientists; and, offers an immediate rating of individual papers by the authors' peers, and an important complement to the indirect assessment provided by the journal impact factor.

181. Faculty of 1000 / BioMed Central Within the F1000, the entire field of biology is divided into 16 subject areas (‘Faculties’)(e.g., ‘Biochemistry,’, Cell Biology, ‘Microbiology’). Each ‘Faculty’ is subdivided into three (3) to twelve (12) ‘Sections,’ (e.g., Biochemistry: Biocatalysis, Molecular evolution, Protein folding), with each section comprised of between 10 to 50 faculty members. F1000 seeks to invite the best internationally known scientists in each represented field and to involve both experienced and younger investigators.

182. “Peer review is a quality-control and certification (QC/C) filter necessitated by the vast scale of learned research today. Without it, no one would know where to start reading in the welter of new work reported every day, nor what was worth reading, and believing, and trying to build one’s own further research upon.” Stevan Harnad, “Free at Last: The Future of Peer- Reviewed Journals,” D-Lib Magazine 5 no. 12 (December 1999) http://www.dlib.org/dlib/december99/12contents.html

183. SEIZE THE E! Embrace the potential of the digital environment to facilitate access, retrieval, use, and navigation of electronic scholarship.

184. OPEN NAVIGATION

185. New Age Navigation: Innovative Interfaces for Electronic Journals Gerry McKiernan The Serials Librarian Fall 2003

186. SUMMARY. While it is typical for electronic journals to offer conventional search features similar to those provided by electronic databases, a select number of e- journals have also made available higher-level access options as well. In this article, we review several novel technologies and implementations that creatively exploit the inherent potential of the digital environment to further facilitate use of e-collections. Gerry McKiernan, “New Age Navigation: Innovative Interfaces for Electronic Journals,” The Serials Librarian, Fall 2003. http://www.coleonline.us/serialslibrarian/

187. http://www.highwire.org

188. Topic Map

192. Secondary Screen

193. Topic Map

194. http://www.inxight.com TopicMap is based on the Hyperbolic Tree SDK for Java, licensed from Inxight Software, Inc., a spin-off company from Xerox PARC, and leading provider of Unstructured Data Management solutions for accessing, analyzing and delivering information.

195. Semio Automated categorization software technology http://www.entrieva.com/

196. Kohonen Self-Organizing Maps http://websom.hut.fi/websom/milliondemo/html/root.html

197. Kohonen Self-Organizing Maps http://websom.hut.fi/websom/milliondemo/html/1_cx5.html

198. Kohonen Self-Organizing Maps http://websom.hut.fi/websom/milliondemo/html/2_gx10.html

199. Kohonen Self-Organizing Maps http://websom.hut.fi/websom/milliondemo/getnd.cgi?32323

200. Kohonen Self-Organizing Maps http://websom.hut.fi/websom/cgi-bin/getfile.cgi/comp.ai/39340

201. http://www.springer.de/books/toc/3540679219-c.pdf

202. “There are some excuses, but at the bottom it will be seen to be the sluggishness of human nature and its superstitious cleavage to old habits.” Stevan Harnad, “Free at Last: The Future of Peer- Reviewed Journals,” D-Lib Magazine 5 no. 12 (December 1999) http://www.dlib.org/dlib/december99/12contents.html

203. “… [In] the digital world, the evaluation process stands ready to be reinvented in a clear, rational way by the relevant research communities themselves.” Jean-Claude Guédon, In Oldenburg’s Long Shadow: Librarians, Research Scientists, Publishers, and the Control of Scientific Publishing. (Washington, D.C.: Association of Research Libraries, 2001), 54. http://www.arl.org/arl/proceedings/138/guedon.html

204. “The Medium is the Message … And the Method.” With apologies to Marshall McLuhen Marshall McLuhen

205. THANK YOU FOR YOUR ATTENTION!

206. Alternative Peer Review: Quality Management for 21 st Century Scholarship Gerry McKiernan Science and Technology Librarian and Bibliographer Iowa State University Library Ames IA USA gerrymck@iastate.edu

207. OPEN MIND

208. REVISED VERSION 1.01 July 27, 2003 11:30 AM