Presentation is loading. Please wait.

Presentation is loading. Please wait.

Open access and research evaluation Thursday 19th October 14.15-15.45 Session 4: Quality Assessment Open Scholarship 2006: New Challenges for Open Access.

Similar presentations


Presentation on theme: "Open access and research evaluation Thursday 19th October 14.15-15.45 Session 4: Quality Assessment Open Scholarship 2006: New Challenges for Open Access."— Presentation transcript:

1 Open access and research evaluation Thursday 19th October 14.15-15.45 Session 4: Quality Assessment Open Scholarship 2006: New Challenges for Open Access Repositories The University of Glasgow 18-20 October 2006 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Institute for Research Information and Quality Assurance Stefan Hornbostel iFQ Institut für Forschungsinformation und Qualitätssicherung Institute for Researchinformation and Quality Assurance Godesberger Allee 90 D-53175 Bonn www.forschungsinfo.de

2 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Relations between OA and research evaluation political / rethorical function to spread OA (to maximize the impact of their articles authors should archive!) OA offers a better way to evaluate research performance? (Compensation of shortcomings of traditional bibliometric Indicators) OA makes it possible to construct new indicators to measure a completely different dimension (usage, reading, …) OA changes scientific behaviour (more easily accessibility, sooner availability, quick circulation, high quality).

3 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Relations between OA and research evaluation (1) political / rethorical function to spread OA (to maximize the impact of their articles authors should archive!) Research evaluation practices are used to convince authors of OA advantages (higher Citationrates, higher attention) The analysis covers 1,494 publication venues containing at least 5 online and 5 offline articles. For 90% of venues, online articles are more highly cited on average. On average there are 336% more citations to online articles compared to offline articles. Lawrence, S. "Online or Invisible" Nature 411 (6837): 521, 2001.

4 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Mandated online RAE CVs Linked to University Eprint Archives, The Funding Councils should mandate that ……. the full text of every refereed research paper, publicly self-archived in the university's online Eprint Archive and linked to the CV for online harvesting, scientometric analysis and assessment. This will increase the uptake and impact of UK research output, by increasing its visibility, accessibility and usage, and set an example for the rest of the world that will almost certainly be emulated, in both respects: research assessment and research access. Harnad, S., Carr, L., Brody, T. and Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives. Ariadne 35.

5 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 The proportion of articles for which their authors provide OA is likely to increase dramatically now, in part because of the mounting evidence for the impact advantage OA confers. Brody, T. and Harnad, S. (2004) Comparing the Impact of Open Access (OA) vs. Non-OA Articles in the Same Journals. D-Lib Magazine 10(6).

6 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 For Example: Cream of Science Target demonstrat(ing) that scholars are willing to deposit their materials in a repository, thereby also increasing the awareness of other scholars Method DARE partners selected ten of their prominent scientists and made their complete publication list, with as much full text available as possible Selection Criteria there was no objective criteria to define 'prominent',…but most of the DARE partner used a formal method; selection by the Executive Board and/or using a letter from the Dean to invite academics to be part of Cream Martin Feijen and Annemiek van der Kuil 2005: A Recipe for Cream of Science: Special Content Recruitment for Dutch Institutional Repositories', Ariadne Issue 45. Researchers are enthusiastic about the initiative. A number of them already refer to www.creamofscience.org on their own websites for a complete overview of their work. They regard Cream of Science as a hallmark of quality. Dare.net 17.05.2005www.creamofscience.org

7 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Kurtz et al. claim that there are (at least) three possible, non-exclusive effects that cause the effect of higher citation rates. Open Access Postulate (OA): Because of free, unrestricted access, papers are read more easily and therefore get cited more frequently Early Access Postulate (EA): Papers offered as e-print are available sooner and therefore gain primacy and additional time in press, and therefore they get cited more often Self-selection Bias Postulate (SB) or (Quality Postulate) : The most important -- and therefore most citable -- papers are posted on the Internet Kurtz, Michael J., Eichhorn, Guenther, Accomazzi, Alberto, Grant, Carolyn, Demleitner, Markus, Henneken, Edwin, Murray, Stephen S. (2005). The Effect of Use and Access on Citations, Information Processing and Management, Vol. 41, Issue 6, p. 1395-1402. OA a causal factor ?

8 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 no relationship between fulltext arXiv downloads and the number of article citations until an article has been downloaded more than 400 times consistent with the detailed work of Henk Moed on the journal Tetrahedron Letters, which suggests that a relatively small group of both highly cited and frequently downloaded papers are responsible for the weak, though statistically significant, correlation between downloads and citations Philip M. Davis / Michael J. Fromerth (2006): Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles? Open Access Postulate

9 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Early Access Postulate We have failed to find support for the Early View postulate and have provided evidence that many highly-cited articles were deposited into the arXiv after formal publication. Partial regression plot of prepublication days on citation, controlling for the number of days since publication and the effect of the journal. Articles deposited in the arXiv after formal publication are highlighted red. Philip M. Davis / Michael J. Fromerth (2006): Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles?

10 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 approximately 15% of the articles in a journal collect 50% of the citations If authors are indeed depositing their best papers, Open Access citation advantage may be an artifact, not the cause of a citation advantage Self-selection Bias Postulate (SB) or Quality Postulate

11 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Problems: Open Access (green and gold), Open Archive, Self Archiving may produce different quality distributions J. Testa and M.E. McVeigh. 2004.The Impact of Open Access Journals. A Citation Study from Thomson ISI, www.isinet.com/oaj.www.isinet.com/oaj

12 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 OA offers a better way to evaluate research performance? (Compensation of shortcomings of traditional bibliometric Indicators) These new possibilities may be interesting seen in combination with the known inherent bias of research evaluation using citation analysis. ….. Tove Faber Frandsen (2006): Open access-based resources in research evaluation The Funding Councils should mandate that … all UK research-active university staff must maintain:.... (II) the full text of every refereed research paper, publicly self-archived in the university's online Eprint Archive ….. This will give the UK Research Assessment Exercise (RAE) far richer, more sensitive and more predictive measures of research productivity and impact, for far less cost and effort (both to the RAE and to the universities preparing their RAE submissions)… Harnad, S., Carr, L., Brody, T. and Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives. Ariadne 35.

13 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 ISI / WOS coverage by main field EXCELLENT (> 80%) Good (60-80%)Good (40-60%)MODERATE (<40 %) Biochem & Mol BiolAppl Phys & ChemMathematicsOther Soc Sci Biol Sci ~ HumansBiol Sci ~ Anim & Plants EconomicsHumanities & Arts ChemistryPsychol & PsychiatEngineering Clin MedicineGeosciences Physics & AstronSoc Sci ~ Medicine & Health Quelle: Henk v. Moed, Evaluation of Research Performance and Funding Programme in Social Sciences; Tagung: Norface Workshop on Research Programme Development and Management 7.02.06, Bonn: http://www.norface.org/norface/files/file_20060207014936Henk%20Moed.pdfhttp://www.norface.org/norface/files/file_20060207014936Henk%20Moed.pdf

14 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Comparison of Mean Citation Rates Between Freely Available Articles and Those That Are Not Freely Available DisciplineMean (open) Mean Standard Error (open) Mean (not open) Mean Standard Error (not open) Difference in Means Philosophy1.600.4911.100.230.500 Political Science2.200.4771.180.3531.016 Electronical and electronic engineering 2.350.4491.560.275.798 Mathematics1.600.2700.840.230.762 Antelman, Kristin (2004) Do Open Access Articles Have a Greater Research Impact?. College & Research Libraries News 65(5):pp. 372-382.

15 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 In electrical engineering, computer science and system engineering, 34 % of the respondents know of this form of publication; the highest percentage here is among computer scientists, at 51 %. In mechanical engineering and production engineering, 31 % of the scientists said they knew about open access journals, compared to just 13 % in the area of heat energy technology and process engineering. In the areas of civil engineering and architecture this type of publication is completely unknown. Deutsche Forschungsgemeinschaft (2005): Publishing Strategies in Transformation? Results of a study on publishing habits and information acquisition with regard to open access

16 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Deutsche Forschungsgemeinschaft (2005): Publishing Strategies in Transformation? Results of a study on publishing habits and information acquisition with regard to open access In the humanities and social sciences, it is mostly social and behavioural scientists who use the option of secondary open access publication for their academic studies. While they make around 9% of the articles published in journals available for free on the internet, this is done by only 3% of the humanities scholars surveyed. In the engineering sciences it is the computer scientists who stand out, with two-thirds of their articles in conference proceedings, 46 % of their journal articles, almost 42 % of their contributions to edited volumes and nearly 24% of their monographs republished on the internet.

17 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Deutsche Forschungsgemeinschaft (2005): Publishing Strategies in Transformation? Results of a study on publishing habits and information acquisition with regard to open access

18 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Richer cross-repository services, that is services that overlay multiple repositories Cross-repository scholarly communication workflows Examples of such value-chains include: The transfer of (parts of) digital objects from digital repositories to parties that provide discovery-oriented services over (parts of) the digital objects. The transfer of digital objects from digital repositories to parties that provide value-added services over the digital objects. The introduction of natively machine-readable and machine-actionable bibliographic citations. The re-use of datasets (from different repositories) as the basis for building a new dataset or for writing a publication. AUGMENTING INTEROPERABILITY ACROSS SCHOLARLY REPOSITORIES A meeting sponsored and supported by Microsoft, the Andrew W. Mellon Foundation, the Coalition for Networked Information, the Digital Library Federation, and the Joint Information Systems Committee. April 20 – 21, 2006, New York, NY Ithaca, NY and Los Alamos, NM - The Open Archives Initiative (OAI), with the generous support of the Andrew W. Mellon Foundation, announces a new effort as part of its mission to develop and promote interoperability standards that aim to facilitate the efficient dissemination of content. Object Reuse and Exchange (ORE) will develop specifications that allow distributed repositories to exchange information about their constituent digital objects. These specifications will include approaches for representing digital objects and repository services that facilitate access and ingest of these representations. The specifications will enable a new generation of cross-repository services that leverage the intrinsic value of digital objects beyond the borders of hosting repositories.

19 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 OA changes scientific behaviour (more easily accessibility, sooner availability, quick circulation, high quality) Scholarly communication has changed rapidly over the last decade as scientists now publish their findings in e.g. open access journals and preprint servers. Bauer & Bakkalbasi (2005) The communication system is changing primarily due to the use of new technology. Friend (2004)

20 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 But: only c. 20% of the number of papers published annually are open access

21 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Poeschl,U. 2006, Peer Review Revisited, 16.05.2006, Ms.

22 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 OA makes it possible to construct new indicators to measure a completly different dimension (usage, reading, …) Web indicators based on usage data are a way to obtain unique information of the utility and scholarly acceptance of OA research output, but these new user oriented web indicators are still in an early experimental phase. The rationale of these approaches is strong but standardization of web metrics is at the beginning. …. there is still much to be done in evaluation of OA research. In addition we need further approaches to build robust measures and metrics especially for OA publications/documents. Following the thesis science is turning to e-science … we conclude e-science will need new stable e-indicators or a combination with traditional indicators to appropriately take e-only research output into consideration. Philipp Mayr (2006): Constructing experimental indicators for Open Access documents.

23 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 First, journal relationships in the RGN network seem to be valid and representative of the community whose downloads have shaped the network. Second, structural journal impact metrics derived from the RGN deviate strongly from the ISI IF. Third, the applied structural impact metrics correlate strongly with the ISI IF when calculated over the AGN, indicating they do validly operationalize journal impact, if we honor the assumption that the ISI IF does. Fourth, the AGN and RGN networks overlap to some degree, but exhibit striking differences. Johan Bollen / Herbert Van de Sompel / Joan A. Smith / Rick Luce (Los Alamos National Laboratory) 2005: Toward alternative metrics of journal impact:A comparison of download and citation data. The rendered journal impact rankings were compared to those derived from the Reader Generated Network, and the ISI IF. Our results indicate the following: Darmoni, Roussel, Benichou, Thirion, and Pinhas (2002) compare journal usage frequency to the ISI IF for a medical DL collection. They define a Reading Factor (RF) which consists of the ratio of a particular journals download frequency to the total downloads of all journals as recorded in the DLs logs. The authors report a low and statistically insignificant correlation between the observed RF and the ISI IF for the same set of journals.

24 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Standards! OA research evaluation should follow the classical methods of bibliometrics: Measuring Research activity P amount of Publications P% Percentage of worldwide Publications (international comparison) or national Publications (national comparison) Measuring Impact Camount of citations Pp%percentage of publications not cited until now Cmax Maximum of Citations C/P Citation Rate C/P/FCSm (mean field citation score) Relative Citation Rate (related to field) (C/P)/JCSm (mean journal citation score) Relative Citation Rate (related to journal)

25 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Thank you!

26 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.10.2006 Difficulties of OA research evaluation Only fits to publications in traditional databases - especially ISI Difficult for publications which are not covered by ISI Missing subfield information Metadata No standardised metadata No gold standard for the evaluation of open access papers No best practices of hundred percent covering Units Main units for OA research evaluation are inexplicit Different kinds of publications Different motives for OA publishing -> bias!

27 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.20.2006 It was found that during the first 3 months after an article is cited, its number of downloads increased 25% compared to what one would expect this number to be if the article had not been cited. Moreover, more downloads of citing documents led to more downloads of the cited article through the citation. An analysis of 1,190 papers in the journal during a time interval of 2 years after publication date revealed that there is about one citation for every 100 downloads. A Spearman rank correlation coefficient of 0.22 was found between the number of times an article was downloaded and its citation rate recorded in the SCI. When initial downloads - defined as downloads made during the first 3 months after publication - were discarded, the correlation raised to 0.35. However, both outcomes measure the joint effect of downloads upon citation and that of citation upon downloads. Correlating initial downloads to later citation counts, the correlation coefficient drops to 0.11. Findings suggest that initial downloads and citations relate to distinct phases in the process of collecting and processing relevant scientific information that eventually leads to the publication of a journal article. Statistical relationships between downloads and citations at the level of individual documents within a single journal Henk F. Moed Journal of the American Society for Information Science and Technology Volume 56, Issue 10, Pages 1088 - 1097 Published Online: 31 May 2005

28 Institut für Forschungsinformation und Qualitätssicherung Hornbostel 19.20.2006 These results furthermore raise questions regarding the validity of the ISI IF as the sole assessment of journal impact, and suggest the possibility of devising impact metrics based on usage information in general. Toward alternative metrics of journal impact: A comparison of download and citation data. Johan Bollen Department of Computer Science, Old Dominion University, 4700 Elkhorn Ave., Norfolk VA 23529 Herbert Van de Sompel Research Library, Los Alamos National Laboratory, Los Alamos, NM, 87554 Joan A. Smith Department of Computer Science, Old Dominion University, 4700 Elkhorn Ave., Norfolk VA 23529 Rick Luce Research Library, Los Alamos National Laboratory, Los Alamos, NM, 87554 Preprint submitted to Elsevier Science 17 May 2006 Similarly, Darmoni, Roussel, Benichou, Thirion, and Pinhas (2002) compare journal usage frequency to the ISI IF for a medical DL collection. They define a Reading Factor (RF) which consists of the ratio of a particular journals download frequency to the total downloads of all journals as recorded in the DLs logs. The authors report a low and statistically insignificant correlation between the observed RF and the ISI IF for the same set of journals. These results show that journal download frequency within a local DL community does not correspond to the ISI IF, which raises questions regarding the ISI IFs validity as the sole indicator of Ig among a specific community of readers. 4


Download ppt "Open access and research evaluation Thursday 19th October 14.15-15.45 Session 4: Quality Assessment Open Scholarship 2006: New Challenges for Open Access."

Similar presentations


Ads by Google