1. 3. Challenges of bibliometrics DATA: many problems linked to the collection of data FIELDS – DISCIPLINES : various classifications, not satisfactory INDICATORS : which ones are valid/needed to quantify research performance? MAPPING : to assess potential rather than past performances (metrics) workshop

2. 3. Challenges FIELDS – SCIENTIFIC DISCIPLINES Various classification (OCDE, national, ISI, Scopus, …) Problems of coverage Academic structure might be quite different from the classification of the database Context : a research assessment exercice (ULB; school of economy and business) Comparison between bibliometrics and peer-review approaches to measure publications in economics Issue 1: retrieval of all publications relevant to economics from the ISI-Thomson database (Web of Science) Issue 2 : matching the knowledge-based sorting with the academic delimitation of disciplines in this university

3. Agricultural economics & policy - Fisheries - Geosciences - Water resources - Engineering (environmental) Area studies - Demography - Gerontology - Nursing - Psychology (3 sub-domains) - Ethnic studies - Family studies - Computer science (3 sub-domains) - Communication - Education & educational research And also... What does ISI database show about the sciencific production in economics of a Belgian university (ULB) ? Retrieval of ULB publications (2001-2008) for all ISI categories related to economics (JEL classification) For each category, manual sorting of publications 4 categories with 100% articles in economics 26 categories with at least 10% articles in economics 20 other relevant categories with less than 10% articles

4. 3. Challenges of scientometrics INDICATORS: choice is critical! Repeatable, based on transparent methodology (allow the institutions to reproduce the methodology in-house) Capable of identifying comparable levels of research quantity and/or quality across disciplines Quantitative (scientific production) vs qualitative (citations) Impact on researchers behaviour Main drawbacks no control for the size of the institution databases not complete for SHS Australian case study (Harzing, in Economics & Bysiness, 2005) Indicators used by the government for funding : till 1993, competitive research grants; since1994: # publications Consequence : high publication quantity (# papers) but slow publication quality (citations per paper)

5. 3. Challenges of scientometrics INDICATORS Which indicator is a valid and robust measure of research quality for a discipline? FNRS study of co-publications (2005-2008) h-index : measure scientific productivity and apparent scientific impact m-index : measure the quality of the h « best » papers

6. 3. Challenges Indicators (a tool for improvment) To feed research policy and improve performances Size versus production Production versus quality Univ E Univ C Univ K Univ G Univ D Size Number of papers Univ E Univ C Univ K Univ F Univ G Univ H Univ I Univ J Univ A Univ D Univ B 02000400060008000 Number of papers Number of citations per paper PHYSICS & ASTRONOMY

7. 6. Pending questions Research evaluation is usefull But Needs to be carefully handled And Raises several questions Impact of evaluation on research activity, strategy (researchers, universities) Impact of evaluation on research funding (disciplines?) Risk to miss emergence of teams, themes (bibliometrics assess the past) Choice of methodology (peer review? bottom-up?) How to feed the research part of the (probably unavoidable) rankings with indicators developed as a support of research policy Possible role of Eurohorcs