Presentation is loading. Please wait.

Presentation is loading. Please wait.

Giorgio Sirilli ISSiRFA - CNR. Giorgio Sirilli ISSiRFA - CNR.

Similar presentations

Presentation on theme: "Giorgio Sirilli ISSiRFA - CNR. Giorgio Sirilli ISSiRFA - CNR."— Presentation transcript:

1 Giorgio Sirilli ISSiRFA - CNR

2 Giorgio Sirilli ISSiRFA - CNR

3 Our subject: Evaluation

4 Knowledge: the bundle

5 There are many different types of knowledge and knowledge holders. Not all knowledge is scientific, much knowledge is practical and contex specific Society is interested in its: Production Transmission Use Knowledge

6 New public management Evaluation Research Assesment Exercise (RAE-REF) Bibliometrics CNR Department SSH Research evaluation: the ladder

7 University teaching research third mission The learned society Research agencies research problem solving management

8 New Public Management Catchwords Accountability Value for money

9 The rationale of performance funding is that funds should flow to institutions where performance is manifest: performing institutions should receive more income than lesser performing institutions, which would provide performers with a competitive edge and would stimulate less performing institutions to perform. Output should be rewarded, not input. Herbst, 2007 Performance based research funding systems The rationale

10 Performance based research funding systems are national systems of ex-post university research output evaluation used to inform distribution of funding Performance based research funding systems

11 Criteria Research (teaching excluded) Evaluation ex-post (ex-ante excluded) Research output Research funding must depend on the results of evaluation National system (internal university evaluations excluded) Share of university funding dependent on Performance-Based Research Funding Systems

12 CountryShare (%)of what Australia6Total revenue Italy2Block grant New Zeland10Block grant Norway2Total funding Slovak Republic15Total funding UK25Research support Hicks D., Reseach Policy (2012)

13 Share of university funding dependent on Performance-Based Research Funding Systems Hicks D., Reseach Policy (2012) The distribution of university research funding is something of an illusion It is the competition for prestige that creates powerful incentives within university systems Performance-based research funding systems aim at excellence: they may compromise other important values such as equity and diversity

14 Carried out by CNR in 2009-2010 Evaluation of structures, not individuals Methodology -General panel -Panels (26) composed of 156 scientists -Documentation provieded by Institutes -Site visit -Final score Lessons: The General Panel recommended that in the future the two groups of scientific areas (natural sciences and engineering, social science and humanities) remain separated. Experience showed in fact that differences in objectives, research methodologies, evaluation criteria suggest to be deal with them separately Evaluation of CNR Institutes

15 VQR (Evaluation of the Quality of Research) Habilitation of university professors bibliometric areas non-bibliometric areas Both exercises largely based on bibliometrics ANVUR

16 Knowledge: the bundle

17 When two types of evaluation do not match

18 Internationally oriented journal articles Books National journals Enlightenment literature Hicks D., Research Policy (2012) Four types of publications

19 International journal literature WoS/Scopus Books High impact National literature Local context Enlightenment Knowledge transfer Other means of transfer The four literatures Hicks D., Research Policy (2012)

20 Science Social Science Journal articles – counted in WoS/Scopus indicators The 3 other literatures – not counted Other: books, edited books, book chapters, monographs, reports, creative works Sources: Bourke et al,1996; Pestana et al., 1995; Winterhager, 1994 81-85% 45-61% Journal literature Problem: missing output

21 Percentage of journal articles indexed in WoS Ossenblok et al. 2012 Norwegian social science – WoS coverage by field Ossenblok et al. 2012 Sciences

22 Genuine scholarly research in any area leads to internationally relevant results Social sciences are embedded in national contexts, national issues are important, subtle theory is developed in national languages Academics write in EnglishSocial scientists more often publish in their own language Academics are international in orientation Social scientists are more oriented to their own national literature than scientists SSCI coverage of national literatures inadequate The problem with national literature: Lower level of critique and peer review, questionable quality National literature

23 Burnhill & Tubby-Hille, 1994, UK social science Sciences Enlightenment literature by field

24 Research evaluation NSESSH BibliometricsHelpfulNot very helpful Scientific literatureInternationally orientedNationally oriented Peer reviewImportantVery important Habilitation of professorsBibliometrics + peer review Peer review + bibliometrics CostExpensiveMore expensive TimeLimitedLonger

25 Cultural identity 15 Institutes and 336 researchers/technologists SSH Cultural heritage 4 Institutes and 110 researchers/technologists SSH + NSE CNR Department of Cultural Identity and Heritage

26 CNR Departments - Researchers and technologists by training (percentage) Cultural identity (15 Institutes) Cultural heritage (4 Institutes) 33% 38% 46% 42% 10% 3% 18% 10% Significant differences in specialisation of researchers between former Departments

27 CNR Departments – Research products (2007-2011) (percentage) Significant similarities in types of products between former Departments

28 CNR Departments – Research products (2007-2011) (percentage) Structural difference between SSH Departments and a NSE Department

29 CNR Departments – Journal articles (2008-2010) 36% 11% 53% 46% 23% 31% Structural difference between former Departments in terms of output

30 Contributo in rivista Articolo in rivista Traduzione in rivista Nota a sentenza Scheda bibliografica Abstract in rivista Abstract/Comunicazione in rivista Abstract/Poster in rivista Rassegna e recensione Contributo in volume Contributo in volume (capitolo o saggio) Traduzione in volume Prefazione/Postfazione Breve introduzione Voce (in dizionario o enciclopedia) Scheda di catalogo Contributo in atti di convegno Abstract/Poster in atti di convegno Comunicazione a convegno Abstract/Poster in convegno Abstract/Comunicazione in atti di convegno Libro Monografia o trattato scientifico Traduzione di libro Edizione critica Commento scientifico Concordanza Indice Pubblicazione di fonti inedite Antologia e manuale scolastico Dizionario e thesaurus Bibliografia, repertorio, glossario Catalogo monografico Manuale tecnico z Curatela Brevetto Altra tipologia Composizione Performance Banca dati Software Cartografia Commento giuridico Rapporti progetti di ricerca Rapporti tecnici, manuali, carte geologiche e tematiche e prodotti multimediali Rassegne, recensioni, prefazioni Altre pubblicazioni Rapporti finali progetti di ricerca Materiale didattico Rapporti di commissioni Rapporti statistici Note interne, manuali e guide Prodotti multimediali Risultati di valorizzazione applicativa Progetti Disegni e design Rappresentazioni Mostre ed esposizioni Manufatti, prototipi d'arte e relativi progetti Rapporti tecnici/preprint/working paper Data set Altro prodotto 25

31 Other products: 12 out of 25 (2001-2011) Products for the market are marginal: other categories should be added Market oriented Academically oriented

32 International journal literature Books National literature Enlightenment Other means of transfer CNR products in data base

33 Evaluation of SSHs: some comments Knowledge is our concern Do not forget the knowledge bundle and its various facets

34 2113 0-11.10 Chemist 2113 0-11.50 Chemist, analytical 2113 0-11.40 Chemist, corrosion 2113 0-11.40 Chemist, crystallography 2113 0-11.20 Chemist, detergents 2113 0-11.20 Chemist, dye 2113 0-11.20 Chemist, food 2113 0-11.30 Chemist, glass 2113 0-11.30 Chemist, inorganic 2113 0-11.20 Chemist, leather 2113 0-11.30 Chemist, metallurgical 2113 0-11.40 Chemist, nuclear 2113 0-11.20 Chemist, organic 2113 0-11.20 Chemist, paint 2113 0-11.20 Chemist, petroleum 2113 0-11.90 Chemist, pharmaceutical 2113 0-11.40 Chemist, physical 2113 0-11.20 Chemist, plastics 2113 0-11.20 Chemist, polymer 2113 0-11.50 Chemist, quality control 2113 0-11.20 Chemist, rubber 2113 0-11.20 Chemist, textile 2111 0-12.10 Physicist 2111 0-12.50 Physicist, acoustics 2111 0-13.50 Physicist, astronomy 2111 0-12.80 Physicist, atomic 2111 0-12.20 Physicist, ballistics 2111 0-12.60 Physicist, electricity and magnetism 2111 0-12.70 Physicist, electronics 2111 0-12.30 Physicist, heat 2111 0-12.20 Physicist, hydrodynamics 2111 0-12.40 Physicist, light 2111 0-12.20 Physicist, mechanics 2111 0-12.80 Physicist, molecular 2111 0-12.80 Physicist, nuclear 2111 0-12.50 Physicist, optics 2111 0-12.20 Physicist, rheology 2111 0-12.90 Physicist, solid-state 2111 0-12.50 Physicist, sound 2111 0-12.90 Physicist, theoretical 2111 0-12.30 Physicist, thermodynamics 3415 4-31.20 Representative, sales/technical XXXXXXX Researcher: MISSING 1315 5-10.30 Restaurateur International Classification of Occupations (ISCO) 2219

35 Evaluation of SSHs: some comments Knowledge is our concern Do not forget the knowledge bundle and its various facets Why evaluate? Eveluation is a complex business, and bibliometrics as a shortcut Much information on outputs of SSH is not covered by bibliometrics SSHs are structurally different from NSEs: different types of evaluation Extending the SSH coverage of literature is OK, but there are inherent limits Significant differences amongst CNR SSH Department Institutes deserve further analysis More experimentation needs to be done in SSH: in the meantime be cautiuous

36 Research evaluation in SSH

37 Thank you for attention

Download ppt "Giorgio Sirilli ISSiRFA - CNR. Giorgio Sirilli ISSiRFA - CNR."

Similar presentations

Ads by Google