1. Jan Dvořák, Jiří Souček : The R&D Efficiency Evaluation in the Czech Republic CRIS2008 – Maribor, Slovenia – June 2008 InfoScience Praha, s.r.o.

2. 2 Agenda The Czech R&D IS (JD)‏ The Efficiency Evaluation in the CR (JD)‏ Efficiency Evaluation in general (JS)‏

3. 3 The R&D Information System of the Czech Republic A CRIS covering the R&D subsidized from the State Budget of the Czech Republic Run by the Czech R&D Council Contains data on  Funding providers, Programmes, Calls  R&D Projects (since 1994, ~30,000)‏  Institutional Research Plans (since 1999, ~880)‏  R&D Results – publications, patents, technologies (since 1998, ~530,000 records, ~420,000 unique results)‏ Data from funding providers

4. 4 http://aplikace.isvav.cvut.cz

5. 5 The R&D Efficiency Evaluation The setup  The R&D IS contains all results of all research that has been supported from public sources in the CR  The State is the predominant supporter of basic research in the CR The „investor's“ point of view  Results per unit funding Runs in 2005, 2006, 2007

6. 6 Results → Points Results are assigned points Example from the 2007 Evaluation  Article in an impacted journal → 5 + 15 x IF / median( IF ) over discipline using data from ISI JCR  Article in a non-impacted, reviewed periodical → 1 (Czech & Slovak) 2 (other languages)‏  Monograph →12.5 (Czech & Slovak) 25 (other languages)‏  Proceedings contribution → 0.1 (Czech & Slovak) 0.2 (other languages)

7. 7 Results → Points Example from the 2007 Evaluation (cont'd):  Patent → 500 (EPO, USTPO, JPO) 50 (other issuer, incl. the Czech Industrial Property Office)‏  Verified technology, breed → 75  Prototype, software → 25 Special handling of Social Sciences and Humanities The point-assignment rules are agreed upon by the R&D Council members

8. 8 Results → Points The point assignment has been evolving E.g. monograph:

9. 9 Point distribution

10. 10 Point distribution M:N relationships need a distribution factor  result : research activity → result points divided uniformly  research activity : institution → result points divided in the proportion of State Budget funding

11. 11 Index Points( X ) / Cost( X ) → Index( X )‏  Institution  Programme  Funding provider  Whole R&D Index( X ) / Index( Whole R&D ) → Relative Index( X )‏

12. 12 Relevant data selection Sliding window: Evaluation in year Y: Research activities ending in [ Y-5, Y-1 ] Infrastructure projects excluded

13. 13 Sample 17,969 1821 235 516The whole Czech state (co)funded R&D totals 27,920 221564 040The Czech public universities sector totals 101%83%78%28,11 52442 790Institute of Chemical Technology Prague 80%87%69%22,31 67637 318Brno University of Technology 112%133%139%31,31 96561 476Masaryk University, Brno 56%80%69%15,62 93345 750Czech Technical University in Prague 130%112%125%36,35 923214 847Charles University in Prague Eval. 20 07 Eval. 20 06 Eval. 20 05 IndexState funding [MCZK] Result poin ts Relative index (within the public universities sector) ‏ Evaluation 2007University name

14. 14 Sample

15. 15 Points by result type

16. 16 Changes in 2008 Modified point-assignment rules  Esp. for articles in impacted journals Institutions evaluated by results  Institution gets points for a result in the proportion of the authors affiliated with it to the total number of authors  Part of the “Reform of the system of research funding by the State”

17. 17 General principles of the Evaluation  Systematic part (the logic of R&D)‏  Results evaluated by the point-system  Dividing points to individual projects  Summing up points to Institutions  Calculation of indexes and relative indexes  Specific parameterization for points-systems  Depends on the type of the result  Using impact factor of the journal  Using properties / attributes of the result  Needed data on objects in R&D  Projects, institutions, results  Links projects – results  Links institutions – results

18. 18 Principles or the impersonal (objective) evaluation Is determined by rules (not by opinions)‏ Impartiality - does not depend on opinions Non-refutability – depends on hard facts Is possibly imperfect, but the inaccuracy is random and suppressed by the aggregation (25 results  5 times better precision)‏ Systematic inaccuracies can be traced and corrected Basic characteristics  Non-refutability  Impartiality

19. 19 The importance of the impersonal (objective) evaluation Serves not only for R&D Council, but  For all workers in R&D field  They can independently judge the contribution of others (projects, teams, institutions, …)‏  Expected reception of a published paper is approximately equal to the impact factor of the journal (it is not necessary to wait many years for citations to accumulate)‏  Impersonal evaluation gain more weight than the peer review, since it is non-refutable

20. 20 Needs of ERA (European Research Area) In the R&D information infrastructure & evaluation  Accessibility of the information on projects, funds, calls, results (also on the national level)‏  Accessibility on the data supporting the evaluation The way :  Integration of data from national CRISes (just in progress)‏  Building evaluation data system Efficiency of the allocation of resources  In economics – market forces  In research area - there are, in principle no market forces The role of market forces should be substituted by the impersonal (objective) evaluation.

21. 21 Our vision The creation of the consortium focused to evaluation of R&D results (in the 7 FP – the infrastructure of the research) WE ARE LOOKING FOR PARTNERS AND COOPERATION! If you are interested in R&D evaluation please contact us!

22. 22 Thank you for your attention Jan Dvořák jan.dvorak@infoscience.cz Jiří Souček jiri.soucek@infoscience.cz Martin Souček martin.soucek@infoscience.cz Richard Papík richard.papik@ff.cuni.cz