1. ISSI 2011 Conference Durban, 4-7 July 2011 R&D evaluation of the Italian CNR institutes: A missed opportunity? Giorgio Sirilli Institute for the Study of Regionalism, Federalism and Self-Government National Research Council

2. zObjectives of the exercise zMethodology zResults zDiscussion and policy implications Outline

3. Objective of the exercise zevaluate the past activity of the research institutes in order to: yverify the matching between their original mission and the present research lines ymeasure the quality of the scientific results in the international context; yidentify the weaknesses of the system and the ways to tackle the problems zidentify the perspectives of the institutes, in order to: yimplement the necessary organizational changes yincrease the value of existing potentialities ydevelop the most promising research lines able to attract more financial resources in a context of budget restraints

4. Features of the exercise z117 Institutes evaluated zTime: April 2009 – May 2010 zGeneral Panel composed of 16 distinguished scientists and technologists belonging to the Italian scientific community (one for each of the macro-areas in which CNR operates), responsible for the overall management and coordination of the process and for the drafting of the final report to be delivered to the CNR board of directors z26 Thematic Panels composed of 156 scientists (40% foreigners or Italian expatriates)

5. Methodology zQuestionnaire filled out by Institutes’ director yPersonnel yPublications yDissemination yIntellectual property yTeaching yExternal funding yInfrastructure zSite visit zDiscussion within the TP (score) zEvaluation by GP

6. Methodology zMaximum score to be assigned by each TP to each parameter with the proviso that the sum should be 100 yPublications yDissemination yEditorial activity yPatents yTraining yExternal funding yInfrastructure

8. Average score assigned by TP

9. The “severity index” For each TP the ratio between the average evaluation of 26 panels and TP’S average.

10. Severity index

11. An example of final evaluation from GP

12. General Panel report zThe performance of the CNR institutes was quite good: ythe average score of institutes in the areas of natural sciences and engineering was 73 ythe average score of institutes in social sciences and humanities was 82 ythe difference was deemed not to be significant zGP noted that Institutes are characterised by a very high heterogeneity zGP recommended that in the future the two groups of scientific areas (natural sciences and engineering, social science and humanities) remain separated zGP recommended that TPs avoid visit too few institutes

13. Pros and cons of the exercise PROS zfirst internal evaluation carried out by CNR zthe process was a credible one, given the presence of international experts zthe evaluation process was welcomed by researchers zthe results were deemed to be potentially used for improving the scientific network and for promoting the carrier of institutes’ researchers zthe evaluation was a good opportunity to start collaboration between those evaluated and the evaluators

14. Pros and cons of the exercise CONS zthe emphasis of the exercise was placed on the scientific dimension, while little attention was paid to the management of the institutes zthe report put too much emphasis on the quantitative dimension producing basically a “league table” of the institutes zthe time between the period of reference of the information supplied to the TPs (2003-2007) and the site visit was too long zthe institutes’ staff had no chance to comment on the results in an iterative process zthe periodicity of the exercise was not specified zthe GP’s final report gave little guidance to the agency’s governing body in terms of proposals for future restructuring of the scientific network

15. A comparison with other European agencies z periodicity zcoverage zself-evaluation of institutes zscope of the evaluation zchoice of peers zsources of information zsite visits zinvolvement of the institutes evaluated zemphasis of the evaluation in the final report zexpected role of evaluation in decision making process zfollow-up of results

17. CNR evaluation compared with other European agencies zoccasional exercise zfocus on the scientific dimension zno involvement of researchers in the selection of peers and no chance for them to discuss the outcome of the evaluation zheavy emphasis on the quantitative dimension zno strategic recommendations formulated to be adopted by the agency’s governing board zexercise not formally inserted in the agency’s strategy formulation

18. Discussion and policy implications zThe results have not yet been used for policy purposes zOveremphasis on the quantitative dimension and quite general recommendations zThe “gold mine” of information was not adequately exploited zA missed opportunity? Why? yEvaluation is not formally embedded in the CNR decision making process yCNR is a sticky organisation yFinancial situation yReforms over the last few years

19. Thank you for your attention Questions?