1. Meta-evaluation of the Performance Evaluation System of Public Research Institutes in Korea Chan Goo Yi (Pukyong National University, Korea ; changoo@pknu.ac.kr)changoo@pknu.ac.kr Jang Jae Lee (Korea Institute of Science and Technology Evaluation and Planning (KISTEP) ; jjlee@kistep.re.kr)jjlee@kistep.re.kr Yong Soo Hwang (Korea Science and Technology Policy Institute (STEPI) ; yshwang@stepi.re.kr) yshwang@stepi.re.kr AEA Evaluation 2009 Conference Nov. 11-14, Orlando, Florida

2. 2 Contents 1. Introduction 2. Research Method and Framework 3. Korean Research Institute Evaluation 4. Current Practise & Meta-evaluation 5. Discussion for Future Development 6. Conclusion 7. References

3. 3 1. Introduction Background - In Korea, evaluation system of research institutes introduced in 1999, and transferred into the performance evaluation system in 2005 - Arguments among various stakeholders such as CEO, researchers and evaluation panel, whether evaluation can contribute the quality enhancement of R&D results the development of management system

4. 4 1. Introduction (con.) Research Purpose - To meta-evaluate (1) the rationality of evaluation system itself and (2) the appropriateness of its current practise - To discuss policy alternatives for development of the evaluation system itself and its implementation

5. 5 2. Research Method & Framework Research Method - In-depth interview with 109 stakeholders conducted between March and May 2008 99 internal stakeholders : CEOs(10), managers in administrative dept.(27), principal investigators(30), researchers(32) 10 external stakeholders : evaluation panels from university(4), industry(3) and public research institute(3) - Meta- evaluation approach : Evaluation of evaluation system and practical process

6. 6 2. Research Method & Framework (con.) Research Framework : Components of In-depth Interview & Meta-evaluation Implementation(4) Paradigm(2) Purpose Object Utilization(2) Impact Type Panel Interval Method Indicator

7. 7 3. Korean Research Institute Evaluation Brief History of Institute Evaluation System - Introduction period (1999-2002) similar system operated among research councils research achievements < management achievements - Diversification period (2002-2005) improved representing characteristics of individual member research institutes research achievements ≒ management achievements - Development period (2006-current) transferred performance evaluation system focusing rather outcome or impact than output research achievements > management achievements

8. 8 3. Korean Research Institute Evaluation (con.) R&D Governance in Korea : 3 Research Councils System - Korea Research Council for Fundamental S&T (KRCF) 13 member research institute under the Ministry of Education, Science & Technology - Korea Research Council for Industrial S&T (ISTK) 13 member research institutes under the Ministry of Knowledge Economy - National Research Councils for Economic, Humanities and Social Sciences (NRCS) 23 member research institutes under the Office of the Prime Minister

9. 9 3. Korean Research Institute Evaluation (con.) Framework of Current Evaluation System FieldPartItemIndicator Research Achieve. (70%) Individual Performance Goals (50%) Objective 1 Sub Objective 1.1 Sub Objective 1.2 Objective 5 Sub Objective 5.1 Common Items (7) Elective Items (10) Indicator 1.1.1 Indicator 1.1.2 Indicator 1.2.1 Indicator 1.2.2 Indicator 5.1.1 Selected by each institutes autonomously Comprehensive Performance Goals (20%) Manage. Achieve. (30%) Responsibility & Innovation R&D & Resource Management 3 indicators

10. 10 4. Current Practise & Meta-evaluation (1) Evaluation Paradigm Evaluation Purpose - Formal : future development strategies, research performance enhancement, accountability, program/management improvement, knowledge transfer etc. - Actual : R&D program/project development, internal management system improvement (Meta-evaluation) - Incompatible between formal purposes and actual - Focused rather short-perspective evaluation purposes

11. 11 4. Current Practise & Meta-evalu. (con.) Evaluation Object - Research Achieve.(70%)vs.Management A.(30%) Each field divided into sub parts and items - Covering both ‘basic R&D program’(grant fund) and ‘national R&D program’(competition fund) (Meta-evaluation) - Too many evaluation objects Not differentiating among evaluation objects - Absent of essential objects for core evaluation purposes such as ‘future development strategy’ - Lacks of consensus for core evaluation objects among stake-holders

12. 12 4. Current Practise & Meta-evalu. (con.) (2) Evaluation Implementation Evaluation Panel - Panel from university, research institute and industry, & comprising all domestic experts Research : individual panel for each institute Management : common panel for all institutes (Meta-evaluation) - Panel members’ professionalism limited Professionals in sub performance goals of R&D project rather than peer reviewer or upper performance goal in R&D program - A few lacks of considering international excellency

13. 13 4. Current Practise & Meta-evalu. (con.) Evaluation Interval - Until 2007 : evaluating all institutes every year - From 2008 : separating research achievement and management one Research A. : 3 years ; Management A. : 1 year (Meta-evaluation) - Too often evaluated and burden to institutions Main factor negative affecting other components such as evaluation purpose, object, utilization - Resulted in more focusing the visible and short term outputs rather than long term and comprehensive outcomes or impact

14. 14 4. Current Practise & Meta-evalu. (con.) Evaluation Method - External panel conducts full evaluation process - Evaluating 4/5 member institutes as one group in a comparative perspective with others - Combination of the ‘review of performance report’ submitted by each institute and the ‘site visit’ for four or five hours in individual institute (Meta-evaluation) - Focused on literature review rather than site visit - Evaluation in a comparative/relative perspective not an absolute one, in particular for the research performance

15. 15 4. Current Practise & Meta-evalu. (con.) Evaluation Indicator : Focused on Research - Each Program(50%) vs. Comprehensive R&D(20%) Performance goals -> Objects -> Indicators - Each institute suggests their own goals/indicators Evaluating performance level targeted in advance (Meta-evaluation) - Disconnection of performance goal and core R&D activities - Indicator pool limited for comprehensive R&D performance goal - Indicators more fit for research program rather than development one

16. 16 4. Current Practise & Meta-evalu. (con.) (3) Evaluation Utilization Evaluation Impact - Evaluation findings directly fed back to the basic R&D(grant fund), but indirectly did to national R&D(competition fund) - Feed back to internal management system (Meta-evaluation) - Evaluation findings more effecting management system rather than R&D management process - Lacks of consensus of impact among stakeholders Insiders : low, Outsiders : relatively high

17. 17 4. Current Practise & Meta-evalu. (con.) Type of Evaluation Utilization - Formal : future development strategy, R&D/ management improvement, R&D prioritization and budget allocation, best practise, CEO ’ s annual pay adjustment, policy suggestion - Actual : budget reallocation of basic R&D, adjustment of CEO ’ s annual payment, best practise (Meta-evaluation) - Limited and confined evaluation utilization - Instrumental utilization is further actual type than conceptual one

18. 18 5. Discussion for Future Development Outline of Discussions for Developments - Institutional Approach (IA) Development/amendment of evaluation system itself and related systems at the level of research councils or the government Long-term and institution-based perspective - Operational Approach (OA) Improvement of evaluation practise and process under the current system short-term and operation-based perspective

19. 19 5. Discussion for Future Development (con.) (1) Evaluation Paradigm Evaluation Purpose - Need to transfer from internal responsibility to external one (OA) Because current system has more contributed internal responsibility/ management rather than external one - More focusing responsibility for external stakeholders (OA) In particular, responsibility for citizen

20. 20 5. Discussion for Future Development (con.) Evaluation Object - Rearrangement of current evaluation object (IA) Whether competitive national R&D program be included or not? - Re-setting the evaluation objects in the longer perspective, such as; (IA) Long-term vision/strategy of the institution Future potentials and R&D infrastructures Risk management in related public sector

21. 21 5. Discussion for Future Development (con.) (2) Evaluation Implementation Evaluation Panel - Setting up individual/independent panel for each institution, from single panel for all (IA) - Enhancement of professionalism of panel (OA) Recruiting more field experts like as industries Extension of job term from 2 to 4/5 years Career management system of panel members - Adoption of international experts panel (OA)

22. 22 5. Discussion for Future Development (con.) Evaluation Interval - Extension of evaluation interval from 1 year to 3 to 5 years (IA) Linkage with CEO’s term in office Same or different evaluation interval between research results and management results Consideration of characteristics of research fields; such as emerging technology vs. long- term basic science

23. 23 5. Discussion for Future Development (con.) Evaluation Method - Enforcement of evaluation method in the way of absolute perspective (IA) In particular, evaluation of research results - Focusing in-depth review in research lab (IA) Extending evaluation period to 3 to 4 days Reviewing first-hand materials(research note) Interview and discuss with researchers - Introduction of cross-cutting review Among related institutions/organizations in public sector (IA)

24. 24 5. Discussion for Future Development (con.) Evaluation Indicator - Improvement of method of establishing performance goals/indicators (OA) Closer connectivity between performance goal/indicators and core R&D activities More changeable and creative goals/indicators - High linkage between performance goals and internal performance management system (OA) For example, BSC, MBO, ISO 9001, KM etc - Increase of indicator pools for comprehensive performance goals (OA)

25. 25 5. Discussion for Future Development (con.) (3) Evaluation Utilization Evaluation Impact - Enforcement of feed back system of evaluation findings to R&D management (OA) High collaboration with related other agencies for national R&D management - Extension of scope and target group of evaluation utilization (OA) From top manager to all employees

26. 26 5. Discussion for Future Development (con.) Type of Evaluation Utilization - Activation of long-term and conceptual evaluation utilization (OA) Setting up vision/mission Planning R&D strategy Disseminating the best practise Producing policy information/knowledge etc - Informing the multiple type of evaluation utilization to all stakeholders (OA) In particular, in-site researchers

27. 27 6. Conclusion Summary of the research - In basic, both internal and external stakeholders consider the current system somewhat useful for R&D management and organizational management - They also suggest policy alternatives for the development of certain components of evaluation system and its current practise Some are the system itself, others are the common limits of evaluation system of public sector in general in Korea

28. 28 6. Conclusion (con.) Implication - Policy alternatives for developments of evaluation system for research institutes in terms of system itself and current practise - Policy knowledge/ information for analysis and re-establishment of the governance of public research institutes

29. 29 6. Conclusion (con.) Limitations and Further Works - Discussion of policy alternatives a little bit lacks specific and detailed matters in some individual components Scope and depth of policy alternatives in certain components is too broad - Need for more detailed action plan in each evaluation component, based on the findings of this meta-evaluation/research

30. 30 7. References Further Information for Korean S&T and Evaluation System - National Science and Technology Council (http://www.nstc.go.kr)http://www.nstc.go.kr - Ministry of Education and Science and Technology (http://www.mest.go.kr)http://www.mest.go.kr - Korea Research Council for Fundamental S&T (http://www.krcf.re.kr)http://www.krcf.re.kr - Korea Research Council for Industrial S&T (http://www.istk.re.kr)http://www.istk.re.kr - Korea Institute of S&T Evaluation and Planning (http://www.kistep.re.kr)http://www.kistep.re.kr