1. IN-DEPTH EVALUATION OF R&D PROGRAM IN KOREA Seung Jun Yoo, Boo-jong Gill, Woo Chul Chai

2. Contents 1. Overview of Public R&D Program Evaluation 2. Objective 3. Procedure 4. Methodologies 5. Utilization of Evaluation Results 6. Challenges and Discussion 2

3. Overview of Public R&D Program Evaluation 1 - Players *NSTC (National Science & Technology Council), MOSF(Ministry of Strategy and Finance) MOSF …… MEST MOEMKE MIFAFF MW KISTEP (Evaluator) Evaluation Supporting Groups R&D programs of each ministry Taking Charge of R&D Evaluation and Budget Allocation Agency NSTC Report 3

4. Overview of Public R&D Program Evaluation 2 - R&D program management process R&D BudgetSurvey/Analysis Evaluation Programs/Projects implemented In-depth Corrections* Feedback Evaluation strategy & Data collection Self → Meta Recommendations to program ministries Evaluation process Correction process 4

5. Overview of Public R&D Program Evaluation 3 - Overview of Public Finance Program General Public Finance Programs SOC, Agriculture, Health/Welfare, National Defense, Industry/Energy, etc. Informatization Programs Information Infrastructure Information Utilization & Impact R&D Programs Science and Technology R&D Human Resources Development, Infrastructure, etc. 5

6. 6

7. Overview of Public R&D Program Evaluation 4 - In-depth evaluation & Self/Meta evaluation R&D Program Evaluation In-depth Evaluation Self → Meta Evaluation - ~ 10 programs with evaluation issues - logic model, evaluation questions, in- depth analysis, communications, recommendations, coordination, etc. - depends on evaluation questions - 1/3 out of all programs (70 programs out of 207 programs, ‘09) - self evaluation by each ministry meta evaluation by MOSF/KISTEP - depend on indicators with weight 7

8. Objective of the Evaluation To increase the efficiency and effectiveness : find out and diagnose the problems at all aspects : improve the program by applying evaluation results 8

9. Procedure 1 - 4(5) steps 1. Evaluation Strategy logic model, evaluation questions data collection plan, methodologies 2. Analyze & Interpret collect data data analysis & interpret 3. Results coordination interim & final results interview & coordination 4. Utilization of the results 4 types management action plan 0. Selecting target program 9

10. Procedure 2 - 7-month schedule (depends on the cases) - (month 0) : selected by selection committee based on special issues, etc. ¶ In-depth evaluation procedure for selected program(s) - month 1 : form evaluation group, gather program(s) data, study target R&D program(s), find major evaluation questions - month 2 : develop logic model and methodologies - month 3/4 : perform in-depth analysis (relevance, efficiency, effectiveness, program design & delivery, etc) 10

11. Procedure 3 - month 5 : interview (researchers, program managers, etc.) - month 5 : report interim evaluation result (MOSF, ministries) - month 6 : report final evaluation result & recommendations Large program : ~ 10 months Specific needs for short-term : 2 ~ 3 months (specify the needs → perform evaluation) 11

12. Methodologies Types of Evaluation Methods SatisfactionImportance-Performance Analysis (IPA) Qualities (Paper, Patent, etc.) Paper : Impact Factor, Times Cited, etc. Patent : Technology Transfer, Value, Citation, Survival Analysis, etc. Policy, StrategySystem Dynamics, etc. Portfolio (Investment, etc. ) BCG Growth Share Matrix, GE/McKinsey Matrix, etc. 12

13. Methodologies_ IPA Performance (high) Performance (low) Concentrate Here!Keep Up Good Work! Low Priority!Possible Overkill! Importance (low) Importance (high) Source: Martilla, J. & James, J. C.(1977), Importance-perfomance analysis. Journal of Marketing, 41(1) : 78 13

14. Methodologies _Survival Analysis Survival Analysis Idea/Pilot Study → Project → Paper ↔ Patent → Technology Transfer Modified from: DeVol, R. & Bedroussian, A.(2006), Mind to Market: A Global Analysis of University Biotechnology Transfer and Commercialization. Milken Institute Following Study 14

15. Methodologies_ System Dynamics Source: Ahn, N. (1999), A system dynamics model of a large R&D program, MIT Press Yoo, S. et al. (2009), In-depth Evaluation of Health & Medical R&D Program in Korea, KISTEP 15

16. Methodologies_ Portfolio Analysis 16

17. Utilization of Evaluation Results Types of Correction Management Action Improve Program Delivery System Correct the inefficient system in program delivery to achieve the goal(s) of the program more efficiently Coordinate Budget Allocation Increase or cut the budget corresponding to insufficient or unnecessary investment, respectively to achieve the goal(s) of the program more efficiently Improve Research Environment Improve regulation or act to achieve the goal(s) of the program more efficiently Consult Program Planning Provide better ideas or guidelines to plan new or following programs 17

18. Challenges and Discussion Evaluation as R&D Management Tool Efficient budgeting Improve goal achievement Self/Meta In-depth Measure the achievement according to performance plan Diagnose problems and correct to improve the efficiency/effectiveness Efficient R&D Management 18

19. Challenges and Discussion UCI concept among stakeholders : Understanding – Change - Improvement is important for raising the accountability (responsibility + acceptability) of evaluation : Understanding = communication with the facts : Change = 4 types of corrections : Improvement = efficiency & effectiveness 19

20. Thank you! Seung Jun Yoo, PhD biojun@kistep.re.kr www.kistep.re.kr