1. Objective: Estimating Government R&D Program Efficiencies as a part of our assistants to the Government R&D Budget Compilation Process Programs are units, comprising the Government R&D budget Korean Government adopted Digital Budget & Accounting System, Top-down Budgeting, Performance Based Budgeting several years ago. Measuring Program Efficiency assists the budgeting process. - Efficiency is used in Program Performance Evaluation - Yr n-1 Performance of a Program affects its Budget Size in Year n+1 in the Top-down budget process. Objective: Estimating Government R&D Program Efficiencies as a part of our assistants to the Government R&D Budget Compilation Process Programs are units, comprising the Government R&D budget Korean Government adopted Digital Budget & Accounting System, Top-down Budgeting, Performance Based Budgeting several years ago. Measuring Program Efficiency assists the budgeting process. - Efficiency is used in Program Performance Evaluation - Yr n-1 Performance of a Program affects its Budget Size in Year n+1 in the Top-down budget process. Our Challenges prior to Measuring Program Efficiency (PE) 1/3 of R&D Programs get In-depth Performance Evaluation per annum  Strategy : Tracking project performances and aggregating these statistics by program Program budgeting for Yr n+1 with performance information from Yr n-1 while programs end, merge, regroup, and are reorganized  Strategy: Human Labor to match projects of Yr n-1 to appropriate programs in Yr n+1 Collect Project Statistics and Obtain Proper Input & Output Measures by Program, and Assign Appropriate Categories for each Program  Strategy: Use Experts to Validate these statistics Our Challenges prior to Measuring Program Efficiency (PE) 1/3 of R&D Programs get In-depth Performance Evaluation per annum  Strategy : Tracking project performances and aggregating these statistics by program Program budgeting for Yr n+1 with performance information from Yr n-1 while programs end, merge, regroup, and are reorganized  Strategy: Human Labor to match projects of Yr n-1 to appropriate programs in Yr n+1 Collect Project Statistics and Obtain Proper Input & Output Measures by Program, and Assign Appropriate Categories for each Program  Strategy: Use Experts to Validate these statistics Computing Efficiencies and Finding Efficient Programs/Program Types Used Input Oriented BCC Model (Banker et al, 1984) Tobit Regression Analysis to find Factors(Program Types) of efficiency Computing Efficiencies and Finding Efficient Programs/Program Types Used Input Oriented BCC Model (Banker et al, 1984) Tobit Regression Analysis to find Factors(Program Types) of efficiency This work was supported by the Korean Ministry of Strategy and Finance(MoSF) Use of Efficiency Measure and Tobit Regression Analysis in Budget Process [We can suggest] 1.[…] officers in charge of R&D programs efficient program planning and budgeting strategies 2.[…] officers in NSTC and MoSF increase/reduce the budget size of relatively efficient/inefficient R&D programs in the budget allocation and compilation process (A Program with good performance gets budget increment whereas one with poor performance gets deduction) 3.[…] program planning schemes and adjust the program budget size accordingly as a part of policy analysis. 4.[…] the midterm R&D budget projection as a part of settling new budget systems Use of Efficiency Measure and Tobit Regression Analysis in Budget Process [We can suggest] 1.[…] officers in charge of R&D programs efficient program planning and budgeting strategies 2.[…] officers in NSTC and MoSF increase/reduce the budget size of relatively efficient/inefficient R&D programs in the budget allocation and compilation process (A Program with good performance gets budget increment whereas one with poor performance gets deduction) 3.[…] program planning schemes and adjust the program budget size accordingly as a part of policy analysis. 4.[…] the midterm R&D budget projection as a part of settling new budget systems Shortcomings of DEA from the statistical inference perspective : Statistical Uncertainties are not take into account. Curse of Dimensionality: the more input/output measures, the more DMUs have efficiency 1. Efficiency measures are results of a Nonparametric Method, valued between 0 and 1, difficult to link regression analysis methods, assuming some distributional properties (In quest of adopting nonparametric regression analysis schemes while using the efficiency measures as response). Shortcomings of DEA from the statistical inference perspective : Statistical Uncertainties are not take into account. Curse of Dimensionality: the more input/output measures, the more DMUs have efficiency 1. Efficiency measures are results of a Nonparametric Method, valued between 0 and 1, difficult to link regression analysis methods, assuming some distributional properties (In quest of adopting nonparametric regression analysis schemes while using the efficiency measures as response). Statistical Challenges : Data Matching & Classification & Statistical Inferences Challenge 1: matching projects in Yr 2008-2010 to a program with budget request in Yr 2012 Challenge 2: obtaining coherent statistics from this matching Challenge 3: classifying programs into various categories based on project information and program plans  Strategy: Experts reduce errors in measuring inputs/outputs for more accurate efficiency measures Challenge 4: measuring efficiency with multiple inputs and outputs  Strategy: Data Envelop Analysis (DEA) has been employed (Farrell, 1957; Banker et al, 1984) Challenge 5: incorporating statistical and systematical uncertainties - survey data have various uncertainties - inference on groups of efficiency measures based on different types of outputs, results in different error structures/distributions (basic research does not produce patents and royalties).  On going work to tackle this statistical challenge Statistical Challenges : Data Matching & Classification & Statistical Inferences Challenge 1: matching projects in Yr 2008-2010 to a program with budget request in Yr 2012 Challenge 2: obtaining coherent statistics from this matching Challenge 3: classifying programs into various categories based on project information and program plans  Strategy: Experts reduce errors in measuring inputs/outputs for more accurate efficiency measures Challenge 4: measuring efficiency with multiple inputs and outputs  Strategy: Data Envelop Analysis (DEA) has been employed (Farrell, 1957; Banker et al, 1984) Challenge 5: incorporating statistical and systematical uncertainties - survey data have various uncertainties - inference on groups of efficiency measures based on different types of outputs, results in different error structures/distributions (basic research does not produce patents and royalties).  On going work to tackle this statistical challenge References: written in Korean are omitted Banker, Charnes & Cooper (1984) Management Science vol.30(9) pp. 1078 – Farrell (1957) with discussions. J. of Royal Statistical Society A vol. 120 pp.11- Shah & Shen (2007) “A Primer on Performance Budgeting,” World Bank References: written in Korean are omitted Banker, Charnes & Cooper (1984) Management Science vol.30(9) pp. 1078 – Farrell (1957) with discussions. J. of Royal Statistical Society A vol. 120 pp.11- Shah & Shen (2007) “A Primer on Performance Budgeting,” World Bank Brief R&D Budget Process for FY 2012 [Ministries submit] proposals on their priorities in R&D (Oct. 31, 2010) […] midterm R&D program plans with estimated budget (Jan. 31, 2012) Ministry gets R&D budget ceiling by the midterm plans (Apr. 30, 2012) […] detail budget requests by program (Jun. 30, 2012). Budget (Re)Allocation by NSTC (Jul. 31, 2012) NSTC finalizes the R&D budget deliberation (Sept. 15, 2012) MoSF submits the budget deliberation to the National Assembly (Oct. 2, 2012)  We (KISTEP) make contributions to these yellow steps Brief R&D Budget Process for FY 2012 [Ministries submit] proposals on their priorities in R&D (Oct. 31, 2010) […] midterm R&D program plans with estimated budget (Jan. 31, 2012) Ministry gets R&D budget ceiling by the midterm plans (Apr. 30, 2012) […] detail budget requests by program (Jun. 30, 2012). Budget (Re)Allocation by NSTC (Jul. 31, 2012) NSTC finalizes the R&D budget deliberation (Sept. 15, 2012) MoSF submits the budget deliberation to the National Assembly (Oct. 2, 2012)  We (KISTEP) make contributions to these yellow steps On Data Envelop Analysis(DEA) to measure Program Efficiency: Program Input: Expenditure in 2008-10 - only coherent input measure across R&D programs Program Outputs by the Program Type - Basic Research(BR): No. of well received Papers (R 2 nIF ≥ 1.0) [=Papers] - Fundamental Research(FR): No. of Registered Patents [=Patents] & Papers - Applied and Development Research(ADR) : Patents, Tech Transfer Fees, Royalties, and Sales Increases - SME R&D aids(SME): Same as ADR outputs These Programs are Decision Making Units (DMUs) for DEA. On Data Envelop Analysis(DEA) to measure Program Efficiency: Program Input: Expenditure in 2008-10 - only coherent input measure across R&D programs Program Outputs by the Program Type - Basic Research(BR): No. of well received Papers (R 2 nIF ≥ 1.0) [=Papers] - Fundamental Research(FR): No. of Registered Patents [=Patents] & Papers - Applied and Development Research(ADR) : Patents, Tech Transfer Fees, Royalties, and Sales Increases - SME R&D aids(SME): Same as ADR outputs These Programs are Decision Making Units (DMUs) for DEA. Year2008200920102011 No. Ministries2930 No. Programs486473483 - No. Projects374493947139179 - R&D Expenditure (trillion won) 10,99412,41513,683 14,890 (budget) KISTEP currently survey and analyze all Gov. funded R&D projects including their performances during 2011. Projects are funded through appropriate Programs during each fiscal yr. Scientific Papers and Patents are the examples of project outputs By aggregating these outputs and inputs, project performance statistics are analyzed at KISTEP These statistics are not fully used for program budgeting KISTEP currently survey and analyze all Gov. funded R&D projects including their performances during 2011. Projects are funded through appropriate Programs during each fiscal yr. Scientific Papers and Patents are the examples of project outputs By aggregating these outputs and inputs, project performance statistics are analyzed at KISTEP These statistics are not fully used for program budgeting Performance based Budgeting : Allocation of funds to achieve programmatic goals and objectives as well as some indication or measurement of work, efficiency, and/or effectiveness Links between performance information & budget allocation Measures: Inputs, Outputs, Effectiveness, Efficiency, Workloads Efficiency = Output/Input Performance based Budgeting : Allocation of funds to achieve programmatic goals and objectives as well as some indication or measurement of work, efficiency, and/or effectiveness Links between performance information & budget allocation Measures: Inputs, Outputs, Effectiveness, Efficiency, Workloads Efficiency = Output/Input 1.1 tr. won ~ $1 mil R&D TypesBRFRADRSME No. of DMUs23345419 Efficiency Dist’nExtreme bipolarMild bipolarExtreme bipolar MinistriesAll MESTMEST > MKEMKE > all othersSMBA > MKE bipolar: many low PE while many PE close to 1./Note we did not use bimodal Results from Tobit Regression Analysis is that Programs comprised of 1.bottom-up projects are more efficient than those of top-down projects 2.individual projects are more efficient than those of group projects & Preplanning shows more efficient than Programs without preplanning. Results from Tobit Regression Analysis is that Programs comprised of 1.bottom-up projects are more efficient than those of top-down projects 2.individual projects are more efficient than those of group projects & Preplanning shows more efficient than Programs without preplanning.