1. China’s National R&D Program and Countries Comparison
Performance Assessment of Government-funded R&D
China’s National R&D Program and Countries Comparison
Prof. Yan FANG
China National Center for
Science and Technology Evaluation
ITEP Symposium, Korea Oct. 20, 2004

2. Main Topics China National Program and Government Input
Study of Performance Assessment of Government-funded R&D
Practice and Comparative Review for Countries Evaluation

3. China National program and Government Input
The Aims of National R&D Programs
To make direct R&D input by government;
To guide R&D input on demand of the State;
To build up the environment for R&D.
The input mainly distributed to MOST, NNSF and CAS by Central Government.

4. Distribution and Input of National R&D Programs
Unit: 100 Million RMB
Start-up
Implement Agency
Key Technologies R&D Program
1983
MOST 15 35
45.2 50
National Basic Research Program
(973 Program)
1997.3 — 12 21
till (2003)
National High-Tech R&D Program (863 Programs)
1986.3 59
150
National Natural Science Fund
1986.2
NNSF
5.72
15.88
44.7
100
Knowledge Innovation Program
1998.6
CAS 54
200
This slide show the China priority national programs and input in different period.
You will know the distribution of national program in 3 sectors mentioned above.

5. R&D Program Charged by MOST (2003)
‘3 plus 2’ national R&D program system
total projects in 9633
3 core R&D programs (1662 projects)
863 program (high-tech) (1296 projects 13.4%)
Key technology program( 339 projects, 3.5%)
973 program (basic research) (27projects, 0.3%)
2 group programs
Condition construction program (818, 3.5%)
Environment construction (R&D industrialization) (7153 projects, 74.3%)

6. The tendency of total R&D input from 1991-2002

7. Total Input of R&D The total input increased 10 times since 1991;
The total amount of 153 billion RMB in 2003
Ratio of input/GDP increasing from 0.7%(1991) to 1.23%(2002) and into a growth period; in 2003 , increasing to 1.31%, increasing rate is about 20%.
Same period, government appropriation/ expenditure is unchanged around 4%.
The increasing of R&D input result from government advocacy on hi-tech strategy and economy development.

8. The change of total R&D input from 1991-2002
Unit: 100 million RMB
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
R&D input (Yuan)
150.8
209.8
256.2
309.8
348.7
404.5
509.2
551.1
678.9
895.7
1042.5
1287.6
Increase (%)
28.9
6.6
0.9
0.6
9.5
24.9
10.9
26.0
17.9
16.4
23.5
R&D/GDP
(%)
0.70
0.79
0.74
0.66
0.60
0.68
0.83
1.0
1.09
1.23
This is amount of total R&d input in last decade for your reference.

9. Investing body has changed
From government-dominated to dual bodies of government and company;
Government cut back from 79% to 33.4% ( );
Company increase from 18%-38.9%( )
till 2003, company input up to 62.4%.
In addition to the change of R&D input, the role of investing body is also changed. In total R&D input, the percentage of government input cut back from 79% of 1985 to 33% of 2000, vice versa, the percentage of company input is increasing to 39%.

10. Investing body has changed%
government
company
others
1985
79.0
18.0
3.0
1987
60.9
39.1 －
1988
56.6
43.4
1990
54.9
23.4
21.7
1994
57.7
32.4
9.9
1995
50.0
35.0
15.0
2000
33.4
38.9
This table and picture reveal what I say in detail.

11. R&D expenditure by sectors in2003
R&D institute: 38.5%
Company: 62.4%
University: 10.5%
Others: 1.2%

12. Ratio of R&D cost/sale for Chinese company
The Ration of R&D cost/Sale for Company Unit: 100 million
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
R&D
cost
(A)
58.6
76.1
95.2
122.0
141.7
160.5
188.3
197.1
249.9
353.6
442.3
560.2
Sale
(B)
11937
15235
19077
23904
30831
33553
36297
37463
41912
49847
58511
67452
A/B
(%)
0.491
0.500
0.499
0.510
0.460
0.478
0.519
0.526
0.596
0.709
0.756
0.831
This is the ratio of R&D cost to saleroom in large-middle size Chinese company.

13. Ratio of R&D cost/sale for Chinese company

14. Why government-funded R&D performance should be assessed
Government as the key player in R&D resource allocation;
To address the problems of low efficiency in R&D investment;
To conduct R&D activities from an outcome-oriented to performance-oriented perspective
To steer the supporting R&D role in the social and economic development.

15. Why government-funded R&D performance should be assessed
To receive direction from taxpayers and report to them the information associated with returns, effectiveness and impacts in government R&D investments.
To enable government affairs to be more open and transparent to the public.
The public resources can be optimized and the flow can be monitored and regulated.

16. State of the art of R&D performance assessment
Specific context of each country makes the conduct of the definition and methodology different.
Common ground to ensure the pursuit:
To provide a systematic institution
To establish professional organizations
A typical example - GPRA formulated by US Congress in 1993.

17. The pre-study for R&D performance assessment
Purpose:
To extend R&D evaluation from internal management to serve public financing.
To shift the insight from a program process management focus to an outcome-oriented focus.
To ensure the optimization of public resources.

18. The pre-study for R&D performance assessment
Evaluation body:
performance assessment needs a diversified evaluation group composed of government agencies, professional evaluation organizations and social communities, especially the stakeholders.

19. The pre-study for R&D performance assessment
Indicator system:
To integrate social, economic and environmental development strategies;
To show that performance of government accountability is in place;
To focus on degrees of public satisfaction.

20. The pre-study for R&D performance assessment
Resulting feedback:
A feedback system for utilization of evaluation results.
The annual government budget linked to the results of performance assessment.
Transparent results accessed by public hearings for hearing opinion and establishing monitoring mechanisms.

21. The pre-study for R&D performance assessment
Role of performance assessment:
A need for “buildup learning organization” over and above the role of monitoring, selecting and phasing-out processes.
To seek new solutions, improve performance, effectiveness, service quality and a higher degree of public satisfaction.

22. The pre-study for R&D performance assessment
Institutional design:
Faced with a challenge to China’s “power-centered” cultural value.
To provide a legal guarantee for institutional assessment.
To formulate enforcement policies, such as
whom to report to for the result
inter-relationship between affiliated evaluation entities and government administration.

23. The pre-study for R&D performance assessment
Pre-study Approaches
Concept buildup
Technology approach
Institutional arrangement
Implementation mechanism

24. Comparison Study among Japan, Korea and China
Share the same circle of cultural origin;
Similar institutional government functions;
Evolution of R&D evaluation in same stage;
Take examples of METI, KISTEP and NCSTE

25. Legal Status of R&D Evaluation
Japan:
1997: “State Implementing Evaluation Guidelines for R&D Programs” by METI
2001: “Act of Policy Implementation Evaluation for Government Administrations” by Cabinet.
2002, “Evaluation Guidelines for Public R&D” by Prime Minister
Consequently, the R&D evaluation in Japan ultimately entered into legislative framework.

26. Legal Status of R&D Evaluation
R.O.Korea: (e.g. KISTEP)
1999: KISTEP follows the tasks of evaluation and reviews national R&D programs.
2001: evaluation function reinforced with “Basic Law of Korean S&T” takes effect;
the R&D evaluation was also developed by other government agencies and institutes, such as ITEP.

27. Legal Status of R&D Evaluation
China:
1994: initial exploitation of S&T evaluation
1997: NCSTE was officially established.
Present: no legislative regulations
Internal documents issued by MOST
Tentative Management Measures on S&T Evaluation (2000).
Uniform Standard for S&T Evaluation

28. Comments on China Situation
Some government agencies have started to introduce the evaluation system.
The system was, to a certain extent, used for
placing a spotlight on management,
a tool to avoid responsibilities
an approach to cope with inquiries from outside.
The evaluation culture faces a challenge to the power-centered culture.
Government legislation is the fundamental guarantee.

29. Purpose and Function of R&D Evaluation
Japan:
to reflect the social and economic demand for R&D;
to improve efficiency and impact of R&D programs;
to get support and understanding of the public;
to optimize the distribution efficiency of resources;
to reform and improve R&D management.

30. Purpose and Function of R&D Evaluation
R.O.Korea:
public pay more attention given to the rational distribution of scarce resources.
review policies to improve the accountability and transparency of government;
reduce distortions in the process of government R&D;
KISTEP’s functions:
to strengthen evaluation and planning of R&D programs;
to review budget of National R&D programs
seen as a useful coordinating instrument.

31. Purpose and Function of R&D Evaluation
China:
new breakthroughs for S&T planning management;
to improve quality and efficiency of decision-making a pressing demand.
advocating an evaluation tool integrated with macro management to facilitate efficiency / effectiveness.
S&T evaluation, as a tool, has played a role in allocating resources within MOST.

32. Comments Internal evaluation External evaluation
The different evaluation dimensions need different institutional systematic design.
The system refers to
the transparency of the evaluation result,
the nature of independence of the evaluation body
neutrality of the evaluation process.

33. Evaluation Criteria for R&D (METI)
R&D Program Criteria
alignment with national goals and policies;
the relevance of R&D program objectives;
availability of R&D operating system;
importance of program outcomes vs. program objectives;
commercial value of R&D outcome.
R&D Projects Criteria
a suitable reason for government investment;
prospects for and planning of the project;
management capability of R&D institutes;
forecast of completed R&D outcomes;
ripple effect and social economic impact.

34. Evaluation Criteria for R&D (KISTEP)
National R&D program:
relevance of program with development objectives;
program implementation efficiency;
availability of program operation;
adjustment of scope of program;
the possibility to integrate with other programs.
For R&D projects
to identify whether the project is out-of-date, unwanted, excessive or can be merged with other projects.

35. Evaluation Criteria for R&D (NCSTE)
Policy criteria : R&D activities vs. development policy;
Objective criteria: relevance of program vs. R&D objective;
Performance criteria: outputs /outcomes vs. objective;
Efficiency criteria: can outputs be maximized by minimum inputs?
Impact criteria: long term positive / negative impact from R&D;
innovation criteria: typology/degree of technology innovation;
Productivity criteria: can R&D facilitate the productivity?
Equity criteria: whether R&D leads to unequal distribution of resources, interest or costs.

36. Comment Different management brings different functions:
METI’s concentrates on industrial and commercial application and social and economic ripple effects.
KISTEP’s provides annual support to budget alignment of R&D programs execution /arrangement.
NCSTE provides specific design for evaluation framework according to The Uniform Standard for S&T Evaluation, which is a minor difference compared with the other two.

37. Classification of evaluation targets
Japan (METI)
R&D system structure, program, project (most application), R&D field and organization;
Evaluation framework is more systematic.
Korea (KISTEP) – Two Departments
One assists NSTC in charge of R&D budget allocation among different government agencies.
Other focuses on evaluation for MOST resource only.
The overall coordinating mechanism / procedure: evaluation --- budget review --- resource allocation.
This slide show the classification of evaluation targets in three agency.

38. Classification of evaluation targets
China (NCSTE)
Practically, the main R&D evaluation covers program, project and institute.
Anti-evaluations for projects account for the majority
Post evaluation and follow up evaluation still in the minority (863 program and innovation) .
A lack of understanding of establishing a learning tool and improving government responsibility.
Pragmatism is the typical guiding principle for evaluation management.

39. Comment
In practice, the targets mainly focused on project evaluation are due to quantity and methods developed.
Framework and continuity in METI’s evaluation is more apparent and systematic.
KISTEP’s dual roles of both internal and external evaluation provide a demonstration on performance assessment.
NCSTE needs to improve the overall R&D evaluation system by integrating internal management with performance assessment.

40. Future directions for China R&D evaluation
Push forward national legislative process to put S&T evaluation in place;
Develop a comprehensive, continuous and effective S&T evaluation system;
Intensify fundamental studies of R&D evaluation;
modification of “Uniform Standard for S&T Evaluation”;
improvement of R&D evaluation databank;
Continuously advocate new concepts of evaluation and facilitate training programs for evaluation institutes;

41. Future directions for China R&D evaluation
Establish global networks through international cooperation and workshops in China;
Develop new methodologies to improve the quality of evaluation;
Establish feedback system for effective allocation of government resources;
Improve transparency of evaluation results and take responsibility on behalf of taxpayers.

42. Thank You Prof. FANG Yan Tel: 86 10 88232484 Fax: 86 10 88232615
Address: P.O.Box 3814 Beijing, P.R.China,