1. Review of Quantitative Monitoring System of Chinese Academy of Sciences ZHENG Haijun, GUAN Zhongcheng, WANG Biaoxiang, WU Jianmei Management Innovation and Evaluation Research Center, CAS Institute of Policy and Management, CAS AEA 2013, Washington DC

3. About IPM and ERC of CAS IPM Found in June 1985 Devoted to the studies of development strategy, development and reform policy, public administration, S&T management and state-of-the-art theories and methodologies of related disciplines Faculty: 138 ERC of CAS (also affiliate to IPM) Faculty: 14 Mission Management Innovation and Evaluation Research, institutes evaluation

4. Quantitative Monitoring is Necessary

5. Quantitative Monitoring Peer Review Quantitative Monitoring(QM) ： widely used management tool for international institutions NIH MPG INRIA SoSP

6. Quantitative Monitoring in CAS

7. Development of Quantity Monitoring of CAS

8. Blue Book Ev. White Book Ev. Yellow Book Ev. Comprehensive Quality Ev. Major Outcomes Oriented Ev. >Innovation Capacity >No more rank >Key indicators +Innovation Capacity >No more rank

9. Blue Book Evaluation （ 1993-2001 ） QM=Evaluation Totally Quantity Evaluation 1993-1999: rank within one system 1998-2001: rank within three category series Basic Research Applied Research Development and High Tech Research Industrialization

10. CategoryParent IndicatorChild Indicator Performance Evaluation Basic Research Performance Natural Science Awards Papers (amount and per amount) Citations Other Publications Applied Research Performance S&T Progress Awards and Invention Award Patents (Applied and Granted) Development and High Tech Research Industrialization Institution Company's Total Profit Tax Development Income Commercialization of Research Findings Blue Book Evaluation （ 1993-2001 ）

11. CategoryParent IndicatorChild Indicator Status Evaluation Research Status Capability of Undertaking Government and Local Governments' Missions Outcomes (amount and per amount) International Cooperation (Funds and # of Programs) Talents Academician of CAS Proportion of Senior Researchers under 45 years old Proportion of holding PhD’s and Master’s Degree # of Graduate Students Funding Input Funds Funds per Researcher Outside funding Equipment and labs Advanced Equipment Proportion of value of Advanced Equipment from 90’ in all Equipment # of Opening Labs and Engineering Center Blue Book Evaluation （ 1993-2001 ）

12. Evaluation results were revealed according to indicators Rank by score into four categories A, B, C and D within each indicators Performance and Status in forms of Category Results of one institute from different years are not comparable Blue Book Evaluation （ 1993-2001 ）

13. White Book Evaluation （ 1999-2001 ） Objective completion status + basic, strategic and prospective contribution Objective completion status Ev., Peer Review S&T objective Ev. Contribution of work Influence to subject, etc. Management objective Ev. Does fixed staff turnover rate exceed 5%? Does the usage of Special Funds for KIP exceed 60%? Etc.

14. White Book Evaluation （ 1999-2001 ） Evaluation of basic, strategic and prospective contribution: QM

15. White Color Book （ 1999-2001 ） Parents IndicatorsChild Indicators Undertaking of Strategy S&T Mission National Major Programs International Programs PapersHigh IF journals by subjective Major Lectures of International Conference Plenary Lectures Invited Lectures Significant social and economic benefits Research Outcomes transfer Start-up owner's equity Major Consultation Reports Excellent labs IP(Issued patens, Software copyrights) Talents Academician and Major Projects investigators, etc. S&T Consultants International Talents Young Talents Major AwardsNational Awards

16. White Book Evaluation （ 1999-2001 ） Qualitative evaluation has been introduced to White- Book evaluation Evaluation = QM+Peer Review Scores, not classification results Turned qualitative evaluation results into quantitative data, then got the ranking Scores based on all quantitative data Characteristics of institutes disciplines is not enough, and QM is still a primary method

17. Yellow Book Evaluation （ 2002-2004 ） Started to focus on quality over quantity Evaluation System Major Innovation Contributions Evaluation, Peer Review Basic Indicators Evaluation, conducted by CAS headquarter bureaus Classification Oriented Evaluation, for all institutes in CAS basic research high technology research and development resources, environment and sustainable development Industrialization

18. Yellow Color Book （ 2002-2004 ） SeriesIndicatorsRemarks Basic Research PapersTop journal Major International Conference Lectures S&T AwardsNational Science Awards Research ProgramsNSFC, 973, etc. Major Innovation Contributions Resources, Environment and Sustainable Development PapersTop journal Major International Conference Papers Consultant Reports Adopted by national leaders or the central ministries and commissions, with substantive instructions S&T Awards National or Provincial awards of Science Awards, Invention Awards, and Progress Awards StandardsInternational, national, or industry standards Research ProgramsNSFC, 863,973, etc. Major Innovation Contributions

19. SeriesIndicatorsRemarks High Tech Research and Development Patents Software Copyright New medicines, new pesticides, new crop varieties S&T Awards National Progress Awards and Invention Awards StandardsInternational, national, or industry standards Research outcomes transferAmount to the account Research programs863, 921, etc. Major Innovation Contributions Industrialization Owner’s equityTotal Owner’s equity Research outcomes transferAmount to the account Research programs Major Industrialization Programs and Enterprises Programs Major Innovation Contributions

20. Yellow Book Evaluation （ 2002-2004 ） Evaluation = QM + Peer Review (similar to White Book Evaluation) Started to pay more attention to major innovation contributions according to series subject’s specialty Monitoring results is reflected as score within each category series

21. Comprehensive Quality Evaluation (2004 - 2011) Ten key process are included in the evaluation self-evaluation of institutes strategic planning for next stage peer review for existing outcome comprehensive analysis of previous evaluation Communication review on-site assessment president office conference etc. Final conclusions and opinions are based on results of diverse contents of strategic decisions Qualitative Evaluation, and QM play an important part in decision making

22. Before 2004 quantitative monitoring gradually returns to its original function a powerful tool to compare among institutes by providing year-based information cannot display developing trend of institutes within certain years in a more comprehensive way Comprehensive Quality Evaluation (2004 - 2011)

23. Innovation Capacity Indicator (ICI) Monitoring

24. Design of Innovation Capacity Indicator (ICI) Law of Comparative Advantage, David Ricardo, 1817 In economics, comparative advantage refers to the ability of a party to produce a particular good or service at a lower marginal and opportunity cost over another. Even if one country is more efficient in the production of all goods (absolute advantage in all goods) than the other, both countries will still gain by trading with each other, as long as they have different relative efficiencies.

25. Indicators Status Outcomes Impact Weights Weights of indicator are based on simulation of years historical data Evaluate the importance of parents indicators Set up the price according to data of a certain year Design of Innovation Capacity Indicator (ICI)

26. How to Choose Indicators CAS has 100+ institutes, and covers most of disciplines Indicators should reflect common characteristics of institute respective expertise and features

27. How to Choose Indicators Expert group discussion, and approval of most institutes 7 Indicators to monitor

28. StatusOutcomesImpact Researches Funds Excellent Talents High Quality Papers IP Completed Major Programs S&T Awards Consultant Reports

29. To some degree, the indicators of ICI consider different characteristics of different institutes excellent talents and scientific research fund are necessary and common indicators high quality paper is the common indicator to reflect the output of fundamental research institutes How to Choose Indicators

30. Intellectual Property is an important output indicator in high technology institute Completed Major Programs can reflect how it meets the demand of the nation In addition, the characteristics indicators are included in the child indicators National Natural Science Awards National Technological Invention Awards National S&T Progress Awards How to Choose Indicators

31. Number of outcomes increasing rapidly How to Choose Indicators

32. Higher Quality Requirements in Choosing Indicators For example, papers published in journal whose impact factor is among top 15% within this discipline (JCR&SJCR) How to Choose Indicators

33. Weights of Indicators Management experts and strategy experts made a group decision making on the weights of ICI, based on evaluation orientation, management experience and a large amount of simulation computation IndicatorsWeights Excellent Talents20% Research Funds10% High Quality Papers20% IP15% Completed Major Programs25% Major Consultant Reports5% Major Research Awards5%

34. Using the weights of 7 indicators, we can determine the price of parent indicators based on the data of 2003 and 2004. Weights of child indicators, determined by its importance One 973 project worth 5 points ， and one NSFC program worth 2 points. Weights of Indicators

35. Parent indicatorsChild Indicators ICI Excellent Talents, 20% Top talents Graduate education Research Funds, 10%Funds from outside High Quality Papers, 20% IP, 15% Patents Software Copyrights Standards Major Consultant Reports, 5% Completed Major Programs, 25% NSFC MOST Other Programs Major Research Awards, 5% Major International Awards National Awards Provincial Awards

36. Computation

37. How to Consider Efficiency total amount of seven indicators are divided by the number of innovation positions or regular budget Regular funding for KIP If Parent Indicator represent money # of Positions for KIP If Parent Indicator does not represent money

38. ICI for CAS Great support for management of CAS Institutes can also recognize their positions among similar institutes Development trend of CAS and its institutes in recent years

39. ICI for CAS

40. Conclusion

41. QM systems fits the improvement of the research quality of CAS Institutes Rapidly promoted the outcomes of CAS ICI is a useful management tool

42. Limitations of ICI Some indicators is bad for some institutions High Quality Paper Small discipline ICI does not reflect all the characteristics of an institute Mainly common indicators Some meaningful works are difficult to quantify Large-scale scientific facility Scarce Resources

43. Conclusion Peer review is getting more attentions from CAS ICI becomes a tools of monitoring, neither ranking or comparing among institutes 2012, Major Outcomes-Oriented Evaluation, QM is an important part of the evaluation system Monitoring Key indicators + ICI Focus on diagnostic function of their own development

44. ZHENG Haijun haijzheng@casipm.ac.cn GUAN Zhongcheng guan@casipm.ac.cn WANG Biaoxiang xwbill@casipm.ac.cn WU Jianmei wujianmei@casipm.ac.cn