1. Stakeholder Involvement and the Choice of Science & Technology Policy Outcome Evaluation Methods Presenter ： Shan-Shan Li STPI, NARL Presentation on 2009 AEA conference 2009.11.14

2. 2 Outline  Introduction  Methodology  Analysis  Research Fundings  Conclusions

3. 3 Introduction

4. 4 The importance of stakeholder Stakeholder involvement plays an indispensable part in science & technology management. Stakeholder involvement can… –help to clarify the root causes of problems and to develop effective strategies for addressing them during different phrases of the S&T management process Evaluation methods are the means for solving such problems or meeting stakeholders’ requirements.

5. 5 Comparison of 10 globally used policy outcome evaluation methods Sources: Summarized and organized from Fahrenkrog, Gustavo, Polt, Wolfgang, Rojo, Jaime, Tubke, Alexander, & Zinocker, Klaus. (eds.) 2002

6. 6 Research questions Stakeholders’ demand-side requirements Stakeholders’ supply-side issues which S&T policy outcome evaluation methods are most suited for measuring stakeholders’ interests and in assessing their impacts in each phase of publicly funded S&T programs? which S&T policy outcome evaluation methods are most appropriate to use in measuring their interests and in assessing their impacts in each phrase of publicly funded S&T programs?

7. 7 Stakeholders in S&T management Profit Organization S&T Funding Organization Internal Environment Competitors Contractors End user Reviewers Mass media Government/de cision makers Consultants Customers/co mmunity Industrial strategic Partners Research team Administrative personnel Researchers External environment The internal environment: which is the environment that is related to S&T management and assignment, used to describe individuals or a group of stakeholders that participate directly in the internal S&T environment, such as customers, contractors, etc. The external environment: This is formed by species elements and is related to the S&T management including technology, law, economy, politics, physics, labor supply and demand, culture and environments, etc.

8. 8 Methodology

9. 9 Methodology funders A funders B S&T executive institutions A cooperative institutions A cooperative institutions C cooperative institutions B cooperative institutions D the end-users utilizing the final products or services of S&T research A the end-users utilizing the final products or services of S&T research C the end-users utilizing the final products or services of S&T research B the end-users utilizing the final products or services of S&T research D Allocation of fundingScience community’s research output, Outcomes and impact Research fields’ collaborationsocietal impact (impact of research utilization) the end-users utilizing the final products or services of S&T research E S&T executive institutions B S&T executive institutions C Sources: Academy of Finland (2006) we will use the 4 phases explained in the Academy of Finland’s report to further derive the issues that the stakeholders are concerned with in order to draw a strategic plan for the future

10. 10 The properties of the research funding impact phases PhaseConcernsProperties Allocation of public funds Can be seen from the point of view of a nation, sponsoring organization itself, applicants and impact  Understand the impact caused by the approval of the funding and the allocation ratio Outputs and outcomes of S&T communities The impact of the modality caused by scientific research  Number of research output and impact elements  Number of research output provided by the research community to non-academic users for utilization and development  The impact of the research executor’s participation on the research program Cooperation of research communities Cooperative system between research organizations or between research organization and research users  Scientific community in the form of knowledge spillovers effect  The formation of international cooperation network  The level of knowledge flows between research communities and research users  Simple calculation of cooperation agreement  Visualization of network framework  Investigate formation of cooperative relations Impact of research utilization The impact of research results is often not formed by a single program, institution, enterprise etc. but rather, it’s the stakeholders that cause the impact, such as social, economic etc.  Evaluation on relevance of the users’ researches  Target creation of the research result commercialization  Investigation on the change in users system  Great impact caused by the evaluation research results Sources: Academy of Finland (2006)

11. 11 The 4 groups of stakeholders within S&T management Academic system Industrial strategic partner Government Investigator Contractor Competitor Decision-maker Research team Administrators Consultants Profit Organization (e.g. insurance) End users Customers Academic staff Community Sponsors S&T R&D management environment S&T R&D implementation system+ Cooperative network system S&T R&D result users

12. 12 Stakeholders’ supply-side and demand-side Phases StakeholdersMain concerns Supply-sideDemand-sideSupply-sideDemand-side Allocation of public funds  Funders  S&T research and development implementation system  S&T research and development management process  S&T research and development planning implementation detail  Reasonability of the funds allocation  Comparison between actual expenditure and budgeted amount  Externality problem  Academic research institute financial enhancement  Support and promote high-quality research  Investigation of system fairness and transparency  Communication process Outputs and outcomes of S&T communities  S&T executive institutions  Cooperation network system  Customers/ community/ end-users  Resource allocation  External network connection  Implementation process efficiency  Schedule arrangement and planning  Future expected outcome  Output produced meet expectation  Cooperation direction  Utilization of research results  Academic staff  S&T research and development energy development Cooperation of research communities  Cooperation network system  Academic staff/ end-users  Cooperation direction and system  The role of the system  The level of relevance between industry, government, academic and research  Energy upgrade Impact of research utilization  Cooperation network system  Academic staff/end users  Community  Increase production  Enhance economic growth  Create legal environment  Level of result commercialization  The impact that result commercialization has on the industry  Spillover effect  Products sold at a profit and impact on employment  Benefits of technology  Outstanding innovation procedures provide feedback to the S&T research and development result  Research utilization

13. 13 Analysis

14. 14 Through the analysis of the two dimensional matrix of ： –stakeholders’ interests –evaluation methods Comparisons (demand-side and supply-side) are shown in p.16, p.17. Analysis

15. 15 Direction of discussion : example PhasesSidesMeaningsMethods public fund allocation supply-sideFunders (including governments, decision makers, contractors, competitors, etc.) are the main stakeholders who provide S&T related funds for some publicly funded executors (including S&T institutions and end- users). Under the circumstance, they have own selective criteria on who is more appropriate institution performing publicly funded S&T programs, including program management efficiency, appropriateness and relevance of programs, rationality of funds allocation, whether the funds additionally improve S&T institutions’ financial situation or not, or whether the funds support and promote excellent research. Take expert review, a qualitative method public fund allocation demand-sideS&T institutions (including team members, administrators, program officers/managers, consultants, etc.) are the main stakeholders who want to apply for publicly funds. Their concerns include the efficiency of communication between managers and members, and the transparency and fairness of the auditing mechanism. Take expert review as an example also to discuss these issues.

16. 16 Comparisons between stakeholders, and evaluation methods (Demand-side) Issue formAllocation of public fundsOutputs and outcomes of S&T communities Cooperation of research communities Impact of research utilization StakeholdersS&T research and development implementation system Cooperation network system Academic staffResearch result usersCommunity Concerning issuesInvestigation of system fairness and transparency Communication process Cooperation direction S&T research and development energy development Research utilization/ safety/ function/ quality Energy upgradeResearch development utilization/ safety/ function/ quality Evaluation methods Macroeconomic Modeling & Simulation * Micro-econometric Models *** Cost-benefit Analysis *** Case Studies ****** Benchmarking ** Expert Review ***** Control Group *** Innovation Survey *** Network Review **** Foresight/Technology Assessment **

17. 17 Comparisons between stakeholders, and evaluation methods (Supply-side)

18. 18 Research findings

19. 19 Macroeconomic modeling and simulation demand side supply side The funders’ concern about the externalities problem created by research resources (e.g. knowledge spillover effect or industrial innovation spillover effect etc.) The community concern about impact of R&D input on the economy (e.g. economic output and employment creation etc.) It can be applied together to measure policy’s direct outcome and between research fields, inter-industries spillover effects’ direct outcome.

20. 20 Micro econometric model This method can directly measure the outcome of public policies towards company or individual project. supply side demand side In order to answer questions about rationality of fund allocation and external outcome etc. and also help S&T research and development implementation system into comparing similar projects implementation results, for S&T research and development results users, the model can evaluate researches for instance R&D activities’ achievements (e.g. relationship between R&D budget and company’s ability to earn profit) or spillover effect of individual project’s R&D activity towards industries etc As for the community, evaluation of the effect caused by government’s science and technology policy measures can be done using the method (e.g. implementation and authentication results of scientific projects)

21. 21 Cost benefit analysis It calculates the socio-economic impacts when the implementation plan is carried out, and through the cost benefit analysis result, suitable project, case or policy can be selected. It helps the manager to think about the cost and benefit in a more structural and systematic way. supply side demand side It is suitable in measuring the appropriateness of the resources and funds allocation and the efficiency and the outcome of the project implementation It can evaluate academic staff’s S&T research and development energy development, research output quality and also S&T research and development result users and the community in the level of research output utilization

22. 22 Expert panels/ Peer review They are able to not only carry out ex-post measurement, but also ex-anti evaluation. supply side demand side experts can use the rationality of the expected budget allocation to further analyze the ex-anti evaluation section. It can play an important part in terms of S&T research and development implementation system’s investigation fairness & transparency, communication process and the utilization of the energy development. with research result development and utilization, they can also provide credible and effective suggestions.

23. 23 Control group approaches supply side demand side This method is mainly under the control of other variables, explore the effect single manipulated variable has on different samples; the main manipulated variables are sponsorship funds and applicants. It is appropriate to apply control group approaches on measuring and evaluating private enterprises economic activities for example: the impact of governmental department’s S&T research and development grants on private enterprises R&D, the impact of government fiscal measures (lowering taxes), and the effectiveness of loaning between emerging companies and banks etc. It is able to help provide analytical information on the demand-side’s fund allocation issues and also additionality effect, such that academic staff’s energy development can be further analyzed.

24. 24 Innovation surveys supply side demand side It mainly focus on the evaluation of the corporations’ innovation activities. Funders can monitor whether policy tools can help to achieve product innovation target or not through funds grant formula or rationality of funds allocation Stakeholders will be very much concern about economic and industrial growth; level of commercialization and product sales due to innovation etc

25. 25 Network analysis supply side demand side It is mainly a relationship structure between each node and also the behavioral change result at each node through the end of the decision- making process. Cooperation network system’s stakeholders have always been the target of the network analyze explore. As a result, necessary information can be obtained through network analysis; also it can explore the condition of knowledge distribution It can fully perform its function when looking at the fund allocation communication process and network system cooperation direction on the demand-side. It explores the interaction and state between each node; therefore the academic staff’s S&T research and development energy development and users’ energy enhancement on the demand-side can also be analyzed and explored

26. 26 Foresight/ technology assessment supply side demand side The transfer of foresight and technology onto the evaluation is mainly a comprehensive outline of the whole fund allocation direction and S&T research and development project content. The stakeholders on the supply-side will care most about the S&T research and development planned implementation procedures and rationality of the allocation of the public funds It would be more interested in knowing the safety of the research and development result and whether or not future demand is met etc

27. 27 Field/ Case study supply side demand side The survey method is qualitative and is suitable for those innovative projects that haven’t been analyzed in depth yet and is also suitable on the policy measures that can have complicated socio-economic impacts, though it is not very appropriate to use it to test the causal relationship. Due to explore the details of S&T implementation. It is suitable for some issues, Investigation of system fairness and transparency, Cooperation direction, S&T research and development energy development, Research utilization/ safety/ function/ quality, Energy upgrade, Research development utilization/ safety/ function/ quality In order to respond the S&T impact. It is the application of these issues, Investigation of system fairness and transparency, Cooperation direction, S&T research and development energy development, Research utilization/ safety/ function/ quality, Energy upgrade, Research development utilization/ safety/ function/ quality

28. 28 Benchmarking supply side demand side It can apply on product, service, procedure, method, framework, and the whole organization etc. In the process of benchmarking, target selection is very important. The choices of indicators can be applied in some issues, Investigation of system fairness and transparency, S&T research and development energy development

29. 29 Conclusions Stakeholders are a determinate element in the choice of outcome evaluation methods. The level of attention paid towards each issue will depend on each stakeholder ultimate goals causing a constant need of coordination in conflicted S&T policy decision makings and evaluation systems. A combination of multiple S&T policy outcome evaluation methods is necessary. Government organizations can carefully formulate a suitable policy through interacting with the stakeholder ; resolving concerning issues and planning future development of the policies together, in order to achieve a realm of ‘mutual benefit’. The identification of stakeholders indeed have influence on the whole process of evaluation. When we make the choices of evaluation methods, we need consider the indispensable element, the stakeholders.

30. Li, Shan Shan ssli@mail.stpi.org.tw http://www.stpi.org.tw/STPI/index.htm ssli@mail.stpi.org.tw http://www.stpi.org.tw/STPI/index.htm

31. 31 Thanks For Your Attention!