Download presentation
Presentation is loading. Please wait.
Published byNora Dorsey Modified over 8 years ago
1
page 1 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Tailor-made evaluation concepts for innovation policy learning Research and the Knowledge Based Society – Measuring the Link 24 th May 2004, NUI Galway, Ireland page 1 Stefan Kuhlmann (ISI; UU), Jakob Edler (ISI) Copernicus Institute for Sustainable Development and Innovation
2
page 2 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Scope of innovation policy evaluation Four poles of evaluation missions and approaches Two opposed examples Summative, quantitative poles example: Relationship between R&D collaboration, subsidies and patenting Formative, qualitative poles example: Assessment of policy instruments supporting "competence centres" Conclusions Overview
3
page 3 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Typical R&D evaluation issues and questions (Source: Arnold/Guy 1997, 72) Appropriateness: Was it the right thing to do? Economy: Has it worked out cheaper than we expected? Effectiveness: Has it lived up to the expectations? Efficiency: What’s the return on investment (ROI)? Efficacy: How does the ROI compare with expectations? Process efficiency: Is it working well? Quality: How good are the outputs? Impact: What has happened as a result of it? Additionality: What has happened over and above what would have happened anyway? Displacement: What hasn’t happened which would have happened in its absence? Process Improvement: How can we do it better? Strategy: What should we do next?
4
page 4 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Impact dimensions of public research and innovation spending
5
page 5 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Summative and formative evaluation Summative Evaluation systematic, indicator based mainly ex post - or interim - measurement and assessment of the performance of programmes (including projects) to assess the programme design, implementation management and the leverage of funding and to learn for future approaches Formative Evaluation systematic consulting, moderating, assessing activities seeking to assist policy makers, programme managers and programme participants throughout the whole life cycle of funding programmes to make all actors involved learn and (re-)adjust and thus contribute to the overall success (and/or improvement and/or termination) of programmes and funded structures and to learn for future approaches.
6
page 6 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Evaluation methods, quantitative and qualitative Quantitative: Statistical data analysis Innovation Surveys: basic data describe the innovation process, using descriptive statistics Benchmarking: comparisons based on a relevant set of indicators across entities Quantitative: Modelling methodologies Macroeconomic modelling and simulation: broader socioeconomic impact of policy interventions Microeconometric modelling: effects of policy intervention at the level of individuals or firms Productivity analysis: impact of R&D on productivity growth at different levels data aggregation Comparison group approach: effect on participants using statistical sophisticated techniques Qualitative and semi-quantitative methodologies Interviews and case studies: direct observation of naturally occurring events to investigate behaviours in their indigenous social setting Cost-benefit analysis: economic efficiency by appraising economic and social effects Expert panels/peer review: scientific output relying on the perception of peer scientists Network analysis: structure of cooperation relationships and consequences for individuals and their social connections into networks Foresight/ technology assessment: identification of potential mismatches in the strategic efficiency of projects and programmes Source: Polt, W. et al., RTD Evaluation Toolbox, http://epub.jrc.es/evaluationtoolbox/start.swf
7
page 7 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Evaluation Matrix: Matching policy instruments and methods Source: Polt, W. et al., RTD Evaluation Toolbox, http://epub.jrc.es/evaluationtoolbox/start.swf
8
page 8 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Four poles of evaluation missions and approaches quantitative qualitative formativesummative Measurement of policy assumptions, outputs and effects Need for … robust operationalisation (sophisticated) methodologies reliable and encompassing data Analysis of policy context and governance Need for … awareness of diversity of actors' perspectives methodology mix
9
page 9 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning R&D personnel Internal R&D expenditures Expenditures für knowledge transfer, fees, licences, standards documents External R&D, technical consulting Investment in R&D-intensive equipement, mate rials, components Knowledge stock Fundamental research Applied research Experimental developement Standardisation (Technometric) characteristics, innovation counts R&D-intensive goods: employment, production growth, factor productivity Various foreign trade indicators market shares Resource indicators R&D personnel Internal R&D expenditures Expenditures für knowledge transfer, fees, licences, standards documents Output Indicators: (Technometric) characteristics, innovation counts R&D-intensive goods: employment, production growth, factor productivity Various foreign trade indicators market shares Summative, quantitative poles - S/T indicators and stages of innovation Intangible functions Measurable functions Measurable feed-back R&D results indicators Patent citation Patent application Scientific publication Literature citation Idea, theory, discovery Technical design Product design, innovation Imitation, improvement, diffusion, exploitation, disposal
10
page 10 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Summative, quantitative poles - example: Relationship between R&D collaboration, subsidies and patenting Recent evaluative study of D. Czarnitzki (ZEW), B. Ebersberger (VTT GTS) and Andreas Fier (ZEW): The Relationship between R&D Collaboration, Subsidies and Patenting Activity: Empirical Evidence from Finland and Germany (Preliminary version to be presented at the IIOC 2004, Chicago, IL) Focus of this evaluative study: Summative question: Investigation whether public R&D subsidies in Finland and in Germany have a positive impact on the innovation output (effects of public incentives and R&D collaboration on innovative output of companies measured by their patenting activity). Quantitative approach: Treatment effects analysis to assess whether policy and/or collaboration yield a positive benefit in terms of patent activity, with a sample of German an Finnish firms. Study applies an econometric matching taking a possible selection bias into account.
11
page 11 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Analysis of public funding, collaboration & patent outcome Descriptive statistics (based on CIS data) Source: Czarnitzki, Ebersberger and Fier, 2004
12
page 12 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Summative, quantitative poles – results of example Results for Germany: Public funding and collaboration (and both) lead to improved innovative performance This hypothesis is not supported for firms that receive R&D subsidies for individual research Results for Finland: Firms actually collaborating and receiving funding, would exhibit less patenting activity if the goverment had not subsidized those firms In this case, firms might not be able to raise enough capital to maintain their high innovation efforts Source: Czarnitzki, Ebersberger and Fier, 2004
13
page 13 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Summative, quantitative poles – conclusions from example Quantitative summative evaluation provides information about relevant measurable outputs and effects; information can be highly likely and quite sophisticated Quantitative summative evaluation has only limited potential to explain causality of measured effects to explore other (indirect) effects, like 'behavioural additionality', learning A formative analysis/evaluation of economic and policy context would help to understand differences and promising starting points for improved policies.
14
page 14 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Formative, qualitative poles: Innovation stakeholder arena as context National research ministry Other national ministries Regional govern- ments National parlia- ment EU Com- mission Multi- national companie s SME asso- ciations Industrial asso- ciations Uni- versities National research centers Research councils Contract research institutes Consumer groups Environ- ment groups Differing interests, perspectives and values Competition for funds No dominant player? Contested policies Need for alignment, otherwise: exit Evaluation... as formative learning medium
15
page 15 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Formative, qualitative poles - example: Assessment of policy instruments supporting "competence centres" Recent evaluative study of Jakob Edler, S. Bührer, V. Lo, C. Rainfurth (Fraunhofer ISI) and S. Sheikh (KMU Forschung Austria), Future of competence centre programmes (K plus and K ind/net) and future of competence centres, Karlsruhe/Vienna 2003 (Study on behalf of two Austrian Federal Ministries) Focus of this evaluative study: Formative question: Strategic advice with respect to the future development of two competence centre support programmes (K plus and K ind/net): Differences of the appropriateness of the two progs? Fit of the two progs' targets and implementation? (Prevailingly) qualitative approach: Evaluation as 'critical friend' of policymakers and stakeholders, questioning policymakers' hypotheses and supporting decisionmaking. Information base: 'Good guess' drawing upon structured interviews, document analysis, structural data, survey of international policy experiences.
16
page 16 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Key for evaluation: understandings the basic concepts of the two progs … Cooperation of industry and science for research and innovation Assumption: cooperation too low Financial incentive for cooperation needed Additionality of support for cooperation Increase of R&D expenditure of companies More R&D results, more risk-taking, speeding-up Learn how to cooperate ('behavioural additionality') Public policy designed as multi-actor, multi-measures programme (MAP)
17
page 17 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Exploration and evaluation of policy rationales, context and governance – two profiles Driven by innovation (Kind/net) Driven by knowledge generation (Kplus) TypeClose to marketClose to basic research RationaleProject-orientedCommunity of practice-oriented Purpose of participation (funding) Overcome firm-internal barriers for cooperative market-oriented R&D Creation of new cooperation structures; upgrade and broadening of research Cooperation cultureOriented towards well-known partners Oriented towards most excellent partners Time horizonShort-term resultsMedium-term, knowledge creation
18
page 18 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Formative, qualitative poles – results of example Overall assessment: two different programme approaches justified, to be better profiled Results and recommendations for Kind/net: Develop clear profile as innovation programme; adapt funding level (below research funding) Improve programme management (e.g. transparency) Results and recommendations for Kplus: Provide stable funding and transparent rules Involve local authorities Extend inter-centre collaboration
19
page 19 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Formative, qualitative poles – conclusion from example Advanced innovation policy instruments are increasingly complex (MAP) Problem: strategic fit of policies – approach, instruments, implementation Formative evaluation as a source of strategic intelligence, providing evaluative inputs for reflexive, incremental policy-development needs qualitative understanding of rationales, context and governance including multiple perspectives of different actors and levels Formative, qualitative evaluation approaches are indispensable, quantitative and summative inputs (e.g. on outputs and performance) are very helpful
20
page 20 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning General principles of strategic intelligence Principle of participation: strategic intelligence realises the multiplicity of actors’ and stakeholders’ values and interests involved in innovation policymaking (multiple perspective approach). Principle of "objectivisation": strategic intelligence "injects objectivised" information into the policy arena, i.e. the results of policy/strategy evaluations, foresight exercises or technology assessment, and also of analyses of changing innovation processes, of the dynamics of changing research systems and changing functions of public policies. Principle of mediation and alignment: strategic intelligence facilitates debates and "discourses" between contesting actors in related policy arenas, thus mediating and "moderating", supported by "objectivised" information to be "digested" by the struggling parties. Principle of decision support: strategic intelligence requires forums for negotiation and the preparation of policy decisions.
21
page 21 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Contact: s.kuhlmann@isi. fraunhofer.de j.edler@isi.fraunhofer.de Info: www.isi. fraunhofer.de
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.