1. page 1 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Tailor-made evaluation concepts for innovation policy learning Research and the Knowledge Based Society – Measuring the Link 24 th May 2004, NUI Galway, Ireland page 1 Stefan Kuhlmann (ISI; UU), Jakob Edler (ISI) Copernicus Institute for Sustainable Development and Innovation

2. page 2 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning  Scope of innovation policy evaluation  Four poles of evaluation missions and approaches  Two opposed examples  Summative, quantitative poles example: Relationship between R&D collaboration, subsidies and patenting  Formative, qualitative poles example: Assessment of policy instruments supporting "competence centres"  Conclusions Overview

3. page 3 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Typical R&D evaluation issues and questions (Source: Arnold/Guy 1997, 72) Appropriateness: Was it the right thing to do? Economy: Has it worked out cheaper than we expected? Effectiveness: Has it lived up to the expectations? Efficiency: What’s the return on investment (ROI)? Efficacy: How does the ROI compare with expectations? Process efficiency: Is it working well? Quality: How good are the outputs? Impact: What has happened as a result of it? Additionality: What has happened over and above what would have happened anyway? Displacement: What hasn’t happened which would have happened in its absence? Process Improvement: How can we do it better? Strategy: What should we do next?

4. page 4 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Impact dimensions of public research and innovation spending

5. page 5 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Summative and formative evaluation  Summative Evaluation  systematic, indicator based  mainly ex post - or interim - measurement and assessment of the performance of programmes (including projects)  to assess the programme design, implementation management and the leverage of funding and to learn for future approaches  Formative Evaluation  systematic consulting, moderating, assessing activities  seeking to assist policy makers, programme managers and programme participants  throughout the whole life cycle of funding programmes  to make all actors involved learn and (re-)adjust  and thus contribute to the overall success (and/or improvement and/or termination) of programmes and funded structures and to learn for future approaches.

6. page 6 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Evaluation methods, quantitative and qualitative  Quantitative: Statistical data analysis  Innovation Surveys: basic data describe the innovation process, using descriptive statistics  Benchmarking: comparisons based on a relevant set of indicators across entities  Quantitative: Modelling methodologies  Macroeconomic modelling and simulation: broader socioeconomic impact of policy interventions  Microeconometric modelling: effects of policy intervention at the level of individuals or firms  Productivity analysis: impact of R&D on productivity growth at different levels data aggregation  Comparison group approach: effect on participants using statistical sophisticated techniques  Qualitative and semi-quantitative methodologies  Interviews and case studies: direct observation of naturally occurring events to investigate behaviours in their indigenous social setting  Cost-benefit analysis: economic efficiency by appraising economic and social effects  Expert panels/peer review: scientific output relying on the perception of peer scientists  Network analysis: structure of cooperation relationships and consequences for individuals and their social connections into networks  Foresight/ technology assessment: identification of potential mismatches in the strategic efficiency of projects and programmes Source: Polt, W. et al., RTD Evaluation Toolbox, http://epub.jrc.es/evaluationtoolbox/start.swf

7. page 7 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Evaluation Matrix: Matching policy instruments and methods Source: Polt, W. et al., RTD Evaluation Toolbox, http://epub.jrc.es/evaluationtoolbox/start.swf

8. page 8 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Four poles of evaluation missions and approaches quantitative qualitative formativesummative  Measurement of policy assumptions, outputs and effects  Need for …  robust operationalisation  (sophisticated) methodologies  reliable and encompassing data  Analysis of policy context and governance  Need for …  awareness of diversity of actors' perspectives  methodology mix

9. page 9 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning R&D personnel Internal R&D expenditures Expenditures für knowledge transfer, fees, licences, standards documents External R&D, technical consulting Investment in R&D-intensive equipement, mate rials, components Knowledge stock Fundamental research Applied research Experimental developement Standardisation (Technometric) characteristics, innovation counts R&D-intensive goods: employment, production growth, factor productivity Various foreign trade indicators market shares Resource indicators R&D personnel Internal R&D expenditures Expenditures für knowledge transfer, fees, licences, standards documents Output Indicators: (Technometric) characteristics, innovation counts R&D-intensive goods: employment, production growth, factor productivity Various foreign trade indicators market shares Summative, quantitative poles - S/T indicators and stages of innovation Intangible functions Measurable functions Measurable feed-back R&D results indicators Patent citation Patent application Scientific publication Literature citation Idea, theory, discovery Technical design Product design, innovation Imitation, improvement, diffusion, exploitation, disposal

10. page 10 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Summative, quantitative poles - example: Relationship between R&D collaboration, subsidies and patenting  Recent evaluative study of D. Czarnitzki (ZEW), B. Ebersberger (VTT GTS) and Andreas Fier (ZEW): The Relationship between R&D Collaboration, Subsidies and Patenting Activity: Empirical Evidence from Finland and Germany (Preliminary version to be presented at the IIOC 2004, Chicago, IL)  Focus of this evaluative study:  Summative question: Investigation whether public R&D subsidies in Finland and in Germany have a positive impact on the innovation output (effects of public incentives and R&D collaboration on innovative output of companies measured by their patenting activity).  Quantitative approach: Treatment effects analysis to assess whether policy and/or collaboration yield a positive benefit in terms of patent activity, with a sample of German an Finnish firms. Study applies an econometric matching taking a possible selection bias into account.

11. page 11 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Analysis of public funding, collaboration & patent outcome Descriptive statistics (based on CIS data) Source: Czarnitzki, Ebersberger and Fier, 2004

12. page 12 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Summative, quantitative poles – results of example  Results for Germany:  Public funding and collaboration (and both) lead to improved innovative performance  This hypothesis is not supported for firms that receive R&D subsidies for individual research  Results for Finland:  Firms actually collaborating and receiving funding, would exhibit less patenting activity if the goverment had not subsidized those firms  In this case, firms might not be able to raise enough capital to maintain their high innovation efforts Source: Czarnitzki, Ebersberger and Fier, 2004

13. page 13 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Summative, quantitative poles – conclusions from example  Quantitative summative evaluation provides information about relevant measurable outputs and effects; information can be highly likely and quite sophisticated  Quantitative summative evaluation has only limited potential  to explain causality of measured effects  to explore other (indirect) effects, like 'behavioural additionality', learning  A formative analysis/evaluation of economic and policy context would help to understand differences and promising starting points for improved policies.

14. page 14 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Formative, qualitative poles: Innovation stakeholder arena as context National research ministry Other national ministries Regional govern- ments National parlia- ment EU Com- mission Multi- national companie s SME asso- ciations Industrial asso- ciations Uni- versities National research centers Research councils Contract research institutes Consumer groups Environ- ment groups  Differing interests, perspectives and values  Competition for funds  No dominant player?  Contested policies  Need for alignment, otherwise: exit Evaluation... as formative learning medium

15. page 15 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Formative, qualitative poles - example: Assessment of policy instruments supporting "competence centres"  Recent evaluative study of Jakob Edler, S. Bührer, V. Lo, C. Rainfurth (Fraunhofer ISI) and S. Sheikh (KMU Forschung Austria), Future of competence centre programmes (K plus and K ind/net) and future of competence centres, Karlsruhe/Vienna 2003 (Study on behalf of two Austrian Federal Ministries)  Focus of this evaluative study:  Formative question: Strategic advice with respect to the future development of two competence centre support programmes (K plus and K ind/net): Differences of the appropriateness of the two progs? Fit of the two progs' targets and implementation?  (Prevailingly) qualitative approach: Evaluation as 'critical friend' of policymakers and stakeholders, questioning policymakers' hypotheses and supporting decisionmaking. Information base: 'Good guess' drawing upon structured interviews, document analysis, structural data, survey of international policy experiences.

16. page 16 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Key for evaluation: understandings the basic concepts of the two progs …  Cooperation of industry and science for research and innovation  Assumption: cooperation too low  Financial incentive for cooperation needed  Additionality of support for cooperation  Increase of R&D expenditure of companies  More R&D results, more risk-taking, speeding-up  Learn how to cooperate ('behavioural additionality')  Public policy designed as multi-actor, multi-measures programme (MAP)

17. page 17 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Exploration and evaluation of policy rationales, context and governance – two profiles Driven by innovation (Kind/net) Driven by knowledge generation (Kplus) TypeClose to marketClose to basic research RationaleProject-orientedCommunity of practice-oriented Purpose of participation (funding) Overcome firm-internal barriers for cooperative market-oriented R&D Creation of new cooperation structures; upgrade and broadening of research Cooperation cultureOriented towards well-known partners Oriented towards most excellent partners Time horizonShort-term resultsMedium-term, knowledge creation

18. page 18 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Formative, qualitative poles – results of example  Overall assessment: two different programme approaches justified, to be better profiled  Results and recommendations for Kind/net:  Develop clear profile as innovation programme; adapt funding level (below research funding)  Improve programme management (e.g. transparency)  Results and recommendations for Kplus:  Provide stable funding and transparent rules  Involve local authorities  Extend inter-centre collaboration

19. page 19 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Formative, qualitative poles – conclusion from example  Advanced innovation policy instruments are increasingly complex (MAP)  Problem: strategic fit of policies – approach, instruments, implementation  Formative evaluation as a source of strategic intelligence,  providing evaluative inputs for reflexive, incremental policy-development  needs qualitative understanding of rationales, context and governance  including multiple perspectives of different actors and levels  Formative, qualitative evaluation approaches are indispensable,  quantitative and summative inputs (e.g. on outputs and performance) are very helpful

20. page 20 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning General principles of strategic intelligence  Principle of participation: strategic intelligence realises the multiplicity of actors’ and stakeholders’ values and interests involved in innovation policymaking (multiple perspective approach).  Principle of "objectivisation": strategic intelligence "injects objectivised" information into the policy arena, i.e. the results of policy/strategy evaluations, foresight exercises or technology assessment, and also of analyses of changing innovation processes, of the dynamics of changing research systems and changing functions of public policies.  Principle of mediation and alignment: strategic intelligence facilitates debates and "discourses" between contesting actors in related policy arenas, thus mediating and "moderating", supported by "objectivised" information to be "digested" by the struggling parties.  Principle of decision support: strategic intelligence requires forums for negotiation and the preparation of policy decisions.

21. page 21 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning Contact: s.kuhlmann@isi. fraunhofer.de j.edler@isi.fraunhofer.de Info: www.isi. fraunhofer.de