Presentation on theme: "Rheinisch Westfälisches Institut für Wirtschaftsforschung 1 Conceptual Challenges of Evaluating the Impacts of a Sizable Technology Programme Portfolio:"— Presentation transcript:
Rheinisch Westfälisches Institut für Wirtschaftsforschung 1 Conceptual Challenges of Evaluating the Impacts of a Sizable Technology Programme Portfolio: The Promotion of Industrial Collective Research (IGF) Verena Groß, Bernhard Lageman, Michael Rothgang (RWI Essen) Session C: Portfolio Evaluation
Rheinisch Westfälisches Institut für Wirtschaftsforschung 2 Content A Portfolio Evaluation B About IGF C Some Questions Concerning the IGF Program Design D Evaluating IGF: the Challenge E Evaluation Design
Rheinisch Westfälisches Institut für Wirtschaftsforschung 3 A Portfolio Evaluation Portfolio evaluation aims at assessing rationality, programme-specific institutional solutions, effects and efficiency of complex programmes Two central dimensions: –Analyzing the structures, procedures, effects of different components of the programme –Comparative analysis of the programme with respect to similar programmes Methodological solutions? –Evaluation research offers a lot of relevant quantitative and qualitative solutions –Evaluation studies do not lead always to convincing results –Portfolio evaluation remains a very challenging task
Rheinisch Westfälisches Institut für Wirtschaftsforschung 4 B About IGF Old programme: IGF – state-funded collective industrial research: a technology programme introduced in the early 50 th of the German Ministry of Economics The German Federation of Industrial Research Associations "Otto von Guericke" (AIF) is commissioned with the programme execution B.1 Old programme - modern features
Rheinisch Westfälisches Institut für Wirtschaftsforschung 5 B About IGF Modern features: development of industry-wide research networks projects should mainly promote SMEs precompetitive research and results available for every firm few ex ante restrictions on the topics of the research projects project ideas are developed bottom up B.1Old programme - modern features
Rheinisch Westfälisches Institut für Wirtschaftsforschung 6 B About IGF total budget since the early fifties 2 billion Euro (2005 prices) nearly 15,000 collective research projects actual annual budget ca. 90 million, which is to be increased up to 120 million in 2010 600 to 700 new collective research projects per year average project cost about 138,000, project costs reaching from 50,000 to 350,000 normal project period is two years B.2 Costs and projects
Rheinisch Westfälisches Institut für Wirtschaftsforschung 7 B About IGF 103 research associations (in 1954 on 17) in most sectors and technology fields that network: -enterprises (~ 50,000 SMEs) -~ 700 research institutes Examples: Association for Cooperative Research for Hot Dip Galvanizing Research Association of the German Food Industry Society for Chemical Engineering and Biotechnology B.3Industrial Research Associations, Institutes and Enterprises
Rheinisch Westfälisches Institut für Wirtschaftsforschung 8 B About IGF Industrial Research Associations Enterprises Research Institutes Phase 2:Discussion of competing research ideas and elaboration of proposals within research asscociations Phase 3:Presentation and official submission of proposals to AIF Phase 4:Evaluation of proposals and funding recommendation to the Ministery Phase 5:Final decision about funding by the Ministery B.4Project Genesis Phase 1:Identifying research problems and development of research ideas
Rheinisch Westfälisches Institut für Wirtschaftsforschung 9 CTwo Questions Concerning IGF Programme Design C.1 Question 1: Bottom-up vs. Top-down Approach (I) A) The traditional programme philosophy and institutional construction Bottom-up: the IGF decide themselves about relevant technological fields and research projects according to the principle of self-organization B) Alternative approach: Top-down: the relevant technological fields could be defined by expert bodies working on behalf of the Ministry What is the optimal solution: A or B or a mixture of both?
Rheinisch Westfälisches Institut für Wirtschaftsforschung 10 CTwo Questions Concerning IGF Programme Design C.2 Question 1: Bottom-up vs. Top-down Approach (II) Clear strengths of the bottom-up approach Good chances for identifying real needs of SMEs in branches where industrial research associations are active The university institutes are bound to look for active and passive SME support for their research Incentives for promoting transfer activities are immanent to IGF structures
Rheinisch Westfälisches Institut für Wirtschaftsforschung 11 CTwo Questions Concerning IGF Programme Design C.3Question 1: Bottom-up vs. Top-down Approach (III) Bottom-up in practice: project portfolio Sectoral project distribution: large part of project funding for old industries (estimated 40% over the last ten years) Strong weight of traditional technologies Up to the 90 th no high-tech-projects, since 2002 high-tech- projects make about 10 to 15 % of the total budget Bottom-up in practice: institutional structures The challenge of combining well functioning institutional structures with flexibility
Rheinisch Westfälisches Institut für Wirtschaftsforschung 12 CTwo Questions Concerning IGF Programme Design C.4Question 1: Bottom-up vs. Top-down Approach (IV) New developments in IGF: New research associations were established in the 90s, specialized in high-tech and cross-sectional technological fields A portion (presently about 15 %) of financial means for high- tech projects is reserved The traditional key of distributing funds to the research associations is going to be abolished
Rheinisch Westfälisches Institut für Wirtschaftsforschung 13 CTwo Questions Concerning IGF Programme Design The logic behind IGF: Limited financial, material and personal resource base of SMEs leads them to underinvestment in R&D IGF as compensation for structural weaknesses of SMEs –by finding solutions for technical problems which arise in SME production processes –by inspiring SMEs to engage in R&D activities themselves –by creating innovation and research networks between SMEs C.5 Question 2: Target Group SMEs (I)
Rheinisch Westfälisches Institut für Wirtschaftsforschung 14 CTwo Questions Concerning IGF Programme Design SME participation revisited: The IGF definition of SMEs is rather broad (turnover < 125 Mio., < 500 employees) compared to EU definition An increasing part of dependent SMEs Securing an adequate share of SMEs in IGF bodies and project advisory boards seems to be a problem The question whether IGF projects strengthen the innovative capacity of SMEs has to be convincingly answered The basic concept is to questioned: What is the target group? - Isolated, single SMEs or SMEs in industrial innovation systems? C.6 Question 2: Target Group SMEs (II)
Rheinisch Westfälisches Institut für Wirtschaftsforschung 15 CTwo Questions Concerning IGF Programme Design C.7Conclusion Evaluation task: Do the new regulations improve IGF performance by strengthening the part of modern highly developed technologies in the project portfolio? better enabling SMEs to take actively part in the IGF system?
Rheinisch Westfälisches Institut für Wirtschaftsforschung 16 D Evaluation IGF: the challenge D.1 Identifying economic project results Complications Project results are by definition (and in practice!) a public good, that means usable by everybody –Proof of concrete implementation activities of IGF project results in single firms only is of limited value for evaluation –No firm which uses IGF results has to render an account of this –Results become part of the general body of technological knowledge, they may not be identifiable as IGF results by users A time lag between closing date of IGF projects and practical uses reaching from 3 to 10 years
Rheinisch Westfälisches Institut für Wirtschaftsforschung 17 D Evaluation IGF: the challenge D.2 Identifying the relevance of projects for SMEs Complications The idea that projects are relevant because they have been carried out under the IGF regulations is to be questioned Bottom-up has to be weighted against the factual selectivity of the system –articulation of SME needs depends on the presence and power of research associations and –the engagement of research institutes which are specialized in the respective technological field Comparing economic relevance of different innovations
Rheinisch Westfälisches Institut für Wirtschaftsforschung 18 E Our Evaluation Design E.1 Levels of Analysis and Components Micro Level: Investigating genesis, realization and application of research results on the project level Meso Level: Analyzing the activities of IGF actors and bodies: –industrial research associations –the role of AiF and its central bodies –research institutes which participate in IGF projects –IGF projects in the framework of sectoral innovation systems Macro Level: Comparing IGF with other technology programmes oriented towards SMEs Analysing the IGF project portfolio in the framework of the new and old programme design
Rheinisch Westfälisches Institut für Wirtschaftsforschung 19 E Our Evaluation Design E.2 Instruments (I) Interviews: Expert interviews with: –Representatives of (all) industrial research associations (over 4 years) –Persons in charge of single projects, selected by chance –Independent experts evaluating project proposals and results –Representatives of central IGF bodies Surveys: Different enterprise surveys: –General survey of industrial firms with respect to their knowledge on and estimation of IGF –Project oriented surveys with respect to estimated relevance of the projects and the use of project results
Rheinisch Westfälisches Institut für Wirtschaftsforschung 20 E Our Evaluation Design E.3 Instruments (II) Case studies of the role of IGF in sectoral innovation systems: The selected sectors represent: –traditional (but developed) technologies like food and furniture industries –far developed technologies but well established industrial products (like mechanical engineering industries and the automotive sector) –high-technology fields or high-end cross-sectoral technologies Quantitative analysis: Econometric Analysis of IGF programme data and the results of our surveys Comparing the participation of SMEs in different technology programmes, as far as relevant data are accessible
Rheinisch Westfälisches Institut für Wirtschaftsforschung 21 Thank you for your attention!