The Dutch Innovation voucher Brussels Oct 15, 2010 The Dutch innovation voucher and evaluation issues Marc Van der Steeg CPB Netherlands Bureau for Economic Policy Analysis The Hague, The Netherlands

The Dutch Innovation voucher Brussels Oct 15, 2010 The evaluation problem A two-way relation ► causal: policy leads to more innovation ► correlation: innovative firms make more use of innovation policy instruments How to disentangle these two relations? ► add covariates to the regression equation ► do highbrow econometrics (e.g. matching) ► or... Controlled experiment ► experimental group and control group ► random allocation ► difference is causal impact

The Dutch Innovation voucher Brussels Oct 15, 2010 The Dutch innovation voucher pilot 2004 Goal: “Lead them to water and pay them to drink” (Angrist et al., 2006) ► introduce SMEs to public research institutes ► [market-oriented incentives for research institutes] Characteristics ► credit note, worth max EUR ► for SME’s only ► no own contribution required ► application-oriented research questions ► placed with a defined group of institutes ► valid for half a year ► no restrictions on level of question or technology ► 100 vouchers available ► LOTTERY if demand exceeds supply

The Dutch Innovation voucher Brussels Oct 15, 2010 Example of voucher project “Biodiesel from Africa” ► Seeds of tropical plant “Jatropha” can be used for production of biodiesel ► However, current oilpresses less efficient than for production of biodiesel from coleseed: 40 % of oil lost ► Diligent Energy Systems used an innovation voucher and asked Technical University of Eindhoven to improve efficiency of process –PHD-student carried out experiments –Plans to do further research on adaptation of coleseed press for “Jatropha”

The Dutch Innovation voucher Brussels Oct 15, 2010 Research questions What is the effect of the innovation voucher on: 1.the commissioning of assignments to public knowledge institutes ► Direct effect (input additionality) ► Persistence effect (behavioural additionality) 2.Innovation (output additionality) ► Product innovations ► Process innovations

The Dutch Innovation voucher Brussels Oct 15, 2010 Data (1) 1,044 applications on September 17th, 2004 Lottery: 100 winners, 944 losers Telephone interviews May, 2005 ► 100 winners ► 500 randomly selected losers ► questions about actual behaviour ► questions about counterfactual behaviour Response rate ► 71 winners (71%) ► 242 losers (48%) Second round of telephone interviews in September 2006

The Dutch Innovation voucher Brussels Oct 15, 2010 Data (2) No significant differences between winners and losers in background characteristics: ► firm size (e.g. 19 versus 16 employees) ► region ► sector We can confidently attribute any differences in outcomes to the voucher policy instrument ► however, we still add all available covariates to regression equation

The Dutch Innovation voucher Brussels Oct 15, 2010 Analysis (1): Direct effect Data; during voucher period: ► 62 out of 71 (= 87%) winners commissioned a project ► 20 out of 242 (= 8%) losers commissioned a project Effect estimates ► 13% of the vouchers not used (= (71-62)/71) ► 8% crowding out (= 20/242) ► 79% additional assignments (= 62/ /242) ► standard errors are small Counterfactual behaviour ► 76% winners say: without voucher, fewer projects ► 86% losers say: with voucher, more projects Conclusion: 8 out of 10 vouchers additional

The Dutch Innovation voucher Brussels Oct 15, 2010 Analysis (2): Persistence and innovation output effect 1.Persistence: no effect ► Voucher winners do not carry out more assignments than voucher losers in one and a half year after voucher period. 2.Innovation output ► indications for small effect on process improvements ► no significant effects on realization of new or better products and new processes

The Dutch Innovation voucher Brussels Oct 15, 2010 Crucial issues for design and evaluation of voucher scheme 1.Identify measurable goals of voucher 2.Randomization is crucial for evaluation! 3.Sufficient vouchers and size of control group 4.Collect pre-treatment characteristics / behaviour of firms ► via application form; administrative datasets 5.Cooperation to evaluation obligatory ► also for control group of losers in lottery! 6.Avoid abuse ► e.g. print assignment to copy shop of university ► which market failure does the voucher solve for?

The Dutch Innovation voucher Brussels Oct 15, 2010 The Dutch voucher instrument after 2004 pilot 2 new larger pilots in 2005: 1000 vouchers Definitive instrument since 2006: ► around 6000 vouchers annually ► Split in small and large vouchers 2010: introduction of private vouchers

The Dutch Innovation voucher Brussels Oct 15, 2010 Other evaluation activitities (1) Dutch cohesion policy (1) 2007 Dutch government introduced plans to improve 83 disadvantaged neighborhoods Substantial budget of 350 million euro per year Neighborhood action plans with goals to improve housing, jobs, education, integration and safety.

The Dutch Innovation voucher Brussels Oct 15, 2010 Other evaluation activitities (1) Dutch cohesion policy (2) Ranking of neighborhoods on various socio- eoconomic indicators: worst 83 have been selected Evaluation design: ► Exploit ranking with cut-off at neighborhood 83 ► Exploit availability of before and after information on wide range of relevant outcome indicators Compare development in 1-83 with that in Local diff-in-diff design: versus

The Dutch Innovation voucher Brussels Oct 15, 2010 Dutch cohesion policy (3) Effect on primary school test scores

The Dutch Innovation voucher Brussels Oct 15, 2010 Other evaluation plans: Innovation and science policy Innovation loans ► Selection of proposals on basis of certain criteria / judgements ► Compare performance of just selected versus just not selected companies Scholarships for talented researchers “Veni & Vidi grants” ► Selection of researchers on basis of ranking of proposals by comittee ► Compare scientific output and careers of just selected versus just not selected applicants

The Dutch Innovation voucher Brussels Oct 15, 2010 Contact Report on innovation vouchers: 58/ Contact: