1. 2011 Evaluation of the Innovation Voucher Programme in Poland: Results, Methods and Research Opportunities Polish Agency for Enterprise Development Brussels, April 14 th 2011 Jacek Pokorski Aleksandra Jaskólska

2. Presentation 1. Innovation Voucher Programme 2. About the evaluation 3. Beneficiaries characteristics 4. Methodology 5. Key findings 6. The most important recommendations 7. Lesson learned - research opportunities

3. About the Programme Pilot project 2008-2010, will be continued in 2011 Financed from domestic funds The main objective: initiation of contacts between micro and small entrepreneurs and science institutions For micro and small enterprises For research & development services provided by scientific institutions Vouchers worth about 3750 euro (15 000zł) Innovation Voucher Programme

4. SCIENTIFIC INSTITUTION (service provider) Programmes scheme PAED BENEFICIARY FUNDS APPLICATION REPORT ON THE SERVICE AGREEMENT CHOICE Innovation Voucher Programme SERVICE REPORT ON THE SERVICE

5. The evaluation focused on the first two editions of the Programme (2008, 2009) Research duration: september – november 2010 The primary evaluation goal was to assess the effectiveness, utility and sustainability of the programme results We wanted to know more about: Characteristics of applicants, beneficiaries and projects carried out under the 1 st and 2 nd edition, Programmes influence on starting business cooperation between enterprises and scientific institutions, Identification of the effects, Looking for proposals for the Programmes modifications About the evaluation

6. Methodology DESK RESEARCH PAED REPRESENTATIVES INDIVIDUAL IN - DEPTH INTERVIEWS BENEFICIARIES INDIVIDUAL IN- DEPTH INTERVIEWS APPLICANTS WHO WERE GRANTED THE SUPPORT BUT EVENTUALLY REFUSED IT INDIVIDUAL IN-DEPTH TELEPHONE INTERVIEWS UNSUCCESSFUL APPLICANTS INDIVIDUAL IN- DEPTH TELEPHONE INTERVIEWS SCIENTIFIC INSTITUTIONS INDIVIDUAL IN-DEPTH TELEPHONE INTERVIEWS BENEFICIARIES COMPUTER ASSISTED TELEPHONE INTERVIEW EXPERT REVIEW INDIVIDUAL IN -DEPTH INTERVIEWS Inception report Final report N=256 N=10 N=13 N=3 N=2

7. The avarage enterprise age was 9.6 years 66% enterprises were functioning longer than 5 years 55.1% enterprises income was less than 250.000 euro, 16.8 % - from 250.000 to 1 mln euro Only 6% Beneficiaries decided to invest their own funds in their projects (projects above 4 575 euro [18 300 zł]) N=256 Beneficiaries characteristics INDUSTRY 29,2% PROFESSIONAL, SCIENTIFIC AND TECHNICAL ACTIVITY 22,9% TRADE AND REPAIRS 15,9% INFORMATION, COMMUNICATION 12,5% OTHER 19,8% BENEFICIARIESS SECTORS

8. Projects objective Beneficiaries were implementing projects which objectives were… N=256 Key findings %

9. 9 The use of projects results 79.9% Beneficiaries declared that they use projects results (N = 167) 75,9% enterprises currently produce products introduced or developed within the Programme 78,5% use technology introduced or developed within the Programme 94,7% sell product certified within the Programme 90% exploit analysis developed within the Programme 83,3% apply solutions supporting enterprises management implemented within the Programme Key findings

10. 10 Effects of the Programme Almost 58% of Beneficiaries declared positive influence arising out of their participation in the Programme and c.a. 35% underlined that positive results should occur in the future. According to the Beneficiaries opinions, the value of the Programme is related with: improved quality of the offered products (almost 68% of opinions about the very high or high impact of such factor), positive impact of participation in the Programme on the Beneficiarys image (approx. 61%), competitiveness and market position (57.2%), better management (55.5%). The Programme had lower influence on enterprises income, nevertheless such influence was regarded as considerable (32.5% of respondents declared high or very high importance of the supported projects role in this aspect). Key findings

11. 11 N=256 96.1% Beneficiaries were satisfied or very satisfied with cooperation with scientific institutions (services providers), 91.4% declared that no problems appeared during cooperation. A major group of the Beneficiaries (approx. 41%) continued cooperation with scientific institutions upon completion of the service financed under the Innovation Voucher Programme. 46% declared various plans concerning cooperation established with scientific institutions in the future in relation to new research & development projects. Among them almost half (c.a. 54%) intends to launch such cooperation within the next 6 months. Key findings Cooperation with the scientific institutions We are going to start cooperation… Up to 3 months Within 3- 6 months Within 6- 12 months DK 28.0%26.3%37.3%8.5% N=118

12. Recommendation 1 The period for implementing the project should be at least 4-6 months, which shall facilitate smooth performance and limit the number of beneficiaries withdrawing from the Programme. Recommendation 2 Works should be commenced to launch a sub-programme offering grants of a higher value – approx. 7500 – 12500 euro (PLN 30000 – 50000 ). If such sub-programme is launched, it shall be absolutely crucial to allow for a longer period of project implementation, approx. 6 months. In case of such sub-programme: A requirement should be introduced for the entrepreneurs own contribution in the project This shall help minimize the risk of projects of questionable usefulness. The beneficiarys own contribution cannot be excessive and should range from 10 to 20%. Recommendation 3 As the higher amount of grant shall stir greater interest in the sub-programme, consideration should be given to qualitative evaluation criteria included in the evaluation of applications so as to avoid the necessity of closing the enrollment process too quickly. Unfortunately, the introduction of such criteria may extend the evaluation process; hence, such criteria must be as simple as possible. Important Medium Key findings and the most important recommendations The evaluations general conclusion is that Innovation Voucher has positive influence on direct contacts and cooperation between micro and small enterprises and scientific institutions.

13. WHY? Programme has many underlying assummptions, which are not explicitly shown in the Programmes documentation. There is a risk, that the methods we used didn't revealed all the Programme's causal chain failures. WHAT? Programme's assumptions (not explicitly made) HOW? Mapping out the programmes theory (reconstruction), In-depth desk research In-depth qualitative methods An expert panel (involving experts which were implementing similar projects in other MSs) Better reconstruction of the Programme's theory Focusing on projects quality and their sustainability Estimating impact in a more rigorous way What could be done to improve measurement and conclusions ? Better reconstruction of the programme's theory Lesson learned

14. Research opportunities Focusing on projects quality and their sustainability WHY? We know that projects are useful for beneficiaries but we know little about their quality. The Programmes requirements for service providers (scientific institutions) were not very strict, that is why projects quality verification process was limited. We also have problems with assessing how works the long-term sustainability of the Programme. WHAT? Projectss quality and their long–term influence on enterprises HOW? Case studies (selected projects) Focus Group Interviews Benchmarking (involving similar programmes implemented in other MSs)

15. Research opportunities WHY? We do not know to what extent the Programme itself stimulates cooperation between micro and small enterprises and scientific units. We rely on information about the Programmes effects declared by the beneficiaries, that is why we need more rigorous measurement relating reliable counterfactual situation. WHAT? programmes effects HOW? Control group (problems: limited capability – for example those applicants, who were granted the support but eventually refused it or other scientific institutions clients) Estimating impact in a more rigorous way

16. Other current PAEDs evaluations of pro-innovative programmes The Innovation Barometer – on going evaluation of 13 Measures of The Innovative Economy OP, 2007-2013, addressed to polish companies and the Business Support Org. a systematic approach to tracking the situation of beneficiaries after receiving the support (from programmes theory, outcomes indicators, tools for research to multi-evaluation scheme implementation - with net effect measurement in certain Measures IE OP) Long-term impact evaluation of pro-innovative investment grant scheme, implemented during the previous financial perspective 2004-2006 the first results are available on www.parp.gov.pl (a year after The SOP Improvement of the competitiveness of enterprises completion)www.parp.gov.pl Evaluation of the quality of pro-innovative services for enterprises, delivered by the Business Support Org. under The Innovation Economy OP, 2007-2013. the wide range of the mystery shopping techniques have been applied (evidence-based feedback from potential beneficiaries)

17. Thank you for your attention Polish Agency for Enterprise Development 81/83 Pańska Street 00-834 Warsaw, Poland www.parp.gov.pl For more information on the evaluation in the PAED, please contact us: jJacek_Pokorski@parp.gov.plAleksandra_Jaskolska@parp.gov.pl