1. Do European Social Fund labour market interventions work? Counterfactual evidence from the Czech Republic. Vladimir Kváča, Czech Ministry of Labour and Social Affairs Oto Potluka, University of Economics, Prague

2. Up next…  Introduction  The Intervention  Methods Used and A Few Results  Conclusion  Discussion Contents

3. CIE HRE counterfactually evaluates EU support in the CZ  CIE HRE – Counterfactual Impact Evaluation of the Operational Programme Human Resources and Employment  Examines the impact of employee training (funded by the European Social Fund) on companies’ employment.  Research in progress (2008-2010, 2011.)  See http://dm.fba.vse.cz/projects/counterfactual- impact-evaluation-of-the-op-hre/ for more info. http://dm.fba.vse.cz/projects/counterfactual- impact-evaluation-of-the-op-hre/ Introduction

4. We are looking for the effect of companies training employees  Grants for employers to train employees with the aim to prevent unemployment via improving companies’ competitiveness and peoples’ skills.  Grants between €40,000 and €600,000.  Any company in the Czech Republic eligible, employees in Prague excluded.  There are 31 604 firms in the sample, consisting of non-applicants (n= 28 974), successful applicants (n= 1447), and rejected applicants (n= 1183). Intervention 4

5. The selection procedure rules out projects below 65 points Intervention StepDone byOutput Formal check (signatures, compulsory annexes, eligibility, etc.) Ministry officer responsible for a particular call for proposals YES / NO Content appraisal (quality of project proposal assessed according to several criteria) Two referees randomly selected. If opinions too divergent, a third one is added, and the most extreme opinion eliminated. Points, if average < 65 points  NO, if average >= 65  YES. Selection commission (checks previous steps and project ranking, can propose changes in budget) Commission of external stakeholders and ministry officials Checks previous steps, can change project status from YES to NO (never vice versa). Ranks projects by number of awarded points, and decides the cut-off point for supported projects. (Depending on available money Grant contract signedMinistry official and project representative Project author refuses  project is not started (happens rarely).

6. Counterfactual methods create comparison groups, in many ways  In essence, counterfactuals create comparison groups to answer the question “what would have happened without the support.”  We use a whole range of counterfact. methods: Regression discontinuity Differences in differences and propensity score matching Instrumental variables  Expected outcome is the change in number of employees. Methods

7. Regression discontinuity examines projects around a cut-off point  Regression discontinuity exploits a selection procedure cut-off at 65 points.  The assumption is that projects barely above 65 points are as good as those barely below. Methods – RDD 1 0 Probability that project is supported Points from assessment 10 20 30 40 50 60 70 80 90

8. 8

9. 9

10. According to RDD, support for small and medium companies works better Significance: * denotes 10%, ** 5%, *** 1%. Results – RDD Employment (2008-2010) Total difference in number of employees caused by support Difference in number of employees caused by support (company mean) OP HRE costs for one additional employee (EUR) Small firms5 468** 7.33** (1.3, 13.3) 3 900 Medium firmsinsignif. 11.37 (-6.4, 28.4) n/a Large firmsinsignif. -3.19 (-12.2, 6.2) n/a

11. PSM matches comparable companies from another sample  Propensity score matching creates a comparison group according to various observable characteristics, i.e. companies similar by size and industry and region, etc.  We draw the comparison group companies from (i) all other eligible companies, (ii) eligible and interested, but rejected, companies. Method – PSM and DiD

12. DiD compares the supported and comparison groups over time  Difference in differences assumes a similar trend among the treatment and comparison group (see next slide).  In our analysis, we created the comparison group using PSM and then used DiD.  Results: in small and medium companies there is a larger effect on employment. Methods – PSM and DID

13. contrafactual Average change in number of employees per company, 2008 – 2010. Results for companies of all sizes pooled. Treatment group before and after: 366 - 352 = 14 Control group before and after: 192 - 164 = 28 Impact = 28 - 14 = 14. Difference before Difference after Impact 13 14 jobs per company saved on average 338

14. Employment (2008-2010) Firm Size Total difference in number of employees caused by support Mean difference in number of employees caused by support (per company) OP HRE costs for one additional employee (EUR) PSM supported: rejected (1447 : 1183 companies) Small + 3 357** + 4.50** (0.5, 8.5) 6 367 Medium + 3 071** + 6.90** (1.1, 12.7) 4 226 Largeinsignif. -20.14 (-46.1, 6.1) insignif. PSM (supported: uninterested) Small + 3 775** + 5.06** (0.6, 10.6) 5 659 Medium + 4 980** + 11.19** (4.5, 17.9) 2 606 Largeinsignif. -3.12 (-11.1, 5.1) insignif. 14

15. IV uses the personal biases of project referees  Instrumental variables exploit the difference in average points over all projects a referee gives (his different "strictness" / "generosity")  We look at similar projects which were (randomly) alocated different referees. Identity of referee influences the chance of support, and thus also the chance of final impact. Methods - IV

17. Even a good method can fail to provide numerical results, but those aren’t everything  No significant numerical results.  Real lessons learned, however: we are trying to minimize the role of referees for the next period, and we are also aware of the need to select and train referees better, so as to standardize their approach to applications.  We will test whether the weather influences referees’ decisions. Results – IV

18. Small and medium companies seem to be a better investment  Support of small and medium companies has significantly larger returns on the number of employees (similarly see Mouqué, 2012).  On average, 1.2 persons per project are employed in its implementation.  Using the results in focusing future calls for proposals (2014+).  Expecting relevant results for profits and sales, a questionable causal mechanism within our time frame. Conclusions Mouqué, D. (2012), What are counterfactual impact evaluations teaching us about enterprise and innovation support,http://ec.europa.eu/regional_policy/sources/docgener/focus/2012_02_counterfactual.pdf

19. Conducting the evaluation is just the beginning, we must also use it  Further discussion and presentation to the evaluation communities in both the Czech Republic and the EU.  Looking for other programmes to evaluate.  Evaluation timing is important. Email: vladimir.kvaca@mpsv.cz, potluka@vse.czvladimir.kvaca@mpsv.czpotluka@vse.cz Conclusions