1. Evaluation of an ESF funded training program to firms: The Latvian case 1 Andrea Morescalchi Ministry of Finance, Riga (LV) 10-11 March 2015 L. Elia, A. Morescalchi, G. Santangelo Centre for Research on Impact Evaluation (CRIE) Joint Research Centre, European Commission crie@jrc.ec.europa.eu

2. CRIE: What is it & Who is in it? What? Centre for Research on Impact Evaluation Joint DG EMPL-DG JRC initiative Established in June 2013 Support to MS and DG EMPL to set up the necessary arrangements for carrying out Counterfactual Impact Evaluations (CIE) of ESF funded interventions Who? Researchers with a background in Economics and Statistics 2

3. at DG JRC  in its Ispra (Italy) site  3 CRIE: Where?

4. Difficult economic times  Even more pressing need for the EU to demonstrate the achievements of ESF and other financial instruments 2014-2020 programming period:  shift toward impact evaluation DG EMPL’s initiative to help MS in this shift toward impact evaluation... CRIE in collaboration with DG JRC 4 How was CRIE born?

5. 1. The intervention 5

6. The intervention – 1st call Type: training to employees of firms. The type, duration and number of trainings can vary over firms and employees within firm. Motivation: improve the performance and competitiveness of firms by enhancing the skills and the productivity of their employees. Eligibility: associations formed by at least 5 firms belonging to the same industrial sector can apply. Duration: March 2009 - April 2010. Participants: 14 projects selected for a total of 111 firms 6

7. The intervention – 2nd call Duration: January 2011 – July 2015. Very similar to 1 st call except: 7 1.total turnover of project co-applicants should exceed 142.29 million Euro in the last year 2.the applicant association should have been at least 3 years in operation 3.Larger group of recipients (>1,000 firms)

8. 2. Related literature on training to firms 8

9. Human capital theory Returns to on-the-job training estimated by: individual productivity measures wages: proxy for individual productivity employment prospects: results of enhancing individuals’ skills and competencies (Latvia: Dmitrijeva and Hazans, 2007) overall firm level outcomes (non-CIE methods) value added or sales (absolute or per-capita value): measures of firm performance Training characteristics to be taken into account: Type Hours Duration Costs 9

10. Explanatory variables Firm Capital Labor R&D Sector of activity Size Location Workforce (shares) Occupation Education Experience Age Gender Worker turnover 10

11. 3. The data 11

12. Data description Data sources: (1)project database from the Investment and Development Agency of Latvia (IDAL): # of employees receiving training; (2)business data collected from the State Revenue Service; Period: Yearly firm level data available for 2008-2013 Variables: Firms’ age, # of employees, location, NACE code, (after- tax) turnover, production costs, profit, salaries, assets, capital investments, fixed assets, long-term intangible investments 12

13. Data description – 4 main firms groups 1.Treated in 1 st call: 111 2.Treated in 2 nd call: 1,118 3.Never treated: 30,557 (nearly 40% of the country total), 4.Treated in both calls: 38, not used in the analysis Sample adjustments: missing data and outliers removed from the sample: group (1) drops to 66 firms starting 2 nd call before December 2010 are removed: group (2) drops to 391! 13

14. Data description – sample statistics 14

15. Data description – profit per employee 15

16. Data description – turnover per employee 16

17. 4. Methodology 17

18. DiD combined with PSM Technique: combination of Difference-in-differences (DiD) with Propensity Score Matching (PSM). PSM: used to select a control group of firms similar to treated firms in terms of observed characteristics. DiD: applied on the sample of treated and untreated peers. 18

19. Propensity score matching Propensity score matching (PSM) is used for selecting control units from two comparison groups in 2008 Variables used for matching: capital investment, long-term intangible investments, production costs, pre-program profit and turnover (all normalized by the number of employees), age, number of employees, NACE sector of activity and geographical location Four algorithms are employed: one-nearest neighbor, with and without replacement, three-nearest neighbors and kernel matching 19

20. Difference-in-differences (DiD) 20

21. 5. Results 21

22. Results for profit per employee 22 Short-term impact

23. Results for (log) turnover per employee 23 Short-term impact

24. 6. Limitations 24

25. Results to be taken with extreme caution… 1.Information on the kind, duration and number of training courses taken by employees is not available 2.exact start and end date of treatment not available 3.strong selection into treatment 4.association identifier not present in the data 5.information not available at the individual level 6.size of the treatment group is very small 7.only 1 year available before the intervention 25

26. 7. Concluding Remarks 26

27. Insights and recommendations The limitations encountered in the present evaluation make it difficult to discuss the replicability and the scaling up of the intervention under scrutiny. Future availability of more detailed data and information will be critical to produce more reliable and robust impact evaluations 27