Feedback from Peer Review on 'Counterfactual Impact Evaluation' Tamara SMETANOVÁ Ministry of Labour and Social Affairs, Czech Republic Kamil VALICA DG EMPL – Impact Assessment, Evaluation Unit ESF Evaluation Partnership meeting, Rome, 26 November 2014

Content 1. Mutual Learning Programme – Peer Review 2. Pilot Counterfactual Impact Evaluation in the Czech Republic 3. Lessons learnt at Peer Review

1. Mutual Learning Programme Peer Review

Mutual Learning Programme Supporting tool of the Commission to encourage exchange of good practice between countries Peer Review: Gather host country and peer countries to discuss a topic Support each official by an independent expert Encourage networking and identify opportunities for mutual learning and policy influence

Peer Review in Prague Hosted by the Czech Ministry of Labour and Social Affairs, 6-7 October 2014 Participants: Belgium, Croatia, Estonia, Greece, Ireland, Italy, Lithuania, Portugal, Slovakia, Spain, Norway and Turkey The evaluation by participants was very good (it was not counterfactual…)

2. Pilot Counterfactual Impact Evaluation in the Czech Republic

Pilot CIE in the Czech Republic The intervention: Support Area ADAPTABILITY €470 million assigned for ADAPTABILITY in About 25 % of all finances intended for HREOP (Human Resources and Employment Operational Program) Pilot CIE projects € 282 mil Support Area 1.1 € 470 EUR Other Priority Axis € mil

The pilot CIE methodology Various methods has been used, tested and compared The pilot trial. An afford to increase awareness of CIE in the CR. Evaluation of impact on companies, not individuals Company-level data.

Methods used Instrumental Variables The grant projects were evaluated by external appraisal experts (AE). Taking advantage of differences in strictness of appraisal experts. The identity of appraisal experts was used as an instrument. Regression Discontinuity Design Taking advantage of selection procedure. Threshold of 65 points for proposals to be considered for approval. Comparing successful and unsuccessful, both close to the threshold. Difference-in-differences Driven by lack of unsupported subjects in a system call „Educate yourself“. Supported companies were matched to non-applicants.

Data for CIE Company-level data Business Register (Czech Statistical Office) 34 thousands companies in the CZSO sample (Non-applicants, Successful applicants, Unsuccessful applicants) Substantial time lag in data availability Monitoring system MONIT 7+ Sample Survey data Individual-level The access to individual-level data has been limited so far. Data from CZECH SOCIAL SECURITY ADMINISTRATION and from EMPLOYMENT OFFICE are supposed to be used in the future.

Results Positive impact on employment in middle and large companies. Larger impact of technical trainings than soft skills trainings. Other results mostly insignificant or in one case even contradictory. However it may take more time for the impact to occur. Sustainability of jobs might be an issue as some portion of the created jobs may be directly related to project administration. Regarding the promotional and education role of the project; we perceive highly successful.

Lessons learned One of the approaches (IV) exploited the differences in "strictness"/"generosity“ of AE. positive with respect to scientific reasons negative with respect to fair selection of projects As part of the methodology the selection procedure was examined carefully. We are trying to minimize the influence of AE assigned on success/failure of proposed project in selection procedure.

The CIE Follow-up Counterfactual approach Is to be applied on extended data set covering longer time period. Qualitative approach The qualitative part will complement the CIE results. Evaluates impact even on employees of supported companies. Provides deeper insight into the mechanism of the intervention. To utilize the experiences with pilot CIE Methodology Data (availability, limitations, structure,…) Increase awareness of CIE methods Individual-level data are supposed to be available for the next programming period.

3. Lessons learnt at Peer Review

Lessons learnt 1 Need of advance planning – ideally when intervention designed Ensure timely access to public datasets and motivate data-holders through inclusion in steering committees No CIE is better than poor CIE: aim for good CIE Develop strong evaluation culture through training to stakeholders (evaluators, policy makers, politicians and the media)

Lessons learnt 2 Need to disseminate results understandable to the stakeholders/public CIE shall first lead to reflection on possible improvement rather than uninformed exclusion of certain target groups or dismissal of intervention Need to combine CIE with qualitative evaluation methods Consideration needs to be given to randomised control trials