The Role of Evaluation in 2014 – 2020 period: Czech ESF experience Vladimír Kváča Ministry of Labour and Social Affairs

Presentation overview Current experience: Evaluative culture Evaluation designs Proposals for a better future

Current experience Evaluative culture

Capacity building in evaluation process 4 Evaluation Mandate Decision Evaluation Design / ToR Evaluation Design / ToR Evaluation Report Evaluation Execution Recommendation Internal eval. staff Consultants Management 4

Current experience Evaluation design

end of project evaluation Typical Design: Post-test only of project participants X P Project participants 6 Based on: Jim Rughs presentation, Stockholm, Semptember 2011

baseline end of project evaluation Comparison group post project evaluation Longitudinal Quasi-experimental P 1 X P 2 X P 3 P 4 C 1 C 2 C 3 C 4 Project participants midterm 7 Source: Jim Rughs presentation, Stockholm, Semptember 2011

baseline end of project evaluation Pre+post of project; no comparison P 1 X P 2 Project participants 8 Source: Jim Rughs presentation, Stockholm, Semptember 2011

end of project evaluation Comparison group Post-test only of project and comparison X P C Project participants 9 Source: Jim Rughs presentation, Stockholm, Semptember 2011

baseline end of project evaluation Comparison group Pre+post of project; post-only comparison P 1 X P 2 C Project participants 10 Source: Jim Rughs presentation, Stockholm, Semptember 2011

baseline end of project evaluation Comparison group Quasi-experimental (pre+post, with comparison) P 1 X P 2 C 1 C 2 Project participants 11 Source: Jim Rughs presentation, Stockholm, Semptember 2011

No need to go always for longitudinal design with control group. But there are many stronger designs than one-group, post- test only…

Proposals for a better future

Steps for a better future 1 Clear and explicit definition of programme/operations goals that are attributable to it. Preferrably in terms of well-being of people. Be aware of contribution x attribution difference.

Higher work intensity sustained Parents work intensity increased Companies want to share kindergartens costs with parents Prosperity of Companies Kindergartens used Companies employ parents Parents want to work more Parents want to send children in Kindergartens built Kindergarten is affordable (costs, transport) for parents Increased employment rate Inspired by Jim Rugh. Area of control / attribution Area of decreasing influence / contribution

Steps for a better future 2 Go for a better design than one group post-test only. Evaluation design should be known before the operation starts, incl. draft of evaluation question. This requires monitoring system to be adjusted to evaluation needs and to be able to collect baseline data. From current practices it is obvious that large parts of evaluation budgets are spent on re-doing insufficient or not well planned monitoring – this takes money and time.

Steps for a better future 3 Evaluation should be regarded as one of the main instrument to allow necessary responsible flexibility in OP execution / planning. Monitoring and a system of indicator-based targets answers the question are we doing the things right? Evaluation should periodically check if we are doing the right things – do the originally expressed assumptions still hold? Are the programme goals still relevant? Are the targets/indicators free of perverse incentives? What change is caused by us?

Steps for a better future 4 Actively build the evaluative culture – at the level of external consultants, at the level of programme internal evaluation specialists and at the level of programme management as well.

Thank you for your attention