1. ESF Evaluation Partnership Meeting – 13 November 2013
Conclusions of the preparatory study for the ex post evaluation of the ESF
ESF Evaluation Partnership Meeting – 13 November 2013 1

2. Agenda Background of the study: research topics Activities final phase
Conclusions
Testing phase
Proposals for work-packages 2

3. Research topics (according to ToR)
How to assess the effectiveness, efficiency and the socio-economic impacts of ESF
How to make use of available information and evaluation methods
How to design evaluation ‘work packages’
Specific attention to:
Geographical aggregations
Thematic organisation
The role of governance
The role of Europe 2020
Empirical instead of theory based approach 3

4. Activities final phase
Feedback with experts (validation)
Questions / feedback EMPL and further development WP
Check shared actions (some shifts / reallocations)
Further analysis on indicators per shared actions
8 (country) testing reports (general confirmation)
BE, DE, ES, IT, LT, NL, PL, UK
From 2 -> 3 work packages
Input for further development work packages
Revised version interim report (second look on indicators; counterfactual evaluations)
Draft final report

5. Conclusion I: How to evaluate effectiveness, efficiency, impact
Theory based versus empirical based approach
Effects: Gross versus net effects, individuals versus systems and structures, possbilities of counterfactual evaluations
Efficiency: financial data on lowest aggregation level?
Sustainable impact: see effects

6. Conclusion II: thematic classification
Different entry points: Priority themes (article 3 regulation), CoE, Fields of activity
Shared actions is a good starting point
EU2020, addressing some additional themes
Cross cutting issues seems not to be a good starting point

7. Conclusion III: geographical aggregations
This study attempted to cluster MSs making use of a wide range of sources on EU and national/OP level.
Two routes to follow: thematic and data availability
Concluding that:
MSs can be clustered in 3 geographical aggregations: - Nordic/ Continental/ Anglo-Saxon MSs - Southern MSs +PL - Central/Eastern European MSs +CY
MSs can be clustered according to topics which get extra attention: - LLL + target groups (AT, IE, LU, NL); - Human capital/institutional development (CY, EL, IT, BG, EE, HU, LT, LV, RO, SK); - Entrepreneurship (BE, DE, DK, FI, FR, ES, PT, MT, CZ, PL, SI)
Difference between MSs in data availability

8. Conclusion IV: data availability
For >80% of the ESF budget, data availability is assessed moderate to high
Data available for a substantial amount of actions on 'Access to Employment' and 'Improving Human Capital‘
Micro-data which might be useable for ex post evaluation in about 16 OPs
In 21 OPs information on final recipients has been linked with other (administrative) databases. In 9 more this seems to be an option
Repeating a counterfactual analysis, which could make results comparable, would be an option for 10 OPs (out of the 43 analysed) 8 8

9. Conclusion V: indicators
Average 2.5 indicator per action
50 % completely grasp the action and 44 % partly
Similar indicators for several actions; under different themes
Hardly any baseline / target
The same indicator is in some cases output, in other cases result
A2E, HC and SI score best (result and impact) 9 9

10. Results of testing phase (1)
Confirmation of possibilities to use micro-data
Annex XXIII is available
Some MSs do not provide information on activity in which the final recipient participated;
Only average costs per participant
No clear cut off dates of participation
Effect data mostly include employment figures
Micro data is accessible, but:
cost time, depending on competences administrator;
cost money;
there is data protection regulation

11. Results of testing phase (2)
Linking administrative systems is possible in all countries.
Most of the 8 countries provide the opportunity to link databases based on social security ID (BE, NL, IT, ES, LT). DE and LT based on name search.
Possibilities for constructing control groups, also including more socio-economic indicators.
Evaluations often do not cover the shared actions or the theme completely. They often do not focus on results but on outputs. They may be useful, however, for explanation building / telling the story
In all 8 countries there are possibilities of surveying (addresses are available)
Types of outcomes expected (employment effect best to measure)
Conclusion for work packages: positive

12. Developing work packages: criteria
ESF budget share of (clusters of) actions
Data availability
Country coverage
Political relevance
All clusters of actions have been scored on these criteria 12

13. Proposal of work packages
Social inclusion: special focus on employability of vulnerable groups
Human Capital; special focus on youth
Access to employment: special focus on youth, self employed and women 13

14. Work package 1: Social Inclusion
Theme, target groups, instruments
Actions focused on final recipients
Focus on vulnerable groups
Pathways to employment for vulnerable groups as well as interventions directed at social integration
Motivation
Social inclusion is a political priority
Large ESF budget share (15 %)
Large number of actions across MSs (100+)
Relatively many output and result indicators across countries
Micro data available in some countries
Counterfactual evaluations and/or leaver surveys are available in some countries
Linking ESF final recipients with administrative data in some countries
Please see slide 29 14 14

15. Expected results All Member States / EU level:
Insight in allocations and expenditure for social inclusion
Insight in the number of disadvantaged target groups addressed by the ESF (Annex XXIII)
For selected Member States
(AT, BE, DE, FI, FR, LT, LV, LU, NL, UK)
Overview of the kinds of actions implemented
Insight in effectiveness and efficiency of these actions
Insight in impacts (indicators, evaluation studies, statistics, additional data collection)
In MSs where ESF is subsidising a large share of national expenditure in the field of social inclusion, macro economic analysis is possible (new MSs)
Identification of best practices by case studies
In red: something missing 15 15

16. Work Package 2: Human Capital
Theme, target groups, instruments
Actions focused on final recipients
Focus on youth
Motivation
Human capital development element 2020 Smart Growth Strategy
Large ESF budget share (approx. 31%)
Large number of actions across MSs (250+)
Certain similarity of approaches across countries
Indicators show important similarities
Reasonable amount and quality of data 16

17. Expected results All Member States / EU level:
Insight in allocations and expenditure for human capital
Insight in the number of participants/target groups addressed by the ESF (Annex XXIII)
Insight in actions in the field of systems and structures in convergence regions.
For selected Member States
(BG, CZ, DK, FR, HU, IE, IT, LV, MT, PL, PT, RO)
Overview of the kinds of actions implemented
Insight in effectiveness and efficiency of these actions
Insight in impacts (indicators, evaluation studies, statistics, additional data collection)
In MSs where ESF is subsidising a large share of national expenditure in the field of human capital, macro-economic analysis is possible (new MSs)
Identification of best practices by case studies
In red: something missing 17 17

18. Work package 3: Access to employment
Theme, target groups, instruments
Actions focused on final recipients
Special focus on:
Unemployed, in particular young people
Self-employment
Women
Motivation
Combating youth unemployment is a political priority
Large ESF budget share (34%)
Large number of actions across MSs (275+ actions)
Relatively many output and result indicators across countries
Micro-data available in some countries
Counterfactual evaluations and/or leaver surveys are available in some countries
Linking ESF final recipients with administrative data is possible in some countries 18 18

19. Expected results All Member States / EU level:
Insight in allocations and expenditure for access to employment, including
active and preventive measures
self-employment and business startup
Insight in the number of unemployed, youth, self-employed, women addressed by the ESF (Annex XXIII)
For the selected Member States
(BE, BG, DE, EL, ES, HU, PL, RO, SE, SK, UK)
Overview of the kinds of actions implemented
Insight in effectiveness and efficiency of these actions
Insight in impacts (evaluation studies, statistics, additional data collection)
the larger the share of ESF contribution in total public spending, the more likely a macro analysis could become productive
Identification of best practices by case studies
In red: is that true? 19 19

20. Questions ?
I would suggest to limit to these three questions, whereby the 2nd is intentionally very open, that means it can be interpreted as asking for themes but also methodologies. 20