1. Presentation to ESF Evaluation Partnership
ESF Ex-post Evaluation: Supporting the integration of disadvantaged groups into the labour market and society
Presentation to ESF Evaluation Partnership
13 March 2015

2. Overview Aims, definitions and questions for the evaluation
Method of approach
Research results so far
Next steps in the finalisation of evaluation

3. Aims of the evaluation
To assess the achievements of ESF investment in supporting the integration of disadvantaged groups into the labour market and society – including:
The distribution and effectiveness/efficiency of ESF use;
The socio-economic impacts resulting and their sustainability; and
Good practice/key success factors – to inform future investment.
Two main components:
Review of the ESF investment in ESF investment in supporting the integration of disadvantaged groups into the labour market and society across EU; and
Detailed assessment within 8 Member States
Key areas of investigation:
Mapping target groups and activities
Effectiveness and efficiency – expenditure, outputs and results
Assessing gender sensitivity, sustainability and CAV

4. Key parameters in the evaluation
Focus on Priority Axes related to the Social Inclusion policy field (certain overlap with other EU level ex-post evaluations, especially concerning Access to Employment)
OP architecture – assess below the level of Priority Axes and analyse achievements of constituent actions and interventions
The cut-off date for OP data:

5. Method of approach Horizontal 27 Member State overview :
Review of strategic choices, expenditure, outputs and results (at the level of Priority Axes)
Data sources: SFC data, analysis of AIRs 2013
In-depth assessment in 8 MS (AT, CY, FI, LT, LU, NL, RO, UK):
Analysis of overall strategy for ESF investment into SI
Mapping below the Priority Axes: actions and interventions
Assessment of effectiveness, efficiency, sustainability, gender sensitivity, Community added value and good practices (58 interventions in 8 MS)
Drawing conclusions on the past and identifying lessons for the future
Data sources: AIR 2013, monitoring documents and monitoring system extracts, interviews in the MAs, IBs and other stakeholders

6. Framework of clusters Cluster 2c
Individual level
Cluster 1
Supporting and enabling actions (preparatory)
Cluster 2a
Advice, counselling, guidance and training
to support labour market integration
Cluster 2b
Actions which have employment as an output
Cluster 2c
Actions aimed at sustaining employment
Cluster 3
Pathway approaches (multi-faceted provision, which can feature combinations of measures provided in Clusters 1&2)
Systemic level
Cluster 4
Systemic measures influencing systems, institutions or cultural contexts (incl. capacity building actions & activities to tackle underlying discriminatory attitudes)

7. Research findings to date – the overall strategy of ESF SI investment in 8 in-depth MS (1)
SI interventions positioned differently within OPs; even when the main SI interventions are contained within specific PA, disadvantaged or vulnerable groups also feature as target groups in other PAs.
Total funding for SI Priority and sub-priority axes would vary just under 10% in LU to over 20% in AT and FI. Lack of data to compare ESF SI funding with national expenditure on similar activities, but available evidence suggest that a key role of OP to support SI interventions in LT and RO.

8. Research findings to date – the overall strategy of ESF SI investment in 8 in-depth MS (2)
A wide range of disadvantaged or vulnerable target groups identified, incl:
young people (including those NEET, early school leavers and those at risk);
individuals with disabilities or limiting health conditions (incl. mental health issues, individuals with alcohol or drug addiction issues)
migrants and individuals from ethnic minority communities (including Roma);
Other individuals at distance from the labour market (incl. LTU and the economically inactive, older workers, refugees and asylum seekers, offenders, ex-offenders and lone parents).

9. Research findings to date – the overall strategy of ESF SI investment in 8 in-depth MS (3)
Also emphasis and focus on the needs of women seeking to access, and progress within, the labour market.
Selection of target groups based on the analysis of the situations in each Member State
Most of OP have been adapted, with a greater focus on youth and individuals recently made, or at risk of, redundancy) as result of the economic crisis
Thematic priorities, objectives and target group selection well aligned with EU priorities set in the ESF Regulation, Lisbon and subsequent Europe 2020 strategies, and the 2008 Recommendation on active inclusion (esp. inclusive labour markets).
Overall intervention logics and rationale underpinning each OP found to be internally coherent, but challenging to implement in a context of recession.
Most OP set targets concerning measurable achievement (e.g. number of persons guided towards employment) rather than measuring soft results which can be an issue for social inclusion groups – targets not set for all output or result indicators

10. Research findings to date – assessment of selected interventions (1)
Effectiveness:
The financial implementation rate (the relationship between the allocated and expenditure to date) varies from 5.6% to over 100%
A number of success factors identified: good financial planning and management between the MA, IB and the project partners; well-established partnerships and delivery models; an ability to attract the target groups by offering appropriate and well-delivered services; and meeting real identified needs on the ground
Next phase of research will assess any differences between types of actions, target groups addressed, countries etc

11. Research findings to date – assessment of selected interventions (2)
Efficiency:
Challenging to measure (data does not capture the intensity of support, indirect participants also supported); effort to measure cost per output and cost per result;
The cost per output varied between the interventions, appears to reflect the intensity of support
Next phase of research will assess any differences between types of actions, target groups addressed, countries etc
Sustainability:
Insufficient information about sustainability of result for individuals – lack of measurement of ‘soft results’ or pathways poses significant issues for social inclusion actions
A significant number of activities are sustained, but primarily with further ESF funding

12. Research findings to date – assessment of selected interventions (3)
Gender sensitivity:
Gender sensitivity approach must start from programme planning and must understand the factors behind gender disadvantage and multiple disadvantage. Little evidence of this in OPs which can partly be the result of lack of data
Little evidence of a deeply embedded gender sensitive approach to the intervention planning, implementation and the delivery of activities aimed to tackle the existing gender inequalities
Few gender sensitive targets
Sex disaggregated monitoring data is generally available, but not all MAs can provide Annex XXIII data at project level which impacts ability to assess this for individual interventions
There are some examples of measures targeted only at women – usually specific sub-groups (e.g. women returners, women from certain ethnic groups etc)

13. Research findings to date – assessment of selected interventions (3)
Community added value:
Volume effects: most visible impact, particularly in countries with limited domestic funding for ALMP (especially during the crisis)
Scope effects: reaching new or under-represented target groups, increasing the scope of mainstream ALMP for disadvantaged groups; particularly in context of crisis helped to target difficult to reach groups, which might otherwise fall ‘between the cracks’
Role effects: contributes to further spread and emphasis of individualised and holistic approaches for target; groups requiring complex support
Process effects: better partnership building among labour market partners at local/regional level which outlast lifetime of the project; systemic actions often generate outputs which can be mainstreamed

14. Research findings to date – lessons learnt
Policy choices:
Reinforce increasing emphasis across the board on individualised approaches, particularly for difficult to integrate groups
In some countries emphasis on particular system approaches/system building (e.g. intermediate labour markets; measures/assessments to support reintegration of disabled individuals)
Target groups: no significant gaps emerge, but some OPs more detailed than others particularly in identification of multiple disadvantage (also some lack of gender sensitive approaches)
Programming: focus on pathways towards employability and improving support services for SI target groups
Implementation:
Implementation lessons focus around appropriate approaches to reach out to and interact with marginalised and hard to reach groups and valuing outcomes along a pathway; maintaining flexibility in the choice of specific support measures suitable for each individual; low threshold access; voluntary participation and peer group referral. For SI target groups, sustainability of support is particularly critical

15. Research findings to date – lessons learnt
Monitoring:
Further standardise the definitions, interpretations and consistent collection and reporting of monitoring data (outputs and results)
Better data on sustainability of results and evidence base
Need for development of approaches to measure soft results and measurements of outputs and results along a pathway to integration, as results for SI projects can otherwise be ‘lost’
Evaluation:
Evaluation evidence remains too limited; including from leaver surveys; little information on sustainability of results and efficiency
Plan evaluations and the resulting data needs at the start of the programming period
Importance of Counterfactual Impact Evaluation

16. Next steps in the finalisation of evaluation
Our work agenda:
Provide the global analysis in the draft final report in April 2015
Revise and review the draft final analysis and submit the final report in May 2015
Your inputs:
Help with data validation in 8 in-depth MS
Comments on the Interim Report no. 2