1. Ex-ante evaluation: major points and state of play
Kamil Valica
Unit A.3 Impact Assessment and Evaluation
DG Employment, Social Affairs and Inclusion
Brussels
ESF Evaluation Partnership Meeting
12th November 2013

2. Changes to Art. 48(3) CPR (1) (based on trilogue compromise)
Few additions and one new point to be assessed
Additions:
48(3)(a)
the contribution to the Union strategy for smart, sustainable and inclusive growth, having regard to the selected thematic objectives and priorities, taking into account national and regional needs and potential for development as well as lessons drawn from previous programming periods

3. Changes to Art. 48(3) CPR (2) (based on trilogue compromise)
the consistency of the selected thematic objectives, the priorities and corresponding objectives of the programmes with the Common Strategic Framework, the Partnership Agreement and the relevant country specific recommendations adopted in accordance with Article 121(2) of the Treaty on the Functioning of the European Union and where appropriate at national level, the national reform programme
New Point:
48(3)(n)
measures planned to reduce the administrative burden of beneficiaries

4. Result Orientation - important factors
Programme design
Intervention logic
Ex-ante evaluation (EAE)
Interactive process
Iterative process
Indicators
On-going evaluation
Implementation
Monitoring
Ex-post evaluation

5. Ex-ante evaluation – a few major points
Intervention logic
Indicators
Timing

6. Clear intervention logic (1)
Challenges and needs
Country-specific recommendations
National Reform Programme
European semester analyses
Consistent translation into thematic objectives and investment priorities
Funding priorities
Linkage to Europe 2020 Strategy

7. Clear intervention logic (2)
Specific objectives (SO)
Precise definition (of change)
Should be more specific than the investment priority about target groups, problem area or structure, procedures, institutions, etc. it seeks to change
Should logically link to the proposed actions
Measurement (result indicators)
Types of actions
The most appropriate interventions to achieve SO (the right "action mix"?)
The choice based on an analysis of problems
Measurement of their outputs (output indicators)

8. Clear intervention logic (3)
Logical framework / Theory of change
Clarifies the intervention logic
Demonstrates causal links between:
Types of actions -> outputs -> intended results / specific objective
Description of how planned actions will contribute to specific objectives is often lacking in first OP drafts reviewed.
Helps identifying for which results/outputs, not captured by common indicators, programme-specific indicators should be established
Output indicators
Result indicators

9. Relevant and clear indicators
Ex-ante evaluators should consult carefully draft guidance on ESF Monitoring and Evaluation and assess:
Relevance
Result indicators should capture a change in the situation of supported participants or entities
Clarity
Indicators should have a clear title and an unequivocal and easy to understand definition

10. Indicators with realistic targets and identified data sources
Ex-ante evaluators should also assess:
Target values – are they realistic?
Consult background papers for ESF target setting
Data collection
Will data be collected and reported on time?
Have necessary arrangements been put in place to collect and store micro-data of participants?
Are existing databases used as data sources?
(NB: reduction of admin. burden for beneficiaries)
Are procedures in place to ensure quality of data?

11. Good timing is necessary to allow for interactive and iterative process
To allow for an interactive process, contractors need to be found on time to accompany the design of programme from its first draft
To allow for an iterative process (work in stages, revisiting of earlier stages on basis of evaluation recommendations), also programme design should start on time to allow for sufficient time space to evaluate and take on board recommendations

12. Ex-ante evaluations of ESF programmes – state of play
Based on a questionnaire of 20 MS (AT, BE, BG, CY, CZ, DK, EE, EL, ES, FI, HU, IR, LV, MT, NL, PL, PT, RO, SI, UK) – information from DG EMPL country geodesks received by 24/09/2013
Is ex-ante evaluation running?
YES
11 MS
Some EAEs started already in 2012 (4 MS)* NO
6 MS
1) Contractors found, but EAEs have not started yet
2) Some regions preparing procurement, no information on other regions
3) EAEs were planned to start in 1st half of Sept 2013, but not sure if they have started
4) Winning bidder selected, but not yet awarded
5) Invitation for tender published
6) EAE awarded, but is expected to be initiated in Nov 2013 (after finalisation of draft PA)
YES/NO
3 MS
1) For 1 OP running, for 2 OPs expected start in Dec 2013
2) For the majority of OPs running, for 1 OP even finalised; for some regional OPs procurement launched in July/Aug 2013
3) Not running for national OP, procurement launched (some regional OPs) or contractors found and EAE running (other regional OPs)
* Date of start not asked, based on open answers

13. Ex-ante evaluations of ESF programmes – other observations
Based on a questionnaire of 20 MS (AT, BE, BG, CY, CZ, DK, EL, EE, ES, FI, HU, IR, LV, MT, NL, PL, PT, RO, SI, UK) – information from DG EMPL country geodesks received by 24/09/2013
In general, the EAEs carried out in an interactive way
Challenges:
Time constraints
- Waiting for elaboration of other strategies
Parallel process of programming and EAE
Frequently changing OP drafts make EAE a complicated exercise and difficult to keep relevance of EAE interim reports; nevertheless valid EAE recommendations taken on board.
Delay in public procurement