Presentation on theme: "Guidance document on ex-ante evaluation"— Presentation transcript:
1 Guidance document on ex-ante evaluation 15 March 2012Joint presentation to the ERDF and ESF Evaluation Partnership Meeting
2 Ex ante evaluation: our approach What is different?Common Regulation & multi-fund programmesStronger focus on results / Europe 2020 strategyMore detailed Article 48Links with Partnership Contracts, SEAOur common approach:Stick to Regulation (article 48 and related articles)-as proposed by CommissionFocus on ex ante evaluators' roleWhen necessary, give examples of differences between ESF/ERDF
3 Ex ante evaluation Guidance Programme strategyQ&AIndicators, monitoring and evaluationConsistency of financial allocationContribution to Europe 2020Strategic Environmental AssessmentEvaluation Process
4 Programme strategy Consistency of programme objectives Coherence Horizontal principlesLinkage between supported actions, expected outputs and resultsQ&A
6 Consistency of programme objectives Step 1: The ex ante evaluator should assess the challenges and needs identified by the programme.Step 2: The evaluator should examine if these challenges and needs have been consistently translated into thematic objectives, priorities, investment priorities and corresponding specific objectives.
7 Step 1: Challenges & needs (1) National & regional programmes: the evaluators should primarily base their assessment on the NRP, country-specific recommendations, European semester analysesIn addition:National ERDF sectoral programmes: specific sectoral analysis & territorial prioritiesRegional programmes: specific regional situation and needsSpecific challenges of sub-regional or functional areas or of specific target groupsHorizontal principles (Art. 7 & 8 CPR)Justification of specific regional challenges and needs
8 Step 1: Challenges & needs (2) The ex ante evaluator should assess whether the identified challenges and needs are consistent with the analysis of disparities and development needs in the Partnership Contract.The evaluators should also base their assessment on the Common Strategic Framework.They should examine the prioritisation of the identified challenges and needs and suggest reviewing it, if appropriate.
9 Step 2: Programme objectives thematic objectivesinvestment prioritiesSpecific objectivesWhile appraising the consistent translation ofthe identified challenge & needs into programmeobjectives, the evaluators should focus on:Their appropriate weight in the investmentprioritiesPrecise formulation of specific objectivesRationale and evidence for omitting major challenges or needsThe evaluators may recommend complementary analyses
10 Coherence Internal coherence External coherence The evaluators should assess:Relationship between specific objectivesCoordination mechanisms in multifund programmesExternal coherenceThe evaluators should examine:Analysis of programme contribution to other strategies/programmes and of the influence of these on the expected results of the programmePlanned integrated territorial approaches
11 Horizontal principles The evaluators should assess the following elements under each principle (Art. 87(3) CPR):Equality between men an womenArrangements to integrate the gender perspective at the level of the programme and operationsContribution to its promotionAnti-discriminationSpecific actions in preparation andimplementation of programmeSustainable developmentSpecific actions in the selection of operationsOpinion of national equality bodies
12 Linkage between outputs and results Clear intervention logic: condition for good programming and evaluations!Recommendation: use a logical framework for each priority axis or investment priorityThe evaluator should examine:- What is the expected change?How will outputs contribute to intended results?Will the proposed actions effectively lead to these outputs?What other factors could influence the expected results?Would evidence suggest other approaches?
13 Linkage between outputs and results Are the planned form of support the most effective? What is the rationale?(grants, prizes, repayable assistance and financial instruments and a combination: Art. 56)Will the actions effectively meet the needs of specific territories or target groups?
14 Indicators monitoring and evaluation Relevance and clarity of proposed indicatorsBaseline and target valuesSuitability of milestonesAdministrative capacityData collectionEvaluationConsistency of financial allocationContribution to Europe 2020Q&A
15 Indicators, monitoring and evaluation Art. 24 CPR- For each priority axis, indicators to assess progress towards achievement of objectives- Basis for monitoring, evaluation, review of performanceIncreased focus on results: role of well designed indicators and evaluationsNew focus on territorial integrated approaches: reflected in monitoring and evaluations where relevantCommon indicators compulsory
16 Indicators, monitoring and evaluation Relevance of indicatorsOutput indicators reflect the main operationsResult indicators are responsive to the policy (values evolve with development of operations)Result indicators reflect the most important intended changeCommon indicators - ERDF: used where relevant to operations and specific objectives / ESF: monitored under all investment prioritiesResults may be close to the intervention, short or long term: depends on the intervention logic, on the intended change
17 Indicators, monitoring and evaluation InterventionOutput indicatorResult indicatorERDFIncubator services for start upsNumber of incubated start upsIncrease of the share of start ups in regional statisticsIncrease of employment in young enterprises (3 years after incubation)Immediate result indicatorLonger-term result indicatorESFTraining of individuals who are inactive, not in education or trainingNumber of supported participants (who are inactive, not in education or training)Number of inactive participants who are newly engaged in job searching upon leaving the interventionParticipants who are in employment 6 months after leaving
18 Indicators, monitoring and evaluation Clarity of indicatorsUnequivoqual title, clear definitionNormative interpretationRobustness and transparenceRobustness: reliability of data source, representativeness of samples (survey)Data sources for baselines and target values + definitions are publicly availableSee ex ante conditionality (Annex IV CPR)
19 Indicators, monitoring and evaluation Baseline and target valuesAvailability of baseline data for result indicators?latest available data (ERDF and ESF Regulations)advice on sources and methods to collect data when necessary, or research to establish baselinesRealistic targets?Compared to actions and form of support, financial allocationsFor output indicators, compared to references (previous programmes or sectoral norms)For result indicators, taking into account other factors
20 Indicators, monitoring and evaluation Performance framework: a subset of indicators for each priority axisSuitability of milestonesDo they capture essential information on the progress of a priority?Can they be achieved at the review points?Will data be available for 2017/2019 progress reports?
21 Administrative capacity The ex ante evaluator should assess whether adequate human resources and administrative capacity for the management of the programme are in placeIncluding: Have possible previous bottlenecks at the level of MA and IB been tackled?The evaluator may advice on how to address any issues
22 Data collectionThe ex ante evaluator shall assess whether the data collection procedures are suitable. Whether they:Respond to all requirements for monitoringincluding: is data available on time, in the required form and quality? How will the data be collected and stored? Will administrative data bases be used to reduce admin. burden? Etc.Provide the data needed for evaluationsincluding: data to i.a. assess effectiveness, efficiency and impact? What are the mechanisms to ensure good quality data? Etc.
23 Evaluation The ex ante evaluator may advice: On possible evaluation needs for on-going evaluationsOn evaluation methodologies and resulting data needsOn setting up the evaluation plan
24 Consistency of financial allocation The ex ante evaluator shall assess the consistency of the financial allocation on the basis of the identified challenges and needs and the proposed actions and selected forms of supportThe evaluator may assess whether the different funds are adequately combined and contribute to integrated approaches
25 Contribution to Europe 2020 Taking account of the proportionality principle and against the background of national and regional needsthe ex ante evaluator shall assess the programme’s potential contribution to Europe 2020 in the light of its thematic objectives and investment priorities.
26 Process Strategic Environmental Assessment Timing Interactive and iterative processPartnership and multi-level governanceEvaluation methods and financingFinal report and publicationAn independent processWhat requirements for evaluators?Q&A
27 Strategic Environmental Assessment A new link with Ex ante:"shall incorporate, where appropriate, the requirements for SEA…"- "Where appropriate": direct impact on environment"Incorporate": does not mean that the ex ante evaluator has to do SEA"Requirements": environmental report, consultations, information, monitoringHow to incorporate SEA: still under discussion
28 Timing Ex ante evaluation & Partnership contracts PC includes (Art. 14):Summary analysis of ex ante evaluations justifying selection of the thematic objectives and the indicative allocation of FundsSummary of main results expected for each thematic objectiveDevelopping ToR … now- PC and OPs adopted Oct.-Dec. 2013- 6 months negotiations: draft PC & OP April-June 2013Programme elaboration 6-8 months: starts Aug.-Oct. 2012Calls for tender launched mid 2012
29 Interactive and iterative process Work in stagesTiming and interactionSteering group29
30 Partnership and multilevel governance Art. 5 CPRStrengthening with the objective of:Increasing legitimacyBuild on wider expertise and knowledgeEnsure collective commitment and shared understanding of expected resultsThe evaluators should review if the relevant stakeholders are involved in the preparation of the programme as well as in its implementation, monitoring and evaluationThe evaluators should base their assessment on the Commission Staff Working Document “Elements for a Code of Conduct on Partnership”30
31 Evaluation methods and financing Mix of methods:- TBE: literature review, workshops, interviews, focus groups, peer reviewsInvolve partners in programme designCost:- number of evaluation questionsmethodsExtra tasks (quantification of baselines, complementary analyses)
32 Final report and publication Submitted with the OP to the Commission- Main evaluation methods used- Changes and improvements to programme- Final assessement of OP- Strategic Environmental AssessmentExecutive summary (+ in English)Made available to citizens
33 An independent process Article 47 CPR: "carried out by experts functionally independent of the authorities responsible for programme implementation"No hierarchical link with the entity responsible for implementation: internal expert may contributeBalance between an interactive/iterative and independent process: critical jugements on the different elements of the programme and its coherence
34 What requirements for evaluators? Knowledge of EU and national documentsDeep knowledge of contextSituation of areas or target groups with specific needsSituation of economic sectors- Able to judge quality of socio-economic analysisExperience in evaluationsData needs for future impact evaluationsRelevance of indicators within the intervention logic (thematic evaluations)