1. 1 Jela Tvrdonova, 2014

2.  Strategic approach to rural development  Common approach to evaluation: legal background and CMEF  Monitoring and evaluation – what we are talking about?  What is evaluated?  Intervention logic in evaluation  Indicators  Evaluation questions  CMEF guidance 2

3. Focus on limited number of objectives Competitiveness – Environment – Quality of life Axes connected with strategic objectives Strong and dynamic agri-food sector Agriculture and forestry with high added value Employment and growth in rural areas Improvement of the governance in rural areas and mobilisation of the endogenous potential

4. Exact definition of objectives in Strategic Guidelines for Rural Development, Council Regulation 1698/2005 (art. 77-87), EC Regulation 1974/2006 (art. 60-62), Health check, National strategic plans and EU strategic monitoring Single framework for all program interventions CMEF: Common monitoring and evaluation framework – Handbook, Annexes, Guidance notes Baseline indicators at the program start Suitable combination of input, output, result and impact indicators, Common evaluation questions for all RD programs Guidance for ex ante, mid- term, ex post and ongoing evaluation

5.  Monitoring – measurement of immediate outputs and results at measure and axis level (different from strategic monitoring)  Evaluation - measurement of long-term effects/impacts of intervention, within the program development context (rural areas) 5 Accountability, comparability and lessons learned for next interventions

6.  On-going process which monitors the gradual implementation of the program at the level of financial inputs, physical outputs and results  Subject of annual reporting  Instruments – common and programme specific indicators linked to outputs and results,

7.  Since 2010 and every other two years  Focus of the EC on the progress of national strategic plans and its objectives implementation  Assesment of the contribution of national plans to the EU Rural Development Strategy

8. On-going evaluation is based on CMEF and country´s own methodology to carry on the evaluation during the program implementation Process based on annual reporting on result/impact indicators including periodical exercises: ◦ Ex – ante: evaluation of relevance of the planned intervention and optimizing of the RDP budget, ◦ Mid-term: for the assessing the progress related to the specific and overall/program objectives, ◦ Ex post : for the assessment of impacts and acievements towards objectives. WILL PROGRAMME WORK AS IT SHOULD? IS PROGRAMME WORKING AS IT SHOULD? DID PROGRAMME WORK AS EXPECTED?

9.  Good programme is the basis for the high quality of monitoring and evaluation  Normally the monitoring and evaluation is linked to programme intervention logic  But also specific topics, such as: ◦ Technical assistance ◦ Administrative arrangements ◦ Delivery mechanism 9

10. Intervention which logically responds to the most important needs of the targeted area 4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation10 Hierarchy of objectives: overall, specific, operational Hierarchy of expected effects: impacts on the territory, results on the supported beneficiaries immediate outputs at project level

11. 4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation11 Intervention logic of EU rural development programmes

12. RDP Intervention logic Context, its description SWOT and needs assessment Relevance Efficiency Overall objectives EU/MS Programme level Specific objectives EU/MS Axis level Operational objectives EU/MS Measure level Impacts Results Outputs Effectiveness Measures, projects and their management and implementation Inputs Coherence EU policy objectives Complementarity Source: EENRD 2014

13.  tools to measure the programme effects and achievements towards expected objectives by measures or the whole programme  should be Specific, Measurable, Achievable, Relevant for the programme, and Timely (SMART) Indicators should be filled with quantitative statistical data; however in some cases, indicators might also be linked to qualitative assessments or logical assumptions 13

14.  Baseline indicators: they relate to general socio- economic context of the programme area (context- related baseline indicators) and to the state of the economic, social or environmental situation in direct relation with the wider objectives of the programme (objectives-related baseline indicators)  Financial execution (input) indicators: they refer to the budget or other resources allocated to the programmes  Output indicators: measure activities directly realized within programmes 14

15.  Result indicators: measure the direct and immediate effects of the intervention within the group of programme beneficiaries and provide information on changes that have taken place  Impact indicators: refer to the benefits of the programme both at the level of the intervention but also more generally in the programme area. They are linked to the wider objectives of the programme 15

16. Inputs Activities Result Ouputs Impact

17.  A common set of baseline, output, result, and impact indicators for the RDPs (Art. 62 Reg. 1974/2006) “shall form the Common Monitoring and Evaluation Framework (CMEF)’’ (Annex VIII lists the common indicators) 17

18.  Since common indicators may not fully capture all effects of programme activities, it is necessary to define additional programme specific indicators (see Guidance notes A & K), which relate to programme specific issues.  They are developed if: ◦ Programme contains programme specific objectives and common indicators are not sufficient to capture achievements ◦ There is the need to evaluate specific issues such as delivery mechanism etc. 18

19.  Define the focus of evaluations  Demonstrate the progress, impact, achievements, effectiveness, efficiency and relevance of rural development policy Common Monitoring and Evaluation System distinguishes EQs: a) Common Evaluation Questions for Rural Development (CEQ-RD) b) Programme-Specific Evaluation Questions (PSEQs) 19 Source: EENRD 2014

20. 20 Demonstrate the contribution of programme interventions Evaluation of matters relevant at the EU policy level Encourage the assessment of programme results and impacts Enhance comparability across RDPs COMMON E QS FOR RURAL DEVELOPMENT Evaluation of specific RDP related topics PROGRAMME SPECIFIC EQ S Evaluation of matters relevant to the programme- specific policy Source: EENRD 2014

21. 21 Evaluation questions and judgment criteria Policy objectives Indicators Triangular consistency between objectives, questions and indicators which allows for: Less EQ and more targeted towards policy Clearer formulation of EQ and harmonization of all terms used More evidence to answer EQ Better planning the evaluation Source: EENRD 2014

22. 1. Common Evaluation Questions  Linked to RD policy objectives  Causal-effect questions (To what extent..?) 22 2.Common judgment criteria  Set the foundations to assess the success of intervention  Formulate explicit judgments on the basis common terms  Facilitate the identification of data, information and analysis needed 3.Common RD indicators  Additional information when necessary 1. Development of CEQs linked to the RD policy objectives 2. Development of common judgment criteria 3. Identification of relevant common rural development indicators Source: EENRD 2014

23. Handbook Annex 1:  Guidance notes - A: Choice and use of indicators, B: On-going evaluation, C: Mid-term evaluation Annex 2:  Guidance notes – D: Hierarchy of objectives, E – Measure Fiches Annex 3:  Guidance notes – F – K, Common indicators Fiches Annex 4  Guidance J – O Other guidances 23

24.  Original set of 150 common evaluation questions of CMEF has been simplified and reduced to the essential demand for knowledge from the European perspective. 24 3 groups of CEQs Programme-related: Community strategic priorities Health Check objectives 7 impact indicators TA and NRN Efficiency of RDP resources Leader approach-related: Community strategic priorities: Employment, diversification and governance Leader approach LAG contribution to LDS and RDP Measure-related: Contribution of the measure to its axis objectives Other measure effects and contribution to other axes objectives Source: EENRD 2014

25. 25 Evaluation QuestionIndicators Evaluation questions – cross-cutting Evaluation questions Axis/sector specific Output indicators Impact indicators Result indicatorsAxis based baselines, objective related, contextual Baselines and their quantification, Target levels Horizontal baselines, objective related, contextual RDP objectives Overall strategic objectives Axis specific objectives Measure/operatio nal objectives Baseline values of output indicators Targets for impact indicators Targets for results indicatiors Targets for output indicators Common and program spec. Eval. questions

26. CMEF provides guidance for MAs in setting up indicators at each level of intervention and baseline indicators Annex 3 of the CMEF Handbook provides detailed description of all common indicators (Guidance notes F – K): ◦ F: COMMON INDICATOR LIST (overview of all common indicators) ◦ G. BASELINE INDICATOR FICHES (detailed description of indicators) ◦ H. OUTPUT INDICATOR FICHES ◦ I. RESULT INDICATOR FICHES ◦ J. IMPACT INDICATOR FICHES 26

27. Each indicator fiche contains the following elements:  Type of indicator  Related measures  Measure Codes  Definition of the indicator  Subdivision  Unit of measurement  Level of collection  Responsible actor for collection  Collection method/good practice  Sources  Registration frequency 27

28. Guidance on practical use of indicators in monitoring and evaluation of RDPs  Guidance note E provides the following on use of indicators within each particular measure: ◦ Measure Code ◦ Rationale of the measure ◦ Target group ◦ Target area ◦ Common indicators ◦ Link rationale of the measure and indicators ◦ Evaluation questions 28

29. 29 Mainly Managing Authorities Introduces the process of the ex post evaluation, steps to be conducted and role of evaluation stakeholders Explains also specificities with respect to NRN Part I Mainly Evaluators Discusses and explains intervention logic, evaluation questions, indicators, methods and data Part II Toolbox Provides additional practical tools for ex post evaluation preparing, implementation and reporting. Part III For everyone Introduces ex post evaluation and its role in policy cycle Explains the scope and focus of ex post evaluation, legal requirements and common evaluation elements Introduction Source: EENRD 2014

30. RDP Intervention logic SWOT and needs assessment Efficiency Overall objectives EU/MS Specific objectives EU/MS Operational objectives EU/MS Impacts Results Outputs Measures and their implementation Evaluation questions Indicators Methods Data EU policy objectives Basis of evaluation Attribution of impacts Collection of evidence Focus of evaluation Measurement tools Source: EENRD 2014

31. Thank you for your attention jela@ruralevaluation.eu jelatvrdonova@gmail.com