1. IPA Funds Programme Management 12-19 sept. 2011 1 Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına Yönelik Teknik Yardım Technical Assistance on Institutional Building for the Implementation of RCOP in Turkey This project is co-financed by the European Union and the Republic of Turkey Evaluation: main concepts Ankara, 19 September 2011 Gerbrand van Bork: TAT Key Expert 2 (Economist and EU Expert) Gerbrand.vanBork@ecorys.com

2. 2 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 After this session you will After this session you will : Know the objectives and types of evaluation Know where to find info (Evalsed Guide) Have insight in the organization of evaluations Have insight in the communication process and how to use of evaluations

3. 3 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Evaluation & policy life cycle Problem analysis Policy formulation Policy implementation Ex ante evaluation Ex post evaluation Monitoring Problem analysis Policy formulation Policy implementation Monitoring Interim evaluation

4. 4 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Why evaluation?

5. 5 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Evaluation is an useful instrument to manage the implemtation of an OP IPA EVALUATION OBLIGATIONS

6. 6 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 IPA Regulation Art. 22 – Evaluation Commission shall Regularly evaluate results and efficiency of policies and programmes and effectiveness of programming Ascertain whether the objectives have been met Formulate recommendations to improve future operations

7. 7 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 IPA Implementing Rules All components Improve quality, effectiveness and consistency of assistance, and strategy and implementation of programmes Ex-ante evaluation of MIPDs Programmes subject to ex ante, interim and, where relevant, ex post evaluations At least one interim evaluation of each programme Ex post evaluation is Commission responsibility

8. 8 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 IPA Implementing Rules Components III & IV Ex ante evaluations to be carried out by beneficiary country for each programme – to be annexed to operational programme Evaluations to be carried out by beneficiary country when monitoring reveals departure from the original goals

9. 9 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Type of evaluations 1.Ex ante evaluation: will we spend this wisely? 2.IPA: interim evaluation: are we spending this wisely? 3.Ex post evaluation: have we spent this wisely?

10. 10 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Theory Evaluation is independent judgement of interventions according to their results, impacts and needs they aim to satisfy

11. 11 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Theory A B Relevance: do we want to go from A to B ? (need versus objective) Effectiveness: do we meet our objective (do we arrive at B)? Efficiency: at what cost do we arrive at B?

12. 12 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Key evaluation issues

13. 13 IPA Funds Programme Management 12-19 sept. 2011 Evaluation Criteria Relevance – The appropriateness of programme objectives to adress the problems and the consistency with other policy objectives. Did the programme meet an identified need at the design stage? Does the need still exist? Efficiency – How well inputs/means have been converted into results in terms of quality, quantity and time. Effectiveness – the achievements of objectives in terms of outputs and immediate results What has been achieved as outputs that will lead to change? Impact – The effect of the programme on the overall objective (or needs/problems) To what extent has the effectiveness contributed to immediate/intermediate/wider change? Sustainability – Assessment of the likelihood of benefits produced by the implementation of the programme to flow after external funding has ended, with particular reference to beneficiary ownership, policy support, institutional and management capacity. Relevance – The appropriateness of programme objectives to adress the problems and the consistency with other policy objectives. Did the programme meet an identified need at the design stage? Does the need still exist? Efficiency – How well inputs/means have been converted into results in terms of quality, quantity and time. Effectiveness – the achievements of objectives in terms of outputs and immediate results What has been achieved as outputs that will lead to change? Impact – The effect of the programme on the overall objective (or needs/problems) To what extent has the effectiveness contributed to immediate/intermediate/wider change? Sustainability – Assessment of the likelihood of benefits produced by the implementation of the programme to flow after external funding has ended, with particular reference to beneficiary ownership, policy support, institutional and management capacity.

14. 14 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Sustainability –Typical Questions Are the benefits sustained institutionally? Are there funds secured to ensure the benefits are available after the programme ends? Are beneficiaries able to afford maintenance of the (physical or institutional) infrastructure created? What support has been provided by relevant local strategies, policies? How well are interests of women/men represented?

15. 15 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Defining Evaluation Questions Defining Evaluation Questions Defining evaluation questions Defining evaluation questions is an essential part of the start-up of any evaluation exercise. The questions should be part of Terms of Reference for the evaluation. The question should be about the main evaluation criteria and should add specific questions such as: Ex-ante: Is the RCOP consistent with EU 2020 strategy? Etc. Interim or ex-post: What were the reasons behind limited absorption? Ex-post Did the programme have any impact on IT use of companies in the target region?

16. 16 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Evaluation & policy life cycle

17. 17 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Evaluation principles of the European Commission Evaluation as an integrated part of programme management; Sufficient resources (staff, funds and skills) should be made available; Standards for quality: relevance, access to information, stakeholders interests; reliability, transparency & objectivity Results should be publicly available

18. 18 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Relevance of objectives, feasibility, risks, impact logic, desirable impacts 6 steps: 1.Analysis of strengths, weaknesses and potential of the Member State, region or sector concerned 2.Analysis of previous evaluation results 3.Assessment of the strategy 4.Quantification of objectives 5.Evaluation of expected impact 6.Quality of management & implementation arrangements 1. Ex ante evaluation

19. 19 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 1. Ex ante evaluation (2) Interactivity: action and reaction 19 Ex ante evaluator Reaction ………... Reaction Assessment Action ………... Action Final version OP Ex ante evaluator Reaction ………... Reaction Assessment Responsible programming body Action ………... Action Final version OP

20. 20 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 2. Interim Evaluations in practice Corrective actions, development A useful management tool for programme management A useful communication tool between EC and member state / region Only limited possibilities for impact assessment as programmes are not finished (results are maximum)

21. 21 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 2. Interim evaluations in practice (2) For IPA at least one interim evaluation on the programme Based on evaluation plan OP Evaluation undertaken and designed in accordance with the internal demands of the MS Strategic or operational Achieving data that cannot be obtained by monitoring Evaluation obliged: - in case of significant departure from initial goals OP, - when OP revisions are proposed (financial, content, implementation)

22. 22 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 InputsOutputOutcome Audit & Control Monitoring Evaluation of quality, management, effects Evaluation versus monitoring or auditing

23. Evaluation methodologies (see Evalsed) 23

24. 24 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Organisation of an evaluation Internal or external evaluator? Terms of Reference (ToR) -Involvement of group? -Circulate (various) draft versions - Approval of the ToR Outlines of the ToR determined by: - Evaluation questions - Budget available Tendering evaluation

25. 25 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Terms of Reference Content: - Evaluation questions Process: - Organisation of the evaluation process (Steering Group) - Required skills, qualifications and expertise of evaluation team - Time planning, including intermediate results - Selection criteria on basis of which will be selected

26. 26 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Group Exercise Requirement: Drafting a ToR to contract a team for the interim evaluation of the RCOP. What are the specific evaluation questions you will ask for RCOP interim evaluation? Time: Preparation - 30 minutes Group presentation - 5 minutes

27. 27 This project is co-financed by the European Union and the Republic of Turkey IPA Funds Programme Management 12-19 sept. 2011 Websites Website with working documents (guidance documents): http://ec.europa.eu/regional_policy/sources /docoffic/working/sf2000_en.htm Quality standards on Evalsed: http://ec.europa.eu/regional _policy/sources/docgener/evaluation/evalsed/index_en.ht m Useful website with formats, etc: http://ec.europa.eu/ europeaid/evaluation/methodology/egeval/index_en.htm Practical handbook on ongoing evaluation for SF programmes / Territorial Cooperation programmes: http://www.interact-eu.net/ongoing_evaluation/326