1. Evaluation Expert Committee 2nd meeting SYNTHESIS OF THE ANNUAL PROGRESS REPORTS FOR 2007 CONCERNING ONGOING EVALUATION Irina RAMNICEANU Helpdesk Evaluation Manager

2. Evaluation Expert Committee 2nd meeting 2 What is the purpose of the synthesis? OUTLINE OF THE PRESENTATION How was the work organized? General findings Specific findings (per reported topic) Role of the Helpdesk 23 June 2009

3. Evaluation Expert Committee 2nd meeting 23 June 20093... help improve ongoing evaluation systems in the MS WHAT IS THE PURPOSE OF THE SYNTHESIS?... enhance the quality of evaluation reporting... identify and promote good practice in conducting and reporting about evaluation activities The Evaluation Helpdesk prepares it annually, to... 323 June 2009

4. Evaluation Expert Committee 2nd meeting Report ongoing evaluation activities APR summary Section Ongoing evaluation APR Section Ongoing evaluation APR Section Ongoing evaluation APR Section Ongoing evaluation APR National/Regional MCEC (Desk Officers) EC and MS APR Section Ongoing evaluation 1.…. 2.…. 3.…. 4.Ongoing ev 5.…. 6.…. 7.…. APR 1.…. 2.…. 3.…. 4.Ongoing ev 5.…. 6.…. 7.…. 423 June 2009 HOW WAS THE WORK ORGANISED? synthesis

5. Evaluation Expert Committee 2nd meeting 523 June 2009 The Guidance Note B – Evaluation Guidelines A USEFUL REFERENCE

6. Evaluation Expert Committee 2nd meeting Few of the reports for 2007 follow closely the indicative structure They vary in quality, length and information content 23 June 20096 GENERAL FINDINGS

7. Evaluation Expert Committee 2nd meeting...but 2007 was a special year: first time such reporting required reports to focus on the ongoing evaluation systems most of the RDPs approved late in the year 23 June 20097 The number of RDPs approved, by quarter GENERAL FINDINGS

8. Evaluation Expert Committee 2nd meeting 8 25% of the programmes deal with the evaluation questions and indicators 50% of the programmes refer to administrative preparations 823 June 2009  Type of ongoing evaluation system  Actors involved (including a steering group)  Contracting evaluators: TORs and beyond SETTING UP THE EVALUATION SYSTEMS

9. Evaluation Expert Committee 2nd meeting 9 Most programmes (plan to) fully outsource their evaluation activities, as a “single package” Alternatives refer to: multiple tenders and minimal outsourcing (only mandatory steps) 923 June 2009  Evaluation activities bundled together and outsourced as one set  Leads to improved continuity and consistency of evaluation throughout the programming period SETTING UP THE EVALUATION SYSTEMS But difficult to draw a clear and comprehensive picture EU-wide

10. Evaluation Expert Committee 2nd meeting 10 23 June 2009 SETTING UP THE EVALUATION SYSTEMS Recommendations for the MS: Cover all the key components of the evaluation system Be clear and specific about how the ongoing evaluation is implemented (type, duration, scope of contracts etc.) Clarify what has been achieved and what is still planned In 2008, include information on your ongoing evaluation system if not already done so in 2007 For 2008 and 2009, give particular attention to the administrative preparations for the MTE

11. Evaluation Expert Committee 2nd meeting 11 Limited focus in the 2007, but increasing in 2008+ 1123 June 2009  Ex-post 2000-06 covered in 10 reports  Ex-ante 2007-13 referred to in 20 reports ONGOING EVALUATION ACTIVITIES Already some reference to methodological issues Confusion about reporting on “borderline activities”...... and wide-ranging scopes and levels of detail

12. Evaluation Expert Committee 2nd meeting 12 23 June 2009 ONGOING EVALUATION ACTIVITIES Recommendations for the MS: Ex-post 2000-06 only if clear relevance and follow-up in 2007-13 Only a concise reference to the activities of the ex-ante evaluation Clearly state any activities supporting evaluation undertaken or planned (e.g. thematic studies) You may refer to methodologies developed or planned for the evaluation of your RDP Include the evaluation capacity building activities Note: data collection activities are detailed under a distinct chapter

13. Evaluation Expert Committee 2nd meeting 13 Data collection described in 30% of the reports 1323 June 2009  Focus on responsible parties and coordination  Typically MA pools data together, but decentralised models also available  Any differentiation by data type? SYSTEMS FOR DATA COLLECTION AND MANAGEMENT Data management systems in 15% of the reports Limited information about data collection methodologies  Development in progress, responsible bodies mentioned  Update on progress  Institutions responsible

14. Evaluation Expert Committee 2nd meeting 14 23 June 2009 Recommendations for the MS: Highlight progress and/or difficulties encountered with regard to data collection (including methodologies) Distinguish, as appropriate, between types of data collected Describe institutional arrangements for data collection and management only when set up; continue reporting changes (if any) Explain arrangements made to develop/update IT systems ONGOING EVALUATION ACTIVITIES

15. Evaluation Expert Committee 2nd meeting 15 Networking activities 1523 June 2009  Very little information in 2007  Most developed (or reported on) in multi-programme countries OTHER TOPICS Difficulties encountered and need for future work  Only a few insights available in 2007  Systems and methodologies for data collection, e.g. enhance usability of IT systems, check functionality  Methodological aspects, e.g. HNV indicator, determine net intervention effects etc.

16. Evaluation Expert Committee 2nd meeting 16 23 June 2009 Recommendations for the MS: Report progress against previous year(s) only, without repeating past information Clearly demarcate activities carried out from activities planned [You may consider adding a timetable of planned activities] Be clear and concise in your wording Do not make extensive references to legal texts or guidelines OTHER TOPICS

17. Evaluation Expert Committee 2nd meeting 17 Support the EC desk officers in charge of the APRs (including through training) 1723 June 2009 ROLE OF THE HELPDESK Provide detailed recommendations in each annual synthesis report