Assessment of the Evaluation Culture Approach

Introduction PM&E project - Scope of the work Evaluation Culture Assessment Why do we need it? Agreement on the methodology

Topics of Discussion Evaluation - background What was already achieved in Romania Further steps taken under the present TA contract Discussion and agreement on the methodology to be employed in assessing the evaluation culture in Romania

Evaluation - background Evaluation Advisory Group (EAG) “birth date” - 25 July 2002 composition mandate first meeting - 27 March 2003

Evaluation - background Objectives of EAG To promote the development of medium-term National Strategies for building local monitoring and evaluation capacities (which would also cover the national public funds) To support candidate countries (via exchange of experience) in their preparation of draft short-term Action Plans for development of local monitoring and evaluation capacities within the framework of decentralisation of monitoring and evaluation. To prepare a Guide to Good Practices in Monitoring and Evaluation Capacity Building, including national strategies, monitoring and evaluation models, capacity building strategies and means to boost monitoring and evaluation capacity.

Evaluation - background Content of the GUIDE Monitoring and evaluation models in Member States and the Candidate Countries - Key output: Establishing a typology of good practices of national strategies for monitoring and evaluation. National Strategies - Key output: Establishing a typology of good practices of national strategies for monitoring and evaluation. Local capacity building - Key output: A survey of good practices in evaluation capacity building.

Evaluation - background EAG-WG1 would research and report on National Strategies and Monitoring and Evaluation Models in order to develop and implement an approach which identifies current good evaluation practice in Candidate Countries and Member States to provide guidance on the implementation of good practice to assist Candidate Countries individually and collectively to recommend contributions for the Guide EAG-WG2 would research and report on the development of a Capacity Building Strategy in order to develop and implement an approach to identify and record good practice in evaluation capacity building strategy in Candidate Countries and Member States to make recommendations to the EAG for consolidation within a handbook of good practice.

Evaluation - background Main conclusions regarding the situation in candidate Countries - WG2-July&Nov 2003 - based on a questionnaire completed and submitted by each country The evaluation function was likely to be a small, under-resourced unit; The main identifiable impact reported has been from the Phare Programme evaluation; Outside Commission requirements, evaluations are rare; Evaluations do not yet make a significant impact on accountability; Institutionalising factors (for example, regulation) are poorly developed; In the short term, evaluation activity will remain centralised; Candidate Countries have anticipated the need for training; Programme/ project monitoring is focused on the Commission’s Phare IE process.

Evaluation - background Milestones Representatives of several Candidate Countries were able to make presentations of their National Strategies in November 2003 The Guide to good practices of evaluation capacity building was issued in draft by EMS on 15 January 2004 Two issues remain for most Candidate Countries: the political will and national evaluation capacity, and the two are interrelated.

What was already achieved in Romania ACTION PLAN FOR THE TAKE OVER OF INTERIM EVALUATION FUNCTION FOR PHARE PROGRAMME- First Pillar of Romanian National Evaluation System – drafted and presented in February 2004 On - going developments

Further steps taken under the present TA contract ECU - institutional building/strengthening Awareness rising on evaluation Assessment of the legal framework related to evaluation Assessment of the evaluation culture Elaboration of the National Strategy for Evaluation Capacity building on evaluation supply side preparation of further interventions

Evaluation Culture “The evaluation process is now seen as a major contributor to state accountability and good governance, but its institutionalisation has been slow, with a few exceptions. One of the obstacles to this is a poorly developed evaluation culture: a practice of asking questions, demanding answers and expecting change.”

Evaluation Culture The nine empirical criteria which must be satisfied before an evaluation culture can be said to be fully established: I Evaluation takes place in many policy domains II Supply of domestic evaluators in different disciplines III National discourse concerning evaluation IV Professional organisations V Degree of institutionalisation – Government VI Degree of institutionalisation – Parliament VII Pluralism of institutions or evaluators performing evaluations within each policy domain VIII Evaluation within the Supreme Audit Institution IX Proportion of outcome evaluations in relation to output and process evaluations

Evaluation Culture For Candidate Countries, three further criteria could be added to those identified above: X Monitoring capacity XI Evaluation Diversity: Strategy, Policy, Programme, Projects XII Information flow within government relating to evaluation These are medium term goals against which Candidate Countries interested in developing an evaluation culture should attempt to benchmark themselves.

Drafting of NES Strategy document Completion of Evaluation Culture Assessment – May 19 First draft of NES – May 22-June 30 Presentation of first draft to EWG – July 25 Presentation of final draft to EWG – August 22. Launch of NES – Evaluation Conference to take place in September. Regional road-trip – 8 NES-promotional events to take place in the regions, September-October.