1. Policy and Operations Evaluation Department (IOB) Evaluability Asessment: Preparatory Steps before Starting an Evaluation Prof. dr. Ruerd Ruben Director Policy & Operations Evaluation Department (IOB)

2. 2 Rationale Growing number of pseudo evaluations Repositioning Evaluation in PM&E cycle Concept of ‘evaluability’ (Whiley 1979; Smith 1989) Procedures for Evaluability Assessment (EA) Further steps

3. 3 Pseudo evaluation Growing number of aid projects/ programs. DAC Evaluation Resource Centre (DeRec): > 1000 evaluations MinBuza: substantial decentralized & local (embassy) evaluations Broad definition of evaluation: systematic assessment if and how activities contributed to stated goals

4. 4 Loss of Evaluability Review of Dutch Private Sector Programs  2/3 of executed ‘evaluations’ cannot be used Major reasons: - evaluation agency not fully independent - stated objectives too broad/vague - no clear indicators defined - data at too aggregate level - absence of baseline data - no representative sampling - too general intervention theory

5. 5 Evaluation in the PME Cycle Ex-post assessment of program performance Separation of ‘monitoring’ (internal) and ‘evaluation’ (external) Tracking of evaluation results Ex-ante guarantees for effective monitoring/evaluation Ex post Ex ante EA

6. 6 Theory of Evaluability Systematic process to determine whether program evaluation is feasible. Decide whether an evaluation is useful to provide timely, relevant findings for decision makers. Wholey (1979): initial steps to assess if a program can be evaluated  (a) set of clear objectives (+ side effects) (b) measurable performance indicators Smith (1989): ten steps procedure for Evaluability Assessment  initial diagnostics

7. 7 Evaluability Assessment 1. Identify relevant stakeholders 2. Define boundaries of the program 3. Analyze available program documents 4. Clarify intervention theory (goals, resources, activities, outcomes) 5. Analyze stakeholders perceptions of the program 6. Assess target population(s) 7. Discuss differences in outcome perceptions 8. Determine plausibility of intervention model 9. Discuss validity of the program 10.Decide about continuation (= full evaluation)

8. 8 Key EA Questions 1.Does the program serve the population for whom it was designed? 2.Does the program have the resources (available/used) as scheduled in the program design? 3.Are the program activities being implemented as designed? 4. Does the program have the capacity to provide data for an evaluation?

9. 9 Crucial Steps EA has five crucial tasks that an evaluator must successfully complete: Task 1. Study the program history, design, and operation; Task 2. Watch the program in action (participants) Task 3. Determine the program’s capacity for data collection, management and analysis; Task 4. Assess the likelihood that the program will reach its goals and objectives (consistency) Task 5. Show whether an evaluation is feasibly (will help the program and its stakeholders).

10. 10 Towards EA operationalization Screening of programs with expenditures > € 5 million Develop EA Protocol (e.g. Unifem’s Checklist for Programme Evaluability UNEF Delivery as One United Nations) Systematic review by sector, by channel, by country, etc Ex-ante assessment (part of project approval procedure) Yearly reporting (feedback to program design & operations)