Perspectives from the Independent Evaluation Group Martha Ainsworth and Soniya Carvalho Preparing High-Quality Implementation Completion and Results Reports

Part I: Tips for preparing a high-quality ICR Martha Ainsworth Part II: Project ratings: connects and disconnects Soniya Carvalho

Why is ICR quality important? ► The ICR is an opportunity to learn and to improve effectiveness It’s the official (public) report on the project, its achievements and lessons ► Low quality ICRs make it difficult for IEG to rate the project accurately Unsatisfactory ICR quality is a good predictor of ratings “disconnects” and disputes From 7/1/2006 onward, “when insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant rating”

What are the IEG criteria for ICR quality? ► Results-orientation (ICR should be outcome- oriented, not an implementation narrative) ► Quality of evidence and analysis ► Lessons based on evidence and analysis ► Internal consistency ► Consistency with Bank guidelines ► Conciseness

1. Make the ICR results-driven a.Organize evidence around achievement of objectives (not implementation of components) b.Document the “results chain”: Show the link between inputs, outputs, outcomes, impacts c.Explain the counterfactual (what would have happened without the project), other factors operating, plausible attribution d.Don’t be constrained by the official project indicators e.Show trends over the whole period, before and after the project, as many observations as possible f.Footnote sources of evidence, triangulate data.  Provide the evidence necessary for someone disconnected from the project to be able to rate it!

Inputs  Outputs  Outcomes  Impacts To assess efficacy using the results chain, you need: to be able to link these elements plausibly to understand the “counterfactual”— what would have happened without these activities The Results Chain

Establishing a “counterfactual” ► Timeline of events ► Account for trends in other determinants (For example, other donor support, weather, changes in national policies, household income, changes in prices that could affect demand or incomes) ► Document trends before and after, in project and non-project areas

Distribution of Malaria in Eritrea Gash barka Anseba Northern Red Sea Southern Red Sea Debub Maekel Highest ranking Lowest ranking Example #1: Eritrea HIV/AIDS, Malaria, STI, and TB Control Project

Annual outputs of anti-malaria interventions increased Nyarango et al. (2006) (incidence and rainfall data) National Malaria Control Program (2004, 2006) (intervention data)

New cases of malaria declined Nyarango et al. (2006) (incidence and rainfall data) National Malaria Control Program (2004, 2006) (intervention data)

Rainfall also declined, but malaria continued to decline when rainfall recovered Nyarango et al. (2006) (incidence and rainfall data) National Malaria Control Program (2004, 2006) (intervention data)

Example #2: Russia Health Reform Pilot Project Russia’s abortion rate declined during the project

It declined even faster before the project

It declined all over Russia, without the project and there were no data for the results chain linking project activities to the decline in abortion rate

Example #3: Romania Roads 2 Project /road safety Road fatalities declined in Romania

They declined faster before the Traffic Safety Campaign and there were no intermediate outcome data on safety

2. Show all indicators of efficiency ► ICRs for all investment-type projects (including TA projects) must assess efficiency, not ICRs for DPLs ► Answers the question: Were costs in achieving the objectives reasonable in relation to the benefits and to recognized norms (“value for money”) ► Show evidence of: Cost-benefit, cost-effectiveness, efficient use of project resources, unit rate norms, service standards, least-cost analysis and comparisons, other efficiency indicators Aspects of design and implementation that contributed to or reduced efficiency ► Cost-benefit analysis is not sufficient to assess efficiency; other indicators must be shown If presented, transparent discussion of assumptions

3. Document safeguards ► Safeguard policies not applicable to DPLs ► Applicable safeguard policy, project category (A, B, C, FI), assessment/mitigation plan ► Whether the mitigation plan was implemented ► Findings of third party safeguard reviews ► Updated assessment and revised mitigation plans implemented if physical components generating economic and social impacts were modified.

4. Other quick tips ► Consistency: Make sure the ratings match the text and the numbers are internally consistent ► Conciseness: More evidence is good, as long as it is relevant and concise; if extensive, add an annex. ► Completeness: Ensure that the sections and annexes are complete and accurate ► Candor: Be candid about shortcomings, data quality ► Lessons: Not too many; must be evidence-based and come from the project’s experience; seek to explain variability in outcomes GOOD LUCK!