1. World Bank Operations Evaluation Department Presentation at 2 nd Meeting of DAC Network on Development Evaluation Paris, November 9-10, 2004 Country Assistance Evaluation at OED: Methodology and Challenges

2. Operations Roadmap 2 Operations Evaluation Department Outline of Presentation 1.Background 2.Country Assistance Evaluation Methodology 3.Rethinking the Approach 4.Relevant Developments in Country Evaluations at the World Bank

3. Operations Roadmap 3 Operations Evaluation Department Background uSince FY95, OED has undertaken 70 Country Assistance Evaluations, covering nearly three quarters of the Bank’s operational program. uRoughly two-thirds of Bank country assistance programs have been rated satisfactory. uThe usefulness of country evaluations illustrated by disconnect with aggregate portfolio performance.

4. Operations Roadmap 4 Operations Evaluation Department Satisfactory Project Outcomes do not automatically translate into Satisfactory Assistance Outcomes… SatisfactoryUnsatisfactory Satisfactory53%33% Unsatisfactory7% Country Portfolio Outcomes Outcomes of Bank Assistance Programs CAE Outcome Ratings and Project Outcome Ratings

5. Operations Roadmap 5 Operations Evaluation Department OED Methodology

6. Operations Roadmap 6 Operations Evaluation Department  Country development performance;  Assistance program performance; and  Donor (World Bank) performance The first step is to clarify the object of evaluation…

7. Operations Roadmap 7 Operations Evaluation Department  Country Assistance Programs are assessed across three dimensions:  Development Impact  Products and Services  Attribution Methodology

8. Operations Roadmap 8 Operations Evaluation Department  Loans, Credits, and Grants  Analytical and Advisory Services  Aid Coordination and Resource Mobilization Dimension I: Products and Services (Bottom-up)

9. Operations Roadmap 9 Operations Evaluation Department  Main Country-level Program Objectives are derived from:  The country assistance strategy agreed with the Client.  The evaluator’s ex-post assessment of fundamental development constraints.  Major Ratings Same as in Project Evaluations  Relevance  Efficacy  Efficiency  Outcome  Institutional Development  Sustainability Dimension II: Development Impact (Top-Down)

10. Operations Roadmap 10 Operations Evaluation Department  Bank Performance  Borrower Performance  Partner Performance  Exogenous Factors Dimension III: Attribution

11. Operations Roadmap 11 Operations Evaluation Department  Individual donor attribution not possible using our current counterfactuals.  Bank performance currently measured against our best practice corporate standards. Attributing Program Results Is a Key Challenge

12. Operations Roadmap 12 Operations Evaluation Department  Professional Services  Strategy consistent with key development constraints, and corporate priorities  Quality at entry  Implementation and follow-up  Selectivity  Participation and partnership  Creativity, initiative, and efficiency Bank Performance

13. Operations Roadmap 13 Operations Evaluation Department  Ownership of assistance program  Support for national and international development priorities (MDGs, national development plan, etc.)  Respect for safeguards Client performance

14. Operations Roadmap 14 Operations Evaluation Department  Impact on design of assistance program  Impact on implementation of assistance program Aid Partner Performance

15. Operations Roadmap 15 Operations Evaluation Department  World economic shocks  Events of nature  War/civil disturbances  Other Exogenous Factors

16. Operations Roadmap 16 Operations Evaluation Department Ongoing Review of Methodology…Staff and Board Views Usage Surveys of Board and Bank Staff revealed: uHigh awareness of CAEs and CAE findings uIncorporation of recommendations uFrequent mention in strategies and Board discussions uBut, u Staff views negative overall…timing, accountability, consistency u Board views more positive, but need to focus more on results and less on compliance

17. Operations Roadmap 17 Operations Evaluation Department ….External Evaluator’s view Methodology reviewed by independent, external reviewer: uThe approach may place too much emphasis on instruments rather than impact uDifferent sections of CAE appear to lead to different conclusions, ie project outcomes uFocus shifts from country to bank to country—may be confusing uNeed to ensure consistency of terminology

18. Operations Roadmap 18 Operations Evaluation Department …Other Agencies Methods OED made a brief examination of methods employed by other agencies: uOED has completed the most country evaluations by far uOnly about half the agencies have formal, codified methodologies uOED is only agency to rate; very few attempt to construct counterfactuals or attribute

19. Operations Roadmap 19 Operations Evaluation Department Contemplated Changes to the Methodology… Based on our ongoing self-assessment, we are considering: uMaking the CAEs more-results based, by organizing by objectives u Establishing what the program attempted to accomplish by objective u Analyzing the product dimension by objective rather than overall u Preparing sub-ratings by objectives, disaggregating the overall rating uChanging attribution to the notion of “plausible association” or Bank contribution.

20. Operations Roadmap 20 Operations Evaluation Department Other Relevant Developments in the Bank… More institution-wide attention to evaluating the impact of country programs uMainstreaming of Results-based CASs, which will: u Contain a CAS Completion Report (self-evaluation), to be validated by OED u More clearly set out what the country program intended to achieve uThe Quality Assurance Group (internal assessments) is also moving to higher-level assessments, such as country-wide ESW or lending reviews.