1. ADB’s Evaluation Architecture and Country Evaluations Guidelines Mala Hettige Senior Advisor Independent Evaluation, ADB Presentation at the meeting of Sub-Regional Development Institutions Johannesburg, 27 April 2014

2. Asian Development Bank (AsDB)  Established in 1966 – in Manila, Philippines  67 shareholding member countries  Provides $ 21b in development assistance  About 760 projects  5 regional department and PS department  Managed by MDG, 6 VPs  3000 staff in HQ and 29 RMs + 3 rep. offices  Administrative Budget about $600 million

3. Independent Evaluation @ AsDB  Abides by OECD’s 4 dimensions of independence  DGE reports to the Board through the Development Effectiveness Committee (DEC)  Reports public upon DG-IE approval, discussed under Chair, DEC  DGE appointed/removed by DEC – 5 year non-renewable term.  Work Program and separate budget approved by Board  DGE has authority over staff recruitment  Staff subject to backward and forward conflict of interest policy

4. Major Evaluation Products Annual Reports Country Assistance Program Evaluations Corporate Evaluation Studies Impact Evaluation Study Thematic Evaluation Studies Project/Program Performance Evaluations Technical Assistance Program Evaluations Real- time Evaluation Studies Project Validation Reports Other Evaluation Reports 4

5. Accountability and Learning 5

6. Evaluation Information System (EVIS)  A database of evaluation lessons  Provides users easy access and search capability  Lessons by sector, theme, topic, country etc.  EVIS is the first system of its kind  Accessible through http://evis.adb.org http://evis.adb.org 6

7. Management Action Record System (MARS)  Tracks ADB Management actions on recommendations  IED Validates actions taken by Management  Reports this validation to the Board  Computerized and available to staff and Board  Departments/offices update progress on actions

8. Country Evaluations and ADB Operations Country Partnership Strategy Project Identification /Preparation Appraisal / Approval Implementation Country Evaluation 8

9. History of Country Evaluations in ADB CAPEs (Country Assistance Program Evaluations)  1998 first CAPE – Peoples Republic of China  2002 Mongolia 2002 questioned choices made  2005 Bhutan had recommendations + criteria ratings  2006 Indonesia added ratings by sector  2007 India had contribution to themes+ value added  2007 China had Strategic and Institutional positioning  2008 Sri Lanka considered the implications of conflict  2008 Philippines considered political economy factors  2009 Cambodia top-down + bottom up assessment  2010 Lao existing guidelines applied  2014 Tajikistan assessment of objectives- ongoing

10. Evolving Guidelines for CAPEs in AsDB  From 1998 to 2006 no guidelines  First Country Evaluation Guidelines 2006  Revised Guidelines issued 2010  CAPE/ validation of a country self evaluation mandatory prior to approval of a CPS  Now doing a few revisions to the guidelines  Also preparing validation guidelines

12. Background: GPS on Country Evaluations GPS for CSPE established in 2008 Based on OECD–DAC evaluation principles Draws on the 2007 ECG Review of CSPE A core GPS criteria allows comparability An optional GPS criteria helps accountability and learning

13. Application of GPS: of Country Evaluation Name of study differs and number differs Coordinated with the upcoming country strategy Substituted with validations or learning products About 9-12 months, 25-75 pps, $ 250+, Lending/non lending, about 10 years (2-3 cycles) Application of principles, approach taken differs Overall Rating and Evaluation Criteria followed by about half. But try not to emphasized much

14. Objectives : Accountability – Board on results over a period Learning - engage with stakeholders on future CPS Timing : to feed into the next strategy Coverage : Long enough to assess results and Recent to ensure findings are relevant Key Aspects of CPSE

15. GPS for Country Strategy and Program Evaluation GPS for CSPE established in 2008 Based on OECD–DAC evaluation principles Draws on the 2007 ECG Review of CSPE A core GPS helps allows comparability An optional GPS helps accountability and learning

16. Core Criteria Relevance Effectiveness Efficiency Sustainability Impact Optional Criteria Positioning Coherence Institutional Development MDB Performance Borrower Performance Partner Coordinatio n Evaluation Criteria

17. Relevance GPS : Relevance refers to the degree to which MDB strategy and program were consistent to critical needs and long-term plans of the country

18. Relevance –> Strategic Relevance Can include optional criteria on positioning – how well did MDB respond or anticipate Evolving development challenges Priorities of the government Build on MDB’s comparative advantage Consider support from other development partners Also can include coherence – extent to which there were measures to Foster internal + external synergies within MDB Foster external partnerships that promote cost- effective division of labor through complimentarities

19. Efficiency GPS: Efficiency refers to the extent to which the design and delivery of assistance were most cost effective.

20. Effectiveness GPS: Effectiveness refers to the extent to which the assistance instruments achieved the intentions and objectives set.

21. Sustainability GPS: Sustainability refers to the likelihood that actual and anticipated results will be resilient to risks beyond the program period.

22. Impact GPS: Impact refers to an MDB’s contribution to long-term changes in development conditions.

23. Optional Criteria Institutional Development MDB Performance Client Performance Partnership Performance Other concerns By sector? What about cross sector impacts? By objectives? How to attribute and measure? How to account for unintended impacts?

24. IED Website www.adb.org/evaluation

25. Thank You! https://www.facebook.com/adbevaluation evaluation@adb.org www.adb.org/evaluation https://twitter.com/adbevaluation http://www.youtube.com/evaluationatadb http://www.scribd.com/adbevaluation/ http://www.soundcloud.com/adbevaluation/ Follow Us