1. Presented by CIDA on behalf of the Task Team on Multilateral Effectiveness

2.  2009 ◦ Large Task Team on Multilateral Effectiveness was established to explore and further develop proposal to strengthen information on development effectiveness (DE) of Multilateral Organizations (MOs)  A smaller Management Group(MG) (US, UK, SADEV, CIDA, WB, UNEG & MOPAN) was created to undertake the work more easily  2010 ◦ Methodology and approach were pilot tested (ADB and WHO) under guidance of the MG/Task Team ◦ Draft report submitted to the Network in November  Network provided guidance to the Task Team on completing the pilot test phase 2

3.  Task Team was requested to: 1)Finalize pilot test (ADB and WHO) 2)Build stronger links with MOPAN and examine complementarity of results with MOPAN 2010 assessments 3)Refine methodology and guidelines 4)Further engage with MOs and with all stakeholders 5)Hold a Management/Steering Group meeting in spring 2011 to take stock and prepare response to the Network on lessons and future steps, taking into account developments by MOPAN 3

4.  Technical Team discussed strategies for complementarity and convergence with the MOPAN Technical Working Group  Received MOPAN reports on ADB and WHO and prepared draft paper on complementarity 4  Workshop of the Technical Team (CIDA, SADEV, DFID with MOPAN representative) to discuss possible revisions to pilot test report, paper on complementarity with MOPAN, and methodology and guidelines  ECG and UNEG invited to participate in the MG, however, they reconsidered their participation and provided their regrets  MG (US, UK, SADEV, CIDA, MOPAN) met and agreed on revisions to pilot test report, paper on complementarity with MOPAN, and methodology & guidelines

5. 1. Focused more directly on the implications of the results for the validity and utility of the meta-evaluation methodology 2. Clarified that results reported are illustrative and used only to test the approach and methodology 3. Classified validity of findings for each criteria as illustrated by each pilot case 5

6. 6

7. 7

8.  Proposed approach is workable and can be implemented within estimated time (6-8 months) and resource requirements (USD $125,000) with little burden on MO 8  Where MO produces adequate volume of evaluation reports over 3-4 year period covering significant portion of investments ◦ Approach works well and results can be generalized about MO DE  Where adequate number of evaluation reports is not available and coverage of activities cannot be estimated ◦ Results on DE are interesting but harder to generalize  At the completion of the pilot test there were opportunities to improve methodology by refining process and instruments

9.  Continued participation by MOPAN in the MG and dialogue between Technical Team and MOPAN Technical Working Group  Identified MOPAN Key Performance Indicators (KPI) and Micro-Indicators (MI) that can be compared to approach criteria  Compared results of pilot test to survey results reported by MOPAN 9

10. 10

11.  Where pilot test and MOPAN Survey criteria are comparable (for seven of 19 tested criteria) ◦ Results are often in agreement (e.g. on strength of the evaluation function)  Where results are not in agreement ◦ This can be explained by different time frames and organizational levels examined by each approach (e.g. on the strength of RBM systems)  The two approaches focus on different aspects of MO effectiveness and rely on different information sources  Results are complementary rather than in conflict  Together, they can provide a more complete picture of MO’s overall performance 11

12.  Reorganized and focused criteria more directly on development effectiveness  Clarified the process for selection options for strengthening development effectiveness information  Refined quality assurance criteria 12

13. 1. The achievement of development objectives and expected results (including policy impacts) 2. Cross-cutting issues: inclusive development which is gender sensitive and environmentally sustainable 3. The sustainability of benefits and results achieved 4. The relevance of MO activities and supported projects and programs 5. The efficiency of MO operations in support of projects and programs 6. The use of monitoring and evaluation to improve development effectiveness 13

14. 14 Scenario A MO Reporting on DE is Adequate Scenario B MO Reporting on DE is not Adequate but Evaluation Function is Scenario C MO Effectiveness Reporting and Available Evaluations Inadequate for Reporting on DE Option 1 Rely on MO Reporting Systems Option 2 Conduct a systematic synthesis of information from available evaluations Option 3 Implement actions aimed at strengthening MO Evaluation system and DE reporting Apply the meta- synthesis of evaluation results methodology Preliminary Review Review of essential documentation

15. 15 Evaluation Quality Assurance Criteria A. Evaluation purpose clearly stated B. Report organized, transparently structured, well written C. Evaluation objectives stated D. Evaluation subject clearly described E. Scope of the evaluation clearly defined F. Evaluation criteria clearly identified G. Multiple lines of evidence used H. Evaluation well designed: methods appropriate to the criteria used I. Evaluation findings and conclusions relevant and evidence based J. Evaluation report indicates limitations of the methodologies K. Evaluation report includes specific recommendations

16.  Recognize that the pilot test has been successful and acknowledge that the methodology is available for use by bilateral, multilaterals or others to apply, as desired  In light of the point above, recognize that others can build on the methodology and guidelines with greater participation by members of MOPAN and individual MOs as further work to assess the effectiveness of MOs are undertaken 16  Acknowledge that some donors and groups of donors may move forward to apply the methodology (jointly or individually; sequentially or concurrently with MOPAN)  Explore over time how to institutionalize or formalize the capacity and responsibility of assessing DE of MOs with MOPAN or other organizations  Explore further engagement on approaches and vehicles for assessing the DE of MOs with the MOs

17.  Is the approach and methodology acceptable to the Network?  Can agencies using the approach and methodology indicate it has been endorsed by the Network?  What is the level of interest among member agencies of the Network in leading or participating in reviews of MOs using the approach and methodology (as revised)? 17