1. March 19, 2013 Jennifer Mulik, MA, MPH– Global Director Health jmulik@pactworld.org Measuring Capacity Development Performance

2. The problem “The monitoring and evaluation of capacity building is as much a challenge now as it was two decades ago.” Simister & Smith, 2010, INTRAC

3. Theory of Change for Capacity Development Outputs Change in the internal systems, skills and policies of organizations, networks and systems Outcomes Improved external performance of organizations, networks and systems Impact Improved health, environment and/or livelihoods in the communities served by organizations, networks and systems (a)Stronger local organizations; (b)Improve their performance; and (c)Have greater impact in their target communities. OCA OD Roadmap ISP Tracking OPI Outcome Mapping Most Significant Change Evaluations

4. Defining Performance - Framework IDRC/Universalia CD Outcomes Framework Efficiency Relevance Sustainability Effectiveness Leadership Structure Human resources Program / services Financial Technology Linkages Administrative Political Social / cultural Technological Economic Stakeholder History MissionCulture Incentives Rewards Infrastructure

5. Operationalizing Performance Effectiveness: Results Standards Efficiency: Delivery Reach Relevance: Target Population Learning Sustainability: Resources Social Capital

6. Benchmarks Level 1Level 2Level 3Level 4 Effective Results The organization is in the process of developing outcome level targets for its programs & services. The organization has set clearly defined outcome level targets for its programs & services. The organization has met over 50% of outcome level targets for its programs & services. The organization has met over 75% of outcome level targets for its programs & services. Level 1 Level 2 Level 3 Level 4

7. Global OPI Scores by Length of Partnership

8. Global OPI Scores Time Series

9. Ethiopia OPI Scores by Length of Partnership

10. Organizational Performance Index (OPI) Testing Project The purpose of the OPI Testing Project funded by the Rockefeller Foundation is to check the OPI reliability, i.e. whether the Index generates the same set of scores when it’s reviewed by two people, separately.

11. Kappa statistic inter-score reliability CountryPact 1 v Pact 2Pact 1 v PartnerPact 2 v Partner Vietnam.804.498.556 Ethiopia.706.454.576 South Sudan.605.571.437 Swaziland.504.495.583 Nigeria.415.502.299 Zimbabwe.357.724.587

12. Result Our OPI tool and methodology of application meets the reliability standard