Improving Effectiveness and Outcomes for the Poor in Health, Nutrition, and Population An Evaluation of World Bank Group Support Since 1997 Martha Ainsworth, Independent Evaluation Group, World Bank Presentation for the American Evaluation Association November 12, 2009

Objectives of WB health, nutrition, population projects, ObjectivePercent Improve health status61 Access, quality, efficiency, equity57 Health reform33 Institutional capacity31 Better management18 Improve nutrition10 Population/family planning4

IFC health investments, (54 investments, $580 million)

I.Evaluation questions, scope and methodology II.Main findings and messages III.Evaluation challenges

Evaluation questions 1.How effective has the Bank Group support for HNP been? How can it be more effective in reaching its objectives? 2.Has the support reached the poor? 3.Have monitoring and evaluation improved, and are they contributing to results? 4.What are the lessons from four approaches? Sectorwide approaches; communicable disease control; health reform; multisectoral approaches

Scope ►World Bank Group: World Bank & International Finance Corporation (IFC) ►Support since 1997 ►World Bank: country level projects financed, analytic work, policy dialogue ►IFC: investment projects and advisory services

Building blocks 1. Desk reviews World Bank sector and country strategies (for objectives) and timeline of key events Systematic analysis of objectives, strategies, design, and effectiveness for all 220 projects in health, nutrition, population approved from , based on project design and completion reports (self-evaluations) Analysis of 117 water supply & sanitation projects and 229 transport projects approved from with respect to their health objectives and achievements Inventory of analytic work, content analysis

Country case study or project evaluation 2. Fieldwork Regional balance Low and middle-income Variation in experiences Key themes

Field evaluations are ex post, objectives-based ►Triangulate data to establish plausible causality: ►Establish a results chain (Inputs, outputs, outcomes, impacts) ►Account for other determinants (Other donor support, weather, changes in income, poverty, national policies) ►Document trends before and after, in project and non-project areas ►Variation in results within and across countries to generate lessons ►Sources: existing studies, analysis of available data, interviews with stakeholders

II. Main findings and messages

1. Two-thirds of the Bank’s HNP projects have had satisfactory outcomes… Successful malaria control in Eritrea

…but performance of the Bank’s HNP support has lagged behind that of other sectors Note: IFC’s initial hospital investments performed poorly, but more recently they perform better than IFC projects in other sectors

Performance of the Bank can be improved by… ►Strengthening risk assessment and mitigation at the design stage, especially political economy ►Reducing project complexity, especially in low- capacity environments, by greater selectivity, prioritization, sequencing of activities ►Up-front institutional analysis ►Intensified supervision ►Better monitoring and (especially) evaluation as a management tool

2. Targeting the poor is important… ► In most regions, government health spending is not pro- poor Source: Global Monitoring Report 2009

…but accountability for reaching the poor and evidence that the poor have benefitted is weak ►Only half of the WB project portfolio has a poverty focus; 13% had objectives that mention the poor; 7% had equity objectives ►Of the 12 completed projects with objectives targeting the poor, only 2 actually measured outcomes for the poor ►Of 20 Poverty Assessments in countries with high fertility, only one had a chapter or sub-chapter on population ►Share of analytic work on HNP and poverty has declined ►Share of Poverty Assessments (FY00-07) w/health chapters dropped from 80% to 58%, nutrition from 28% to 12%.

3. Monitoring has improved, but shortfalls in monitoring and evaluation are reducing effectiveness ►Increasing share of projects have monitoring indicators and baseline data, but more than half of outcome indicators have no baseline value (FY07) ►A third of HNP projects finance pilot interventions or impact evaluations; only about 4% actually evaluate. ►If there’s no evaluation design in the Project Appraisal Document, then the evaluation doesn’t take place.

To ensure that the poor benefit… ►Target support to the poor and monitor outcomes ►Increase support to reduce high fertility; expand support to reduce malnutrition ►Expand IFC support for private investments with greater social benefits ►Manage complexity, improve quality ►Increase the impact of other sectors on HNP outcomes ►Improve the results focus and better governance through evaluation Recommendations To be more effective…

III. Evaluation lessons and challenges Challenges ►Multitude of objectives and countries, changes over a decade ►In an ex-post mode, limited by what the project collected ►Establishing the counterfactual ►Getting ownership of the findings in non-health sectors Lessons ►Time the evaluation to have maximum impact; don’t wait 10 years! ► Try to answer a few important questions well ►Contrast good and bad experiences to understand what works and why ► Share and discuss intermediate findings and outputs ► An External Advisory Panel adds credibility to the results

HNP evaluation website