Complementary Approaches to Evaluating Social Safety Nets at the World Bank: Evaluation questions and approaches AEA meetings 2010 Jennie Litvack and Victoria Monchuk IEG World Bank 1

What are Social Safety Nets (SSNs)? ►Non-contributory transfers, targeted, in some way, to the poor and vulnerable ►Such as: Unconditional or conditional cash transfer programs Food and in-kind transfers Food and energy subsidies School feeding programs Scholarships Public works programs Non-contributory /social pensions 2

What functions do Safety Nets serve? ►Protect the poor and vulnerable from shocks-- individual or systemic shocks ►Redistribute resources by increasing the consumption of the poorest in society and reduce chronic poverty and inequality ►Improve investments in human capital by the poor ►Protect the poor and vulnerable from negative effects that can accompany macro reforms 3

Safety Nets and the World Bank ►1980s: Bank started supporting SSNs – structural reform ►1990s: Transition to market economies of former Soviet Union ►1998: Asia Financial Crisis ► : Debt crisis in Latin America, Turkey ► : Food, fuel and financial crisis 4

Safety net lending has increased 5

Safety nets in all regions 6

Evaluation purpose Accountability: $13 billion spent ►Assess how effectively the Bank has helped countries develop SSNs to protect the poor and vulnerable and mitigate risk Learning: New strategy in preparation ►What lessons can be learned for informing future support 7

Evaluation questions ►How effective, relevant and sustainable have Bank supported projects been in LICs and MICs? ►How relevant and effective has overall Bank assistance been in helping countries establish sound safety nets? ►How can the effectiveness and relevance of Bank support be explained and understood? ►What lessons can be derived for future support? 8

Evaluation approach ►Large and complex evaluation scope World Bank thematic area ►Multiple evaluation questions Describe what Bank is doing Assess how relevant and effective it is Assess what final results have been produced Understand why things work, why not, how improve? Need to develop evaluation building blocks 9

Evaluation building blocks ►SSN portfolio reviews Lending, Trust Funds, Analytical work and Technical Assistance 244 lending projects coded and analyzed Analyzed trends in Bank engagement over time, by: region, country income level, SSN instruments, SSN functions, project results framework, performance, etc. Also for Trust Funds and analytical and technical work 10

Evaluation building blocks ►Country case studies 30 case studies analyzed (25 randomly selected desk-based, 5 purposely selected field-based), 3 in-depth Project Performance Assessment Reviews Looking beyond projects–Bank program in country context Review CASs, lending projects, analytical work, as well as semi-structured interviews with staff, other donors and clients (in field-based cases) Examined the relevance and nature of the Bank’s engagement; country’s application of social risk management framework (SP Strategy, 2001); and the results in countries 11

Evaluation building blocks ►Food, fuel and financial crisis (FFF) Staff Survey On-line survey sent to World Bank staff working on social protection/SSNs in all client countries Focused on countries' experiences with SSNs in the context of the FFF crises, as well as World Bank SSNs assistance to countries in response to FFF crises 12

Evaluation building blocks ►Impact evaluations of safety nets Literature Review: 137 impact evaluations reviewed to examined evidence re what works and under which conditions; distribution of benefits; sustainability of benefits; and generalizability of results. Sub-sample of 25 programs supported by the Bank. Two new rigorous impact evaluations–Colombia and Pakistan–examining longer term impacts of conditional cash transfers (CCTs) 13

Evaluation building blocks ►Background studies Political economy of SSNs programs Decentralization and SSNs: challenges and strategies for the Bank SSNs in low-income countries A review of Public Works Programs Development Policy Loans for Economic Reforms: Bank experience with SSNs to mitigate distributional impacts on the poor World Bank matrix structure and how it affects assistance for SSNs 14

Matrix mapping each evaluation question with building blocks ►Relevance Portfolio reviews (lending, analytical and technical work) Case studies Background papers on political economy, public works programs, SSNs in low-income countries, etc. Application of social risk management strategy, etc. ►Effectiveness/understanding effectiveness Portfolio review (design details, results frameworks, project ratings) Case studies Impact evaluations, etc. 15

The “Matrix” 16

Main take-aways: ►Develop building blocks ►Triangulate to answer questions ►Evaluation will be available spring 2011 worldbank.org/ieg 17