IPDET Lunch Presentation Series Equity-focused evaluation: Opportunities and challenges Michael Bamberger June 27,

Why equity matters Many programs do not reach/benefit the poorest and most vulnerable sectors Widely-used indicators [MDGs, HDIs, economic growth] are based on averages and conceal gap between the poorest [bottom 20%] and the rest of the population is not being reduced The poorest groups are invisible Poor and vulnerable groups have special needs that are not addressed Major gaps continue to exist between men and women No society has achieved full gender equity 2

Defining equity Equity refers to fairness It examines unequal outcomes (e.g. income, health, access to electricity) to determine if differences are due to: Unavoidable circumstances (e.g. floods, war, international economy, limited public resources) Free choice (some people may prefer less working hours and more leisure) Avoidable circumstances = inequity: Unequal opportunities (family background, education, poverty) Contextual factors (economic, political, social, environmental etc) 3

4 Opportunities Education Health Security Housing Outcomes Contextual factors Economic Labor markets Transportation Political Socio-cultural Environmental Psychological Education Employment Income Health and life expectancy Political participation and voice Unavoidable Aging Natural disasters? War? Economic cycles? Geographic location? EQUAL UNEQUAL and inequitable UNEQUAL But not inequitable Avoidable Free choice? UNEQUAL EQUAL Linking opportunities and outcomes in equity analysis +/- effects on equity

Determinants of inequity Income Location Ethnicity, caste, religion Gender Disability Legal systems 5

Why so many evaluations do not address equity Political reasons Equity analysis challenges indicators of progress (MDGs, HDI etc) Governments do not wish to give equal benefits Roma, refugees, sexual orientation, ethnic minorities Socio-cultural Society may support marginalizing “undesirable” groups Methodological 6

Methodological challenges for evaluating equity Lack of disaggregated data Many outcomes are sensitive or difficult to identify and measure Unanticipated outcomes Complex processes of behavioral change Equity evaluation requires new methodologies Findings can be threatening 7

Equity focused evaluation Assessing the relevance, effectiveness, efficiency, impact and sustainability of interventions on equitable development Identifying which groups benefit most and least from programs and policies Assessing effectiveness of policy frameworks and identifying policy gaps 8

Some equity-focused evaluation frameworks Theory of Change Contribution analysis Realist evaluation Social exclusion analysis Gender analysis Systems analysis Bottleneck analysis Policy gap analysis 9

Benefits of systems analysis for equity evaluation Programs are embedded in existing social and political systems with historical traditions and linkages among stakeholders Different actors have different perspectives on the program Boundaries – open or closed New interventions cause contradictions and conflict Outcomes determined by how conflicts resolved 10

Bottleneck supply and demand framework: factors affecting use of services by vulnerable populations 11 Use of services by vulnerable population Supply side factors Coverage Budgets and available resources Demand side factors Knowledge, Attitudes and Practice of vulnerable groups Community ownership Culturally acceptable services Culturally sensitive staff distance cost of travel and fees available transport time poverty Contextual factors Economic Political Institutional Legal and administrative Environmental Socio-cultural

Country and project level tools for equity analysis Multi-dimensional poverty index (UNDP) Quintile analysis Public expenditure incidence analysis Gender inequity analysis Access to public services [e.g. Citizen Report Cards] Participant observation Participatory group techniques [e.g. Wealth Ranking] 12

Evaluators’ Equity Checklist Who are the poorest and most vulnerable groups in your program? What constraints affect their participation? Does your evaluation address these constraints? Does your sampling frame cover all of the target population? Are you sure about this? Who does and does not use project services? Why? Are there any constraints on women’s participation? Will your evaluation design detect unanticipated and negative outcomes? Are you sure about this? 13

A final reality check Do you really understand who are the vulnerable and marginal groups? Are you really sure their situation will be covered by the evaluation? Will your data collection instruments really capture the situation of invisible groups who do not want to be interviewed, who are socially marginalized or in fear of persecution? 14

15