1. Roma Inclusion: Monitoring and Evaluation for Results November 2010, Brussels, Belgium – DG REGIONAL POLICY Joost de Laat (PhD), Economist, Human Development

2. 2 Key Messages More systematic M&E of Roma policies and programs will support Roma inclusion There is considerable M&E experience to build on There are existing M&E partnerships that can be strengthened Collective action can achieve this institutionalization of M&E for results

3. Outline The Case for Monitoring and Evaluation WB Monitoring for Roma Inclusion WB Impact Evaluations for Roma Inclusion Moving Forward

4. 4 Why Monitor and Evaluate Examples of scale-up of proven programs Opportunidades (PROGRESA) conditional cash transfer – Mexico Balsakhi remedial education – India Various governments have institutionalized M&E systems that systematically conduct rigorous evaluations – e.g. Mexico, USA, Colombia, Chile

5. 5 Why Monitor and Evaluate  Supports learning about programs  Helps re-designing and improving programs  Encourages comparison among programs and supports efficient use of public funds  Improves planning, operation and budgeting  Promotes transparency and accountability  Generate public support to continue or to scale-up proven programs

6. 6 WB Monitoring for Roma Inclusion Inputs Human Resources Regional Level: Departments such as Human Development (HD) for Europe and Central Asia(ECA) Central (anchor) units such as the World Bank Data Group: Statistical Development and partnership (DGSDP) team Partnerships with academics, governments, civil society Financial Resources Trust Funds: e.g. Nordic Fund – Roma health study WB budget: e.g. Economic benefits Roma inclusion study

7. 7 WB Monitoring for Roma Inclusion Activities Implement Roma booster samples (w/ partners) Regional survey (2011; with EC, UNDP) Bulgaria (2010, 2011; with OSI) Czech Republic (2008; CZ govt) Serbia (2005) Leverage national surveys with information on Roma Romania Family Budget Survey Serbia Laborforce Survey Undertake diagnostics. E.g. “Roma Inclusion: an Economic Opportunity” study (2010)

8. 8 WB Monitoring for Roma Inclusion Policy implications from study Issue: Large employment and wage gap Diagnostic: Male laborforce participation is as high as majority Female laborforce participation is lower than majority Vast majority do not depend on social assistance Vast majority do not have secondary education Returns to secondary education among Roma are high

9. 9 WB Monitoring for Roma Inclusion Policy implications from study Skills: Roma want to participate in the labor market but many lack necessary skills Employment activation of adults (e.g. job search asst.) Improve secondary completion (e.g. mentors) Improve school readiness (ECD) Social assistance: protects the poor Consider linking it with skill acquisition (e.g. adult training, school participation of children)

10. 10 Impact Evaluations for Roma Inclusion Implement and evaluate Multiple policy options to address Employment activation School completion School readiness Which policy actions are most (cost-) effective? Combine programmes and projects with evaluations Find out what works best Generate public support to continue or to scale- up proven programs

11. 11 WB Impact Evaluations for Roma Inclusion Inputs Human Resources Regional Level: ECA Impact Evaluation cluster Central (anchor) units such as the Development Impact Evaluation Initiative (DIME) Partnerships with academics, governments, civil society Financial Resources Trust funds: e.g. Spanish Impact Evaluation Fund (SIEF); WB budget Client government self-financing

12. 12 World Bank Impact Evaluation Clusters 11 Impact Evaluation Clusters (DIME/SIEF): 1.Conditional Cash Transfers 2.Early Childhood Development 3.Education Service Delivery 4.HIV/AIDS Treatment and Prevention 5.Local Development 6.Malaria Control 7.Pay-for-Performance in Health 8.Rural Roads 9.Rural Electrification 10.Urban Upgrading 11.ALMP and Youth Employment 12.Roma Inclusion cluster

13. 13 Impact Evaluations for Roma Inclusion Ongoing World Bank IE work in Europe Roma Related: EU Parliament Roma Pilots on Early Childhood Education and Care, and Microfinance Bulgaria - social inclusion project - ECD Serbia - local service delivery – school grants Macedonia – conditional cash transfer Other: Active Labor Market Policies: Latvia (public works), Serbia (self-employment grants), Turkey (job training)

14. 14 Impact Evaluations for Roma Inclusion Counterfactual methodologies used Ex ante methodology: 1.Randomized evaluations (e.g. Macedonia, Turkey) 2.Double-difference (DD) methods (e.g. Bulgaria) Ex post methodologies: 3. Propensity score matching (PSM) (e.g. Latvia) 4. Regression discontinuity (RD) design (e.g. Serbia) 5. Instrumental variable (IV) methods Proper comparison group: treatment outcomes must be identical to comparison group outcomes, if the treatment group did not participate in the program

15. 15 Way Forward: M&E for Results Strengthen existing Roma M&E partnerships National governments European Commission and multilaterals Academics and policy institutes Civil Society Institutionalize evidence and policy action link Systematic monitoring and diagnostics Systematic rigorous impact evaluations Systematic knowledge sharing Strengthen links between evidence, program design, and funding

16. 16 Conclusion More systematic M&E of Roma policies and programs will support Roma inclusion There is considerable M&E experience to build on There are existing M&E partnerships that can be strengthened Collective action can achieve this institutionalization of M&E for results

17. 17 THANK YOU!

18. 18 Impact Evaluation Resources World Bank (2010) “Handbook of Impact Evaluations” by Khandker et al. www.worldbank.org/sief www.worldbank.org/dime www.worldbank.org/impactevaluation http://ec.europa.eu/regional_policy/sources/ docgener/evaluation/evaluation_en.htm http://ec.europa.eu/regional_policy/sources/ docgener/evaluation/evaluation_en.htm www.povertyactionlab.org http://evidencebasedprograms.org/