Why Evaluate? Evaluating the Impact of Projects and Programs, Beijing, China April Shahid Khandker World Bank Institute

Rationale for evaluation Evaluation §Helps identify and measure the results §Helps identify the causal link between intervention and results §Provides a systematic and objective assessment of program impacts §Helps determine if interventions are relevant and cost- effective § Promotes accountability, evidence-based policymaking, and learning.

When to evaluate? §Evaluation can only be done after a certain time and requires thorough investigation §It is a snapshot against some benchmark at a point in time of intervention

What to evaluate? Operational versus impact evaluation Operational evaluation (OE) measures cost efficiency of program implementation but does not establish causality. Impact evaluation (IE) measures economic efficiency in delivering services and seeks causality. Not all programs should be subject to evaluation: Exception:.. §A program of strategic importance §Contributes to knowledge gaps §Innovative design

How to evaluate? §Impact evaluation hinges on determining what would have happened if the program had not existed. §Good practice involves a comparison of outcome before and after intervention with those with and without intervention §The problem is finding a counterfactual – various methods are available §Findings sensitive to the method; alternative methods desirable

Country experiences: §Policymakers ask for feedbacks to determine whether to upscale a program Micro-finance program in Bangladesh §Policymakers required to determine what works and how does it work? Thailand village fund Indonesia urban poverty project Social development fund of Egypt §Government needs to determine the innovative nature of the program design? Integrated versus sectoral intervention: The case of social fund in Egypt

Country experiences (continued): §Government needs to focus on possible consequences of any policy change §Subsidized education policy for girls in Bangladesh and Pakistan T §Teacher Incentive structure in Kenya §Government needs to determine if an economy wide change caused any negative effect: §Impact of financial crisis in East Asia §Impact of China’s accession to WTO

Chinese experience The Grazing Ban component in the China’s Loess Plateau Watershed Rehabilitation Project: Initially by a few village committees in the project area; Concerns: poor people’s capability of adapting the change, enforcement, etc Impact Evaluation: Free grazing is one of the major cause of soil erosion on the plateau; Grazing ban led to pen-feeding and the introduction of improved breeds of livestock; Together with other measures, the soil erosion started to improve; Farmer themselves get the incentive to enforce the ban. Eventually it was adopted on a countrywide basis by many counties widely beyond the project counties.

China’s Poverty Reduction Nearly 400 million people were lifted out of poverty from 1981 to 2001 (World Bank 2005b, using $1/day poverty line), representing a decline of poverty incidence from 53% in 1981 to 8% in 2001(Ravallion, Chen 2004). But the progress has been highly uneven across time and space (Wang 2005). What are the driving factors? Many studies have been conducted: An econometric study showed 46.9% of the total output increase in (one of the fastest growing period) can be attributed to the Household Responsibility System (Lin 1992). Ravallion and Chen (2004) found that nearly half of the poverty reduction that happened before 1985 was the result of the rural reform, which led to a clearly defined property right, more equitable access to land, and less distortion in agri prices. … Results: Government’s focus on Rural Income Growth.

China’s Southwest Poverty Reduction Project §Impact analysis shows: Had a significantly favorable impact on many dimensions What worked and how? Impact Evaluation The labor mobility component—benefits the extremely poor (human capital, remittance, etc) Extensive participation of poor households in all phases—fundamental Emphasis on capacity building, flexibility Etc. Key lessons of the project like new approaches to labor mobility, village deve. Planning, poverty monitoring, etc. have become part of China’s national poverty reduction policy and have been extended to poor counties throughout the country

Monitoring versus Impact Evaluation §Policymakers do not need an impact assessment of all programs and policies §What policymakers need most often is better design and effective delivery of program inputs, which can be done with a better monitoring system in place

What does monitoring do? §Monitoring is a tool that provides regular information on: 1.how things are working 2.how a project is being implemented 3.how project progresses relative to targets

What does monitoring do? (continued) Rationale for Monitoring: §Monitoring holds implementers accountable for delivery of inputs §Provides basis for corrective action §Provides assessment of continued relevance §While monitoring looks at progress relative to targets and assumes there is causality, evaluation seeks causality.

Components of a M&E Strategy §Outcome-based monitoring such as poverty monitoring §Complemented with systematic impact evaluation §Creating a feedback process §Building capacity for monitoring and impact evaluation §Promoting participation