Evaluation: For Whom and for What? Centre for Social Policy Studies University of Ghana Daniel Doh & Stephen Ntow

Outline Introduction Some historical issues Conceptualising impact evaluation Impact evaluation for whom? Evaluation for what? Final thoughts

Introduction Development programming is concerned with how best to improve wellbeing of people in society. In attempting to do this, there is the need to examine how best policies or programmes are working or not working. This brings to the fore the issue of impact evaluation Impact evaluation serves both objectives of evaluation: lesson-learning and accountability

Brief history of impact evaluation Impact evaluation emerged in the 20th century as a result of: advances in statistics and econometrics emergence of political and social needs Started in medicine, education and more recently in economics and political sciences Examples of impact evaluation methods are: Randomised controlled trials Difference in difference analysis In recent times however, there is also the reliance on theory-based evaluation and other participatory (qualitative tools)

Source: CEGA (2010) – The Asian Foundation Impact Evaluation Workshop

What is common to the definitions above? All these definitions share a number of concepts: Social programmes: organised collective efforts to improve social welfare Welfare outcomes: dimension of living conditions/well-being of a population (the MDGs) Counterfactual analysis: a comparison of welfare outcomes with and without the project Impact evaluation is a systematic process of explaining how a development intervention has or has not contributed to changes in welfare of a people

Shifting the Programme Design Paradigm From “We know what is best”…. To “We can learn what is best in this context, and adapt to new knowledge as needed”

Impact Evaluation for Whom? The final product of an impact evaluation is like a buffet dinner served before different stakeholders and interest groups. Each stakeholder or group goes for only what is perceived to be relevant and useful. However, over the years some important stakeholders have not found their menu on the dinner table!

Evaluation for whom? Donor Community Project Beneficiaries Governments/Other Policy Actors CSOs Programme Management Project Beneficiaries Academics

Evaluation for what and When? “to recognize impact and its relevance and to use lessons learned to make the positive impacts even bigger or larger next time” (Daniel Svoboda, 2008) Impact evaluation is part of a broader agenda on evidence based policy making When? Early (As part of programme design) When you introduce change. Scale – up is envisaged

For Who? Government and other Policy Actors Policies Accountability Donors / Development Partner Know how their contributions affect welfare of final beneficiaries satisfy their home country tax payers Development practitioners and Managers Accountability, Tool for efficiency and effectiveness Beneficiaries Measurement and realization development gains. Decision making Civil Society Accountability and Justification Shape informed decisions of their constituents Academics Knowledge Replication

Major Considerations for IE Methodological issues: Credible, Rigorous, uses of evidence and who? Randomized controls or participatory methods? Situational appropriateness: Purpose to method? Some important considerations Clarifying the values Developing and/or testing a theory (Theory o Change) Describing impact and variable Explaining attributions Synthesizing evidence into an overall judgment. Contextualizing the report Reporting findings and supporting its use.

Challenges & suggested solutions Variations in implementation and environment across different sites. Different impacts. Absence of baselines. Extended durations and controlling other factors. Influence of other programs and factors Diverse components Resource constraints

Final thoughts Impact evaluation as a professional practice and an academic exercise has found its eternal place in development practice It is important therefore to brace ourselves with the requisite methodological skills to be able to meet demand This methodology, as much as it adheres to conventional/standard practices must also take into account our socio-cultural context and be made meaningful to the people it affects

Other Definition and References

Rossi, Lipsey and Freeman, Evaluation ‘An evaluative study that answers questions about programme outcomes and impact on the social conditions it is intended to ameliorate’

Development Assistance Committee (DAC) ‘The evaluation of the ‘positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.’

Development Impact Evaluation Initiative (DIME) ‘Impact evaluations compare the outcomes of a program against a counterfactual that shows what would have happened to beneficiaries without the program.’

Independent Evaluation Group (IEG) ‘A counterfactual analysis of the impact of an intervention on final welfare outcomes.’

Thank You