Module 10 Rapid, non-technical review of EFGR-responsive evaluation methodologies Technical Assistance on Evaluating SDGs: Leave No One Behind EvalGender+ Network together with UN Women, UNEG and EvalPartners

Presentation developed by Michael Bamberger and Asela Kalugampitiya based on Chapter 4 of ”Evaluating the Sustainable Development Goals within a “No-one left behind” lens through equity-focused and gender-responsive evaluations”

Outline Why it is important for non-evaluation specialists to understand the basic principles of evaluation design Evaluation levels and types The main evaluation methodologies The integration of EFGR evaluations into the SDGs

1. Why it is important for non-evaluation specialists to understand the basics of evaluation methodology There is no “best” evaluation design Different designs are required to answer different evaluation questions Many evaluators have a preferred evaluation design that they will often recommend even when it is not the most appropriate All evaluation designs have strengths and weaknesses Consequently evaluation managers and other stakeholders have an important role in working with evaluators to select the best design for a particular evaluation

2. Evaluation: Levels and types Evaluations can be conducted at 3 levels Policy level To assess the effectiveness with which policies are implemented and how well they achieve their objectives Program level To assess the effectiveness with which programs are implemented and coordinated How well each component achieves its objectives The effectiveness with the overall program achieves its broad goals and objectives Project level To assess the effectiveness with which projects are implemented and how well they achieve their objectives

There are 4 main types of evaluation Purpose Policy evaluations How well do the design, implementation and outcomes of national and sector policies contribute to EFGR goals? Formative evaluations Providing regular feedback to management and stakeholders on how the design and implementation of SDG projects contribute to EFGR goals Proposing corrective measures Learning lessons Developmental evaluations Similar to formative evaluation but focusing on innovative and emergent projects Assessing how complexity dimensions affect implementation and EFGR goals Summative (impact) evaluations Estimating quantitative (and sometimes qualitative) changes in EFGR indicators Assessing the extent to which these changes can be attributed to the SDG program

3. The main evaluation methodologies Experimental and quasi-experimental designs Statistical designs Theory-based evaluations (including theory of change) Case study methods Qualitative and participatory methods Review and synthesis Complexity responsive evaluations

A. Experimental and quasi-experimental designs Experimental designs [Randomized control trial = RCT] RCTs provide the strongest methodology for assessing project impacts Subjects are randomly assigned to receive the project treatment [water, school meals, small business training etc] or to be in a control group that does not receive the project treatment RCTs require large samples and are quite expensive Works better for simple project designs with few components and few outcomes than for large, complex programs. Ethical and political concerns using RCTs

Simple experimental design [RCT] Before project treatment Intervention After project implemented Project group P1 X P2 Control group C1 C2 Random assignment

Quasi-experimental design [QED] Similar design to RCT but random assignment is not possible and groups are matched using statistics or judgment Less precise than RCTs but can be used in a wider range of situations

Comparing RCTs and QEDs RCTs must be planned and administered before the project begins QEDs can be administered after the project has began RCT are methodologically more precise but QEDs are more flexible Both designs are quite expensive as they require large samples and a high level of research experience

B. Statistical designs Statistical modeling and econometric analysis. Mainly used at the national level to assess the impacts of policies or country-wide programs by comparing experiences with other countries Countries are matched on macro-indicators such as GDP growth, per capita income, investments in different sectors, rate of infrastructure construction. Useful for national programs where it is not possible to identify a comparison group (counter-factual) within the country

C. Theory-based evaluation [theory of change] Approaches such as theory of change are used to explain the steps through which an intervention is intended to achieve its objectives. Identifies the key assumptions on design, implementation strategies and potential constraints which must be tested A good theory of change should also focus on broader social, political, economic, legal and other factors affecting project implementation and outcomes Program impact and effectiveness can be assessed by comparing actual implementation experience with the TOC

Theory-based evaluation [continued] Theories of change (TOC) can be used to provide a framework for every kind of evaluation design TOCs help provide a framework for learning lessons from the experience of project design, implementation and outcomes.

D. Case-based approaches The case is taken as the unit of analysis A case may be an individual, household, community, organization, state or even a country A sample of cases are selected to: be representative of a total population To illustrate particular sub-groups (the most or least successful, outliers etc) The analysis can be qualitative and descriptive or quantitative Case-based methods can be used as stand-alone evaluations or to illustrate and explore groups identified in surveys.

Qualitative comparative analysis (QCA) QCA is a form of case study that has become popular in recent years It requires a higher level of technical expertise but can provide more rigorous estimates of project effectiveness and outcomes QCA identifies: the combination of factors that must be present for a project to achieve its intended outcomes The combination of factors that prevents a project from achieving its intended outcomes Usually requires a sample of at least 15 cases

E. Participatory and qualitative analysis Involves a wide range of stakeholders in design, implementation and interpretation of the evaluation Can be used for methodological reasons or to support a rights-based/ empowerment approach Provides a deeper understanding of the lived-experience of different groups Gives voice to poor and vulnerable groups Analysis of processes Helps understand the influence of contextual factors and processes of social control

Participatory and qualitative analysis [continued] Uses a mixed methods approach combining a wide range of tools and techniques for data collection and analysis Participatory approaches are in-line with the human rights and gender equality evaluations proposed by UNEG

Examples of participatory and qualitative methods Participatory consultative methods such as PRA Outcome mapping and outcome harvesting Most significant change Participant observation Key informant interviews Community consultations and some kinds of focus groups Longitudinal case studies

F. Review and synthesis studies Identification of all evaluations conducted on a particular topic (e.g. the effects of micro-credit on women’s empowerment) Selection of studies that satisfy standards of methodological rigor Often only randomized control trials are acepted May incorporate a theory of change to structure findings Summary of findings and lessons that are statistically sound Provides a useful starting point for designing programs and indicates what kinds of outcomes can be expected – and what is not likely to be achieved

G. Complexity-responsive evaluations Not widely-used but important for EFGR as programs with a gender or equity focus frequently involve dimensions of complexity Complexity-responsive evaluations can largely use familiar evaluation tools described in this session but it is necessary to begin with a complexity diagnostic to understand why and how the program is complex Conclude by reassembling all of the individual evaluations to understand the big picture Complexity is important because often each program component has a positive rating but the overall program fails to achieve its broader objectives

4. Principles for integrating EFGR evaluations into the SDGs Begin with a review of lessons learned from past approaches and evaluations build gender and equality into the theory of change and results framework Develop a checklist of areas where EFGR principles and indicators can be integrated into the evaluation process Begin with rapid diagnostic studies to help understand the EFGR issues which should be addressed Integrate gender and equality into ongoing or planned evaluations