The complex evaluation framework

2 Simple projects, complicated programs and complex development interventions Complicated programs Simple projects blue print producing standardized product relatively linear Limited number of services Time-bound defined and often small target population Defined objectives May include a number of projects and wider scope Often involves several blueprint approaches Defined objectives but often broader and less precise and harder to measure Often not time-bound Context important multiple donors and national agencies country-led planning and evaluation Non linear Many components or services Often covers whole country Multiple and broad objectives May provide budget support with no clear definition of scope or services multiple donors and agencies context is critical Large, complex Small, simple Complex interventions

The Special Challenges of Assessing Outcomes for Complex Programs 3 1.Most conventional impact evaluation designs cannot be applied to evaluating complex programs 2.No clearly defined activities or objectives – General budget and technical support integrated into broader government programs – Multiple activities – Target populations not clearly defined – Time-lines may not be clearly defined

Special challenges continued 3.Multiple actors 4.No baseline data 5.Difficult to define a conventional comparison group

Alternative approaches for defining the counterfactual for complex interventions 1. Theory driven evaluation 3. Quantitative approaches 4.Qualitative approaches 5.Mixed method designs 6.Rating scales 7.Integrated strategies for strengthening the evaluation designs 5

Theory- based approaches strengthening alternative counterfactuals Unpacking complex programs Portfolio analysis Reconstructing baseline data Creative use of secondary data Secondary data Triangulation Qualitative approaches Quantitative approaches STRATEGIES FOR EVALUATING COMPLEX PROGRAMS Rating scales Counterfactual designs Attribution analysis Contribution analysis Substitution analysis Mixed method designs Estimating impacts The value- added of agency X Net increase in resources for a program