1. Module 7 Key concepts Part 2: EVALUATING COMPLEX DEVELOPMENT PROGRAMS
Technical Assistance on Evaluating SDGs: Leave No One Behind
EvalGender+ Network together with UN Women, UNEG and EvalPartners

2. Presentation developed by Michael Bamberger and Asela Kalugampitiya based on Chapter 3 of ”Evaluating the Sustainable Development Goals within a “No-one left behind” lens through equity-focused and gender-responsive evaluations”

3. Outline The dimensions of complexity Complexity is a continuum
Some programs are less complex than they seem
Challenges of evaluating complex programmes
Applying complexity analysis to equity focused and gender responsive evaluation

4. The dimensions of complexity

5. COMPLEXITY ANALYSIS FRAMEWORK
EMERGENT OUTCOMES
SYSTEMS
DEFINING BOUNDARIES
Causal levels and social processes
Complex causality
Dimensions of complexity
Complexity dynamics

6. Dimensions of complexity in development evaluation
EMBEDDEDNESS AND THE NATURE OF THE SYSTEM
Historical, economic, political, socio-cultural, administrative and organizational, climatic and ecological, legal and regulatory context
Norms and beliefs
Interconnectedness, boundaries, dynamics (e.g. path dependence, system shock)
INSTITUTIONS AND STAKEHOLDERS
Governance, funding, coordination, implementation system
Number and diversity of stakeholders (e.g. implementing agencies, donors, politicians, beneficiaries, evaluators)
Stakeholder expectations, demands and ‘theories-in-use’
Conflict, cooperation, evaluation culture
INTERVENTION
Design and purpose (e.g. initial logframe, logic model, theory of change)
Size and scope (e.g. number and types of intervention activities, levels of intervention)
Data coverage, quality and accessibility
CAUSALITY AND CHANGE
Causality (e.g. non-linearity, emergence, feedback loops, multiple pathways)
Attribution and contribution
Theories, mechanisms, models of behavioral change
Implementation
Direct, indirect, intended, unintended, positive, negative effects
challenges in delimitation, sense-making, consensus-seeking, design, implementation and use of evaluations
EVALUATION
Purpose
Time, resources and data
Methodology
Participation and process
Values and ethics

7. The dimensions of complexity
The nature of the intervention (program)
Interactions among stakeholders and participating agencies
Causality and change
The nature of the system within which the program is embedded
Factors affecting the complexity of the evaluation

8. Dimension 1: The nature of the intervention
Complexity of the objectives
Size
Stability of program design
Implementation procedures
Number of services/components
Technical complexity
Social complexity
Duration
Is program design well tested

9. Dimension 2: Interactions among institutions and stakeholders
Budget: is use of funds well defined?
Number of funding and implementing agencies
Number of stakeholders and similarity of interests

10. Dimension 3: Causality and change
The nature of causal pathways (see next slide)
Degree of certainty of outcomes
Agreement on appropriate actions to address problems

11. Simple, linear causal chain
Intervening
variables
Input
Outcome

12. Complex causal chains

13. Examples of complex, non-proportional change
Tipping point
Inertia e f c t e f c t
Inputs
Inputs

14. Dimension 4: Embededness and the nature of the system
How independent is the program of the wider context
Complexity of the processes of behavioral change

15. Some key concepts in complexity analysis
Embededness
Adaptability and self-organization
Emergence

16. Complexity is a continuum
A program can have a high or low level of complexity on each of the 4 dimensions
For operational purposes an agency may use the Checklist for Assessing Levels of Complexity to define cut-off points: rating programs as having high, medium or low levels of complexity on each dimension

17. Interventions that look “complex” may not be
“complex” interventions may comprise a number of “simple” components
Many programs with complex coordination and management mechanisms may have relatively “simple” components and causal chains
Some agencies may find it convenient to claim their programs are too complex to be rigorously evaluated – to avoid critical scrutiny.

18. Challenges relating to program design
Program covers whole country
Multiple components with complex interactions
Interventions intended to cause multiple outcomes
No logical linkages between interventions and high level program goals
Program components operate differently in different settings
Multiple stakeholders and implementing agencies

19. Data availability Agencies collect only limited monitoring data
Difficult to compare data collected by different agencies
No standard data platforms
Different methodologies and reporting systems
No data on process analysis
Need real-time data to monitor change and detect problems

20. The complex nature of causality
Multiple causal pathways
Multiple outcomes caused by different combinations of inputs
Influence of contextual variables
Non-linear causality
Emergent designs
Long duration of programs

21. Real-world constraints
Budget
Time
Data access
Attitudes of stakeholders
Clients have different values and methodological preferences
Political time-tables and evaluation time-tables

22. 5. Applying complexity analysis to equity focused and gender responsive evaluation
Complexity dimensions of equity
Equity outcomes are affected by all of the SDG so there are very complex pattern of interaction
There are multiple stakeholders and implementing agencies often with competing objectives
Social exclusion is determined by the interaction between objective factors such as income, housing and access to services and by political, cultural and psychological factor that maintain existing inequalities.
Equity has a historical dimension which must be understood

23. Complexity dimensions of gender
Gender inequalities and patterns of change are determined by interactions among all of the SDGs
A model of social control is required to understand interactions among political, economic, socio-cultural, legal, ecological and historical factors.
Social control mechanisms include social pressures, sanctions, force and threats of force, violence, legal and religious sanctions
It is also necessary to study subtle patterns of behavioral change
Social control also operates within the family and involves processes that are difficult to observe.

24. Complexity dimensions of gender [continued]
Social media play an important role and must be studied
Many of the processes of empowerment and change in gender roles are intergenerational so the evaluation must have a long time- perspective

25. Resources
Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation & Use ntal_evaluation/applying_complexity_concepts_patton

26. Thank you for attention
For more information, please contact:
Marco Segone, Director, Independent Evaluation Office -UN Women, Co-chair - EvalGender+, Chair – United Nations Evaluation Group
Florencia Tateossian, Evaluation Specialist – UN Women
Asela Kalugampitiya, EvalPartners Executive Coordinator,