1. 1 A Tool for Designing Evaluations of Paradigm Shifts in Complex System Interventions Beverly Parsons InSites www.insites.org www.insites.org bparsons@insites.org Pat Jessup InSites www.insites.org www.insites.org pjessup@insites.org Marah Moore I2i Institute www.insites.org www.insites.org marah@i2i-institute.com Handouts and PowerPoint presentation available at: www.insites.org/AEA2010

2. 2 Session Objective  To build capacity of evaluators to design evaluations of interventions in complex systems

3. 3 Four Phases of Evaluation Collect Data Make Meaning from Data Design Evaluation Shape Practice

4. 4 Tool’s Role in Evaluation Design  Tool represents the intervention’s theory of change and relationship of intervention to larger system/situation  Evaluation designers also need to know: –Whether evaluation is part of intervention (internal) or is separate from intervention (external) –If evaluation is part of intervention, does it participate through Evaluation processes and/or Evaluation findings

5. 5 Session Agenda  Introduction to tool for designing evaluations of interventions in complex systems  Work in small groups to begin tool development  Reconvene to discuss completion of tool and its use

6. 6 The Tool

7. 7 Theory of Change in Paradigms, Structures, and Conditions of Complex Systems Example from Cross-Site Evaluation of Quality Improvement Center on Early Childhood

8. 8 Tool Characteristics  Tool focuses on deep patterns of change in systems (i.e., changes in structures and paradigms)  Tool recognizes multiple system dynamics at play in complex systems

9. 9 Differences from Typical Logic Model  Will almost always depict more aspects of the system than just the intervention  Depicts phases of change over time with variation in rates and patterns across components of the system  Likely to depict the intervention through a different lens

10. 10 Background Concepts

11. 11 What is a System?  A group of interacting, interrelated, and interdependent elements forming a complex whole  A configuration of parts connected and joined together by a web of relationships  The whole is greater than (different than) the sum of its parts

12. 12 The Systems Iceberg Events and Behaviors Patterns Structures Paradigms Conditions What is happening now? How do patterns play out over time and space? What are the drivers? How are they related?

13. 13 Deep System Change  Deep system change involves shifts in paradigms and structures of system boundaries, relationships, perspectives, and dynamics over time and space.  Change is evident in mid-level patterns through data analysis/synthesis.  Change is evident in observable events and behaviors.

14. 14 Paradigm Shift: Risk to Protective Factors Risk Factors Substance Abuse Poverty Stress Protective Factors Parental resilience Social connections Concrete support in time of need Knowledge of parenting and child development Social and emotional competence of child

15. 15 Dynamics of a Social System and Its Context Unorganized (random) Organized (simple, complicated) Self- Organizing (complex, adaptive) Relationships High Predictability Low Predictability Perspectives High Agreement Low Agreement Context

16. 16 Group Task  Succinctly describe the paradigm shift.  Identify the ultimate beneficiary of the paradigm shift.  Identify the likely key points of leverage to shift the system.  Order the likely points of leverage along left side of page beginning with the ultimate beneficiaries of the system and those close by.

17. 17 Phases of Shift  Baseline understanding of fundamentals and system dynamics  Trying out interventions in system  Tipping point to new fundamentals and system dynamics balance  Sustainable adaptive balance of new fundamentals and system dynamics in a shifting context

18. 18 Theory of Change in Paradigms, Structures, and Conditions of Complex Systems Example from Cross-Site Evaluation of Quality Improvement Center on Early Childhood

19. 19 Other Key Evaluation Design Questions  Is the evaluation internal or external to the intervention?  What are the boundaries of the intervention?  What are the focal points of the evaluation?  How will the evaluation attend to the larger system (beyond the intervention)?

20. 20 Example of Evaluation Design

21. 21 QIC-EC Cross-Site Analysis Framework

22. 22 A Tool for Designing Evaluations of Paradigm Shifts in Complex System Interventions Beverly Parsons InSites www.insites.org www.insites.org bparsons@insites.org Pat Jessup InSites www.insites.org www.insites.org pjessup@insites.org Marah Moore I2i Institute www.insites.org www.insites.org marah@i2i-institute.com Handouts and PowerPoint presentation available at: www.insites.org/AEA2010