1. Unpacking the notion of program theory from a complexity lense: Can indigenous and cross-cultural perspectives help? Sanjeev Sridharan and Janet Smylie Centre for Research on Inner City Health University of Toronto Benita van Wyk, Feedback Research & Analytics

2. Outline 1. Why is program theory important and useful? 2. What are the limitations of the standard approach to program theory? The view from evaluation 3. Ideas for a potential research project

3. Conceptualizing and Critiquing Programs: Program Planning and Evaluation Evaluation Literature Indigenous/ Cross-Cultural Literature Identify concepts to enhance program planning and evaluation Implement Concepts in Two Programs Evaluate Programs

4. Basis for dialogue Sustainability Connectivity Dynamics Heterogeneity

5. The important questions What is a program? Does the nature of the causal assumptions make a difference in the way we plan programs and evaluations? Dealing with complexity

6. An Example of a complex intervenion Dealing with complexity

7. An Example: Primary Prevention Have a Heart Paisley

8. The nature of the causal problem Note that the causal assumptions are not clearly specified Most assumptions are implied and of the form: If (A and B and C and D and E and at time T) then good outcomes might result The problem: – Causal contingencies not clearly specified or understood – Dynamic complexity not well understood

9. Another example: The Smoking Ban in Scotland

11. The incompleteness of program theories  The logic model while complex is still incomplete  A policy such as the smoking ban might lead to proximal impacts through perhaps an uncomplicated pathway; however the logic of the causal chain for intermediate and long-term outcomes might be incomplete  Additional “programmatic” inputs might be needed.  Information on the nature of such additional interventions can perhaps come from a synthesis of literature plus additional dialogue with stakeholders with an intimate knowledge of the processes by which programmes works.

12. Defining a complex intervention (MRC, 2000) “The greater the difficulty in defining precisely what exactly are the ‘active ingredients’ of an intervention and how they relate to each other, the greater the likelihood that you are dealing with a complex intervention.”

13. System Dynamic Approaches (Sterman, 2006)  Constantly changing;  Governed by feedback;  Non-linear, History-dependent;  Adaptive and evolving;  Characterized by trade-offs;  Policy resistance: “The result is policy resistance, the tendency for interventions to be defeated by the system’s response to the intervention itself.”

14. Features of complex health interventions (Pawson et al., 2004) The intervention is a theory or theories The intervention involves the actions of people. The intervention consists of a chain of steps These chains of steps or processes are often not linear, and involve negotiation and feedback at each stage. Interventions are embedded in social systems and how they work is shaped by this context. Interventions are prone to modification as they are implemented. Interventions are open systems and change through learning as stakeholders come to understand them.

15. An additional purpose of evaluation for complex interventions Explication  Why?  Multiple components– be explicit about the “active ingredients”  Timeline of impact unclear  Non-uniformity– Contextualizing  Stakeholder involvement: Co-construction of intervention

16. On causality and complexity  Does a cause operate the same way in simple systems as compared to complex systems?  Are there different kinds of causes?  E.g. Famines  Precipitating causes  Amplifying causes  Causes of vulnerability  Innovations in evaluation design and approaches

17. So what do programmes do?  A need for a richer vocabulary on programmes as causes– can we learn from other sciences such as evolution or chemical kinetics?  Different metaphors: Programmes as catalysts, programmes as foundations, programmes as operating systems  Understanding complexity in multiple light – The system-as-cause – Understanding dynamic complexity

18. System-as-Cause Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68. Meadows DH. Leverage points: places to intervene in a system. Sustainability Institute, 1999. Available at. Richardson GP. Feedback thought in social science and systems theory. Philadelphia, PA: University of Pennsylvania Press, 1991. Sterman JD. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.

19. “Solutions” Can Also Create New Problems Meadows DH, Richardson J, Bruckmann G. Groping in the dark: the first decade of global modelling. New York, NY: Wiley, 1982. Merton RK. The unanticipated consequences of purposive social action. American Sociological Review 1936;1936:894-904. Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68. Policy resistance is the tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention itself. -- Meadows, Richardson, Bruckman

20. Unpacking the causal contingencies of change If (A AND B AND C AND D AND E and at time T) then good outcomes might result What does this really mean? Unpacking the complexity of the change process What is missing in our understanding of how programs work?

21. Seeing and understanding connection “ You think you understand two because you understand one and one. You think you understand two because you understand one and one. But you must also understand ‘and’.” You think you understand two because you understand one and one. But you must also understand ‘and’.” -- Sufi Saying

22. 1. The role of sustainability Why sustaining programs might be key to success Going further – sustainability of programs at the individual level – the critical importance of Continuity of Care

23. 2. The limited understanding of heterogeneity Heterogeneity of needs Heterogeneity of mechanisms of change Heterogeneity of interventions

25. 3. The role of connectivity between interventions Far too much focus of most evaluations on a single intervention as the unit of analysis Understanding connectivity between programs

26. Better understanding of ecology of interventions

27. 4. Understanding the relationship between multiple levels of interventions Many complex interventions require programming at the community, neighborhood, school and individual level Multilevel alignment across the interventions

28. 4. Multilevel alignment complexity

29. 5. Understanding Dynamic Complexity Timeline of impact Developmental Synergies

30. Anticipated timeline of impact

31. Dynamic complexity (1)

32. Dynamic complexity (2)

33. 5. Coordination complexity The advantages of partnerships are poorly understood More importantly, collaborative advantages of individual recipients rarely made

34. Coordination complexity

35. Cracks in a network complexity

36. 6. Sequential Complexity Relationship between sequences of interventions poorly understood Most of our methodologies are extremely limited in understanding such relationships

37. 6. Sequential intervention complexity

38. 7. Structural complexity Policy resistance Translating programs to fit within the structures The system-as-cause

39. Next Steps Identify partners Identify concepts from Evaluation and Indigenous/Cross-Cultural Field/Literature that can help enhance the concepts Translate concepts into practical testable experiments

40. Other concepts and ideas? sridharans@smh.toronto.on.ca