1. IUHPE Global Programme: The Public Health Agency of Canada’s Effectiveness Project Marcia Hills, RN, PhD Simon Carroll (PhD Student) Canadian Consortium for Health Promotion Research Sylvie Desjardins, Director, Strategic Policy, PHAC. Rio de Janeiro, Brazil May 12, 2005

2. Overview 1) Typical strategies for assessing effectiveness…systematic reviews and an alternative approach 2) Policy-maker’s need for evidence of effectiveness and frameworks/tools 3) The Public Health Agency of Canada’s Effectiveness Project

3. Why should health promoters care about ‘evidence’ and ‘effectiveness’ Increasing demand on policy/decision makers to provide transparent and explicit justification of decisions Ignoring the resource allocation issue is neither possible nor desirable Valuable lessons are discounted Valuable evidence is ignored What counts as evidence “Gambit of compliance” Leads to poor evaluations, poor data gathering, and a waste of valuable resources (i.e. 10% of funds!)

4. Evaluation… Programmatic Focuses on goals, objectives and outcomes of a particular program Highly contextualized Evaluation – programs, people & practice Concerns practice with people in communities –What is the relationship between the evaluator and those delivering the program? –What should be accepted as evidence upon which to base practice?

5. Effectiveness… Looks for an evidence across programs Driven by desire for evidence-based policy…practice…decision-making Accountability for public funds Decides what constitutes ‘evidence’ and influences how evaluations are conducted

6. Typical Strategies for Assessing Effectiveness…Systematic Reviews Numerical Meta- analysis –Randomized Control Trials –Quantitative –Statistical notion of causality (probability) Narrative Review –Multiple Methodologies (interpretative) –Qualitative –Ignores or rejects causal explanation Different philosophies, assumptions Paradigm “wars”

7. Difficulties with both approaches… Based on “best buy” Accumulation of evidence in similar programs Beliefs about causality that are based on statistics or reactions to it Either context–free or over contextualized lessons

8. Realist synthesis approach (Pawson, 2000) Based on philosophy of science that argues that it is possible to have a logic of comparison that is not statistical Causal powers of an intervention (program) lie in its underlying mechanism/s - its basic theory about how program resources will affect participant’s action in certain contexts NOT …does the program work ? …BUT what are the conditions (context) under which the resources the program offers have an impact on participant’s actions

9. Mechanisms, Contexts, Outcomes Want to discover those contexts that have produced solid and successful outcomes FROM…those contexts that induce failure All cases are important …successful policy depends as much on avoiding previous errors as by “imitating” successes

10. Causality… The language of causality is crucial because it refers to the elements of a programme that make them work…or not work. Improving the priority setting of funders and improving the quality of practice in health promotion demands attention to what causes positive and negative outcomes We need to move beyond the either/or of relying on probabilistic reasoning or retreating into the defensive position of denying the capability of causal assertions We need to conceptualize our phenomena into discernable structures, objects and relations, rather than a obsessive focus on producing data for statistical modeling.

11. Public Health Agency of Canada: Effectiveness Project Goal.. To develop a framework for assessing the effectiveness of federally-funded community initiatives to improve health

12. Individual, Community & Systems Change Framework for Assessing the Effectiveness of Community Initiatives That Promote Health Improvements in Population Health Outcomes While Reducing Health Inequity Impact on Determinants of Health Individual Change Community Change Systems Change CommunityOrganization& Action CollaborativePlanning TransformationalChange Social Economic Political Environment Gender Cultural Contexts

13. Component 1 Mechanisms: Collaborative Planning 1) Meaningful participation of all relevant stakeholders 2) Critical dialogue 3) Shared power 4) Project action planning and evaluation

14. Component 2: Community Organization and Action 1) Ongoing education and training opportunities 2) Evolving leadership 3) Sustained mobilization of resources 4) Critical reflection and systematic monitoring

15. Component 4: Transformational Change 1) Develop and attract champions 2) Generate publicity of project successes 3) Influence Public Policy and Decision-making Bodies 4) Work with relevant social movements and provincial and/or national advocacy groups

16. A Realist Synthesis Approach to Assessing Effectiveness… Allows us to learn from past successes and failures Focuses our attention on the issues so important to community practitioners Uncovers what is often undervalued and discounted Builds credibility for community initiatives by using a theory of causation based on a realist philosophy of science

17. A Snapshot of the NARO Effectiveness Website URL

18. Until one is committed, there is hesitancy, the chance to draw back, always ineffectiveness, concerning all acts of initiative (and creation). There is one elementary truth, the ignorance of which kills countless ideas and splendid plans: that the moment that one definitely commits oneself, then Providence moves too. All sorts of things occur in one’s favour that would never otherwise have occurred. A whole stream of events issues from the decision, raising in one’s favour all manner of unforeseen incidents and meetings and material assistance which no man could have dreamed would come his way. Whatever you can do or dream you can, begin it. Boldness has genius, power and magic in it. Begin it now! Johann Wolfgang Von Goethe