1. Accountability in Health Promotion: Sharing Lessons Learned Management and Program Services Directorate Population and Public Health Branch Health Canada Presentation to the Canadian Evaluation Society June 2, 2003 Vancouver, British Columbia

2. Health Canada, Population and Public Health Branch Presentation Objectives Share lessons on the process and planning elements of an evaluation synthesis for health promotion programs: Evaluation environment and review ‘Umbrella’ approach Methodology/framework Transferable knowledge developed on tools and approaches 2

3. Health Canada, Population and Public Health Branch Federal Context: Measuring for Results Increased role for accountability through ‘Results based’ management Shared accountability/horizontal approach ‘Value for money’, relevance, management practices Increased role for prevention/promotion in health policy Evaluators faced with dual task of developing performance measurement strategies while substantiating health promotion outcomes 3

4. Health Canada, Population and Public Health Branch About the Initiative Obligation to renew authorities under which Grant and Contribution health promotion programs are administered Improving program management practices and knowledge base Evaluation Synthesis for six programs ‘Umbrella’ Results-based Management and Accountability Framework for 15+ programs Horizontal initiative engaging many stakeholders 5

5. Health Canada, Population and Public Health Branch What is the Synthesis? Summary of evaluations – their methodology and results Integrates findings based on topic/functions, versus looking at practices and outcomes through a single program lens ‘One window’ approach to view net effects of programs (evaluation practices and program outcomes) 6

6. Health Canada, Population and Public Health Branch Synthesis vs. Single Evaluation Less opportunity for bias in results; add strength to overall results Future application for improving evaluations Systematic, comprehensive, allows for comparability of large amounts of information Combines formative and summative Enhance credibility of process and product, clearer link to policy Bias unchallenged Time-specific, focus on outputs vs. rigour and data quality Activity/project/issue oriented Transferable frameworks

7. Health Canada, Population and Public Health Branch Basic Steps: Evaluation Strategy  Build the Framework  Build the Team  Develop the Tools 7

8. Health Canada, Population and Public Health Branch Build the Framework Identify stakeholders and built synthesis review team Develop ‘umbrella’ framework and focus the evaluation Propose a research plan (define and pose questions) using common assessment criteria Establish tools for gathering and analyzing evidence Establish consultative process 8

9. Health Canada, Population and Public Health Branch Build the Team Build team and assign functions Management team oversees process Functional task group plans and implements Multi-disciplinary research team Assign primary and secondary reviewers Maintain regular communication and document experience Ongoing validation of review process and outcomes being reported

10. Health Canada, Population and Public Health Branch Project Authority Project Management Team Evaluation Risk-based Audit Renewal  Planning & Product development  Communication, messaging, capacity building  Systematic implementation Executive Management Build the Team and Assign Functions Task Groups 9 Middle Management Program/ Functional Managers

11. Health Canada, Population and Public Health Branch Develop the Tools Evaluation Synthesis framework and work plan Evaluation synthesis review protocol: Literature review of key supporting documents Assessment of review instrument through pilot Instrument validation These tools and the Evaluation Synthesis will be used for the ‘RMAF’ 10

12. Health Canada, Population and Public Health Branch Preliminary Findings: Lessons Process Planning intensive Appetite for reorienting evaluation culture Consensus driven approach building trust into process Creating momentum to build evaluation further into program management practices Reports Output (vs. health outcome) oriented due to context Method/data source quality and availability Guidance on evaluation

13. Health Canada, Population and Public Health Branch Developing common assessment tools with broader applications: Evaluation Synthesis Review ‘Protocol’ Evaluation Framework/Methodology Standard indicators to support integrated approaches Consultation workbook Synthesis research on methods & interventions Documenting experience as a transferable approach Building a ‘community of practice’ and culture of evaluation to be more collaborative Strengthening the evidence base linking to policy and supporting integrated health promotion strategies Added Value 13

14. Health Canada, Population and Public Health Branch Project Challenges Characteristics of population health promotion Level of implementation and ability to prove ‘effective’ Summarizing key achievements of programs (e.g. time, resources, variety) 11

15. Health Canada, Population and Public Health Branch Opportunities Establishing a comprehensive, collective, integrated and structured evaluation strategy Taking stock of evaluation methods and techniques in health promotion; their limitations and possibilities Developing integrated tools and mechanisms for accountability to improve program design and delivery 12

16. Health Canada, Population and Public Health Branch The Way Forward Share lessons learned on process and products: Results of synthesis report Results of ‘RMAF’ Develop generic synthesis model to validate assessments of management practices and their evaluations Maintain focus on building capacity in evaluation… 15