Designing Survey Instrument to Evaluate Implementation of Complex Health Interventions: Lessons Learned Eunice Chong Adrienne Alayli-Goebbels Lori Webel-Edgar Jennifer Robertson Helen Cerigo Anne Philipneri Carly Heung Quynh Huynh Heather Manson The Canadian Public Health Association Conference, May 28 th 2014

PublicHealthOntario.ca Why do we need to study implementation? Implementation is a complex process involving multiple steps at multiple levels by multiple people Only a small fraction of evidence-based interventions are actually implemented in the real world Implementation research helps: Understand interaction with the local context Identify factors that hinder or facilitate implementation 2

PublicHealthOntario.ca Challenges in evaluating implementation There is an increasing body of evidence on factors that affect implementation, but: Different theories/frameworks, different terminology/definitions used Knowledge about relationships between factors is limited Available tools tend to focus on a limited number of factors and have weak reliability and validity 3 How do we decide on what to measure?

PublicHealthOntario.ca Objective of this presentation To share lessons learned from identifying and measuring factors contributing to implementation of the enhanced Ontario Healthy Babies Healthy Children (HBHC) Program. 4

PublicHealthOntario.ca The HBHC Program Aims to help children in Ontario have a healthy start in life and targets at children from the prenatal period to transition to school Funded by Ministry of Children and Youth Services (MCYS) Early 2013 MCYS introduced mandatory enhancements to the HBHC program to be implemented by 36 public health units Three key elements of the enhancements: Funding of 36 screening liaison nurse positions Introduction of a new validated screening tool Introduction of training to use evidence-informed interventions during home- visiting February to April 2013: staggered implementation process 5

PublicHealthOntario.ca Aims of the process implementation evaluation To what extent has the program been implemented as planned? Fidelity with protocol and program goals Reach of target population for each program component Which factors hinder/facilitate the implementation? Multiple program components Multiple HBHC staff and community partners Multiple organizations Different local settings and population characteristics 6 →Administrative database analysis → Surveys & focus groups with HBHC staff

PublicHealthOntario.ca Evaluation framework: multi-level factors contributing to implementation outcomes 7 System Organization (Public Health Unit) Provider Program participants Innovation Contributing Factors Reach Fidelity Local adaptations Impact on program change goals Implementation Outcomes Facilitation Feedback Adapted from: Chaudoir et al., Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implementation Science. 8:22

PublicHealthOntario.ca How did the framework help in developing the survey? Identified several multi-level factors that have shown to contribute to implementation outcomes Encouraged a structured stepwise method to identify relevant constructs to measure for each multi-level factor 1.Review of the implementation science literature to identify measurable constructs at each level 2.Search for existing instruments that measure the identified constructs 3.Expert consultation to identify additional instruments and verify the relevance of the identified constructs 8

PublicHealthOntario.ca Stakeholder engagement To determine which of the identified constructs are most important to measure for the HBHC program implementation Brainstorm groups with HBHC program managers and directors of 36 health units (May 2013) 9 Better understanding of constructs that are most important in the language of the field AND additional factors relevant to consider Question: Based on your knowledge and experience, what aspects of do you think are relevant/important to consider in the evaluation? Multi-voting exercise: to prioritize identified aspects

PublicHealthOntario.ca Constructs examples selected for each multi-level factor Innovation level constructs (Characteristics of the enhanced HBHC program) Adaptability & flexibility Consistency (additional) Complexity Evidence Feasibility Provider level constructs (Characteristics of health unit staff implementing the program) Job stress Job satisfaction Knowledge Professional confidence Preparedness for implementation (additional) 10 Organizational level constructs (Characteristics of the public health units in which the program is implemented) Organizational culture Communication Leadership System level constructs (The external environment and broader socio-cultural context) Community characteristics (additional) Funding and staff resources Policies and mandates (additional) Involvement of external partners (additional)

PublicHealthOntario.ca Development of specific survey items Survey items (open/closed-ended) were developed for all selected constructs in collaboration with program specialist Questions were derived from previously used/validated instruments, where possible Questions often adapted to the specific context of HBHC and the role-perspective (5 versions) For additional constructs we developed new questions using ‘field language’ Pilot-testing with 2 staff from each role 11

PublicHealthOntario.ca Example survey items 12 Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree Prefer not to answer □□□□□□ Innovation level: “The enhanced HBHC program leaves enough room for me to make my own practice decisions” (Adaptability/Flexibility, adapted from Barriers and Facilitators Assessment Instrument ) Provider level: “I feel stressed by my new responsibilities and tasks” (Job stress) Organizational level: “Colleagues in my health unit are willing to innovate and/or experiment to improve clinical practice ” (Organizational culture, adapted from ORCA) System level: “My health unit has sufficient support to facilitate the implementation of the enhanced HBHC program in terms of budget” (Funding resources, adapted from ORCA) Facilitation level: “ Change champions in my health unit support me to implement the program changes relevant to my role” (Change champions) Feedback level: “Management and/or senior leadership in my health unit welcomes feedback from staff regarding the implementation of the new HBHC protocol” (Feedback) Statements with the following response categories:

PublicHealthOntario.ca Conclusions and lessons learned The evaluation framework was useful to inform systematic survey development for evaluating the HBHC implementation Integrating factors from other theories/frameworks into the evaluation framework encouraged a more comprehensive understanding of the implementation process The multi-level nature of the framework allowed for multiple perspectives and the use of multiple data sources Stakeholder engagement was essential to ask the right questions The survey contributed to the limited instruments available to measure factors that hinder and facilitate implementation of the multiple levels 13

PublicHealthOntario.ca Limitations and challenges Our survey instrument used validated tools, but has not been validated The multi-level approach resulted in a long survey (feasibility depends on level of commitment of the sample) Due to a lack of consistency in definitions and terminology for constructs, comparisons with other studies are difficult Richness of data can be challenging in terms of making sense of the results 14

PublicHealthOntario.ca 15