1 Considerations When Providing Technical Assistance on Using Evidence December 13, 2010 Shawna L. Mercer, MSc, PhD, Director, The Guide to Community Preventive Services, Centers for Disease Control and Prevention (CDC) Cancer Prevention and Control Research Network, Prevention Research Center Program, CDC

2 Acknowledgements The material for this presentation on considerations when providing technical assistance on using evidence was provided by: The Cancer Prevention and Control Research Network of the Prevention Research Center Program, CDC

3 Overview ● Assessing the fit of organizational characteristics to match potential program, practice or policy characteristics ● Readiness to implement ● Adaptation

4 Planning & Assessment What’s the problem? Setting Objectives What do we want to achieve? Selecting Interventions What works? Implementing How do we do it? Program Planning Steps Evaluating Did it work? How well? Evaluating Did it work? How well?

5 Planning & Assessment What’s the problem? Setting Objectives What do we want to achieve? Selecting Interventions What works? Implementing How do we do it? Program Planning Steps Evaluating Did it work? How well? Evaluating Did it work? How well?

6 Planning & Assessment What’s the problem? Setting Objectives What do we want to achieve? Selecting Interventions What works? Implementing How do we do it? Program Planning Steps Evaluating Did it work? How well? Evaluating Did it work? How well?

7 Before You Hit the Ground Running ● Consider the goals, objectives and target audience for your proposed program, practice or policy (P 3 ) ● Consider the characteristics of the organization or setting ● Then, select the best one to match those goals, objectives, and audience

8 Why it is important to understand evidence-based options Reading about potential options helps to: ● Consider which matches your goals and audience ● Gauge fit to the community and organization or the need for adaptation ● Review the program’s, practice’s, or policy’s methods, facilitators guide, or implementation protocol to understand the steps for delivery ● Learn the costs of implementation

9 Definition of Fit ● Overall compatibility between a program, practice or policy and:  The audience and community served and/or  The organization that will implement it ● Ideal Match

10 Criteria for Selecting an Evidence- Based Program/Practice/Policy ● Choose an option that is well-matched with:  The health topic  The audience  Setting or organizational capacity Do you have what is needed to implement it?  Delivery methods that fit your organizational objectives & structure Using computer technology Calling participants for follow up Promoting access Making policy changes

11 Matching P 3 to the Organization and Community ● Priorities and values ● Readiness for prevention ● Interface with other delivery methods and strategies in use ● New or existing partnerships ● Availability of technical assistance &/or training

12

13 Readiness for Implementation ● Define phases of implementation  Pre-implementation  Implementation  Maintenance ● Discuss important factors or tasks in each phase

14 Pre-Implementation ● Hiring staff or recruiting volunteers ● Program staff orientation  Intervention overview (e.g., components, outcomes)  Logic model (big picture of program, practice, or policy)  Materials/resources  Logistics

15 Pre-Implementation ● Training  Intervention (e.g., logic, model, core elements)  Logistics for each component  Necessary knowledge about topic  Necessary skills for P 3 (e.g., counseling/education, computer, etc.) ● Technical assistance  Developers or researchers Materials/components Updating or adapting materials/components

16 Pre-Implementation ● Enlist community/stakeholder input  Best outreach/recruitment strategies  Estimate number in target population ● Incorporate previous needs assessment data ● Conduct formative research on any adapted materials  Feedback from expert panel  Focus groups/discussion with target populations  Pilot testing

17 Pre-Implementation ● Plan for program evaluation  Determine evaluation focus and questions (e.g., process, outcome) with stakeholders  Create or modify data collection tools  Discuss dissemination plans

18 Implementation ● Conduct program, policy, or practice ● Track implementation of core elements of P 3 (e.g., which components used, steps taken) ● Collect process measures (e.g., attendance, timeliness of activities, satisfaction with activities, need some for policy, etc.) ● Monitor and evaluate activities

19 Examples of Maintenance Activities If P 3 is successful, consider: ● Referrals for further service, if needed ● Seeking additional funding ● Securing a program champion ● Making it a part of the organization’s standards

20 Learning about the Use of Evidence- based Programs, Practices, or Policies ● The field of translation, implementation science is still young  Will known evidence-based option work in every setting?? ● Document what works or does not work in terms of implementation and outcome measures.  Activity logs (e.g., core elements, satisfaction)  Staff meeting minutes (e.g., staff burden and satisfaction)  Changes in behaviors, environment or policies

21 Fidelity versus Adaptation

22 Adaptation is… … making  Changes  Additions  Deletions  Substitutions to an evidence-based P 3 in order to make it more suitable for a particular population and/or an organization’s capacity.

23 Program Fidelity ● Fidelity: faithfulness to the elements of the program, in the way it was intended to be delivered ● Components of fidelity*:  Adherence to program protocol/implementation guide  Dose or amount of program delivered  Quality of program delivery, and  Participant reaction and acceptance * Rabin, Brownson, Haire-Joshu, Kreuter, Weaver. A glossary for dissemination and implementation research in health. Journal of Public Health Management Practice, 2008, 14(2), 117–123.

24 Fidelity vs Adaptation, Some evidence… ● Review of over 500 studies showed a relationship between the level of implementation (fidelity) and program outcomes ● However, fidelity was below 100%; some adaptation always occurs and there is some evidence that it improves outcomes Durlak, J. A. & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Comm Psych, 41,

25 Core Elements & Key Process Steps ● Core elements*:  Required components that represent the theory and internal logic of the intervention and most likely produce the intervention’s effectiveness ● Key process steps:  Required steps that are conducted to contribute to the intervention’s effectiveness  Critical steps taken in program implementation in the program’s methods section or implementation protocol *Eke, Neumann, Wilkes, Jones. Preparing effective behavioral interventions to be used by prevention providers: the role of researchers during HIV Prevention Research Trials. AIDS Education & Prevention 2006, 18(4 Suppl A):44-58.

26 Green, Yellow & Red Light Adaptations Provides guidance on whether a particular adaptation is … …safe (green) …should be made cautiously (yellow) …should be avoided (red)

27 Things That Can Probably Be Modified ● Names of health care centers or systems ● Pictures of people and places and quotes ● Hard-to-read words that affect reading level ● Wording to be appropriate to audience ● Ways to recruit your audience ● Incentives for participation ● Timeline (based on adaptation guides) ● Cultural preferences based on population

28 Things That Can Probably Be Modified: Proceed with Caution ● Substituting activities ● Adding activities to address other risk factors or behaviors ● Changing the order of the curriculum or steps (sequence)

29 Things That Cannot Be Modified ● The health communication model or theory ● The health topic/behavior ● Deleting core elements or whole sections of the program ● Reduction of program  Timeline  Dosage (e.g., activities, time/ session) ● Putting in strategies that detract from the core elements

30 Key Message: If you choose an evidence-based program to adopt, do not change the core elements (what are they) or key process steps.

31

32 Example with Policy/Process Steps

33 Cancer Prevention and Control Research Network (CPCRN) Training TopicsImportant Concepts What do we mean by evidence-based? Define evidence-based (EB) Continuum of evidence Benefits of using EB strategies/programs Needs assessment & program planning Community and target audience analysis Determinants of behavior Finding an evidence-based strategy or program Sources of EB strategies & programs Selecting a strategy or program (“fit”) Adapting the evidenced-based program to meet your needs Define: adaptation, core elements, fidelity Discuss what can & cannot be changed Evaluating your program Process evaluation (e.g., implementation, fidelity, program adaptations) Outcome evaluation

34 For more information on how to select an evidence-based program or policy that meets the needs and constraints of your community : Kurt Ribisl, PhD Principal Investigator – CPRCN Coordinating Center at UNC Chapel Hill The findings and conclusions in this presentation are those of the presenters and do not necessarily represent the views of CDC.