1. Implementation and Sustainability in the US National EBP Project Gary R. Bond Dartmouth Psychiatric Research Center Lebanon, NH, USA May 27, 2014 CORE Symposium London

2. Presentation Outline Describe the three initial implementation phases of the National EBP Project Summarize methods and findings: 1. Fidelity 2. Implementation Factors 3. Sustainability

3. (1998) Phase 1: Identifying Evidence-Based Practices (EBPs) for People with Severe Mental Illness (1998)

4. 1998 RWJ Conference on Evidence-Based Practices Assertive community treatment (ACT) Supported employment (SE) Integrated dual disorders treatment (IDDT) Illness management and recovery (IMR) Family psychoeducation (FPE) Medication management approaches in psychiatry (MedMAP)

5. Phase 2: Developing EBP Toolkits and EBP Implementation Model (1999-2002)

6. National EBP Project: Rationale Agency leaders, practitioners, and state leaders need user-friendly resource materials Agency leaders, practitioners, and state leaders need user-friendly resource materials BUT information is not enough… BUT information is not enough… Training and consultation also needed Training and consultation also needed

7. EBP Toolkits Contents Scientific review articles and PowerPoint lectures Practitioner manuals Educational videos (DVDs) Handouts for policymakers, consumers, & families Fidelity scales Outcome monitoring software

9. Phase 3: Evaluate Effectiveness of Implementation Model

10. Design of National EBP Project Examined 5 psychosocial EBPs Examined 5 psychosocial EBPs 8 participating states 8 participating states Each state implemented 2 different EBPs in multiple sites Each state implemented 2 different EBPs in multiple sites 53 sites started (49 final sites) 53 sites started (49 final sites) 2 years of qualitative observation of implementation factors 2 years of qualitative observation of implementation factors Fidelity reviews every 6 months Fidelity reviews every 6 months

11. Primary Measure of Implementation Outcome Fidelity: Degree to which a particular program follows the standards for an EBP

12. Correlational Design

13. 1. Fidelity: Methods and Findings

14. Structure of EBP Fidelity Scales Each scale consists of 12-28 items Items rated on 5-point continuum 1 = Not Implemented 5 = Fully Implemented ≥ 4.0 considered good implementation

15. Content of Fidelity Items “Structural” items “Clinical” items

16. “Structural” Fidelity Items Things that can be done by administrative fiat, such as: – –Daily team meetings – –Multidisciplinary staffing – –Low caseload ratio – –Following a curriculum – –Distributing educational handouts No clinical skill needed to implement

17. Clinical Fidelity Items Assessed the provisions of interventions requiring clinical skill Examples: – –Stagewise interventions for dual disorders – –Motivational techniques Clinical items involved qualitative judgments

18. Data Collection for EBP Fidelity Scales Ratings made by two independent assessors Day-long site visit Multiple data sources (interviews, chart review, observation) Fidelity report given afterwards to agencies for quality improvement

19. Finding #1: Usual practice definitely not evidence-based

21. Finding #2: Over Half the Sites Successfully Implemented the EBP

22. National EBP Project: 2-Year Rates of Successful Program Implementation

23. Finding #3: Some EBPs were easier to implement than others

25. Finding #4: Most improvements in EBP fidelity occurred within the first year

27. 2. Implementation Factors: Methods and Findings

28. Implementation Factors: Data Collection Procedures Implementation monitors recorded notes at study sites ~ monthly for 2 years Periodic interviews with key staff Notes and interviews entered as “documents” in qualitative data base

29. Coding System Documents contained codable “events” Each event coded according to – –Type: Barrier, Facilitator, or Strategy – –Content: 26 Dimensions grouped into 5 Domains

30. Conceptual Framework for Implementation Factors

31. Total Number of Units Coded

32. Overall, Which Barrier, Facilitator, and Strategy Domains Were Most Common?

33. Most Frequently Noted Domains of Barriers, Facilitators, and Strategies (Total Sample) Barriers: – –Leadership, Prioritization, Work Force Facilitators: – –Prioritization, Leadership, Work Force Strategies: – –Work Force, Leadership

34. How Do Barriers, Facilitators, and Strategies Affect Fidelity? Correlational Findings

35. Correlations: Implementation Factors over Both Years with 24-Month EBP Fidelity

36. Summary: Barriers Content domains relatively similar across EBPs Did not appear to affect fidelity!

37. Summary: Facilitators EBPs differed substantially Leadership generally had a huge positive effect on fidelity Work Force factors unrelated or negatively correlated with fidelity Feedback structures in place had a positive impact

38. Summary: Strategies EBPs differed on Leadership and Work Flow Work Force (mainly Training!) was unrelated or had a negative effect on fidelity Work Flow and Reinforcement had a positive effect on fidelity

39. Negative Findings for Training: Speculations Didactic training is a poor method for teaching clinical skills Clinical skills needed to implement an EBP are not identical to components of program fidelity Staff turnover also helps explain negative correlation between training and fidelity outcomes

40. Implementation Factors: Conclusions 1. 1. Active, observable Leadership has dramatic impact on implementation 2. 2. A focus on Work Flow (policies, documentation) and Reinforcement (fidelity and outcome monitoring) important strategy 3. 3. Work Force facilitators and strategies had a puzzling negative relationship with fidelity

41. 3. Sustainability: Methods and Findings

42. Methods Interviews with site leaders 2 years after end of 2-year implementation phase (2007) Second round of interviews 4 years later (2011) Close-ended and open-ended questions

45. Barriers to Sustainability

46. Strategies Used to Sustain EBP

47. Factors Influencing Sustainability 1. 1. Stable and adequate funding 2. 2. Fidelity and outcome monitoring 3. 3. Weekly supervision 4. 4. State leadership supportive of EBP

48. Overall Conclusions Systematic efforts at implementation have a powerful effect on successful implementation Some elements in implementation model more effective than others Several factors in common for successful implementation and sustainability