Presentation is loading. Please wait.

Presentation is loading. Please wait.

Phyllis C. Panzano, Ph.D., PI Dee Roth, M.A., Co-PI Bev Seffrin, Ph.D, Senior Consultant Dushka Crane-Ross, Ph.D., Project Manager Decision Support Services,

Similar presentations


Presentation on theme: "Phyllis C. Panzano, Ph.D., PI Dee Roth, M.A., Co-PI Bev Seffrin, Ph.D, Senior Consultant Dushka Crane-Ross, Ph.D., Project Manager Decision Support Services,"— Presentation transcript:

1 Phyllis C. Panzano, Ph.D., PI Dee Roth, M.A., Co-PI Bev Seffrin, Ph.D, Senior Consultant Dushka Crane-Ross, Ph.D., Project Manager Decision Support Services, Inc. Ohio Dept of Mental Health, OPER The Innovation Diffusion and Adoption Research Project (IDARP) ODMH RESEARCH RESULTS BRIEFING 2003 Funded by the ODMH & the Mac Arthur Foundation

2 Ohios Quality Agenda Best Practices Outcomes QI

3 Evidence Base Salience

4 Coordinating Centers of Excellence (CCOEs) Evidence Base Salience Advance Directives MH/Schools MH/Criminal Justice OMAP Family Psychoeducation Cluster-Based Planning MST SAMI-IDDT

5 oUniversity or local partnership oOne Best Practice per CCOE oStatewide service area Structure of CCOEs

6 oPromotion of Best Practices oEducation & training oCapacity development oFidelity measurement oCross-system sharing Role of CCOEs

7 What factors and processes influence the adoption, assimilation, and impact of evidence-based practices by mental health provider organizations? Research Question

8 oCharacteristics of the Best Practice oAdoption Decision & Implementation Process oAdopting Organization oAdopting Organization – CCOE Relationship Independent Variables

9 Research Team Decision Support Services, Inc. Phyllis Panzano, Ph.D Beverly Seffrin, Ph.D. Sheri Chaney, M.A. Vandana Vadyanathan, M.A. Sheau-yuen Yeo, M.A. Ohio Dept of Mental Health Dee Roth, M.A. Dushka Crane-Ross, Ph.D. Rick Massatti, M.A. Carol Carstens, Ph.D. Ohio State University: Fisher College of Business Department of Psychology

10 Theoretical Background o Numerous literatures are relevant o Resulting Assumptions: o EBPs are innovations o Scientific evidence necessary but not sufficient o Upper Echelon Theory relevant o Implementation effectiveness Innovation effectiveness o Factors at many levels impact outcomes o 3 phases: initiation; decision; implementation

11 idarp 100-PIECE JIGSAW PUZZLE COMPLEXITYCOMPLEXITY

12 IDARP Models

13 Model 1: Adoption Decision – Decision making under risk

14 Phase 1: Decision Under Risk IMPLEMENT ADOPTER WAIT & SEE NEVER Perceived Risk of Adopting Capacity to Manage or Absorb Risk Risk-taking Propensity ANTECEDENTSANTECEDENTS More Likely Less Likely LIKELIHOOD OF IMPLEMENTING

15 Model 2: Multi-level Influences on Implementation Success

16 Interested in Two Classes of Outcomes o Measures of Innovation effectiveness: Benefits that accrue to an organization and its stakeholders as a result of implementing an innovative practice (positive consequences for clients, staff, etc.) o Measures of Implementation effectiveness: Accurate, committed and consistent use of practice Accurate, committed and consistent use of practice by targeted employees (assimilation, fidelity, etc.) by targeted employees (assimilation, fidelity, etc.)

17 Expected Link Between Two Classes of Outcomes IMPLEMENTATION EFFECTIVENESS INNOVATION EFFECTIVENESS

18 For example: IMPLEMENTATION EFFECTIVENESS INNOVATION EFFECTIVENESS FIDELITY POSITIVE OUTCOMES

19 Variables at multiple levels are expected to impact these two classes of outcomes

20 Examples of Variables by Level LevelExample ENVIRONMENT System and professional norms System and professional norms IOR (Org with CCOE) Quality of communication Quality of communication ORGANIZATION Learning culture Learning culture PROJECT Re: Organization Re: Decision Re: Implementation Availability of dedicated resources Availability of dedicated resources Commitment to decision to adopt Commitment to decision to adopt Access to technical assistance Access to technical assistance INNOVATION Scientific support Scientific support Experiential evidence Experiential evidence

21 Level 4: Inter-organizational Level 3: Adopting organization Level 2: Project level Level 1: Innovation level Dependent Variables: Implementation effectiveness Innovation effectiveness Level 5: Environment Model 2

22 Model 3: Cross-Phase Effects on Implementation Outcomes

23 INITIATION Decision IMPLEMENTATION Outcomes Time Model 3: Cross-phase effects

24 Experiential Evidence Objective Process Access to Technical Assistance Positive Consequences Time Model 3: Examples of Cross-phase Effects InitiationDecisionImplementation

25 Model 4: Effects of Implementation Variables on Outcomes Over Time

26 PRESENT OUTCOMES Model 4: Effects of Implementation Variables Over Time PAST Implementation PRESENT Implementation TIME

27 PRESENT OUTCOMES Model 4: Examples of Effects of Implementation Variables Over Time PAST Access to Technical Assistance PRESENT Dedicated Resources TIME

28 Methods & Progress to Date

29 Four CCOEs Participating Selection criteria maximize generalizability Selection criteria maximize generalizability 1)Cluster-Based Planning Alliance 2)Multi-systemic Therapy (CIP) 3)Ohio Medication Algorithm Project 4)Integrated Dual Disorder Treatment (IDDT) – New Hampshire - Dartmouth model

30 Research Design o Longitudinal study o Organizations at different stages of adoption o Multiple key informants at each organization o Quantitative and qualitative data o Interviews, surveys & archival data

31 Participating Projects* by Type of Innovation MST OMAP IDDT/ SAMI Cluster Alliance *18 organizations involved in multiple projects; Total of 74 organizations with 91 projects under study.

32 Participating Projects by Stage of Adoption at Time One Implementer De-adopter Adopter Never Wait & see N = 91

33 Participating Projects by Stage of Adoption at Time Two Implementer De-adopter Adopter Never Wait & See N = 50

34 Key Informants by Level at Time One CFO/QA Implementer Decision maker CCOE Community Collaborative N = 369

35 Key Informants by Level at Time Two Implementer Decision maker CCOE Community Collaborative N = 135

36 Findings 6 22 agree Strongly disagree Very satisfied $ 37,500

37 Do the data support our four models?

38 THE TIP….OF THE TIP

39 ___________________________________ POSITIVE CORRELATION As the value of one variable increases, the value of a second variable also increases Median Income Years of Formal Education Lower Higher LessMore + correlation(r = +1.00)

40 ___________________________________ NEGATIVE CORRELATION As the value of one variable increases, the value of a second variable decreases Unemployment Rate Years of Formal Education Lower Higher LessMore - correlation(r = -1.00)

41 ___________________________________ ZERO 0 CORRELATION The relationship between the value of one variable and the value of a second variable is random Years of Formal Education Shorter Taller LessMore Zero Correlation Height x x x x x x x x x x x x x x x x x (r = 0.00)

42 ___________________________________ CORRELATION CANNOT BE DETERMINED because the value of one (or both) variable(s) is constant or almost constant Lower Higher Years of Formal Education = BA, BS Unemployment Rate

43 The Adoption Decision (Model 1)

44 Time 1/First contact data

45 Phase 1: A Decision Under Risk Likelihood of implementing as indicated by Stage Implementer Adopter Wait & See Never Perceived Risk of Adopting Capacity to Manage or Absorb Risk Risk-taking Propensity

46 Antecedents to Risk Perceptions Likelihood of implementing as indicated by Stage Implementer Adopter Wait & See Never Perceived Risk of Adopting Capacity to Manage Risk Risk-taking Propensity ANTECEDENTSANTECEDENTS

47 Antecedents to Perceived Risk Perceived Risk Innovation Level Factors Relative Advantage Scientific Evidence Experiential Evidence Org-Level Factors Knowledge Set Environmental Factors Norms for Adoption -.45

48 Antecedents to Risk Management Capacity to Manage Risk EBP–Level Factors Ease of Use Org–Level Factors Top Mgmt. Support Environmental Factors Environmental uncertainty Craft Skills +.25 Dedicated Resources +.63

49 Antecedents to Risk Propensity Organization-Level Factors Learning Encouragement Managerial Attitude About Change Risk Propensity

50 Summing Up: Model 1 3. Antecedents have implications for action 3. Antecedents have implications for action 1. Adoption decision is a decision involving risk 2. Organizations are more likely to adopt if: Perceived risk of adopting is low Perceived risk of adopting is low Capacity to manage risk is high Capacity to manage risk is high Propensity to take risks is high Propensity to take risks is high

51 Model 2: Implementation Phase Understanding Outcomes of Implementation for Adopters and Implementers

52 Time 2/Second Contact Data

53 Two classes of outcomes 1. Implementation Effectiveness (e.g., fidelity, assimilation) 2. Innovation/practice Effectiveness (e.g., positive outcomes)

54 Is implementation effectiveness related to innovation effectiveness? IMPLEMENTATION EFFECTIVENESS INNOVATION EFFECTIVENESS Reinvention 1 Positive outcomes ? -.64 AssimilationPositive outcomes.61 1 Self report; reflects extent to which practice was modified

55 Level 4: Inter-organizational Level 3: Adopting organization Level 2: Project level Level 1: Innovation level Dependent Variables: Implementation effectiveness Innovation effectiveness Level 5: Environmental Model 2

56 Assimilation: One measure of implementation effectiveness

57 Is Assimilation Explained by Variables at Multiple Levels?

58 DYAD: Communication quality +.45 ORG: Learning culture +.30 Centralization +.43 PROJECT: Dedicated resources +.52 Ease of use +.40 INNOV: Fit w/Tx philosophy+.45 Dependent Variable: Assimilation * * Extent practice seen as part of permanent operations Some Examples

59 Variables at multiple levels are related to reported assimilation.

60 Are views about positive outcomes explained by variables at multiple levels?

61 Positive outcomes Overall positive consequences Overall positive consequences Positive outcomes for consumers Positive outcomes for consumers Positive impact on organizations image Positive impact on organizations image Positive impact on organization functioning Positive impact on organization functioning Overall positive impact Overall positive impact Extent expectations realized Extent expectations realized

62 DYAD: Identification +.40 to +.60 ORG: Risk mgmt to +.40 PROJECT: Perf. monitoring +.52 to +.74 Access to TA +.48 to +.66 Reinvention -.24 to -.49 INNOV: Scientific evidence +.30 to +.60 Dependent Variable: Positive Outcomes Some Examples

63 Variables at multiple levels are related to perceived positive outcomes.

64 Model 3: Cross-Phase Effects on Implementation Outcomes Understanding Effects of Initiation-Phase and Decision-Phase Variables on Implementation Outcomes

65 INITIATION Decision IMPLEMENTATION Outcomes TimeTime Cross-Phase Effects on Implementation Outcomes Model 3: Cross-Phase Effects on Implementation Outcomes Time 1 Time 2

66 INITIATION (Time 1) Decision IMPLEMENTATION Outcomes (Time 2)Outcomes Time Initiation-Phase Effects Model 3: Initiation-Phase Effects

67 Expected Benefits+.44 Relative advantage +.63 Trust CCOE+.46 Results demonstrability+.49 Assimilation Initiation-Phase Effects

68 Expected Benefits+.58 to +.69 Relative advantage +.59 to +.74 Trust CCOE+.38 to +.57 Results demonstrability+.26 to +.51 Positive Outcomes Initiation-Phase Effects

69 INITIATION Decision (Time 1) IMPLEMENTATION Time Decision-Phase Effects Model 3: Decision-Phase Effects Outcomes (Time 2) Outcomes (Time 2)

70 Objective decision +.37 Information access +.34 Internal influence +.28 Organizational commitment +.37 Assimilation Decision-Phase Effects

71 Objective decision +.46 to +.71 Information access +.42 to +.61 Internal influence +.31 to +.46 Organizational commitment +.42 to +.52 Positive Outcomes Decision-Phase Effects

72 Variables in earlier phases can have enduring effects on implementation outcomes.

73 Model 4: Understanding Effects of Implementation- Phase Variables Over Time

74 PRESENT OUTCOMES Model 4: Effects of Implementation Variables Over Time PAST Implementation PAST Implementation PRESENT Implementation PRESENT Implementation TIME

75 Model 4: Effects of Implementation Variables Over Time PAST Top Support PAST Top Support PRESENT Top Support PRESENT Top Support TIME PRESENT OUTCOMES

76 Model 4: Effects of Implementation Variables Over Time PAST Freedom to Express Doubt PAST Freedom to Express Doubt PRESENT Freedom to Express Doubt PRESENT Freedom to Express Doubt TIME PRESENT OUTCOMES

77 Model 4: Effects of Implementation Variables Over Time PAST Access to Technical Assistance PAST Access to Technical Assistance TIME PRESENT OUTCOMES PRESENT Access to Technical Assistance PRESENT Access to Technical Assistance

78 Model 4: Effects of Implementation Variables Over Time PAST Dedicated Resources PAST Dedicated Resources PRESENT Dedicated Resources PRESENT Dedicated Resources TIME PRESENT OUTCOMES

79 Implementation strategies need to be sustained in order to have positive impacts on long-term outcomes.

80 Shifting Gears: Comparing Different EBPs at Time One

81 Do adopting organizations hold similar views about the four practices? Clustering (n = 23) Clustering (n = 23) MST (n = 16) MST (n = 16) OMAP (n = 15) OMAP (n = 15) IDDT/SAMI (n = 16) IDDT/SAMI (n = 16) IDDT/SAMI with Initial Funding (n = 12 * ) IDDT/SAMI with Initial Funding (n = 12 * ) * 9 funded demonstrations; 12 projects

82 Organization: Organizational Commitment MST, IDDT & IDDT-FUNDED > OMAP Strongly Agree Strongly Disagree Clustering IDDT IDDT- Funded MST OMAP

83 Innovation: Experiential Evidence MST, IDDT-FUNDED & IDDT > CBP > OMAP Clustering IDDT IDDT- Funded MST OMAP Strongly Agree Strongly Disagree

84 Innovation: Scientific Evidence MST, IDDT-FUNDED & IDDT > CBP & OMAP Strongly Agree Strongly Disagree Clustering IDDT IDDT- Funded MST OMAP

85 Innovation: Magnitude of Change Required to Implement MST, IDDT-FUNDED & IDDT > CBP & OMAP Clustering IDDT IDDT- Funded MST OMAP

86 Innovation: Fidelity Seen as Crucial to Implementing the Practice MST > OMAP, CBP, IDDT & IDDT-FUNDED Strongly Agree Strongly Disagree Clustering IDDT IDDT- Funded MST OMAP

87 Implementation: Resources* for Initial Implementation No differences Strongly Agree Strongly Disagree * money, personnel & time Clustering IDDT IDDT- Funded MST OMAP

88 Implementation: Resources* for Ongoing Implementation OMAP > CBP, MST & IDDT > IDDT-FUNDED Strongly Agree Strongly Disagree * money, personnel & time Clustering IDDT IDDT- Funded MST OMAP

89 Implementation: Problems Recruiting Staff MST > CBP, OMAP, IDDT & IDDT-FUNDED Great Extent No Extent Clustering IDDT IDDT- Funded MST OMAP

90 Implementation: Reinvention No differences Very Great Extent No Extent Clustering IDDT IDDT- Funded MST OMAP

91 Outcome: Assimilation CBP, OMAP & IDDT > IDDT-FUNDED Very Great Extent No Extent Clustering IDDT IDDT- Funded MST OMAP

92 A peek at interview data

93 Approach Interview team produces transcripts from interviews with multiple agency sources Interview team produces transcripts from interviews with multiple agency sources Qualitative codes attached to text in Atlas Ti Software program such as: Qualitative codes attached to text in Atlas Ti Software program such as: Issue diagnosis & decision process Issue diagnosis & decision process Planning process for EBP implementation Planning process for EBP implementation Facilitators & Barriers Facilitators & Barriers Expected/unexpected, +/- outcomes Expected/unexpected, +/- outcomes Unit of Analysis = mentions or coded phrases Unit of Analysis = mentions or coded phrases

94 Focus of todays look at qualitative data Data collected during time one/first contact with 36 projects (~ 3 interviews per) Data collected during time one/first contact with 36 projects (~ 3 interviews per) Projects X Stage Projects X Stage 18 Implementers 18 Implementers 7 Adopters 7 Adopters 7 Wait & See/Never 7 Wait & See/Never 4 De-adopters 4 De-adopters Projects X EBP Projects X EBP 13 IDDT 13 IDDT 9 Cluster-Based Planning 9 Cluster-Based Planning 7 OMAP 7 OMAP 7 MST 7 MST

95 Glossary of Categories CCOE - relating to the CCOE, its staff and services it provides. CCOE - relating to the CCOE, its staff and services it provides. EBP – perceptions relating to the innovation. EBP – perceptions relating to the innovation. Money - expenses (actual or anticipated), funding of the EBP and financial issues that impact the agency. Money - expenses (actual or anticipated), funding of the EBP and financial issues that impact the agency. Staff - reactions, recruitment, retention and qualifications of staff. Staff - reactions, recruitment, retention and qualifications of staff. System - coordination, collaboration, and interest in Mental Health and other related systems. System - coordination, collaboration, and interest in Mental Health and other related systems.

96 CCOE: Major Themes Facilitators (n = 119) Facilitators (n = 119) Attended CCOE presentation/became aware of CCOE/had previous experience with (n = 55) Attended CCOE presentation/became aware of CCOE/had previous experience with (n = 55) CCOE provides instrumental help (n = 50) CCOE provides instrumental help (n = 50) Positive reaction to CCOE (n = 11) Positive reaction to CCOE (n = 11) Barriers (n = 36) CCOE doesnt understand the agencys issues or constraints (n = 9)

97 CCOE: Mentions by Stage

98 EBP: Major Themes Facilitators (n = 225) Facilitators (n = 225) Received training or information (n = 53) Received training or information (n = 53) EBP is good match with culture, systems, or similar to what we already do (n = 24) EBP is good match with culture, systems, or similar to what we already do (n = 24) EBP might be useful (n = 19) EBP might be useful (n = 19) Barriers (n = 153) Dont know how to proceed – in the dark (n = 24) EBP isnt a good fit to this organization (n = 13) EBP might NOT be useful (n = 13)

99 EBP: Mentions by Stage

100 Money: Major Themes Facilitators (n = 69) Facilitators (n = 69) Received funds (n = 31) Received funds (n = 31) Identified potential funds (n = 24) Identified potential funds (n = 24) There is a potential savings from the EBP (not necessarily for the agency) (n = 6) There is a potential savings from the EBP (not necessarily for the agency) (n = 6) Barriers (n = 115) Agency has financial issues/EBP costs money (n = 80) Funding for the EBP is not sustainable (n = 29) Our funding shrunk/was lost/ended (n = 11)

101 Money: Mentions by Stage

102 Staff: Major Themes Facilitators (n = 122) Facilitators (n = 122) Staff is interested/ supportive/likes the EBP (n = 25) Staff is interested/ supportive/likes the EBP (n = 25) Staff hired for program (n = 18) Staff hired for program (n = 18) Staff thinks the EBP makes sense (n = 8) Staff thinks the EBP makes sense (n = 8) Barriers (n = 171) Recruitment & turnover issues (n = 67) Resistance to EBP, skepticism, lack of interest (n = 44) Competing priorities (e.g. innovation vs. productivity) (n = 15)

103 Staff: Mentions by Stage

104 System: Major Themes Facilitators (n = 106) Facilitators (n = 106) Support and interest in the system (n = 34) Support and interest in the system (n = 34) Collaboration/ cooperation/ communication & integration in the system (n = 33) Collaboration/ cooperation/ communication & integration in the system (n = 33) Barriers (n = 82) Lack of support/no interest (n = 29) Conflict, lack of collaboration between important entities, no communication (n = 29) Competing priorities & turmoil in system (n = 9)

105 System: Mentions by Stage

106 Summing Up: Facilitator/Barrier Analysis - Category Overall, facilitators were mentioned more often than barriers (641:557). EBP: The category with the most mentions of facilitators (225); Staff: The category with the most mentions of barriers (171). CCOE EBP $ Staff System CCOE EBP $ Staff System

107 Facilitator & Barrier Analysis - Phase Facilitators and barriers can usually be identified as occurring during specific phases of the process. Facilitators and barriers can usually be identified as occurring during specific phases of the process. The next analysis separates most of the same mentions in terms of phase in which they occurred. The next analysis separates most of the same mentions in terms of phase in which they occurred.

108 Facilitator & Barrier Analysis – Initiation Phase Initiation Phase – A facilitator or barrier that is anticipated or experienced PRIOR to the adoption decision. Initiation Phase – A facilitator or barrier that is anticipated or experienced PRIOR to the adoption decision. Initiation Phase Facilitators = 229 Initiation Phase Facilitators = 229 Initiation Phase Barriers = 91 Initiation Phase Barriers = 91

109 Mentions during Initiation Phase

110 Early in the Implementation Phase Early Implementation – A facilitator or barrier that is anticipated or experienced AFTER the adoption decision, but before full implementation. Early Implementation – A facilitator or barrier that is anticipated or experienced AFTER the adoption decision, but before full implementation. Early Implementation Facilitators = 166 Early Implementation Facilitators = 166 Early Implementation Barriers = 122 Early Implementation Barriers = 122

111 Mentions during Early Implementation

112 Implementation Phase Implementation – A facilitator or barrier that is anticipated or experienced AFTER the agency begins to implement the EBP. Implementation – A facilitator or barrier that is anticipated or experienced AFTER the agency begins to implement the EBP. Implementation Facilitators = 155 Implementation Facilitators = 155 Implementation Barriers = 250 Implementation Barriers = 250

113 Mentions during Implementation Phase

114 Summing Up: Facilitator/Barrier Analysis - Phase Initiation: Facilitators are mentioned more than TWICE AS FREQUENTLY as barriers. Initiation: Facilitators are mentioned more than TWICE AS FREQUENTLY as barriers. Early Implementation: Adopters & Implementers mention 50% more facilitators than barriers (trend not seen in Wait & See/Never or Deadopters). Early Implementation: Adopters & Implementers mention 50% more facilitators than barriers (trend not seen in Wait & See/Never or Deadopters). Implementation: While there are more barriers than facilitators mentioned throughout the Implementation phase, Deadopters mention nearly FOUR TIMES more barriers than facilitators. Implementation: While there are more barriers than facilitators mentioned throughout the Implementation phase, Deadopters mention nearly FOUR TIMES more barriers than facilitators.

115 So what does all this mean?

116 Major Messages Adoption decision is a risky decision Adoption decision is a risky decision Implementation effectiveness related to but not equal to innovation effectiveness Implementation effectiveness related to but not equal to innovation effectiveness Factors at many levels contribute to success Factors at many levels contribute to success What happens early (e.g., during initiation) can have enduring effects What happens early (e.g., during initiation) can have enduring effects Present implementation climate explains present outcomes Present implementation climate explains present outcomes

117 SOME IMPLICATIONS


Download ppt "Phyllis C. Panzano, Ph.D., PI Dee Roth, M.A., Co-PI Bev Seffrin, Ph.D, Senior Consultant Dushka Crane-Ross, Ph.D., Project Manager Decision Support Services,"

Similar presentations


Ads by Google