1. Organized Delivery System DRAFT Evaluation Plan Darren Urada, Ph.D., Valerie Antonini, MPH, Cheryl Teruya, Ph.D., Elise Tran, Kate Lovinger, M.S., Howard Padwa, Ph.D., Diego Ramirez, June Lim, Ph.D., Doug Anglin, Ph.D., Richard Rawson, Ph.D. UCLA Integrated Substance Abuse Programs CBHDA SAPT+ Meeting March 26, 2015

2. Acknowledgments Thanks to Lynn Brecht, Desiree Crevecoeur, Christine Grella,Yih-Ing Hser, Michael Prendergast, Beth Rutkowski, and Liz Evans for their suggestions and advice.

4. Suggestions & advice appreciated! DRAFT

6. Evaluation Goals Evaluate the Organized Delivery System (DMC, SAPT) in terms of: –Access to care –Quality of care –Coordination of care Within SUD continuum of care With recovery support services With mental health and primary care services – Costs (might be led by DHCS) Help inform implementation via feedback.

7. Realistic Data Goals Use existing data where possible. Align measures with existing or expected future data requirements. Where necessary, supplement with new data collection while attempting to minimize the burden on stakeholders.

8. Design Randomized controlled trials are ideal but impractical here. Pre-Post Comparisons County comparisons (Opt-in vs. Opt-out) Qualitative data

9. State Timeline (DRAFT)

10. Simplest Scenario: No Overlap Between Phases Possible Scenario: Overlapping Phases Likely Scenario: Overlapping Phases and Start Dates Baseline Yr 1 follow-up Phase 1 County A (Phase 1) 20152016 Baseline Yr 1 follow-up County B (Phase 1) Baseline Yr 1 follow-up County C (Phase 2) Baseline Yr 1 follow-up County D (Phase 2) Phase 2 Solution: Examine Counties by Start Date, Not Phases

11. Baseline Yr 1 follow-up Solution: Examine Counties by Start Date, Not Phases County A (Phase 1) Start Date Start Date + 1 Year County B (Phase 1) County C (Phase 2) County D (Phase 2) Baseline Yr 1 follow-up Note: Creates up to 371 time points, with multiple measures at each one.

12. T -1 (now) T 0 (start/baseline) T 1 (end of yr 1) T 2,3,4 ? T end Provider Survey Waiting list q EBPs; staffing  Integrated Practice Assessment Tool (IPAT) and single-item version  Patient Survey Perceptions of care  Quality of life (SF-12)  Provider Survey Waiting list q EBPs; staffing  IPAT  Key Informant Interviews (County Focused) Plans for implementation Perceptions of system (other providers)?  Key Informant Interviews (County Focused) Recommendations for future phases; insights, challenges, lessons learned Patient Survey Perceptions of care  Quality of life (SF-12)  Smaller Provider Survey (if needed) Smaller Patient Survey (if needed) Provider Survey Waiting list q IPAT  Patient Survey Perceptions of care  Quality of life (SF-12)  Key Informant Interviews (County Focused) Data to be Collected by UCLA

13. T -1 T0T0 T0T0 T1T1 T1T1 T 2,3, 4? T end Data CalOMS  OSHPD  Medi-Cal   DATAR Licensed beds NSDUH Chem hospitals, freepsych  KL Capacity/Access Project Data CalOMS  OSHPD  Medi-Cal   DATAR Licensed beds NSDUH Chem hospitals, freepsych  Kate Special Project Data CalOMS  OSHPD  Medi-Cal   DATAR Licensed beds NSDUH Chem hospitals, freepsych  Kate Special Project Data CalOMS  OSHPD  Medi-Cal   DATAR Licensed beds NSDUH Chem hospitals, freepsych  Kate Special Project Data CalOMS  OSHPD  Medi-Cal   DATAR Licensed beds NSDUH Chem hospitals, freepsych  Kate Special Project County/DHCS Audit; EQRs Use of EBPs  Staffing  MOUs  Beneficiary number ASAM (ASI)   ASAM (ASI)   County/DHCS Audit; EQRs Use of EBPs  Staffing  MOUs  Beneficiary number County/DHCS Audit; EQRs Use of EBPs  Staffing  MOUs  Beneficiary number ASAM (ASI)   Existing Data to be Analyzed by UCLA

14. Overview of Measures Access - Has access to treatment increased in counties that have opted in to the waiver? Quality - Has quality of care improved in counties that have opted in to the waiver? Integration & Coordination of Care - Is SUD tx being coordinated within the continuum of care? With recovery support services? With primary care and mental health services? Cost (might be led by DHCS) - Is the waiver cost effective?

15. Access

16. Potential Measures of Access Has access to treatment increased? Capacity –# admission by type of service (CalOMS-Tx) –# of admissions (Medi-Cal claims) –Licensed beds (includes privately funded beds, inpatient detox) –Residential capacity (DATAR) –Newly certified and de-certified sites (SMART6i data) –Capacity by zip code/city and modality Penetration Rates – % beneficiaries receiving services (CalOMS-Tx, Medi-Cal)

17. Access (continued) MAT use –Prescriptions filled for MATs (Medi-Cal claims) –NTPs, outpatient-medicated (CalOMS-Tx) –# of physicians able to prescribe Bup by ZIP/city Time to treatment/between treatment –Transition times between levels (CalOMS-Tx) –Typical wait time by modality (Provider survey, ASAM, CalOMS-Tx) Telehealth –billing for teleheath increased? (Medi-Cal claims) –offer telehealth? (Provider survey)

18. Access (continued) Existence of a functioning/up-to-date beneficiary access number, provider directory for patients –Web searches, calls. –Ability to get an appointment? Other medical services –MH use, ER use, hospital inpatient days (Medi-Cal) –Chemical dependency recovery hospitals and freestanding psychiatric hospitals (OSHPD). Misc. from Provider Survey –After-hours care/urgent care –Interpretation services.

19. Quality

20. Potential Measures of Quality Has quality of care improved? Appropriate placement (ASAM, CalOMS-Tx): –Comparison of ASAM scores and actual placement –Use of continuing ASAM assessments, appropriate transitions –% of referrals with successful treatment engagement (e.g., stepping down)

21. Quality (continued) # of admissions to chemical dependency recovery hospitals and free standing psychiatric hospitals (OSHPD) # of ER and psychiatric emergency visits (Medi-Cal, CSI-MH) # of hospital inpatient days (Medi-Cal, CSI-MH) % of patients who initiated and engaged in AOD treatment (if receiving services from primary care; Healthcare Effectiveness Data & Information Set [HEDIS])

22. Data indicator reports # of grievances (Grievance reports) Workforce (County or DHCS monitoring?): –Staff turnover –Staff to client ratios –Staff /client population ethnic/racial diversity –Staff use of EBPs (fidelity) –Training (e.g., cultural competency) –Staff certification documented Quality (continued)

23. Outcomes AOD use (CalOMS-Tx, patient survey) Social support (CalOMS-Tx, patient survey) Living arrangements/housing (CalOMS-Tx, patient survey) Employment (CalOMS-Tx, patient survey) Quality of Life / Functioning (CalOMS-Tx, patient survey) Use of other services (e.g., ER, mental health) (CSI-MH, Medi-Cal, OSHPD) Quality (continued)

24. Client perceptions of care (Mental Health Statistics Improvement Plan [MHSIP] Consumer Survey) –General satisfaction –Perception of access –Perception of quality and appropriateness –Perception of participation in treatment planning –Perception of outcome of services –Perception of functioning –Perception of social connectedness Quality (continued)

25. (to be adapted)

27. Provider perceptions of service delivery and implementation of the DMC ODS (Provider survey) –Staffing (e.g., counselor-client ratio) and turnover –Use of EBPs –Cultural competency training –Patient centered care (e.g., shared decision making, provision of information about treatment options, family involvement) –Implementation challenges, what’s working well –Training/technical assistance needs –Suggestions for improving implementation Quality (continued)

28. Stakeholders’ (e.g., county level administrators, consumer advocates) perceptions of the quality of services being delivered and implementation of the DMC ODS (Key informant interviews) –Biggest challenges (anticipated/current) –Strategies to address challenges –What’s working well –Communication –Training and technical assistance needs –Recommendations for improvement –Lessons learned Quality (continued)

29. Integration / Coordination

30. Potential Measures of Integration and Coordination of Care Is SUD treatment being coordinated within the continuum of care? With recovery support services? With primary care and mental health services?  TWO Components: 1.WITHIN SYSTEM SUD Continuum of Care Recovery Support Services 2.ACROSS SYSTEMS SUD + MH SUD +PH

31. Integration and Coordination Summary of Data Collection Activities 1.Survey – County Administrators 2.Qualitative Interviews – County Administrators 3.Survey – Provider Executives 4.Patient Survey 5.Considering Qualitative interviews – Providers  Purposive sample from providers with varying levels of care integration 6.Secondary analysis  Quantify referrals to and from primary care and mental health  Quantify referrals across the SUD continuum

32. Component 1 – Within System County level (Administrator survey, interview) –Guidelines for SUD providers regarding establishing MOUs/formal partnerships with other levels of care within the continuum –Procedures for MOU enforcement / monitoring/reporting among providers? **Reference County Implementation plan and initial survey response to guide interview

33. Component 1 – Within System Provider Level (survey) –Levels/Types of Treatment services offered –Treatment Partnerships/MOUs bi-directional agreements and utilization –Recovery Support Service Partnerships What/how do they incorporate RSS in their treatment program Patient navigator services? –Communication Pt. Information Transfer (EHR and data sharing protocols) Interpersonal communication (pt care plan exchanges) –Organizational Dynamics Leadership buy in/mission statements Operational reliability to facilitate the continuum for all patients Patient survey

34. Component 2 – Across Systems County level (Administrator survey and interview) –Communication between departments (scheduled meetings/frequency, etc) –Dept level MOUs (SUD and health plans, and mental health) –Guidelines for SUD providers regarding establishing MOUs/formal partnerships with MH and physical health –Procedures for MOU enforcement / monitoring/reporting among providers **Reference County Implementation plan and initial survey response to guide interview

35. Component 2 – Across Systems Provider Level (survey) –SUD + MH –SUD + PH Assess provider perception of current level of integration/collaboration and goal Implement IPAT survey – Integrated Practice Assessment Tool –Developed by SAMHSA-HRSA –Decision tree leading to a determined Level of Integration/ Collaboration Care (Level 1-6) Patient survey

36. Component 2 – Across Systems SAMHSA-HRSA Six Levels of Integration/Coordination

37. Component 2 – Across Systems After determining IPAT level of Integration… –More in-depth for levels 1-4 addressing the following domains: Systematic Screening for add’l services (including PH, MH, recovery, and patient navigation services) MOUs/referral protocols Array of expertise/services available onsite (co-located or integrated) Proportion of patients that are being coordinated across systems Physical proximity of partnered service providers Separation time between referral and scheduled visit Available patient navigators Engagement of patients and family in care plan

38. Questions Collecting ASAM data Getting CalOMS-Tx summaries back to providers Are there other things counties plan to measure?

39. Health Plan Use of ASAM Criteria Magellan: http://dhhs.ne.gov/behavioral_health/Documents/ASAMChecklist.d oc http://dhhs.ne.gov/behavioral_health/Documents/ASAMChecklist.d oc Cigna: http://apps.cignabehavioral.com/web/basicsite/media/consumer/ed ucationAndResourceCenter/articles/comparisonSummary.pdf http://apps.cignabehavioral.com/web/basicsite/media/consumer/ed ucationAndResourceCenter/articles/comparisonSummary.pdf Beacon: http://www.beaconhealthstrategies.com/private/pdfs/MA%20LOC% 202014%20Final.pdf http://www.beaconhealthstrategies.com/private/pdfs/MA%20LOC% 202014%20Final.pdf Managed Health Network: https://www.mhn.com/static/pdfs/MHNLOC.pdf https://www.mhn.com/static/pdfs/MHNLOC.pdf Note: Sources were compiled by Los Angeles County Evaluation System (LACES) evaluation staff at UCLA ISAP.

40. Use of ASAM Criteria by Washington State Washington Administrative Code Chapter 388-877: Behavioral Health Services Administrative Requirements http://app.leg.wa.gov/WAC/default.aspx?cite=388- 877&full=true http://app.leg.wa.gov/WAC/default.aspx?cite=388- 877&full=true Washington Administrative Code Chapter 388-877B: Chemical Dependency Services http://app.leg.wa.gov/WAC/default.aspx?cite=388- 877B&full=truehttp://app.leg.wa.gov/WAC/default.aspx?cite=388- 877B&full=true

41. See you next time!

42. Questions? Comments? Darren Urada, Ph.D. durada@ucla.edu