1. Annual Site Reports 2006-2007 FY A Training and Overview Los Angeles County Evaluation System: An Outcomes Reporting Program 2008

2. Purpose of this Presentation Overview of the Los Angeles County Evaluation System (LACES). Review the content and purpose of the Site Reports. Discuss those content areas that are the focus of the evaluation. Describe future evaluation activities –Discharge Outcome Report (DOR)

3. Overview of LACES LACES is an eight year collaborative partnership between ADPA, contracted providers, UCLA/ISAP and other county and state representatives. The purpose of LACES is to design, implement, and conduct an evaluation of the adult alcohol and other drug treatment/recovery system in Los Angeles County. On-going evaluation, not a temporary study.

4. Accomplishments of LACES To date LACES has: 1.Fundamentals of the ongoing evaluation system. 2.Designed and implemented evidence-based, standardized evaluation procedures. 3.Determined that alcohol and other drug treatment/recovery is cost effective. 4.Review & improve existing treatment forms and instrument. 5.Designed reports to benefit treatment.

5. First… Some Questions… Have you heard of a site report? Have you seen a site report? Have you had the opportunity to review the contents and findings of the site report with the director or program staff?

6. A Site Report Is… Brief: six pages, double-sided. Info from admission & discharge LACPRS. Demographics, outcomes & performance measures. Information is included on your site and for all sites of the same program type. Annual and some quarterly reports

7. What is the Purpose of the Site Report? How participants are benefiting from treatment. Provide information on program functioning and short-term treatment outcomes. Identify provider performance strengths and areas needing improvement.

8. What the Site Report Does Not Do? Provides “site specific” information – a single program type (e.g., outpatient) for a single address. –The site report does not consider funding source. Provides feedback concerning performance in the focus areas of the evaluation.

9. Contents of the Site Report Executive Summary –Overall participant admission and discharge numbers. –Information on key outcomes and performance. Full Report –Demographic information. –More details on treatment outcomes and performance. –Charts illustrating admission to discharge changes.

10. Executive Summary Single page, stand alone document. Admission to discharge changes. Key Areas include –“Participant Flow” chart. –Treatment Outcomes –Program Functioning

11. Executive Summary Tables Table A: Number of admissions and discharges for the time period covered by the report. Table B: Information on those discharged. –Determine areas of improvement Table C: Admission to discharge changes outcome areas. –Includes participants who were in treatment 30+ days and completed an exit interview.

12. Program Functioning and Treatment Outcomes Areas included in the Site Reports include: Program Functioning –Number of admissions and discharges. –Percent of exit interviews. –Participants with “positive compliance.” Treatment Outcomes –Retention in treatment (average days). –Changes in alcohol and other drug use. –Changes in other areas including health problems, employment, and social support.

13. Program Functioning (Performance) Admission and discharge numbers –Review to ensure ADPA received all submitted records. Positive Compliance –Discharge status = completed treatment or left with satisfactory progress. –Important to determine the effectiveness of the program.

14. Performance cont’d. Exit Interviews –Responses to all discharge LACPRS questions constitutes an “Exit Interview.” –Information from both admission and discharge is required to measure treatment effectiveness. –No exit interviews = No measurement of the participant changes that occurred during AOD treatment.

15. Table B: Participant Flow by Fiscal Quarter and Year to Date (YTD) Type of TransactionTotal by Fiscal QuarterYTD Total FirstSecondThirdFourth Participants Discharged50253020125 Participants with Positive Compliance2530201590 Percent with Positive Compliance50%60%67%75%72% Participants in Treatment at least 30 days 1822101262 Percent in Treatment at least 30 days36%44%50%80%50% Participants in Treatment at least 90 days 1215121150 Percent in Treatment at least 90 days24%30%40%55%40% Participants with Exit Interviews202515 45 Percent with Exit Interviews 40%50% 75% 36% 25125 50% 40%

16. Treatment Outcomes Measures are similar to state and federal government. Retention in treatment for 30/90 days. –In LA County, some participants leave treatment during the first two weeks. –Research shows that without an adequate amount of time in treatment, few improvements are found.

17. Treatment Outcomes, cont’d. Performance Measures cover changes in: –Substance use (alcohol and other drugs). –Criminal justice involvement (days in jail/prison). –Homeless at admission versus discharge. –Positive employment activities (full- & part-time work, job training, and enrollment in school). –Social Support Activities (AA, NA, etc.).

18. Table C: Quarterly Performance Measures Performance MeasureAverage Change from Admission to Discharge This Facility SiteAll Residential Service Sites AdmissionDischargeChangeAdmissionDischargeChange Average Days of Primary Drug/Alcohol Use* 12.30 ‑ 12.3 15.90.3 ‑ 15.5 Average Days of Criminal Justice Involvement 2.40 ‑ 2.4 4.90.2 ‑ 4.7 Average Days in Treatment201117.4 Percent Reported Homeless49.6%4.6% ‑ 45% 47.2%23.9%-23.3% Percent Reported Positive Employment Activities 3.5%8.0%4.5%6.9%22.4%15.4% Percent Reported Social Support Activities 57.0%93.0%36.0%42.6%92.9%50.2% This Facility Site Residential Services All Sites Admission Discharge Change Days of Drug/ Alcohol Use 12.3 0-12.3 Reported Homelessness 49.6%4.6% -45%

19. Full Report Provides an overview of LACPRS admission and discharge information. Section includes: –General demographics. –Primary and secondary alcohol and other drug use information. –Program performance.

20. Demographics Table provides demographic information on participants discharged. It includes demographic information on other sites for the benefit of the reader. Provides information on populations of special interest such as General Relief and homeless.

21. Primary & Secondary Substances Percentage of treatment participants by primary or secondary drug problem. Primary or secondary drug information from other sites is also included. These tables may be used to track changes in drug use patterns for the site.

22. Program Outcomes Section Refers to the four major sections. 1.Discharge status. 2.Changes in alcohol and other drug use. 3.Changes in employment and social support. 4.Changes in health problems and other areas of social functioning.

23. Discharge Status Positive compliance examined separately from negative compliance –Left with unsatisfactory progress Average length of stay demonstrates differences in stay by discharge status –Useful to providers attempting to increase overall retention.

24. Alcohol and Other Drug Use Bar chart shows change in average of days of primary and secondary drug use. Complete abstinence is not expected. Changes may also be found for injected drug use – especially for those sites that have a high percentage of needle users.

25. Chart 1: Admission to Discharge Changes in Drug and Needle Use

26. Health Problems & Social Functioning Changes in jail days, family conflict, & physical health problems. In some instances the number of jail days will be very low. May show an increase in physical health problems due to increased access to health care.

27. Chart 2: Admission to Discharge Changes in Health Problems and Social Functioning

28. Employment and Social Support Changes in days spent employed and engaged in social support activities. Not all sites focus on employment or provide social support activities. Useful to examine the utilization of these activities by the participant.

29. Chart 3: Admission to Discharge Changes in Employment & Social Support

30. Important Performance and Treatment Outcome Measures Completion of treatment/Positive Compliance. Abstinence from/Reduction in drug/alcohol use. Exit Interviews Length of Stay/Enrollment in Treatment (2 benchmarks) –30 Days = Engagement. –90 Days = Retention. All important aspects of the treatment plan.

31. Why Completion of Treatment? Most outside of this field are interested in completion figures. Positive Compliance –Completed treatment. –Left treatment prior to completion with satisfactory progress. Others have adopted this mindset, but many have not.

32. Why Reduction/Abstinence? Silly picture, for a silly question. That is our business. If we cannot demonstrate abstinence or reductions in use, funders will move the money to those who can.

33. Interviews Responses to all discharge LACPRS questions. One of the most important of the evaluation foci. Information from both admission and discharge is required. No exit interviews = No measurement of changes.

34. Why Engagement & Retention? First 30 days are critical to the outcome of treatment. Without adequate treatment, few improvements are found. Funding organizations are beginning to take note of engagement and retention rates.

35. Note on Documentation Accuracy is extremely important. –Length of stay/enrollment very important. –Example – length of stay/enrollment. In 2004-2005, average length of stay was 105 days. But only 40% were in treatment for 30+ days. If this information is inaccurate, others may become suspicious of all information reported.

36. Max LOS/LOE by Funding Source All funding sources support 30- and 90-day length of stay/enrollment benchmarks. FUNDINGOutpatient LOEResidential LOSDay Care LOE CalWORKS365 Days Drug Court730 Days90 DaysN/A Female OffenderN/A180 DaysN/A General Relief180 Days PPN180 Days N/A Prop 36300 Days90 Days204 Days Drug Medi-CalNo max length - approval is need for 180+ days.

37. Average Provider Performance During the last 3 fiscal years, average performance in these areas was… PERFORMANCE AREAOutpatientResidentialDay Care Reduction in Primary Drug Use75.7%97.7%77.9% 30-Day Retention71.8%64.1%68.9% 90-Day Retention48%38.1%40.6% Exit Interviews48.6%73.7%47%

38. Strategies & Suggestions for Improving Outcomes

39. Improving Exit Interview Rates Some ways to increase rates of exit interviews include –Informing the participant of the exit interview when admitted. –Include completion of the exit interview as part of the individualized treatment plan. –Remind the participant when they are nearing completion of their plan of the exit interview.

40. Improving Engagement & Retention? Engage them from first contact Make same day appointments if possible. Appointment cards To encourage forgetful clients. Phone calls To remind clients of appointments. To follow-up with clients who may have dropped out.

41. Other Suggestions?

42. Challenges LACPRS –Provides a lot of good information. –Revisions and monitoring of information will be needed to ensure quality & validity. Site Reports –Also requires revisions to format & content. –Not getting past Executive Directors’ eyes.

43. Your Views on the Site Reports What do you think about the site reports? What have you heard from your directors? Do you have suggestions or recommended changed to the reports?

44. Future of the Site Reports Assess individual sites in the key areas described in this presentation. Develop peer performance standards. Programs that fall short will be offered –Information, –Training, –and other forms of assistance.

45. LACES Phase III Focus on engagement and retention. Continue with community contact. Continue to expand and refine the site reports. –Include Narcotic Treatment Programs. –Develop web-based training and educational modules. Continue the expansion of LACES activities –Discharge Outcome Report

46. Discharge Outcome Reports (DOR) Provides admission and discharge information on a single individual. –A site report for one person. Useful when you need to report treatment outcomes to others. May also be useful when transferring someone to a new level of care outside your agency.

47. The End