Systems of Care Outcomes Michael Dennis, Ph.D. Chestnut Health Systems, Bloomington, IL Presentation at “UT CAN Local Academy 2006 Celebration, Integration and Painting the Vision”, June 5-7, 2006, Salt Lake City, Utah. Rockville, MD: Center for Mental Health Services (CMHS), Substance Abuse and Mental Health Services Administration (SAMHSA) under contract and several individual grants. The opinions are those of the author and do not reflect official positions of the consortium or government. Available on line at or by contacting Joan Unsicker at 720 West Chestnut, Bloomington, IL 61701, phone: (309) , fax: (309) ,

2 Look at the kinds of outcomes that make sense when evaluating a local or state systems of care Shifting from emphasis from terminal outcomes, to performance measures that help you manage and fine tune the process Recognize that there can be multiple measures and perspective Incorporating the common values and principals into the measures Provide a detailed example Goals

3 Performance Measures & Outcomes for Local or State Systems of Care Systems of Care Vision Skills Incentives Resources Action Plan 1. Needs Assessment 2. Capacity Building 3. Program Selection 4. Implementation 5. Evaluation Source: SAMHSA/CSAP Pathways Course Evaluation 101

4 Vision Can staff reliably articulate the local systems “vision”? Do they actually use it to guide decision making? (Ask how to learn from them) Percent of staff who strongly agree that they have used their strengths in their job (most) days Percent of families who strongly agree that staff were trying to help them navigate the system Skills Has the needs assessment identified specific skill sets or knowledge that is needed? Have staff been provided skills training related to above? To what extent are they actually using it at all, with the right people, and to proficiency? Has this led to a measurable change in performance? Examples of Systems Outcomes

5 Incentives Have we developed a set of incentives (recognition, training, trips to professional conferences, money) that staff appear to care about? Are the incentives being used? Have waivers been sought from state or other agencies to facilitate collaboration and reduce inefficiency? Resources Have training and supervision resources been redirected to target the strategic areas for addressing skills deficit and/or building on strengths? Have partnerships with other agencies or local community resources been made to leverage or bring in additional resources? Have formal grants or other proposals been submitted for funds to address gaps and/or start up costs? Examples of Systems Outcomes

6 Action Plan Have we translated the theory/logic model into the critical steps for implementation? Do we have performance measures for each of these step that are reliably measured and available in relatively real time (e.g., weekly, monthly) to help manage the process? Are the measures being implemented and the management reports being used? Can we demonstrate that the management reports helped us to identify a specific problem, that something was changed, and that performance on the targeted measure improved? Examples of Systems Outcomes

7 Washington Circle Group Performance Measures Everyone in Plan or Target Population Percent Screened Percent with Substance Diagnosis Percent Engaged by Treatment System Percent Retained by Treatment System - % with Psychiatry Services - % with Family Services - % stepped down from Residential/IOP Percent Retained or getting checkup/booster 90+ days after intake Identification Engagement Services Continuing Care Initiation Percent Initiating Treatment

8 Standardize screening so that identification of needs is independent of what door you walk through Increased matching based need and strengths of individuals/families and staff/programs Percent targeted for a service who get it? Who come back for more? Who are satisfied with it? Early working alliance/satisfaction Actual rates of referral, receipt of referred services, and engagement in referred services Percent of staff certified on providing a specific procedure Adherence to a set of specific clinical or customer oriented procedures Satisfaction with specific components of care, staff, facilities, scheduling, following up etc Some Other Common Initial Issues

Example: Assertive Continuing Care (ACC) Experiment 183 adolescents admitted to residential substance abuse treatment Treated for days inpatient, then discharged to outpatient treatment Random assignment to usual continuing care (UCC) or “assertive continuing care” (ACC) Over 90% follow-up 3, 6, & 9 months post discharge Source: Godley et al 2002, in press

10 Time to Enter Continuing Care and Relapse after Residential Treatment (Age 12-17) Source: Godley et al., 2004 for relapse and 2000 Statewide Illinois DARTS data for CC admissions 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Days after Residential (capped at 90) Percent of Clients Cont. Care Admis. Relapse

11 ACC Enhancements Continue to participate in UCC Home Visits Sessions for adolescent, parents, and together Sessions based on ACRA manual (Godley, Meyers et al., 2001) Case Management based on ACC manual (Godley et al, 2001) to assist with other issues (e.g., job finding, medication evaluation)

12 Assertive Continuing Care (ACC) Hypotheses Assertive Continuin g Care General Continuin g Care Adherence Relative to UCC, ACC will increase General Continuing Care Adherence (GCCA) Early Abstinence GCCA (whether due to UCC or ACC) will be associated with higher rates of early abstinence Sustained Abstinence Early abstinence will be associated with higher rates of long term abstinence.

13 ACC Improved Adherence Source: Godley et al 2002, in press 0% 10% 20% 30% 40%50%60%70%80% WeeklyTx Weekly 12 step meetings Regular urine tests Contact w/probation/school Follow up on referrals* ACC * p<.05 90% 100% Relapse prevention* Communication skills training* Problem solving component* Meet with parents 1-2x month* Weekly telephone contact* Referrals to other services* Discuss probation/school compliance* Adherence: Meets 7/12 criteria* UCC

14 GCCA Improved Early (0-3 mon.) Abstinence Source: Godley et al 2002, in press 24% 36% 38% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Any AOD (OR=2.16*)Alcohol (OR=1.94*) Marijuana (OR=1.98*) Low (0-6/12) GCCA 43% 55% High (7-12/12) GCCA * p<.05

15 Early (0-3 mon.) Abstinence Improved Sustained (4-9 mon.) Abstinence Source: Godley et al 2002, in press 19% 22% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Any AOD (OR=11.16*)Alcohol (OR=5.47*) Marijuana (OR=11.15*) Early(0-3 mon.) Relapse 69% 59% 73% Early (0-3 mon.) Abstainer * p<.05

16 Questions for breakout session 1.Do you have one or two key processes or needs you are trying to impact? 2.Have you identified capacity and resources to actually deal with them? 3.Have you picked a specific intervention to try to do? 4.Can you out line the small steps that would be necessary for this to work at the client level? 5.What are the performance measures for each step and where will this data come from? 6.How will the leadership team get the feedback, what will the tables/reports look like?