1. Children’s Intensive Services (CIS) Program Evaluation Presented By: Elizabeth Earls – President/CEO Rhode Island Council of Community Mental Health Organizations and Deborah O’Brien – Director, Quality Improvement The Providence Center

2. History and Overview Why Measure Outcome? In the 1990’s, as resource demands for health and social programming increase, State and Federal leaders, managed care entities and consumers demand programs that are accountable and measure the effectiveness of their work The Department of Children, Youth and Families (DCYF) begin to incorporate performance indicators into all of its service contracts. DCYF establishes “Outcome Committees” for its various service sectors with the goal of establishing measurement and accountability.

3. History and Overview Why Measure Outcome? The Community Mental Health Organizations meet with DCYF staff for a number of months and determine that they will use the Child Behavior Checklist (CBCL) as the CIS performance measurement tool. DCYF announced that as part of the planned revisions to the CIS contracts, it would mandate that the CBCL be completed on each child who received services from a CIS program.

4. History and Overview Why the Child & Adolescent Functional Assessment Scale (CAFAS)? DCYF had never established the staff and the technical capacity to collect the data from the CBCL’s. Therefore, they have no ability to analyze or provide feedback to us on the success, or lack of success, in our programs. Some clinicians had reported that their experience with the CBCL indicated that it was not a good measure of outcome.

5. History and Overview Why the Child & Adolescent Functional Assessment Scale (CAFAS)? Through the Council’s Quality Improvement Committee, CMHCs compare their experience with DMHRH and its required Outcome Evaluation Instrument (OEI): DMHRH collects data and reports back to organizations quarterly. The OEI had been approved and utilized by several CMHO’s to fulfill their JCAHO – ORYX process.

6. History and Overview Why the Child & Adolescent Functional Assessment Scale (CAFAS)? The RI Council determines that it will take the lead on this issue in the absence of action by DCYF. The Council sub-committee with both QI directors and Children’s Services Directors. The Committee identifies possible outcome measurement tools to be used in the CIS programs. The tools are reviewed across a determined set of criteria.

7. History and Overview Why the Child & Adolescent Functional Assessment Scale (CAFAS)? Criteria used for the comparison include the following: Target population Domains covered by the Tool Completer of the instrument (staff vs. client) Administration time Cost Norms/Breadth of use Reliability Ability to measure change over time Technical infrastructure requirements Clinical usefulness Face Validity

8. History and Overview Why the Child & Adolescent Functional Assessment Scale (CAFAS)? In February 2000, the Council’s Member Organizations vote to use the CAFAS. It informs DCYF of its decision. DCYF agrees to assist with support of training and recognizes that the CAFAS will replace the CBCL.

9. History and Overview Why the Child & Adolescent Functional Assessment Scale (CAFAS)? High degree of inter-rater reliability Sensitive to change over rating time periods Ability to identify high risk behaviors Sub-scales measure functioning in various domains Identification of specific profile types with children & adolescents Tested extensively and used in National studies: Fort Bragg, Michigan CMHC, CMHS (REACH) Benchmark with other studies

10. History and Overview CAFAS Implementation Council contracts with Dr. Kay Hodges, creator of the CAFAS, to train 40 clinicians as trainers on the CAFAS. The Council has made an important decision to systematically collect data based on what the clinician knows and has complied into this single format. We will collect Statewide data and be able to measure our success and advocate for our programs. We will provide the data to DCYF.

11. History and Overview CAFAS Implementation Implementation Committee is created to establish a uniform protocol across the CIS Programs so that all staff are trained; are determined reliable raters; and are collecting data in a consistent manner.

12. History and Overview CAFAS Protocol Issues being addressed by the Implementation Committee, include but are not limited to: Age of population to be assessed (7 to 17) Level of credentialed staff (masters and bachelors) Administration intervals (1-month and 6-months) Train-the-trainer on CAFAS software Rater training, rater remediation, rater 18-month boosting (345)

13. History and Overview CAFAS Protocol Implementation Committee issues (continued): RI specific manual created for FAQ’s Data export software developed Identify and contract with database consultant for aggregate reporting Satisfaction Measurement (To be determined)

14. CAFAS – Self-Harmful Behaviors Subscale Severe Impairment Severe disruption or incapacitation (30) Moderate Impairment Major or persistent disruption (20) Mild Impairment Significant problems or distress (10) Minimal or No Impairment No disruption of functioning (0) 142 Non-accidental self- destructive behavior has resulted in or could result in in serious self-injury or self-harm (e.g., suicide attempt with intent to die, self-starvation). 146 Non-accidental self- harm, mutilation, or injury which is not life-threatening but not trivial (e.g., suicidal gestures or behavior without intent to die, superficial razor cuts). 149 Repeated non-accidental behavior suggesting self-harm, yet the behavior is very unlikely to cause any serious injury (e.g., repeatedly pinching self or scratching skin with a dull object). 151 Behavior is not indicative of tendencies toward self-harm 143 Seemingly non- intentional self-destructive behavior has resulted in or could likely result in serious self-injury (e.g., runs out in the path of a car, opens car door in moving vehicle), and youth is aware of the danger. 147 Talks or repeatedly thinks about harming self, killing self, or wanting to die. 144 has a clear plan to hurt self, or genuine desire to die.

15. CAFAS CAFAS Domains School Home Community Behavior Towards Others Moods and Emotions Self-Harmful Behavior Substance Abuse Thinking Caregiver Scales

16. CAFAS -Severity of Impairment

17. CAFAS Outcome Results: CIS and Benchmark Programs

18. CIS Outcomes by Length of Treatment

19. Areas of High Risk at Admission to CIS

20. Risk at Admission and Discharge Decrease in high risk behaviors

21. Next Steps Streamline the data import and administration Identify and implement a satisfaction tool Train and implement the Pre-School & Early Childhood Functional Assessment Scale (PECFAS) Analysis of aggregate data for all stake holders