1. Assessing Your System of Care: The System of Care Rating Tool
Ohio System of Care Summit
Beth A. Stroul, M.Ed. President, Management & Training Innovations
September 14, 2018

2. System of Care Development: How Will You Know?
So your state, tribe, territory, or community has developed strategies for implementation and expansion of the system of care (SOC) approach
How will you know how you’re doing? If you’re getting there?
How will you measure
SOC implementation and expansion progress?

3. Development of the Rating Tool
Method to assess progress in SOC implementation and expansion
Builds on previous methods
Developed, pilot tested, revised tool based on feedback
Continuing to refine tool
Version 2.0 launched in 2016
Implementing tool in multiple states
Developed a version for managed care organizations
Step-by-step guide published April, 2017

4. Step-by-Step Guide
For Rating Tool 2.0

5. Rating Tool Purposes Brief, easy to administer, web-based instrument
Designed to assess implementation of the SOC approach in a geographic area – typically a community or region
Dual purpose – evaluating progress and identifying elements requiring improvement to inform resource allocation and technical assistance (TA)
Derives an overall estimate of the “level” of implementation of the SOC approach
Assesses implementation of key elements of the SOC approach at a point in time, which are operationally defined
Can be used as part of evaluation
Identifies strengths in SOC implementation and areas needing improvement

6. System of Care Elements Assessed
Use of a Strategic Plan
SOC Values and Principles
Array of Services and Supports
SOC Infrastructure Components
Commitment of Key Partners to SOC Approach

7. Levels of Overall SOC Implementation
Level 1 – Little or No Implementation
Level 2 – Some Implementation
Level 3 – Moderate Implementation
Level 4 – Substantial Implementation
Level 5 – Extensive Implementation

8. Uses at the Community or Regional Level
Measure progress in SOC implementation while efforts are underway to develop or improve SOCs
Initial assessment
Use at regular intervals to assess progress over time (annually, biennially?)
Identify strengths and areas needing attention, TA, and resource investment while implementing the SOC approach

9. Uses at State, Tribal, Territorial Level
Assess implementation progress statewide, and throughout tribes and territories, with multiple communities or regions, outcomes of expansion efforts
Baseline and subsequent ratings of progress in communities or regions (depends on structure)
Communities or regions complete tool at regular intervals, state determines percentage at each of the 5 levels
Assess all communities or subset depending on expansion approach
Annual use provides measure of progress based on comparison of percentage at each level at each point in time (annually, biennially?)
Identifies strengths, areas needing improvement, and need for investment of resources and TA statewide in particular communities or regions

10. Administering the Rating Tool
Version 2.0 can be use as a self-assessment, with the entire package given to a jurisdiction. Includes all tools to administer and generate reports.
Survey is placed in state or community’s own Survey Monkey Account
Detailed guide
Scripts for s to lead community person and respondents
Excel templates to generate reports
Opportunities for consultation

11. Respondents
Designed for approximately 10+ respondents per community or region – scores are averaged
Can be customized to each community regarding number and type of respondents
Required to fulfill specific types of roles and be sufficiently knowledgeable about the SOC to make a reliable assessment
Local community-level or regional-level directors or managers of services for children with behavioral health challenges
Lead provider agency directors or managers for services children with behavioral health challenges
Clinicians or care managers
Family organization directors or family leaders
Youth organization directors or youth leaders
Local community-level or regional-level directors or managers of services for children with behavioral health challenges from a partner agency
Other key stakeholders identified by lead contact person for a community or region

12. Data Collection and Analysis
Identify a lead contact in each community/region to be assessed
Lead contact for each community/region identifies potential respondents
Each identified respondent is ed a unique URL link from Survey Monkey to complete the rating tool
Tool takes approximately 20 minutes to complete
Assessment can be completed in one sitting or over multiple sessions
Assertive follow-up is needed to maximize response rate
Respondent scores are averaged to determine ratings for each SOC element
Individual responses are confidential
All data is reported in aggregate
Community and state level reports are developed

13. SOC Domains Assessed Strategic Plan Domain Principles Domain
Existence and use of a plan for SOC implementation
Principles Domain
Implementation of SOC values and principles – each operationalized with 4 or 5 indicators
Individualized, wraparound approach
Family-driven approach
Youth-guided approach
Coordinated approach
Culturally and linguistically competent approach
Evidence-informed and promising practices and practice-based evidence approaches
Least restrictive approach
Broad service array
Data and accountability

14. Family-Driven Approach
Families have primary decision making role in service planning and delivery
Families set goals and desired outcomes
Family strengths are incorporated
Families have choice of services and supports
Families have access to peer support
Family organization exists that supports family involvement at system and service delivery levels

15. Individualized, Wraparound Approach
Individualized child and family teams are used (including family, youth, providers, etc.) to develop and implement a customized service plan
Individualized assessments of child and family strengths and needs are used to plan services and supports
Individualized service plans are developed and implemented for each child and family that address multiple life domains
Services include informal and natural supports in addition to treatment
Flexible funds are available to meet child and family needs not financed by other sources

16. Coordinated Approach
Intensive care coordination with a dedicated care coordinator is provided to high-need youth and families
Basic care coordination is provided for children and families at lower levels of service intensity
Care is coordinated across multiple child-serving agencies and systems
Over overall plan of care is created across child-serving agencies and systems (may be more detailed plans for individual systems)

17. Culturally and Linguistically Competent Approach
Culture-specific services and supports are provided
Services and supports are adapted to ensure access and effectiveness for culturally diverse populations
Providers represent the cultural and linguistic characteristics of the population served
Providers are trained in cultural and linguistic competence
Specific strategies are used to reduce racial and ethnic disparities in access to and outcomes of services

18. Least Restrictive Approach
Utilization of inpatient hospitalization is decreased, and it is primarily used for short-term, acute treatment and stabilization when necessary and appropriate
Utilization of residential treatment is decreased, and it is primarily used for short-term stays to achieve specific treatment goals, when necessary and appropriate
Utilization of home- and community-based services is increased
The number of children who are served in settings more restrictive than necessary is reduced

19. SOC Domains Assessed Services Domain
Home and Community-Based Services and Supports – Includes a list of services required in SOC Expansion Grants with addition of newer services and supports typically included in SOCs
Out-of-Home Treatment Services for short-term goals linked to home- and community-based services and supports
No community is likely to have all services and supports, but provides a guide for an optimal array

20. Home- and Community-Based Treatment and Support Services
Screening
Assessment and evaluation
Individualized service planning
Intensive care coordination
Service coordination
Outpatient individual therapy
Outpatient group therapy
Outpatient family therapy
Medication treatment/monitoring
Crisis response services (non-mobile)
Mobile crisis response and stabilization
Intensive home-based services
School-based mental health services
Day treatment
Substance use treatment
Therapeutic behavioral aide services
Behavior management skills training
Tele-behavioral health
Youth peer support
Family peer support
Youth and family education
Respite services
Therapeutic mentoring
Mental health consultation
Supported education and employment
Supported independent living
Transportation

21. Out-of-Home Treatment Services for Short-Term Goals Linked to Home- and Community-Based Services and Supports
Therapeutic foster care
Therapeutic group care
Crisis stabilization beds
Medical detoxification
Substance use residential treatment
Residential treatment
Inpatient hospitalization

22. SOC Domains Assessed Infrastructure Domain
Point of accountability structure for SOC management and oversight
Defined access/entry points to care
Extensive provider network for comprehensive service array
Structure and/or process to manage care and costs for high-need populations (e.g., care management entities)
Structure and/or process for training, TA, and workforce development
Interagency partnerships/agreements
Structure and/or process for measuring and monitoring quality, outcomes, and cost
Structure and/or process for partnerships with family organizations and family leaders
Structure and/or process for strategic communications/social marketing
Structure and/or process for partnerships with youth organizations and youth leaders
Structure and/or process for strategic planning and identifying and resolving barriers

23. SOC Domains Assessed Commitment Domain
Commitment to the SOC philosophy and approach among key stakeholders
Child-serving systems – mental health, child welfare, juvenile justice, education, health, substance use, courts/judiciary, Medicaid
High-level policy and decision makers
Providers
Family and youth leaders
Managed care organizations (MCOs) – for both physical and behavioral health and behavioral health MCOs

24. Types of Reports Excel Templates - used to generate reports
Community or Region Report
Rating of overall level of implementation
Summary ratings in each domain
Ratings for each individual indicator in domains to identify strengths and areas to improve
Progress report
State, Tribe, or Territory Report
Number and percent of communities or regions at each level of implementation
Mean ratings across all communities or regions assessed in each domain
Total level of implementation score and percentage across all communities or regions
Summary rating of overall level of implementation for each participating community or region

25. Community or Region Summary Data Level of Implementation Score
LEVEL OF SOC IMPLEMENTATION SCORE
Score
% of Maximum Score
LEVEL I LITTLE OR NO IMPLEMENTATION ( ) (0% - 20%)
LEVEL II SOME IMPLEMENTATION (101 –200) (21% – 40%)
LEVEL III MODERATE IMPLEMENTATION (201 – 300) (41% – 60%)
LEVEL IV SUBSTANTIAL IMPLEMENTATION (301 – 400) (61% – 80%)
LEVEL V EXTENSIVE IMPLEMENTATION (401 – 500) (81% – 100%)

26. Community or Region Summary Data Domain Scores
SUMMARY DOMAIN SCORES
Score
% of Maximum Score
STRATEGIC PLAN SCORE (MAX = 5)
PRINCIPLES SCORE (MAX = 190)
SERVICES SCORE (MAX = 170)
INFRASTRUCTURE SCORE (MAX = 60)
COMMITMENT SCORE (MAX = 75)
TOTAL SCORE (MAX = 500)

27. State, Tribe, or Territory Summary Data Level of Implementation Across Communities or Regions
LEVEL OF SOC IMPLEMENTATION SCORES ACROSS COMMUNITIES OR REGIONS
# Communities or Regions
% Communities or Regions
LEVEL I LITTLE OR NO IMPLEMENTATION
LEVEL II SOME IMPLEMENTATION
LEVEL III MODERATE IMPLEMENTATION
LEVEL IV SUBSTANTIAL IMPLEMENTATION
LEVEL V EXTENSIVE IMPLEMENTATION

28. MEAN LEVEL OF SOC IMPLEMENTATION SCORE ACROSS COMMUNITIES OR REGIONS
State, Tribe, or Territory Report Summary Data Mean Level of Implementation Across Communities or Regions
MEAN LEVEL OF SOC IMPLEMENTATION SCORE
ACROSS COMMUNITIES OR REGIONS
Score
% of Maximum Score
LEVEL I LITTLE OR NO IMPLEMENTATION ( ) (0% - 20%)
LEVEL II SOME IMPLEMENTATION (101 –200) (21% – 40%)
LEVEL III MODERATE IMPLEMENTATION (201 – 300) (41% – 60%)
LEVEL IV SUBSTANTIAL IMPLEMENTATION (301 – 400) (61% – 80%)
LEVEL V EXTENSIVE IMPLEMENTATION (401 – 500) (81% – 100%)

29. MEAN DOMAIN SCORES ACROSS COMMUNITIES OR REGIONS
State, Tribe, or Territory Report Summary Data Mean Domain Scores Across Communities or Regions
MEAN DOMAIN SCORES ACROSS COMMUNITIES OR REGIONS
Mean Score
Mean %
STRATEGIC PLAN SCORE (MAX = 5)
PRINCIPLES SCORE (MAX = 190)
SERVICES SCORE (MAX = 170)
INFRASTRUCTURE SCORE (MAX = 60)
COMMITMENT SCORE (MAX = 75)
TOTAL SCORE (MAX = 500)

30. Community or Region Name
State, Tribe, or Territory Report Summary Data Individual Community/Region Scores
LEVEL OF SOC IMPLEMENTATION SCORES FOR INDIVIDUAL COMMUNITIES OR REGIONS
Level I: Little or No Implementation = (0 – 20%)
Level II: Some Implementation = (21-40%
Level III: Moderate Implementation = (41-60%)
Level IV: Substantial Implementation = (61-80%)
Level V: Extensive Implementation = (81-100%)
Community or Region #
Community or Region Name
Total Score
(0-500) %
(0-100%) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

31. Detailed Data for Both Reports Individual Community/Region or Mean Across Communities/Regions Assessed
Example for “Individualized, Wraparound” Principle
Principles and Indicators
Implementation of System of Care Principles During the Past 12 Months
Score
0 = Not at all implemented 1 = A little implemented 2 = Somewhat implemented 3 = Moderately implemented 4 = Substantially implemented 5 = Extensively implemented
Individualized, Wraparound Approach to Service Planning and Delivery
Individualized child and family teams are used (including family, youth, providers, etc.) to develop and implement a tailored service plan
Individualized assessments of child and family strengths and needs are used to plan services and supports
Individualized, tailored service plans are developed for each child and family that address multiple life domains and are revised based on progress
Services include informal and natural supports in addition to treatment
Flexible funds are available to meet child and family needs not financed by other sources
SCORE: INDIVIDUALIZED (MAX = 25)

32. SCORE: INDIVIDUAL SOC PRINCIPLES
Detailed Data for Both Reports Individual Community/Region or Mean Across Communities/Regions Assessed
SCORE: INDIVIDUAL SOC PRINCIPLES #
% of Max
Little or not at all implemented = %
Somewhat implemented = 21-40%
Moderately implemented = 41-60%
Substantially implemented = %
Extensively implemented = %
SCORE: INDIVIDUALIZED (MAX = 25)
SCORE: FAMILY DRIVEN (MAX = 25)
SCORE: YOUTH GUIDED (MAX = 25)
SCORE: COORDINATED (MAX = 20)
SCORE: CULTURAL AND LINGUISTIC COMPETENCE (MAX = 25)
SCORE: EVIDENCE-INFORMED ( MAX = 20)
SCORE: LEAST RESTRICTIVE (MAX = 20)
SCORE: SERVICE ARRAY (MAX = 20)
SCORE: DATA AND ACCOUNTABILITY (MAX = 10)

33. Community or Region Progress Report Level of SOC Implementation
LEVEL OF SOC IMPLEMENTATION RATING
Year 1
Year 2
Year 3
Year 4
Year 5 # %
LEVEL I
LITTLE OR NO IMPLEMENTATION
( ) (0% - 20%)
LEVEL II
SOME IMPLEMENTATION
(101 –200) (21% – 40%)
LEVEL III
MODERATE IMPLEMENTATION
(201 – 300) (41% – 60%)
LEVEL IV
SUBSTANTIAL IMPLEMENTATION
(301 – 400) (61% – 80%)
LEVEL V
EXTENSIVE IMPLEMENTATION
(401 – 500) (81% – 100%)
LEVEL OF SYSTEM OF CARE IMPLEMENTATION TOTAL SCORE

34. Community or Region Progress Report Domain Progress
Year 1
Year 3
Year 4
Year 5 #
% of Max
STRATEGIC PLAN SCORE (MAX = 5)
PRINCIPLES SCORE (MAX = 190)
SERVICES SCORE (MAX = 170)
INFRASTRUCTURE SCORE (MAX = 60)
COMMITMENT SCORE (MAX = 75)

35. SOC IMPLEMENTATION PROGRESS ACROSS COMMUNITIES OR REGIONS
State, Tribe, or Territory Progress Report Level of SOC Implementation Across Communities or Regions
SOC IMPLEMENTATION PROGRESS ACROSS COMMUNITIES OR REGIONS
Level of System of Care Implementation Rating
% Communities or Regions
Year 1
Year 2
Year 3
Year 4
Year 5
LEVEL I LITTLE OR NO IMPLEMENTATION
LEVEL II SOME IMPLEMENTATION
LEVEL III MODERATE IMPLEMENTATION
LEVEL IV SUBSTANTIAL IMPLEMENTATION
LEVEL V EXSTENSIVE IMPLEMENTATION
MEAN LEVEL OF SYSTEM OF CARE IMPLEMENTATION TOTAL SCORE ACROSS COMMUNITIES OR REGIONS

36. DOMAIN PROGRESS ACROSS COMMUNITIES OR REGIONS
State, Tribe, or Community Progress Report Mean Domain Progress Across Communities or Regions
DOMAIN PROGRESS ACROSS COMMUNITIES OR REGIONS
Mean % Across Communities or Regions
Year 1
Year 2
Year 3
Year 4
Year 5
STRATEGIC PLAN SCORE
PRINCIPLES SCORE
SERVICES SCORE
INFRASTRUCTURE SCORE
COMMITMENT SCORE

37. Community B

38. Community B Detailed Data
Finding:
Strength – Individualized services
Weakness – Cultural competence
Little or not at all implemented:
Cultural and linguistic competence
Somewhat:
Coordinated
Evidence-informed
Service array
Data and accountability
Moderately:
Family driven
Youth guided
Least restrictive
Substantially:
Individualized

39. Individualized:
Strengths – Individualized service plans, flexible funds
Weakness – Child and family teams
Coordinated:
Strength – Basic care coordination
Weakness – Coordination across multiple agencies, one overall plan of care
Cultural competence:
Weakness in all indicators

40. Finding:
Strengths – Non-mobile crisis, school-based services, day treatment
Weaknesses – Home- and community-based support services
A little available:
Intensive care coordination
Substance use treatment
Youth and family peer support
Respite
Behavioral aides
Behavior management skills training
Mentoring
Moderately available:
School-based
Day treatment
Crisis response (non-mobile)
Remainder somewhat available

41. Finding:
Strengths – Inpatient hospitalization (if used for short-term goals and linked)
Weaknesses – Limited service array, more inpatient services than alternatives
A little available:
Crisis stabilization beds
Moderately available:
Inpatient hospitalization
Remainder somewhat available

42. Finding:
Strengths – Access to care, interagency partnerships
Weaknesses – Most infrastructure elements
Somewhat implemented:
Interagency partnerships
Defined entry points to care
Not at all implemented:
Strategic communications
Remainder a little implemented

43. Finding:
Strengths – Child-serving systems
Weaknesses – Family and youth leaders, decision makers, providers
Little or not at all committed:
Family and youth leaders (20%)
Moderately committed:
Child-serving systems (50%)
Somewhat committed:
Decision makers (40%)
Providers (40%)
Managed care organizations (40%)