1. The Role of Client Outcomes in Evaluating a System of Care Model of Mental Health Delivery
Richard Phillips, Ph.D.
Director of Research and Evaluation,
Idaho Child Welfare Research & Training Center
Eastern Washington University
Patty Gregory, LCSW
Director
Outline for presentation
Introduction to outcomes: different types – Rick
Introduction to system of care for SED children: the child/family centered model (slide #4) – Patty
Type of data required by federal/state guidelines (slide #5) – Rick
Type of data useful to partners to make decisions about a case (slide #6) – Patty
Data/themes collected from the sites: (slide #7 – notes pages
Family centered practice and conflict with standardized data: Rick
The reality of collaboration: Patty
The role of advocacy: Rick
Recommendations: (slide #11 – notes pages)
Ensure that system of care values and principles extends into the bureaucracy – Patty
Work with local sites to establish data collection routines that celebrate their actions, rather than their reporting functions: Rick
Provide evaluation support for local collaboratives so that volunteer hours are spent where the members are most effective – in collaborative functions, not recording or documenting functions: Patty
Provide support for moving to a participatory evaluation model that recognizes that client ownership of outcomes is the single most important function of mental health intervention, and that acknowledges that validity and reliability may not be as important as changes in client behavior - Rick

2. Design On site meeting at each demonstration site for:
information gathering
case staffing observation
training on evaluation issues
conducting focus groups for process data collection
Goal setting process
Council member identification of barriers encountered in reaching the goals.
Critical case review
Goal setting process with the three demonstration sites to identify goals they were trying to accomplish as a community council
Council members were surveyed for an initial goal list, goals were ranked and goals that were most frequently rated were top priorities.
As a parallel process members were asked to identify barriers they were encountering in reaching these goals.
The critical case review methodology was adopted to collect information around the issues faced by community councils. The critical case review process analyzes how individual councils function around one case they identify as having a successful outcome and one case in which they could not identify a successful outcome.
A group interview process captured personal memory and case files to describe the sequence of one case in each category and used the differences to understand how various issues influenced practice.

3. Role of Outcomes in a System of Care
What are outcomes?
Changes in adaptive behaviors
Basic assumption: what people do as a result of professional interaction is the best measure of professional effectiveness
What’s the difference between outcomes and something else?
Outputs: services/activities provided to clients to help (them) improve their condition
Basic assumption: the amount of services is the best measure of professional effectiveness.

4. Kinds of Outcomes Representative Outcomes
Refers to the health or capacity of a system around specific issues or mandates
Represents how an entity or person is doing on a global scale.
Refers to how individual outcomes relate to a larger system.
Serves to help an entity or person assess how they are doing on a global scale.
Examples include:
State average scores on standardized academic achievement tests
Decreases in CPS referrals or re-referrals
Decreases in CAFAS scores

5. Kinds of Outcomes Functional Outcomes
Refers to adaptive behaviors of a person or a program or system as they strive to reach their goals; what people actually do to reach their goals.
Serves to enhance the self-reliance of persons in relation to the goals in question
Example:
Client C visited 3 agencies for the first time and received food and clothing for a week
Social Worker B collaborated with the school counselor to create an early referral form
Student K stayed in school and completed his past assignments

6. Participatory Evaluation: Using Client Progress to Help Clients Progress
Functional Outcomes
What people do as a result of participating in a relationship
Take actions
Change attitudes
Increase performance
Become more able
Representative Outcomes
What happens as a result of aggregated functional outcomes
Increased classroom performance
Increased grades or attendance
Positive changes in CAFAS scores

7. System of Care for Children
Set of shared values and principles
Focus on strengths of family; not on diagnosis of the child
Maintain child in their own home, in their own school, and in their own community
Service array planned with family to meet family identified needs
Non-traditional
Traditional

8. Post Treatment Data
School attendance/conduct for first full quarter after completion of service
# of failing school academic core subject grades out of total possible
CAFAS score
Post Juvenile Probation Assessment score
Disposition of case
Approximate # of hours spent by council in chambers
Approximate # of hours spent by council members in case management
Number of council members involved in direct service on case
Family satisfaction survey results
School attendance/conduct
office referrals
full days of in-school suspension
full days of out-of-school suspension
Disposition of case
out of community placement
out of state placement
in-patient placement

9. Nothing About Us Without Us Parent perspective on functional outcomes
First and foremost, we seek safety
That our children are insured a quality of life as happy productive members of community as measured by:
sharing a ceremony or celebration,
being “in this together”
making it through a school suspension while adjusting to a new medication
We need strengths acknowledged, resources valued and our children treated with love and respect
CrossBear, S. (Spring/Summer 2002) Nothing about us without us, Data Matters, National Technical Assistance Center for Children’s Mental Health, Georgetown University Child Development Center
Parents need to know that those who provide care for their children, first and foremost do no harm.
The first outcome parents seek is that their children live.
Quality of life insures that children have the ability to be happy and productive members of the community
What is the measure of that?
Can be measured by success in school and progress in educational plans. Measured by children’s presence in the juvenile justice system, out of home placements and community-based services.
But it can also be measured by nights at the supper table without conflict.
Measured by family indicators:
Community councils, in order to assist family to make decisions about a case, first need:
Parent participation
Identified strengths of the family and child from multiple arenas in multiple situations
Clearly understand the problem situation from the family’s situation
Support from the agency bureaucracy to support “whatever it takes” in assisting families
Understanding of the limits/constraints and possibilities of community partners
Respect for families and each other in the decision-making process
Full partnership participation

10. Generalized Service Model
Needs Assessment & Diagnosis
Client Presenting Issues
Tx plan and Clinical goals
Services Provided
Periodic Assessment & Service Plan Review
Demonstrates reliance on representational outcomes and on CAFAS and CBCL
Case Closure when Clinical goals met

11. Developing Community Council Model
Council Goals
Family Centered Problem Solving
Client Referral
Tx Goals & Service Plan
Interventions & Assessments
Collaborative Reviews
Case Closure when Council and Family Goals met

12. Conclusion
When the probable results of inaction, and the lack of results of current actions, form the most important reasons for referral to a program, then a conclusion about standardized data can be made.
Any data that are used in predictive sense to direct what the parent should do are data that are seen as harmful to the process of getting the family to engage as partners with the council in setting out a treatment plan for their child.

13. The Reality of Collaboration
Council members defensive because of agency history
Lack of experience with engaging families in a process where the family leads interventions
Lack of understanding of the participation agencies regulations and guidelines
Lack of trust – in the family and in the professional process
Inability to get key partners to the table – not full participation
Lack of available concrete services/programs
Funding constraints

14. The Role of Advocacy
A key measure of council effectiveness is the willingness and ability of council members to integrate council client concerns into their daily agency routines
Advocacy looks like:
making connections with other professionals as council members go about their daily routines
Intervening in one’s own agency to help clients gain access to services
Being supportive by other council members in challenging status quo treatment options
Advocacy does not look like being held accountable for collecting data to show client progress

15. Greatest Challenge: Supporting care providers as they move from an expert, service provider orientation to a facilitating, partnering orientation

16. Recommendations
Ensure that system of care extends into the bureaucracy
Work with local sites to establish data collection routines that celebrate client actions
Provide evaluation support for local councils so volunteer hours are spent in collaborative functions, not recording or documentation functions
Train to a participatory evaluation model that recognizes that client ownership of outcomes may be the single most important function of a mental health intervention
Learn to acknowledge that validity and reliability may be obstructionist constructs for support changes in client behavior
Ensure that system of care extends into the bureaucracy – Patty
Work with local sites to establish data collection routines that celebrate their actions, rather than their reporting functions – Rick
Provide evaluation support for local collaboratives so that volunteer hours are spent where members are most effective – in collaborative functions, not recording or documenting functions (use of data collection and management services in the form of a full-time person) Patty
Provide support for moving to a participatory evaluation model that recognizes that client ownership of outcomes is the single most important function of mental health intervention, and that acknowledges validity and reliability may not be as important as changes in client behavior - Rick

17. Functional Outcome Evaluation
Client Progress
Professional Client set
Goals Goals

18. Bibliography
Burns, B., & Goldman, S., (1998) Promising practices in wraparound for children with serious emotional disturbance and their families. Washington DC: Georgetown University Child Development Center, CASSP technical Assistance Center. Substance Abuse and Mental Health Services Administration. U.S. Department of Health and Human Services
CrossBear, S. (Spring/Summer 2002) Nothing about us without us, Data Matters, National Technical Assistance Center for Children’s Mental Health, Georgetown University Child Development Center
Friesen, B., & Huff, B., (1996) Family perspectives on systems of care. In B. Stroul (Ed.) Children’s Mental Health. Creating Systems of Care in A Changing Society. Baltimore, MD: Brookes Publishing.
Lourie, I.S., & Davis, C. (1999) A Needs Assessment of Idaho’s Children With Serious Emotional Disturbances and Their Families.
Pines, S. (2002) Building Systems of Care: A Primer. National Technology Assistance Center for Children’s Mental Health
State of Idaho (2002). Building on Each Other’s Strengths. Child Mental Health Initiative Grant, GFA: SM CFDA