Presentation on theme: "Richard Phillips, Ph.D. Director of Research and Evaluation,"— Presentation transcript:
1 The Role of Client Outcomes in Evaluating a System of Care Model of Mental Health Delivery Richard Phillips, Ph.D.Director of Research and Evaluation,Idaho Child Welfare Research & Training CenterEastern Washington UniversityPatty Gregory, LCSWDirectorOutline for presentationIntroduction to outcomes: different types – RickIntroduction to system of care for SED children: the child/family centered model (slide #4) – PattyType of data required by federal/state guidelines (slide #5) – RickType of data useful to partners to make decisions about a case (slide #6) – PattyData/themes collected from the sites: (slide #7 – notes pagesFamily centered practice and conflict with standardized data: RickThe reality of collaboration: PattyThe role of advocacy: RickRecommendations: (slide #11 – notes pages)Ensure that system of care values and principles extends into the bureaucracy – PattyWork with local sites to establish data collection routines that celebrate their actions, rather than their reporting functions: RickProvide evaluation support for local collaboratives so that volunteer hours are spent where the members are most effective – in collaborative functions, not recording or documenting functions: PattyProvide support for moving to a participatory evaluation model that recognizes that client ownership of outcomes is the single most important function of mental health intervention, and that acknowledges that validity and reliability may not be as important as changes in client behavior - Rick
2 Design On site meeting at each demonstration site for: information gatheringcase staffing observationtraining on evaluation issuesconducting focus groups for process data collectionGoal setting processCouncil member identification of barriers encountered in reaching the goals.Critical case reviewGoal setting process with the three demonstration sites to identify goals they were trying to accomplish as a community councilCouncil members were surveyed for an initial goal list, goals were ranked and goals that were most frequently rated were top priorities.As a parallel process members were asked to identify barriers they were encountering in reaching these goals.The critical case review methodology was adopted to collect information around the issues faced by community councils. The critical case review process analyzes how individual councils function around one case they identify as having a successful outcome and one case in which they could not identify a successful outcome.A group interview process captured personal memory and case files to describe the sequence of one case in each category and used the differences to understand how various issues influenced practice.
3 Role of Outcomes in a System of Care What are outcomes?Changes in adaptive behaviorsBasic assumption: what people do as a result of professional interaction is the best measure of professional effectivenessWhat’s the difference between outcomes and something else?Outputs: services/activities provided to clients to help (them) improve their conditionBasic assumption: the amount of services is the best measure of professional effectiveness.
4 Kinds of Outcomes Representative Outcomes Refers to the health or capacity of a system around specific issues or mandatesRepresents how an entity or person is doing on a global scale.Refers to how individual outcomes relate to a larger system.Serves to help an entity or person assess how they are doing on a global scale.Examples include:State average scores on standardized academic achievement testsDecreases in CPS referrals or re-referralsDecreases in CAFAS scores
5 Kinds of Outcomes Functional Outcomes Refers to adaptive behaviors of a person or a program or system as they strive to reach their goals; what people actually do to reach their goals.Serves to enhance the self-reliance of persons in relation to the goals in questionExample:Client C visited 3 agencies for the first time and received food and clothing for a weekSocial Worker B collaborated with the school counselor to create an early referral formStudent K stayed in school and completed his past assignments
6 Participatory Evaluation: Using Client Progress to Help Clients Progress Functional OutcomesWhat people do as a result of participating in a relationshipTake actionsChange attitudesIncrease performanceBecome more ableRepresentative OutcomesWhat happens as a result of aggregated functional outcomesIncreased classroom performanceIncreased grades or attendancePositive changes in CAFAS scores
7 System of Care for Children Set of shared values and principlesFocus on strengths of family; not on diagnosis of the childMaintain child in their own home, in their own school, and in their own communityService array planned with family to meet family identified needsNon-traditionalTraditional
8 Post Treatment DataSchool attendance/conduct for first full quarter after completion of service# of failing school academic core subject grades out of total possibleCAFAS scorePost Juvenile Probation Assessment scoreDisposition of caseApproximate # of hours spent by council in chambersApproximate # of hours spent by council members in case managementNumber of council members involved in direct service on caseFamily satisfaction survey resultsSchool attendance/conductoffice referralsfull days of in-school suspensionfull days of out-of-school suspensionDisposition of caseout of community placementout of state placementin-patient placement
9 Nothing About Us Without Us Parent perspective on functional outcomes First and foremost, we seek safetyThat our children are insured a quality of life as happy productive members of community as measured by:sharing a ceremony or celebration,being “in this together”making it through a school suspension while adjusting to a new medicationWe need strengths acknowledged, resources valued and our children treated with love and respectCrossBear, S. (Spring/Summer 2002) Nothing about us without us, Data Matters, National Technical Assistance Center for Children’s Mental Health, Georgetown University Child Development CenterParents need to know that those who provide care for their children, first and foremost do no harm.The first outcome parents seek is that their children live.Quality of life insures that children have the ability to be happy and productive members of the communityWhat is the measure of that?Can be measured by success in school and progress in educational plans. Measured by children’s presence in the juvenile justice system, out of home placements and community-based services.But it can also be measured by nights at the supper table without conflict.Measured by family indicators:Community councils, in order to assist family to make decisions about a case, first need:Parent participationIdentified strengths of the family and child from multiple arenas in multiple situationsClearly understand the problem situation from the family’s situationSupport from the agency bureaucracy to support “whatever it takes” in assisting familiesUnderstanding of the limits/constraints and possibilities of community partnersRespect for families and each other in the decision-making processFull partnership participation
10 Generalized Service Model Needs Assessment & DiagnosisClient Presenting IssuesTx plan and Clinical goalsServices ProvidedPeriodic Assessment & Service Plan ReviewDemonstrates reliance on representational outcomes and on CAFAS and CBCLCase Closure when Clinical goals met
11 Developing Community Council Model Council GoalsFamily Centered Problem SolvingClient ReferralTx Goals & Service PlanInterventions & AssessmentsCollaborative ReviewsCase Closure when Council and Family Goals met
12 ConclusionWhen the probable results of inaction, and the lack of results of current actions, form the most important reasons for referral to a program, then a conclusion about standardized data can be made.Any data that are used in predictive sense to direct what the parent should do are data that are seen as harmful to the process of getting the family to engage as partners with the council in setting out a treatment plan for their child.
13 The Reality of Collaboration Council members defensive because of agency historyLack of experience with engaging families in a process where the family leads interventionsLack of understanding of the participation agencies regulations and guidelinesLack of trust – in the family and in the professional processInability to get key partners to the table – not full participationLack of available concrete services/programsFunding constraints
14 The Role of AdvocacyA key measure of council effectiveness is the willingness and ability of council members to integrate council client concerns into their daily agency routinesAdvocacy looks like:making connections with other professionals as council members go about their daily routinesIntervening in one’s own agency to help clients gain access to servicesBeing supportive by other council members in challenging status quo treatment optionsAdvocacy does not look like being held accountable for collecting data to show client progress
15 Greatest Challenge: Supporting care providers as they move from an expert, service provider orientation to a facilitating, partnering orientation
16 RecommendationsEnsure that system of care extends into the bureaucracyWork with local sites to establish data collection routines that celebrate client actionsProvide evaluation support for local councils so volunteer hours are spent in collaborative functions, not recording or documentation functionsTrain to a participatory evaluation model that recognizes that client ownership of outcomes may be the single most important function of a mental health interventionLearn to acknowledge that validity and reliability may be obstructionist constructs for support changes in client behaviorEnsure that system of care extends into the bureaucracy – PattyWork with local sites to establish data collection routines that celebrate their actions, rather than their reporting functions – RickProvide evaluation support for local collaboratives so that volunteer hours are spent where members are most effective – in collaborative functions, not recording or documenting functions (use of data collection and management services in the form of a full-time person) PattyProvide support for moving to a participatory evaluation model that recognizes that client ownership of outcomes is the single most important function of mental health intervention, and that acknowledges validity and reliability may not be as important as changes in client behavior - Rick
18 BibliographyBurns, B., & Goldman, S., (1998) Promising practices in wraparound for children with serious emotional disturbance and their families. Washington DC: Georgetown University Child Development Center, CASSP technical Assistance Center. Substance Abuse and Mental Health Services Administration. U.S. Department of Health and Human ServicesCrossBear, S. (Spring/Summer 2002) Nothing about us without us, Data Matters, National Technical Assistance Center for Children’s Mental Health, Georgetown University Child Development CenterFriesen, B., & Huff, B., (1996) Family perspectives on systems of care. In B. Stroul (Ed.) Children’s Mental Health. Creating Systems of Care in A Changing Society. Baltimore, MD: Brookes Publishing.Lourie, I.S., & Davis, C. (1999) A Needs Assessment of Idaho’s Children With Serious Emotional Disturbances and Their Families.Pines, S. (2002) Building Systems of Care: A Primer. National Technology Assistance Center for Children’s Mental HealthState of Idaho (2002). Building on Each Other’s Strengths. Child Mental Health Initiative Grant, GFA: SM CFDA