California’s Child Welfare Outcomes & Accountability System: Using Performance Measures to Encourage Improvement Barbara Needell, MSW, PhD Center for Social Services Research University of California at Berkeley The Performance Indicators Project is a collaboration of the California Department of Social Services and the University of California at Berkeley, and is supported by the California Department of Social Services and the Stuart Foundation 1

Child Welfare is a System Rate of Referrals/ Substantiated Referrals Home-Based Services vs. Out of Home Care Reentry to Care Permanency Through Reunification, Adoption, or Guardianship Counterbalanced Indicators of System Performance Use of Least Restrictive Form of Care Rule #1 applies to your Toolkit everybit as much as it does our System in CA—even the best measures can be misleading if they are not used as a part of the entire array of measures that capture performance from beginning to end. Example: reunification rates and their interdependence with both entry rates and reentry rates Length of Stay Positive Attachments to Family, Friends, and Neighbors Stability of Care SOURCE: Usher, C.L., Wildfire, J.B., Gogan, H.C. & Brown, E.L. (2002). Measuring Outcomes in Child Welfare. Chapel Hill:  Jordan Institute for Families,

Background… California’s Child Welfare System Improvement and Accountability Act Became law (AB 636) in 2001 Went into effect in January 2004 with publication of first quarterly report Began with county self assessments and System Improvement Plans (SIPS) that identified key challenges and strengths, based on public data Currently includes all federal measures (17 in CFSR2), augmented by state measures that capture important aspects of performance Participation rates (referrals, substantiations, entries, in care) Sibling placements Key process measures (e.g., child visits, time to investigation) Least restrictive placements Measures added over time We may not like the CFSR2 measures, but they are current mandates with potential penalties attached.

CDSS / CSSR Collaboration Longstanding Interagency Agreement Funding from CDSS* and Stuart Foundation Quarterly Data Reports for CA and counties Dynamic excel documents with data, charts, etc. Data Publicly Available: cssr.berkeley.edu/ucb_childwelfare Site organized around federal measures (with extensions) Full data refresh quarterly Ad hoc data tabulations (filtering capacity) Composite viewer/composite planner * A portion of CDSS funding comes from AOC for work especially relevant to the courts

*Updated with Q4_08 Data

Example… Measure C1.3: Reunification within 12 months (entry cohort) Federal Base Of all children entering foster care for the first time in the 6-month period who remained in foster care for 8 days or longer, what percent were discharged from foster care to reunification in less than 12 months from the date of latest removal from home? Site Extensions Agency, Days in Care, Time Period Views, Episode Count Age, Ethnicity, Gender, Placement Type, Removal Reason Exit Status at 3m, 6m, 12m…120m Subgroup filtering, Count/Percent, Excel Export Strength based approach to CFSR2 measures—we use them but then enhance/expand

*Dates available for Federal (and all) reports despite the fact that this measure was added with CFSR2

*Federal Report

*STSG Report

*Exit status dates available for 2007 entry cohort

**Exit status dates available for 1998 entry cohort

**Filtering options

*STSG Report (%)

**Shows count view

January 2004-July 2009 California CWS Outcomes System: AB636 Measures, % IMPROVEMENT (+) or (–) indicates direction of desired change Updated with Q4_08 Data We believe that publicly available data has contributed to our improvement Decline in Performance Improvement in Performance

January 2004-July 2009 California CWS Outcomes System: Federal Measures, % IMPROVEMENT (+) or (–) indicates direction of desired change 110.4% Updated with Q4_08 Data Decline in Performance Improvement in Performance 18

Federal Standard/Goal July 2009 California CWS Outcomes System: Performance Relative to Federal Standard/Goal Federal Standard/Goal 100% Updated with Q4_08 Data 19

Public Data: Putting it All Out There PROS: Greater performance accountability Community awareness and involvement, encourages public-private partnerships Ability to track improvement over time, identify areas where programmatic adjustments are needed - County/County and County/State collaboration CONS: Potential for misuse, misinterpretation, and misrepresentation Available to those with agendas or looking to create a sensational headline Misunderstood data can lead to the wrong policy decisions “Torture numbers, and they’ll confess to anything” Gregg Easterbrook The Pros and Cons of Data made public: Pros Greater performance accountability (Numbers are there for all to see) Community awareness and involvement, encourages public-private partnerships (Funders can be shown “results”) Allows state and counties to track improvements over time and to identify areas where programmatic adjustments are needed (Counties can see the big performance picture) County/County and County/State collaboration (County A is doing well in X by doing Z. Maybe that would work in our county as well…) Cons: Potential for misuse, misinterpretation, and misrepresentation (e.g., inappropriate county comparisons, treating small insignificant differences as meaningful, taking statistics out of context) Available to those with agendas or looking to create a sensational headline (Accurate statistics can be used to bolster inaccurate arguments (or create attention-grabbing headlines)) “Torture numbers, and they’ll confess to anything” (Gregg Easterbrook) 20

Child Welfare County Data Profiles: Court Data Reports Customized excel reports for judges designed by AOC and CSSR staff Data restricted to court dependent children when possible County/State and County side by side table and graph comparisons % change between timeframes Presents data from child welfare and court sources (filings) Provides links to source reports on CWS/CMS reports website to allow users to further explore data Most data is publicly available—arranged in a way that will be most useful for the courts Ability to “drill down” to subsets may be necessary to understand need and target resources

CSSR.BERKELEY.EDU/UCB_CHILDWELFARE Barbara Needell bneedell@berkeley.edu 510-290-6334 CSSR.BERKELEY.EDU/UCB_CHILDWELFARE Needell, B., Webster, D., Armijo, M., Lee, S., Dawson, W., Magruder, J., Exel, M., Glasser, T., Williams, D., Zimmerman, K., Simon, V., Putnam-Hornstein, E., Frerer, K., Cuccaro-Alamin, S., Winn, A., Lou, C., & Peng, C. (2009). Child Welfare Services Reports for California. Retrieved April 1, 2009, from University of California at Berkeley Center for Social Services Research website. URL: <http://cssr.berkeley.edu/ucb_childwelfare> Presentation Developed by Emily Putnam-Hornstein and Christine Wei-Mien Lou 22