1. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley The Child and Family Services Review Composite Scores: A “Great Start” Barbara Needell, MSW, PhD Center for Social Services Research University of California at Berkeley *Schuerman, J., & Needell, B. (2009). The Child and Family Services Review Composite Scores: Accountability off the Track. Chicago: Chapin Hall at the University of Chicago http://www.chapinhall.org/research/report/child-and-family-services-review-composite-scores-accountability-track The Performance Indicators Project is a collaboration of the California Department of Social Services and the University of California at Berkeley, and is supported by the California Department of Social Services and the Stuart Foundation

2. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley CFSR Overview Review and evaluation of state foster care and adoption services federally funded by Titles IV-B and IV-E of the Social Security Act Carried out by the Children’s Bureau of the Administration on Children and Families (ACF) GOAL: Evidence-based accountability and performance improvement of state child welfare systems

3. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley The CFSR Process States held responsible for meeting “substantial conformity” on safety, permanency, and well-being outcomes and systemic factors The Onsite Review takes place in three sites within the state, and involves a case review of a limited number of cases (65 in 2 nd round) and interviews with CW stakeholders Using the State Data Profile, states develop a Statewide Assessment, a report on child and family outcomes and systemic factors, and submits to the ACF 60 days prior to the Onsite Review ACF sends each state a State Data Profile based on data from the most recent three years with information about the state’s performance relative to national standards and other safety and permanency data States submit data to two systems: the National Child Abuse and Neglect Data System (NCANDS) and the Adoption and Foster Care Analysis and Reporting System (AFCARS)

4. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Flaws in Performance or Flawed Process? States not in “substantial conformity” must develop a Program Improvement Plan (PIP) To date, no state has passed the CFSR at the initial stage – every state has had to develop a PIP Financial penalties may be incurred if the PIP is not successfully completed But there are weaknesses in the CFSR process (we will focus on the National Standards and composite scores)

5. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley State Demographic Variation

6. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Systemic State Differences in Caseload Inclusion Criteria State Child Welfare Caseload Definition Population: Mentally ill and developmentally disabled; juvenile corrections Placement and Status Changes: Trial home visits, runaways, respite care, etc. Payment by State Changes in State Laws/Regulations

7. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Data Quality Weaknesses Database (AFCARS) not designed to measure longitudinal performance, but ACF attempts to stitch together files to construct a pseudo-longitudinal database Prone to record linkage errors and still does not include all data needed to track performance over time Data is not of adequate quality to warrant the severity of consequences

8. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Practice and Policy Conflicts between Measures Placement Avoidance & Rapid Reunification Placement Stability

9. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Equal Count/Weighting of States Despite Differences in Child Population Size CA and VT are weighted equally, although there are 60x as many children served in CA foster care as in VT foster care CA VT

10. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Statistical Method Used is Complicated and Not Superior to a Simpler Approach ACF uses principal components analysis (PCA), which is often used to simplify a large number of similar variables This statistical manipulation does not necessarily produce a better result than a more straightforward approach, such as a simple average Because the composite measure is an abstraction, it is not clear what actions should be taken to improve scores and may even encourage gaming of the system

11. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Arbitrary and statistically inappropriate decisions in using PCA Varimax Rotation Averaging Principal Components

12. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Arbitrary National Standard and Adjustment National standard set at 75 th percentile with no clear justification 75 th percentile is actually that for the normal curve, a theoretical statistical distribution State outcome data are not necessarily normally distributed

13. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Flawed Method to Develop Minimum Improvement CFSR2 requires a percentage improvement by states, based on their baseline performance in a given year Amount of required improvement determined using a complex calculation involving sampling error, which is not justifiable statistically The national standards largely use “p”-- the percent of time an event happens, which should not be used in this context

14. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Example

15. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Summary/Conclusion Flaws and issues with the national standards and composite scores are substantial enough to take a fresh look at “substantial conformity” Note: We have not discussed other areas of the CFSR at all today—the State Data Profile, the Case Record Reviews, the Stakeholder Interviews, etc—and all of these may also be in need of review/reform Not clear that reasonable outcome indicators can be developed from the currently available AFCARS

16. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Summary/Conclusion A fully longitudinal national foster care database is necessary to adequately measure performance (e.g., State Data Center Foster Care Database, California Child Welfare Performance Indicators Project Analytic Assistance for Child Welfare…in North Carolina.) We have the ability NOW to reform the CFSR process, including the performance measures that come from administrative data The development of a functional CFSR process can encourage and foster true improvement in state child welfare systems

17. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley CSSR.BERKELEY.EDU/UCB_CHILDWELFARE Barbara Needell bneedell@berkeley.edu 510 290 6334 Presentation Developed by Christine Wei-Mien Lou