1. Using a Summary Score Approach to Analyze Performance Measures Over Time
Charlie Ferguson, Ph.D.
San Jose State University School of Social Work
Demonstration Session 480 at the American Evaluation Association Annual Meeting
Friday, October 18, 2013, 11:00 a.m. to 11:45 a.m.

2. Overview of the Workshop
Introduction
Overview of California’s Title IV-E Child Welfare Waiver Demonstration Capped Allocation Project (CAP)
Overview of the CAP Evaluation
The Issues
The Solution
Questions

3. Acknowledgements
Administration for Children and Families, Children’s Bureau
California Department of Social Services
Alameda County Department of Children and Family Services
Alameda County Probation Department
Los Angeles County Department of Children and Family Services
Los Angeles County Probation Department

4. Overview of the CAP What was the CAP?
A five-year Title IV-E Child Welfare Waiver Demonstration Project sponsored by the Children’s Bureau.
Who participated in the CAP?
California Department of Social Services
Alameda County Department of Children and Family Services
Alameda County Probation Department—Juvenile Probation
Los Angeles County Department of Children and Family Services
Los Angeles County Probation Department—Juvenile Probation

5. Overview of the CAP California’s Child Welfare System:
Shared governance structure
CDSS: monitoring, oversight, and support
County DCFS: day-to-day administration
California’s Probation System:
County operated, adult and juveniles
Portion of juvenile population in out-of-home placements
Departments seek Title IV-E reimbursement for eligible youth

6. Overview of the CAP What was the purpose of the CAP?
To assist the child welfare and probation systems in developing and implementing alternative services to foster care to bring about better outcomes for children and families.
What was the strategy of the CAP?
Ending the Title IV-E entitlement and capping the funding allocation for five years in exchange for eliminating eligibility restrictions, gaining fiscal flexibility, and the opportunity to generate and retain savings.

7. Overview of the CAP What were the goals of the CAP?
The goals of the Child and Family Services Review process:
(a) improve the array of services for children and families and engage families through a more individualized approach that emphasizes family involvement;
(b) increase child safety without an over-reliance on out-of-home care;
(c) improve permanency outcomes; and
(d) improve child and family well-being.

8. Overview of the CAP: Theory of Change
Suspension of the entitlement
Limit on the allocation
Lifting of the eligibility restriction
Flexibility in spending
Beneficial methodology
to determine capped allocation
Growth rate in portions
of allocation
Five year budget horizon
Opportunity to generate savings
Opportunity to reinvest savings
Opportunity to retain savings
“Extra” allocation
resulting from
the methodology
used to determine
the CAP
Reorientation
of the
system to:
Prevention
Early Intervention
Permanency
Placement in the
most family-like
setting possible
Better
outcomes
for
children &
families
Better
systems-
level
outcomes
Reinvestment
Savings

9. Overview of the Evaluation
Primary Purpose:
To determine whether and how changes in the funding structure for foster care affects the functioning of county child welfare systems and relevant probation systems.
Secondary Purpose:
To assess outcomes for dependent and delinquent children and their families before the CAP began and then during its implementation.

10. Overview of the Evaluation
The CDSS had specific requirements for the evaluation:
No random assignment and (ultimately) no comparison counties.
Limited intrusion on county departments for data collection.
Use publicly available data from the University of California’s CA Child Welfare Indicators Project (data aggregated to the county department level).

11. Overview of the Evaluation
The rationales behind the CDSS requirements:
A general sentiment against the use of random assignment in any situation and not feasible in this instance.
Concerns about identifying appropriate comparison counties and imposition regarding data collection.
Limited political will or desire to strengthen the evaluation.
A desire for consistency in outcome measures across large projects and child welfare reform efforts.

12. Overview of the Evaluation
A separate evaluation conducted in each of the four departments.
Each evaluation consisted of three components:
Process Study
Fiscal Study
Outcome Study

13. Overview of the Evaluation
Outcome Study
Interrupted Time-Series Design: Trend Analysis
Base: July 1, 2002 – June 30, 2007
Cap: July 1, 2007 – June 30, 2012
Participation Rates (four rates)
Child Safety (five indicators)
Exits to Permanency (five indicators)
Placement Stability (three indicators)
Appropriateness and Restrictiveness (two indicators)
Data Source—CA Child Welfare Indicators Project

14. The Issues
A need to reduce the complexity: (i.e., too many line charts)
Multiple variables with multiple indicators across four distinct outcome evaluations.
A change in the capacity of the CA Child Welfare Indicators Project over time allowed for the use of “context” counties.
A desire to increase the rigor of the graphical analyses.
Moving beyond a narrative description of the trend to include multiple vantage points.
A need to make the results and conclusions readily accessible.
Stakeholders need “simplicity” in terms of understanding and use.

15. The Solution The development and use of a Summary Score in the
Analysis of Indicators
Analysis of Outcomes
Overall Assessment

16. The Solution The development and use of a Summary Score for the
Analysis of Indicators
Analysis of Outcomes
Overall Assessment

17. The Solution Analysis of Indicators: Baseline and CAP
Slope—generated in Excel using scatter plot and linear plot with equation function
Celeration—Acceleration, Deceleration, Horizontal Stability
Trend—Yes (trend in the desired direction); No (trend not in desired direction)
National Standard/Goal—Count of Achieved / Possible

18. The Solution
Analysis of Indicators: Comparison between Baseline and CAP
Magnitude—Increase or Decrease between the end of Base and start of CAP (% pts)
Change in Trend between Base and CAP—Yes; No; Continuing
Change in Trend in Desired Direction—Yes; No; Yes Continuing; No Continuing

19. The Solution
Analysis of Indicators: Comparison between Baseline and CAP
Summary Score (inclusive of Base and CAP information)
3 possible pts.; 1 pt. for each of the following if true
(a) A shift in Magnitude in the desired direction
(b) At/Exceeded National Standard/Goal at least once during CAP
(c) Change in Trend in desired direction or continuing in desired
direction

20. The Solution Analysis of Outcomes:
Summary Score—Count & Percent for each outcome variable
Child Safety (comprised of 5 indicators): n & %
Exits to Permanency (comprised of 5 indicators): n & %
Placement Stability (comprised of 3 indicators): n & %

21. The Solution Overall Assessment:
Summary Score—Count & Percent for all outcome variables
Trend Lines—Count & Percent for Base vs. CAP
National Standard/Goal—Count & Percent for Base vs. CAP

22. The Solution Cons Graphical Analyses (i.e., not statistical analyses)
Still somewhat subjective and open to interpretation
Pros
Reduced the complexity of multiple graphical analyses
Relatively simple process
Allowed for analyses at three levels

23. The Solution
Questions?

24. San Jose State University
Thanks!
Contact Information:
Charlie Ferguson, Ph.D.
School of Social Work
San Jose State University
One Washington Square
San Jose, California