1. FosteringCourtImprovement.org Data: Telling a Story … A Walk Through Meaningful Data Analysis Judge Nancy Salyers Judicial Executive and Co-Founder, Fostering Court Improvement Co-Director, Fostering Results Andy Barclay Statistician and Co-Founder, Fostering Court Improvement Barton Child Law and Policy Clinic, Emory University School of Law Presented To: Fostering Court Improvement, A Data Workshop for Decision Makers December 7, 2006

2. FosteringCourtImprovement.org2 Overview How Knowledge of the Past and Present can Help you Shape the Future Data, Data, Data Isn’t Yada, Yada, Yada Court Performance Measures AFCARS The Easy Win vs. Sea Change Getting Real, Getting Local, Getting Context Results that Matter to Kids & Families

3. FosteringCourtImprovement.org3 Shaping the Future Why Data, Data, Data Isn’t Yada, Yada, Yada

4. FosteringCourtImprovement.org4 Lessons learned or missteps repeated? Lunchroom lore or documented results? Shaping the Future

5. FosteringCourtImprovement.org5 Shaping the Future “They always say that time changes things, but you actually have to change them yourself.” Andy Warhol

6. FosteringCourtImprovement.org6 Do you know where your caseload has been? Do you know where it is? Do you know where it is going? Shaping the Future

7. FosteringCourtImprovement.org7 Shaping the Future Am I on the road to success or on a “frolic and detour”? (Only the data know.)

8. FosteringCourtImprovement.org8 Shaping the Future “Georgia Judge leads stampede out of homes.”

9. FosteringCourtImprovement.org9 Shaping the Future “Illinois Judge ‘doesn’t have time’ to return children to parents.”

10. FosteringCourtImprovement.org10 Shaping the Future Take off the blinders. Or Load and Learn.

11. FosteringCourtImprovement.org11 Shaping the Future Where to Start?

12. FosteringCourtImprovement.org12 Court Performance Measures The three leading legal/ judicial organizations, with a grant from the Packard Foundation, have created and pilot-tested court performance measures that complement those used in the CFSR process so that agencies and courts can collaborate in the measurement of their performance and tracking of their own progress in improving safety, permanence, and timeliness for children coming before them.

13. FosteringCourtImprovement.org13 Court Performance Measures (Building a Better Court) Reinforce current federal reform initiatives, including CFSR and PIP’s Helps to identify areas of court reform essential to better outcomes Assist the CIP by providing courts with tools to engage in systematic process of court performance measurement

14. FosteringCourtImprovement.org14 Court Performance Measures “Performance measures are the yardstick by which courts and child welfare agencies can measure the success they are achieving and the progress they are making.” Building a Better Court

15. FosteringCourtImprovement.org15 Court Performance Measures Safety: Are there new allegations while in foster care? Permanency: What was the exit from foster care and how many came back? Due Process: Is there notice and representation? Timeliness: How long between hearings?

16. FosteringCourtImprovement.org16 Already Measurable and Measured (AFCARS) Safety Permanency Timeliness

17. FosteringCourtImprovement.org17 AFCARS Adoption and Foster Care Analysis and Reporting System

18. FosteringCourtImprovement.org18 Uses of AFCARS Award Adoption Incentive bonuses Allot Chafee Independence $$$ Produce annual reports to Congress on child welfare outcomes Conduct IV-E eligibility reviews Assess state’s compliance with federal CFSR performance standards

19. FosteringCourtImprovement.org19 Platform for Collaboration Longitudinal AFCARS provides a platform for rolling out the due process & timeliness measures recommended by Building a Better Court Promotes working partnership between the courts, child welfare agencies and universities, Provides 9 of the 24 measures defined in Building a Better Court, and 9 of the 30 in ToolKit.

20. FosteringCourtImprovement.org20 Fostering Court Improvement What gets measured gets done, If You know it’s being measured.

21. FosteringCourtImprovement.org21 Stories from Medicine : An Easy Win Diagnostic Imaging (Cardiac PET/SPECT) Pre-Screen Very simple questionnaire Predictive Value of the Test Surgery: y/n Angioplasty: y/n Easy sell to cardiologists Very easy sell to insurance companies Bayesian Pre-Screening Tool

22. FosteringCourtImprovement.org22 Stories from Medicine : A Sea Change Major resistance, even when shown computer was right more often, lower mortality Over time (3 years) older docs modified their risk tolerance, less invasive, better outcomes, learned from computer Tripled business, more patients living longer, much lower costs... Automated Reading of Scans

23. FosteringCourtImprovement.org23 Court Stakeholders Have a Right to be Skeptical In medicine, I see examples of better outcomes at lower cost quite often. Have yet to see one in child welfare. Many claims, little evidence Judges, attorneys, CASAs, and caseworkers have good reason to be skeptical: Complex Business Spectacular IT Failures Terabytes of Unused Data

24. FosteringCourtImprovement.org24 Get Real Data reports are only meaningful to you in the context of your job, your reality. We (FCI) have been successful * using data to develop a shared context among local court stakeholder groups. Very different jobs, very different realities, but we need to have some common understandings to do successful * work together (and occasionally alter some realities). * successfully improve an outcome

25. FosteringCourtImprovement.org25 Diverse Realities: Dependency Court Caseloads 3,140 US counties + DC 70% of counties remove less than 1 child per week 50% less than 1 per 2 weeks 7% of counties (234) average more than 1 child removal per business day

26. FosteringCourtImprovement.org26 Get Local Much as I’d like to, we probably can’t do this with just a website. our web reports are not for a general audience. County by county … Judges convene Georgia’s CIP buys lunch I walk through a Powerpoint of charts & numbers from the website.

27. FosteringCourtImprovement.org27 Broad Topic Areas of Court Stakeholder Meetings Removal Rates Reasons for Removal Kids in Care Group Settings IV-E Reimbursement Kids Headed for Adoption Reunification Rate Time to Adoption

28. FosteringCourtImprovement.org28 A Simple Temporal Context Emphasize the Context for every number, nothing without Context. Temporal: Interpret current measures in the context of the past.

29. FosteringCourtImprovement.org29 The Context of Comparable Jurisdictions Emphasize the Context for every number, nothing without Context. Comparables: Interpret in context of comparable counties.

30. FosteringCourtImprovement.org30 The Context of Statewide Rank Emphasize the Context for every number, nothing without Context. Rank: Interpret in context of state or other ranking.

31. FosteringCourtImprovement.org31 The Systemic, Process Context Flow of cases through the system

32. FosteringCourtImprovement.org32 Outcome Framework Context Safety Maltreatment does not occur. Measures: Proportion of children reported maltreated. Maltreatment does not recur. Measures: Proportion of children re-reported within 3, 6, and 12 months; Proportion re- entering foster care. Permanency Children remain in their homes. Measures: Proportion removed from their home. Bonded attachments are maintained. Measures: Proportion of siblings placed together; Frequency of visitation with kin; Proportion in a stable placement. Permanency is timely. Measures: Proportion reunified with family within 12 months; Proportion adopted within 24 months. Well Being Children are healthy. Measures: Proportion with physical health needs met; Proportion with mental health needs met. Children are educated. Measures: Proportion performing at age- appropriate grade level. Note that these outcomes are from the child’s perspective, not the system’s.

33. FosteringCourtImprovement.org33 Track Record: Real Work Done in a Shared Context Douglas County, GA: Removals cut 50% Relative placements increased 10-fold Adoption rate doubled Floyd County, GA: Removals cut 30% Pre-teens in group care down 80% Hall County, GA: Adoptions under 24 months tripled Bartow County, GA: Removals cut 50% Most counties unchanged, but actively dissecting their processes

34. FosteringCourtImprovement.org34 Improving the Product Additional Reports: Safety Report from linked NCANDS-AFCARS data Children Remaining Safe with Family CFSR 2001, CFSR 2006 Illinois Signs of Progress Reports Toolkit (9 Measures) Discovering Best Practices: Composite Measures Causal Linkages from Process Measures to Outcome Measures

35. FosteringCourtImprovement.org35 Loose Ends Process Measures (caseloads, due process, …) are crucial, but only when put in the context of Outcome Measures. Judges care deeply about outcomes. Need bottom-lines to keep us aligned and on task, I propose: Children safe with family Failing that, timely permanency Shift spending from data acquisition to data analysis. Don’t call GeekSquad™ for data analysis. (Don’t equate technology and data analysis)

36. FosteringCourtImprovement.org Questions, Comments... Judge Nancy Salyers Judicial Executive and Co-Founder, Fostering Court Improvement Co-Director, Fostering Results Andy Barclay, Andy@FosteringCourtImprovement.org Statistician and Co-Founder, Fostering Court Improvement Barton Child Law and Policy Clinic, Emory University School of Law Presented To: Fostering Court Improvement, A Data Workshop for Decision Makers December 7, 2006