1. Basics of Performance Measurement Presented by Mark Hardin National Child Welfare Legal Resource Center on Legal and Judicial Issues ABA Center on Children and the Law © MMVI by the American Bar Association

2. 2 WHY DO PERFORMANCE MEASUREMENT? From Reinventing Government: What gets measured gets done. If you don’t measure results, you can’t tell success from failure. If you can’t see success, you can’t reward it. If you can’t reward success, you’re probably rewarding failure.

3. 3 WHY DO PERFORMANCE MEASUREMENT? If you can’t see success, you can’t learn from it. If you can’t recognize failure, you can’t correct it. If you can demonstrate results, you can win public support.

4. 4 Some benefits of performance measurement for courts: Guidance for court process improvements and efficient case flow management; Identification of weaker elements of court performance, encouraging improvement; Evidence whether reform efforts are client effective and also cost effective; Identification of reforms that aren’t working well or have limited benefits which leads to their redesign or elimination;

5. 5 Some benefits of performance measurement for courts: Increased compliance with legal deadlines for court hearings and court decisions; Decreased delays in scheduling hearings; Fewer hearing delays once hearings have started; Hearings in which all parties have a better opportunity to be heard; Better informed court decisions, contributing to improved outcomes for children;

6. 6 Some benefits of performance measurement for courts: Increased communication and collaboration between the court, child welfare agency and other agencies that are present in the courtroom; Increased collaboration between courts, child welfare agencies and service providers;

7. 7 Some benefits of performance measurement for courts: Support of requests for needed funding to support pilot programs, increased staff; Assessment of judicial resources necessary to support a timely and fair process;

8. 8 SOME KEY TERMINOLOGY: Quantitative analysis: Evaluation of phenomena through numerical analysis, based on collecting and calculating measurable data. Examples: Observations involving counting and measuring, surveys asking for numerical or yes/no answers that can be tabulated numerically.

9. 9 SOME KEY TERMINOLOGY: Qualitative (descriptive) analysis: Evaluation of phenomena through observations and descriptions, such as through observations or interviews. Information is recorded and recounted in descriptive rather than numerical terms.

10. 10 SOME KEY TERMINOLOGY: Performance standards: list of principles or expectations for performing a specified professional role, generally formally approved by an official body representing the group to whom the standards apply. Examples: ABA standards for agency attorneys in child welfare cases; Child Welfare League of America Standards for Adoption.

11. 11 SOME KEY TERMINOLOGY: Performance measurement: Broad terminology for scientific process (using methodologies developed or approved by social scientists) of measuring and evaluating performance. Performance measurement may be either ongoing or ad hoc. It may involve qualitative or quantitative analysis or both.

12. 12 SOME KEY TERMINOLOGY: Quality assurance and quality improvement (QA & QI): ongoing evaluation to determine level of compliance with pre-established performance standards or expectations.

13. 13 SOME KEY TERMINOLOGY: Automated performance analysis: Measurement of performance through computer operations. Example: Automated performance measurement of dependency cases.

14. 14 Automated Performance Measurement: Computer generated Every case is counted and calculated Is ongoing and never stops Should be, but isn’t necessarily, an automated byproduct of other work

15. 15 Non Automated Performance Measurement: This is done either ad hoc or periodically (QA/QI) May be quantitative (numerical) or qualitative (descriptive)

16. 16 Non Automated Performance Measurement: Quantitative Based on adding numbers, such as: Measuring times between events Counting numbers of events Scoring multiple choice tests Non automated quantitative measurement often uses carefully identified samples rather than counting every case

17. 17 Non Automated Performance Measurement: Quantitative Methods for non automated judicial performance analysis: Surveys (questionnaires) mailed or handed out File reviews Extracting data from computers ad hoc Court observations (counting frequency of events)

18. 18 Non Automated Performance Measurement: Qualitative Examples: Interviews – open ended, as opposed to numerical, descriptions of observations File reviews (reading about cases and describing them, seeking insights, not counting) – open ended, as opposed to numerical, descriptions of observations Court observations (describing what happens in court, seeking insights) -- open ended, as opposed to numerical, descriptions of observations

19. 19 Non Automated Performance Measurement: Qualitative Qualitative work still involves social science Ask different people the same questions Word questions carefully to avoid ambiguity Use systematic methods to compare what different people said about the same thing Example: Computer sorts to see how different people address the same issue Ideally: Checking back to clear up inaccuracies Use “triangulation” or combining qualitative with quantitative for a full picture of what is being studied

20. 20 ABCs of Automated Performance Measurement: Measures A numerical statement of some significant dimension of performance. Examples: Of those children who are adopted from foster care, the percentage who were adopted within 24 months after their entry into foster care Of those children who a court has adjudicated to have been abused or neglected, percentage of children whose cases were adjudicated within 60 days after the filing of the petition Of those children whose court cases were closed in 2005, the percentage who were the subject of a new abuse or neglect petition in 2006

21. 21 ABCs of Automated Performance Measurement: Data Elements Bits of information needed to fill in a performance measure. Example of measure: Of those children who are adopted from foster care, the percentage who were adopted within 24 months after their entry into foster care Example of data elements: Foster child adopted (A) Date the child entered foster care (B) Date the child was adopted (C)

22. 22 ABCs of Automated Performance Measurement: Business rules (specifications) Instructions to the programmers how to collect and calculate the information to produce a performance measure

23. 23 ABCs of Automated Performance Measurement: Business rules (specifications) Example: Percentage of adopted children adopted within 24 months after entering foster care: Identify each foster child adopted, e.g., in 2005 in specific state, county, etc. (A) For each child subtract the date of entry into foster care (B) from the date of exit (C) – to calculate the number of days. (D) Count the number of foster children adopted in 2005. (E) Of the children in E, count the number of children where (D) was less than 24 months. (F) Divide (F) by (E) for the percentage of foster children adopted in 2005 who were adopted within 24 months of entry into foster care.

24. 24 Automated Performance Measurement by Child Welfare Agencies: NCCANDS Voluntary Information related to CAPTA Reporting, safety, recurrence of abuse and neglect Includes specific data elements

25. 25 Automated Performance Measurement by Child Welfare Agencies: AFCARS Mandatory data elements and measures Information regarding Titles IV-E and IV-B Entry into foster care Length of stay Permanency

26. 26 Automated Performance Measurement by Child Welfare Agencies: CFSR Performance Standards Includes measures But must reach a certain level based on 75 th percentile nationally If fail to reach level, must do two-year “Program Improvement Plan” (PIP) If fail to succeed in PIP, face fines

27. 27 Automated Performance Measurement by Child Welfare Agencies: SACWIS SACWIS (state automated child welfare information system) is a set of functions that computer systems are supposed to fulfill States receive special Title IV-E matching funds to establish management information systems fulfilling SACWIS requirements SACWIS includes many mandatory and some optional functions, including optional court functions

28. 28 SACWIS optional requirement 58: preparation of agency documents for courts Does the system facilitate preparation of all State agency documents for courts, such as petitions, letters, attorney approvals, and supervisory approvals? Does the system interface with one or more court systems? Is duplicate data entry required to generate court documents?

29. 29 SACWIS optional requirement 59: notification of court actions Does the system generate notifications, either manually or automatically of upcoming court actions for all relevant parties?

30. 30 Automated Performance Measurement by Child Welfare Agencies: CFSR Performance Standards for Safety Discrete safety measures and standards CFSR S1A - Recurrence rate of abuse and neglect: Of all children who were victims of a substantiated or indicated maltreatment allegation during the first 6 months of FFY 2004, what percent were not victims of another substantiated or indicated maltreatment allegation during a 6-month period? National Standard = 95.2% or more. CFSR S1B - Rate of child abuse and neglect while the child is in foster care: Of all children in foster care in FFY 2004, what percent were not victims of a substantiated or indicated maltreatment by a foster parent or facility staff member? National Standard is 99.67% or more.

31. 31 Automated Performance Measurement by Child Welfare Agencies: CFSR Composite Performance Standards Combined results of several related measures Combined to avoid unintended incentives, like high percentage of reunifications with low percentage of reentries after reunification Must collectively meet a specific standard

32. 32 Court Improvement Performance Measures: No national data No standards (too early) Focus on: Safety Permanency “Due process” and fairness Timeliness

33. 33 Court Improvement Performance Measures: Safety 1A – Maltreatment while under court jurisdiction “Percentage of children who are abused or neglected while under court jurisdiction.” 1B -- Maltreatment after release from court jurisdiction “Percentage of children who are abused or neglected within 12 months after case closure.”

34. 34 Court Improvement Performance Measures: Permanency 2A – Achievement of Child Permanency Percentage of children in foster care who reach legal permanency by reunification, adoption, or legal guardianship.

35. 35 Court Improvement Performance Measures: Fairness 3A – Number of Judicial Officers Per Case Percentage of cases in which all hearings were heard by one judicial officer. 3G – Presence of Advocates During Hearings Percentage of cases in which legal counsel for parents, children and the agency is present at every hearing.

36. 36 Court Improvement Performance Measures: Timeliness 4A – Time to Permanent Placement Average (median) time from filing of the original petition to permanency. 4B – Time to Adjudication Average (median) time from filing of the original petition to adjudication.

37. 37 To Contact Us Jennifer Renne, (202) 662-1731, rennej@staff.abanet.orgrennej@staff.abanet.org Mark Hardin, (202) 662-1750, markhardin@staff.abanet.orgmarkhardin@staff.abanet.org National Child Welfare Resource Center on Legal and Judicial Issues ABA Center on Children and the Law 740 15th Street, NW Washington, DC 20005-1022 http://www.abanet.org/child/nrclji