1. Evaluation of Ohio’s Title IV-E Waiver Demonstration Project “ProtectOhio” Summary of Findings from the 5-year study September 2003 HSRI, Westat, Chapin Hall, IHSM

2. Green = demonstration PCSA Yellow = comparison PCSA Ohio’s Evaluation Counties

3. LOGIC MODEL Waiver Systems Reform Interventions Improved Outcomes for children & families Any service Any clients $ up front Keep savings Internal organization Services available Financing patterns MC strategies What’s done for what families and children How they go through the system Services received Use of relatives  placement days  permanency  functioning  well being

4. Demonstration Counties’ Response to the Waiver Compared to their own pre-Waiver actions and comparison counties’ concurrent actions, demonstration sites made these changes: –Greater targeted focus on prevention –More attention to outcomes –Overall greater use of managed care strategies –More interagency collaboration –Slower growth in foster care spending –More growth in “other child welfare” expenditures

5. Demonstration sites focused more on prevention… Demonstrations and comparisons had similar levels of insufficiency in array of available services; moderate differences in area of prevention Demonstrations targeted new prevention activities to areas identified as insufficient Demonstration sites expressed stronger commitment to prevention

7. Table 3.3: Number of Service Insufficiencies in Year 3 and Creation of New Service in Year 4 DemonstrationComparison Y3 number of counties having insufficient prevention service 129 Y3 count of all insufficient prevention services across those counties 3728 Y4 number of new prevention services created in the area of insufficiency 2010 New services created as a percent of insufficient services 20 out of 37 = 54% 10 out of 28 = 36% Number of counties who targeted services (targeting = county created services for at least half of their reported service insufficiencies) 94

8. Table 3.4: Prevention Groups* New Spending on non-foster care Strong ThemeNo Strong Theme DemosCompsDemosComps Above the median Group OneGroup Three 6132 Median or below Group TwoGroup Four 3218 *One demonstration and one comparison counties had incomplete fiscal data so they could not be categorized in this table.

9. Demonstration sites gave more attention to outcomes… Gathering more outcomes data: 10 demo and 6 comp reported systematically gathering outcome information from staff (Y4) Using outcomes data in decision-making (Y3) Integrating fiscal & program outcomes information (Y4)

11. Figure 2.6: Use of Fiscal and Program Data (Y4) 0 2 4 6 8 10 12 14 Program Managers Use Fiscal Data Fiscal Staff Use Program Data Demonstration (n=14) Comparison (n=14)

12. Ohio’s Managed Care Strategies Service array (care criteria): new services, shifting focus Financing (capitation & risk): capitated & case rate contracts, use of IV-E Targeting (eligibility): special initiatives, specialized units Case management: unit structure, transfers Provider competition: provider affiliations, rate changes Utilization review: placement reviews, service reviews Information management: automated MIS, mgrs use info Quality assurance: control & enhancement, outcome focus

13. Demonstration sites employ more managed care strategies In Year 2, significantly more use of managed care strategies by demos In Year 4, demos made more use of 7 of 8 managed care strategies Overall rankings: more demos in the “high” group, fewer in the “low” group

14. Table 2.6 Tukey’s Quick Test for Differences in Managed Care Index Score (Year 2) County Demonstration/ ComparisonY2 ScoreCounty Demonstration/ ComparisonY2 Score LorainDemonstration47.69FairfieldDemonstration25.78 HamiltonDemonstration46.09StarkDemonstration25.52 MedinaDemonstration38.68ButlerComparison25.45 RichlandDemonstration36.91HancockComparison24.77 PortageDemonstration35.56HockingComparison24.77 FranklinDemonstration33.60BelmontDemonstration23.53 GreeneDemonstration31.66AllenComparison20.64 CrawfordDemonstration29.56AshtabulaDemonstration20.20 TrumbullComparison29.48WarrenComparison19.75 ClarkDemonstration28.51MahoningComparison17.66 MuskingumDemonstration28.45ColumbianaComparison14.59 MontgomeryComparison28.20MiamiComparison13.62 SummitComparison28.04WoodComparison9.02 SciotoComparison26.69ClermontComparison8.95

15. Table 2.4: Y4 Managed Care Score as a % of Total Possible Score Managed Care Category Demonstration County Average Comparison County Average Financing38%18% Utilization Review58%52% Service array45%52% Case Management49%46% Competition42%29% Targeting40%30% Quality Assurance55%46% Data Management48%37% TOTAL47%39%

16. Counties Grouped by Level of Managed Care Activity, Year 4 Counties with High Managed Care Activity Counties with Moderate Managed Care Activity Counties with Low Managed Care Activity Demo:Comp:Demo:Comp:Demo:Comp: Franklin Greene Hamilton Lorain Medina Richland Montgomery Trumbull Belmont Clark Muskingum Portage Stark Allen Butler Miami Scioto Summit Ashtabula Crawford Fairfield Clermont Columbiana Hancock Hocking Mahoning Warren Wood

17. Demonstration sites have stronger collaboration… Overall, PCSA partners say collaboration with PCSA has become stronger BUT Demo and comparison PCSAs have similarly strong relationships with juvenile court, mental health board

18. Table 3.19: Characterizing the Relationship between Survey Respondents and the PCSA* Relationship StrengthDemonstrationComparison Average Strength of Relationship (scale: -2 to 2) 1.401.11 Rating the Strength of Relationship Very strong/strongstrong *Difference is significant at 0.058

19. Table 3.20: Characterizing the Change in the Relationship between Survey Respondents and the PCSA since 1997* Change in Relationship Strength DemonstrationComparison Average Change in the Strength of Relationship (scale: -1 to 1) 0.890.61 Rating Average Change in the Strength of Relationship Grown strongerStayed the same/grown stronger *Difference is significant at 0.000

20. Relationship between PCSA and Main Partners (Year 4) PCSA relationship with Juvenile Court:  Majority of demo & comp counties have strong relationship (9D, 7C)  Inappropriate referrals continue to be issue in some demo and comp counties PCSA relationship with Mental Health: –Relationships remain strong in most counties, no significant change over Waiver period (9D, 7C) – Inadequate mental health services for children and families, so PCSAs (both demo and comp) develop and purchase own mental health services

21. However, demonstration sites are no different from comparisons in: Family involvement in case decisions – slow growth in family role and team conferencing; Use of managed care contracting – 3 demo; Increased competition in foster care – growth in foster homes and per diem rates; Utilization review and QA – moderate use of formal review processes.

22. Utilization Review (Year 4) Placement review processes: –Pre-placement – 9D, 8C –During placement – 8D, 5C Oversight of non-placement resources: –Pre-service review – 4D, 4C –Subsequent review – 2D, 3C

23. Changes in Patterns of Child Welfare Spending Over Time Both groups of counties experienced growth in paid placement days and in the average daily cost of foster care; BUT Demonstration sites appeared to contain growth in foster care spending more than comparison sites; Demonstration sites appeared to increase other child welfare spending more than comparison sites.

24. Change in Placement Day Utilization Evaluation GroupCounty Growth Baseline- 2002 Evaluation GroupCounty Growth Baseline- 2002 ComparisonWood(38%)DemonstrationMedina6% DemonstrationBelmont(34%)ComparisonScioto6% ComparisonMahoning(29%)DemonstrationGreene17% DemonstrationLorain(26%)ComparisonClermont17% ComparisonButler(25%)ComparisonMontgomery17% ComparisonTrumbull(22%)ComparisonMiami22% DemonstrationMuskingum(21%)ComparisonHocking28% DemonstrationClark(20%)DemonstrationCrawford32% DemonstrationAshtabula(15%)ComparisonSummit32% ComparisonAllen(9%)ComparisonColumbiana44% DemonstrationPortage(5%)DemonstrationFranklin46% DemonstrationHamilton2%DemonstrationFairfield48% DemonstrationRichland2%ComparisonWarren53% DemonstrationStark5%ComparisonHancock97%

25. Changes in Foster Care Spending Evaluation GroupCounty Growth Baseline- 2002 Evaluation GroupCounty Growth Baseline- 2002 ComparisonWood(24%)DemonstrationFairfield47% ComparisonButler(1%)ComparisonMahoning53% DemonstrationPortage2%ComparisonScioto59% ComparisonAllen13%DemonstrationStark59% ComparisonTrumbull17%ComparisonSummit61% DemonstrationRichland18%ComparisonMontgomery67% DemonstrationClark20%ComparisonWarren80% DemonstrationHamilton26%ComparisonMiami92% DemonstrationGreene33%DemonstrationFranklin103% DemonstrationMuskingum40%ComparisonColumbiana131% DemonstrationBelmont43%ComparisonHocking152% DemonstrationAshtabula43%

26. Additional Waiver Revenue Most demonstration counties received more revenue through the Waiver than they would have received through normal Title IV-E reimbursement for foster care board and maintenance expenses.

27. Additional Revenue Received ($1000s) Demonstration Estimated Expenditures Eligible for Title IV-E Foster Care B&M Reimbursement 1998-2002 ProtectOhio Waiver Award 1998-2002 Total ProtectOhio Revenue Available for Reinvestment 1998-2002 ProtectOhio Revenues as a Percent of Total Expenditures Ashtabula$2,504$4,290$1,7867% Belmont$2,627$3,161$5343% Clark$9,753$12,743$2,9906% Franklin$82,084$95,740$13,6562% Greene$4,900$5,038$1381% Hamilton$53,246$76,399$23,1537% Lorain$8,600$12,061$3,4616% Medina$1,726$1,712($14)(0%) Muskingum$5,594$6,219$6253% Portage$5,158$7,883$2,72510% Richland$6,345$6,751$4061% Stark$24,937$26,212$1,2751%

28. Additional Revenue Purchased All but one of the demonstration counties spent its additional Waiver revenue on child welfare services other than board and maintenance payments. As a result, spending on all other child welfare services increased significantly more among the group of demonstration counties than among the group of comparison counties.

29. All Other Child Welfare Expenditures Evaluation GroupCounty All Other CW Services Cumulative Spending as a % of Total Expenditures Evaluation GroupCounty All Other CW Services Cumulative Spending as a % of Total Expenditures DemonstrationPortage0%DemonstrationAshtabula12% ComparisonTrumbull0%ComparisonButler12% ComparisonMiami3%DemonstrationGreene12% ComparisonWood7%ComparisonAllen13% ComparisonMontgomery8%DemonstrationStark13% ComparisonHocking8%ComparisonWarren15% ComparisonSummit9%DemonstrationRichland16% ComparisonScioto10%DemonstrationMuskingum19% DemonstrationHamilton10%DemonstrationClark21% ComparisonMahoning11%DemonstrationBelmont36% DemonstrationFranklin12%

30. Overall, demonstration counties have made more changes in practice than comparison sites because the Waiver: allows PCSAs to spend in different ways makes it easier to change management practices may “grease the wheels” of collaboration …But how did these changes affect child & family outcomes?

31. Outcomes Resulting from Organizational and Fiscal Changes Changes in Caseload Trends Changes in Case Mix Waiver effects on Permanency

32. Abuse/Neglect Incidents Fell During Waiver Period for Both Groups

33. During Waiver Period, Number of Children in Caseloads Increased in Demonstration Counties

34. Number of Children Entering First Placements Increased in Large Demonstration Counties

35. Early Findings: Permanency Trends Both demonstration and comparison counties experienced an increase in the number of children in permanent custody over the first three years of the Waiver. The comparison counties experienced a much larger increase than did the demonstration counties in the number of children in PPLA.

36. Early Findings: Permanency Trends In the first 2 years of the Waiver, both demonstration and comparison county groups increased the percentage of children who experienced no moves in their first placements, to over half of all children in first placements. Nearly 25 percent of children in both groups experienced only one move while in placement, and this percentage increased during the first 2 years of the Waiver. The percentage of children who experienced five or more moves decreased in both groups

37. Early Findings: Permanency Trends During the first 2 years of the Waiver, the number of teens placed in group and residential settings in demonstration counties did not decline. The demonstration counties did not experience greater success than comparison counties at moving children from group and residential settings to less restrictive settings during the first 2 years of the Waiver.

38. Early Findings: Permanency Trends In the first 3 years of the Waiver, both demonstration and comparison county groups experienced an increase in the number of children placed with relatives. However, 9 of 14 demonstration counties experienced an increase, compared with 6 of 14 comparison counties.

39. Use of Kinship Care Varied by County Table 5.4: Most Frequent Approaches to the Use of Relative/Kinship Care* ApproachDemonstration CountiesComparison Counties Informal placement directly with relative MuskingumAllen, Columbiana, Hancock, Hocking, Mahoning, Miami, Trumbull Custody to relative after short timeBelmont, Clark, Crawford, Greene, Medina, Richland, Stark Butler, Clermont, Montgomery, Scioto, Wood Placement with relative, custody to PCSA Ashtabula, Fairfield, Franklin, Hamilton, Portage Summit, Warren Approach not determinedLorainNone

40. Early Findings: Permanency Trends The majority of counties in both demonstration and comparison groups had an increase in the number of new children eligible for adoption subsidy. The overall number of adoptions increased 40% in demonstration counties and 32% in comparison counties during the first three years of the Waiver.

41. Early Findings: Safety Trends During the first 3 years of the Waiver, the recidivism rate for targeted children (those identified with moderate to high risk) declined slightly in the demonstration counties while decreasing considerably in comparison counties. Nine demonstration counties and seven comparison counties experienced decreases in recidivism rates for targeted children.

42. Early Findings: Well-being The balance between in-home and placement services remained stable across the baseline and the Waiver periods, and demonstration counties’ proportions were similar to comparison counties’

43. Changes in Placement Case Mix During the Waiver Table 6.2 Changes in Demographic Characteristics of Children in First Placements in Demonstration Counties, from Pre-Waiver to Waiver Period CountySexAgeRace Male<11-45-1314-17WhiteBlackOther Ashtabula  Belmont  Clark  Crawford  Fairfield  Franklin  Greene  Lorain  Medina  Muskingum  Portage  Richland  Stark 

44. Percentage of Children Placed From Abuse/Neglect Incidents Changed for Several Counties During Waiver Period Table 6.3 Changes in Abuse/Neglect and Disabilities of Children in First Placements in Demonstration Counties, from Pre-Waiver to Waiver Period CountyAbuse/Neglect and Disabilities Sexually AbusedAlleged Victim of Abuse or Neglect Cognitive DisabilitiesPhysical Disabilities Ashtabula  Belmont  Clark  Crawford  Fairfield  Franklin  Greene  Lorain  Medina  Muskingum  Portage  Richland  Stark 

45. Several Counties Increased First Placements with Relatives During the Waiver Table 6.4 Changes in Settings of First Placements in Demonstration Counties from Pre-Waiver to Waiver Period County Setting of Child’s First Placement Residential Treatment Center Group Home Foster Home Nonlicensed Nonrelative RelativeIndependent Living Detention Facility or Hospital Adoptive Home Ashtabula  Belmont  Clark  Crawford  Fairfield  Franklin  Greene  Lorain  Medina  Muskingum  Portage  Richland  Stark 

46. Waiver Effect on the Exit Type from First Placements Overall Table 6.5 Waiver Effects on Exit Types from First Placements for All Demonstration Counties First Placements Ending With: Percentage of Cases Waiver Effect Actual Under the WaiverCounterfactual Projection Reunification45.3456.74-11.40* Custody to Relative18.1214.463.66* Adoption9.26 0.00 Runaway1.040.540.50* Other a 26.2219.027.20* Total100.00 N/A

47. The Waiver Shortened the Overall Median Duration of First Placements Table 6.6 Waiver Effects on Duration of First Placements for All Demonstration Counties First Placements Ending With:Median Placement Duration in MonthsWaiver Effect Actual Under the WaiverCounterfactual Projection Reunification3.042.900.14 Custody to Relative6.566.280.28 Adoption31.7832.00-0.22 Runaway6.969.24-2.28 Any type of exit4.504.90-0.40*

48. Significant County-Level Effects of Where Children Exited from Placement The Waiver significantly increased the percentage of first placements that ended with custody to relatives in six of the 13 demonstration counties and decreased it in one county. The Waiver significantly increased runaways in three counties, although the overall effect was small The Waiver decreased reunifications in the largest county. The Waiver increased adoptions in one county, with no overall effect. The Waiver increased “other” exit types in one county (the largest) and decreased them in two others, for an overall increase

49. County-Level Waiver Effects on Exit Type Table 6.7 County-Level Waiver Effects on Exit Types from First Placements County Exit Type Reunification Custody to Relative AdoptionRunawayOther a Ashtabula  Belmont  Clark  Crawford  Fairfield  Franklin  Greene  Lorain  Medina  Muskingum  Portage  Richland  Stark  Overall 

50. Some County-Level Waiver Effects on Median Duration to Each Exit Type Table 6.8 County-Level Waiver Effects on Median Duration of First Placements County Exit Type ReunificationCustody to RelativeAdoptionRunawayAny Type of Exit Ashtabula  Belmont  Clark  Crawford  Fairfield  Franklin  Greene  Lorain  Medina  Muskingum  Portage  Richland  Stark  Overall 

51. No County-Level Waiver Effect on Reentry Table 6.9 Re-entry After Reunification from First Placement a CountyActual Rate Under the Waiver (%)Median Duration Before Re-Entry (months) Ashtabula26.7615.32 Belmont36.1011.46 Clark27.309.22 Crawford23.0818.40 Fairfield37.448.12 Franklin41.649.84 Greene30.4414.04 Lorain23.7012.88 Medina38.6016.22 Muskingum35.3810.08 Portage31.7011.90 Richland38.4013.46 Stark32.4415.56 Overall37.4412.08

52. Putting it all together: Case Studies of 6 Demonstration sites Clark, Fairfield, Franklin, Lorain, Muskingum, Stark Major organizational changes adopted and outcomes achieved  Offers insights into dynamics of child welfare practice under the Waiver, but cannot be extrapolated to all counties

53. Case Study Findings (1): Lorain and Muskingum Commitment to systemic change appears to have fostered a greater degree of success, in terms of fiscal and participant outcomes, than in other evaluation counties. –More use of managed care strategies –More focus on prevention –Increased spending on non-foster care –Fewer children in PPLA –Quicker adoption, more adoption –More exits to relative custody

54. Case Study Findings (2) Other four counties had less clear-cut results despite initiating a variety of reform efforts. Clark – reformed front-end of the system through expansion of home-based services and collaboration with juvenile court; result was reduced placement days. Franklin – focus on external reform, contracting with private providers; result was reduced LOS in foster care. Fairfield and Stark – less consistent reform agendas during Waiver due in part to major organizational challenges; no significant improvements in outcomes.

55. Overall Results Mix of evidence of significant impact of the Waiver, areas where systematic impact is lacking, and signs of substantive change in individual counties Confounding circumstances: –Funding context (levies, state cutbacks) –County-administered system so county-to-county variation –Lack of management tools –Low financial risk –Small sample sizes; differences among data sets Suggests that the flexibility provided by the Waiver…  facilitates reform efforts where other factors are already conducive,  but may not itself be robust enough to generate fundamental reform of the state’s public child welfare system.

56. The Future Good likelihood of Waiver extension Evaluation focus: More in-depth analysis of foster care utilization More examination of child safety Further analysis of expenditure patterns Targeted investigation of special topics – use of relatives, mental health services, court-referred children, etc.