Summary of Findings from the 5-year study September 2003

Evaluation of Ohio’s Title IV-E Waiver Demonstration Project “ProtectOhio”
Summary of Findings from the 5-year study September 2003 HSRI, Westat, Chapin Hall, IHSM

Green = demonstration PCSA Yellow = comparison PCSA
Ohio’s Evaluation Counties Green = demonstration PCSA Yellow = comparison PCSA

Improved Outcomes for children & families
LOGIC MODEL Improved Outcomes for children & families Waiver Systems Reform Interventions Any service Any clients $ up front Keep savings Internal organization Services available Financing patterns MC strategies What’s done for what families and children How they go through the system Services received Use of relatives  placement days  permanency functioning  well being

Demonstration Counties’ Response to the Waiver
Compared to their own pre-Waiver actions and comparison counties’ concurrent actions, demonstration sites made these changes: Greater targeted focus on prevention More attention to outcomes Overall greater use of managed care strategies More interagency collaboration Slower growth in foster care spending More growth in “other child welfare” expenditures

Demonstration sites focused more on prevention…
Demonstrations and comparisons had similar levels of insufficiency in array of available services; moderate differences in area of prevention Demonstrations targeted new prevention activities to areas identified as insufficient Demonstration sites expressed stronger commitment to prevention

Demonstration Comparison
Table 3.3: Number of Service Insufficiencies in Year 3 and Creation of New Service in Year 4 Demonstration Comparison Y3 number of counties having insufficient prevention service 12 9 Y3 count of all insufficient prevention services across those counties 37 28 Y4 number of new prevention services created in the area of insufficiency 20 10 New services created as a percent of insufficient services 20 out of 37 = 54% 10 out of 28 = 36% Number of counties who targeted services (targeting = county created services for at least half of their reported service insufficiencies) 4

Table 3.4: Prevention Groups*
New Spending on non-foster care Strong Theme No Strong Theme Demos Comps Above the median Group One Group Three 6 1 3 2 Median or below Group Two Group Four 8 *One demonstration and one comparison counties had incomplete fiscal data so they could not be categorized in this table.

Demonstration sites gave more attention to outcomes…
Gathering more outcomes data: 10 demo and 6 comp reported systematically gathering outcome information from staff (Y4) Using outcomes data in decision-making (Y3) Integrating fiscal & program outcomes information (Y4)

Figure 2.6: Use of Fiscal and Program Data (Y4)
2 4 6 8 10 12 14 Program Managers Use Fiscal Data Fiscal Staff Use Program Data Demonstration (n=14) Comparison

Ohio’s Managed Care Strategies
Service array (care criteria): new services, shifting focus Financing (capitation & risk): capitated & case rate contracts, use of IV-E Targeting (eligibility): special initiatives, specialized units Case management: unit structure, transfers Provider competition: provider affiliations, rate changes Utilization review: placement reviews, service reviews Information management: automated MIS, mgrs use info Quality assurance: control & enhancement, outcome focus

Demonstration sites employ more managed care strategies
In Year 2, significantly more use of managed care strategies by demos In Year 4, demos made more use of 7 of 8 managed care strategies Overall rankings: more demos in the “high” group, fewer in the “low” group

Demonstration/ Comparison
Table 2.6 Tukey’s Quick Test for Differences in Managed Care Index Score (Year 2) County Demonstration/ Comparison Y2 Score Lorain Demonstration 47.69 Fairfield 25.78 Hamilton 46.09 Stark 25.52 Medina 38.68 Butler Comparison 25.45 Richland 36.91 Hancock 24.77 Portage 35.56 Hocking Franklin 33.60 Belmont 23.53 Greene 31.66 Allen 20.64 Crawford 29.56 Ashtabula 20.20 Trumbull 29.48 Warren 19.75 Clark 28.51 Mahoning 17.66 Muskingum 28.45 Columbiana 14.59 Montgomery 28.20 Miami 13.62 Summit 28.04 Wood 9.02 Scioto 26.69 Clermont 8.95

Table 2.4: Y4 Managed Care Score as a % of Total Possible Score
Managed Care Category Demonstration County Average Comparison County Average Financing 38% 18% Utilization Review 58% 52% Service array 45% Case Management 49% 46% Competition 42% 29% Targeting 40% 30% Quality Assurance 55% Data Management 48% 37% TOTAL 47% 39%

Counties Grouped by Level of Managed Care Activity, Year 4
Counties with High Managed Care Activity Counties with Moderate Managed Care Activity Counties with Low Managed Care Activity Demo: Comp: Franklin Greene Hamilton Lorain Medina Richland Montgomery Trumbull Belmont Clark Muskingum Portage Stark Allen Butler Miami Scioto Summit Ashtabula Crawford Fairfield Clermont Columbiana Hancock Hocking Mahoning Warren Wood Explain that categories are relative: in prior year, cut-off scores were lower, so that some counties may now be in a lower category than were in prior year -- simply means they did not increase their MC activity as rapidly as other counties, not that they regressed.

Demonstration sites have stronger collaboration…
Overall, PCSA partners say collaboration with PCSA has become stronger BUT Demo and comparison PCSAs have similarly strong relationships with juvenile court, mental health board

Relationship Strength
Table 3.19: Characterizing the Relationship between Survey Respondents and the PCSA* Relationship Strength Demonstration Comparison Average Strength of Relationship (scale: -2 to 2) 1.40 1.11 Rating the Strength of Relationship Very strong/strong strong *Difference is significant at 0.058

Change in Relationship Strength
Table 3.20: Characterizing the Change in the Relationship between Survey Respondents and the PCSA since 1997* Change in Relationship Strength Demonstration Comparison Average Change in the Strength of Relationship (scale: -1 to 1) 0.89 0.61 Rating Average Change in the Strength of Relationship Grown stronger Stayed the same/grown stronger *Difference is significant at 0.000

Relationship between PCSA and Main Partners (Year 4)
PCSA relationship with Juvenile Court: Majority of demo & comp counties have strong relationship (9D, 7C) Inappropriate referrals continue to be issue in some demo and comp counties PCSA relationship with Mental Health: Relationships remain strong in most counties, no significant change over Waiver period (9D, 7C) Inadequate mental health services for children and families, so PCSAs (both demo and comp) develop and purchase own mental health services Overall: Since most counties have levies or other funds that can be used flexibly, they have been able to address coordinated community services and programs without having to rely solely on flexible Waiver funds.

However, demonstration sites are no different from comparisons in:
Family involvement in case decisions – slow growth in family role and team conferencing; Use of managed care contracting – 3 demo; Increased competition in foster care – growth in foster homes and per diem rates; Utilization review and QA – moderate use of formal review processes.

Utilization Review (Year 4)
Placement review processes: Pre-placement – 9D, 8C During placement – 8D, 5C Oversight of non-placement resources: Pre-service review – 4D, 4C Subsequent review – 2D, 3C

Changes in Patterns of Child Welfare Spending Over Time
Both groups of counties experienced growth in paid placement days and in the average daily cost of foster care; BUT Demonstration sites appeared to contain growth in foster care spending more than comparison sites; Demonstration sites appeared to increase other child welfare spending more than comparison sites.

Change in Placement Day Utilization
Evaluation Group County Growth Baseline-2002 Comparison Wood (38%) Demonstration Medina 6% Belmont (34%) Scioto Mahoning (29%) Greene 17% Lorain (26%) Clermont Butler (25%) Montgomery Trumbull (22%) Miami 22% Muskingum (21%) Hocking 28% Clark (20%) Crawford 32% Ashtabula (15%) Summit Allen (9%) Columbiana 44% Portage (5%) Franklin 46% Hamilton 2% Fairfield 48% Richland Warren 53% Stark 5% Hancock 97% Again, in this comparison of change from prior to the waiver to the waiver period, demonstration and comparison counties are found at all points in the distribution.

Changes in Foster Care Spending
Evaluation Group County Growth Baseline- 2002 Comparison Wood (24%) Demonstration Fairfield 47% Butler (1%) Mahoning 53% Portage 2% Scioto 59% Allen 13% Stark Trumbull 17% Summit 61% Richland 18% Montgomery 67% Clark 20% Warren 80% Hamilton 26% Miami 92% Greene 33% Franklin 103% Muskingum 40% Columbiana 131% Belmont 43% Hocking 152% Ashtabula

Additional Waiver Revenue
Most demonstration counties received more revenue through the Waiver than they would have received through normal Title IV-E reimbursement for foster care board and maintenance expenses. Placement reviews: see next slide MIS: full gamut, with demo showing somewhat greater activity around data collection, management, and analysis. Most primarily rely on FACSIS, but some are linking program and fiscal data (3d/4c) and acquiring information systems that allow them to do more sophisticated analysis, computer modeling (2d/1c) CLA: CLA continues to be implement to various degrees in half demo and 2 comp, enabling counties to monitor their use of child welfare resources. (I DIDN’T THINK THERE WERE 2 COMPS…) Delays in installation of PRO-IV Variation in use of main elements of CLA: all cases in certain units vs. a sample of cases Implementation in stages: family assessment (7d/1c), service decisions (4 d/1c), time limited services (4d/1c), workload management (3d/0c)

Additional Revenue Received ($1000s)
Demonstration Estimated Expenditures Eligible for Title IV-E Foster Care B&M Reimbursement ProtectOhio Waiver Award Total ProtectOhio Revenue Available for Reinvestment ProtectOhio Revenues as a Percent of Total Expenditures Ashtabula $2,504 $4,290 $1,786 7% Belmont $2,627 $3,161 $534 3% Clark $9,753 $12,743 $2,990 6% Franklin $82,084 $95,740 $13,656 2% Greene $4,900 $5,038 $138 1% Hamilton $53,246 $76,399 $23,153 Lorain $8,600 $12,061 $3,461 Medina $1,726 $1,712 ($14) (0%) Muskingum $5,594 $6,219 $625 Portage $5,158 $7,883 $2,725 10% Richland $6,345 $6,751 $406 Stark $24,937 $26,212 $1,275 Most demonstration counties received more revenue through the Waiver than they would have received through normal Title IV-E reimbursement for foster care board and maintenance services. Variety of reasons: - how individual county compared to group of control counties - how average Title IV-E eligibility rate compared to eligibility rates for higher cost placements - whether county found other revenue sources for foster care (IV-E admin for purchased foster care, Court agreements)

Additional Revenue Purchased
All but one of the demonstration counties spent its additional Waiver revenue on child welfare services other than board and maintenance payments. As a result, spending on all other child welfare services increased significantly more among the group of demonstration counties than among the group of comparison counties. Area of significant growth in the last year. In year 2, demo counties were substantially more active in this area, this year less difference between demo and comp, but demo remain more focused, developing more units and using outcomes to make management decisions. Examples: QA line for workers with clinical questions, workers offering suggestions for changes during their performance reviews, improving cultural competency by offering field trips to learn about cultural differences, peer review processes, CQI processes... Staff designated to QA: half of demo have a QA unit ( 4 d last year), with 2 c with a unit this year. 2d and 4c have a single worker dedicated to QA. Outcomes: Ds make notably more use of outcomes than comparison: 9 D and only 2 C use 3 or more of the outcome-focused activities 4 demonstration counties and 1 comparison county ranked high on MC QA

All Other Child Welfare Expenditures
Evaluation Group County All Other CW Services Cumulative Spending as a % of Total Expenditures Demonstration Portage 0% Ashtabula 12% Comparison Trumbull Butler Miami 3% Greene Wood 7% Allen 13% Montgomery 8% Stark Hocking Warren 15% Summit 9% Richland 16% Scioto 10% Muskingum 19% Hamilton Clark 21% Mahoning 11% Belmont 36% Franklin All but two of the demonstration counties spent additional Waiver revenue on child welfare services other than board and maintenance payments. As a result, spending on all other child welfare services increased significantly more among the group of demonstration counties than among the group of comparison counties. Most of the additional revenue was spent on county staff and related expenses, and most of that for foster care case management.

…But how did these changes affect child & family outcomes?
Overall, demonstration counties have made more changes in practice than comparison sites because the Waiver: allows PCSAs to spend in different ways makes it easier to change management practices may “grease the wheels” of collaboration …But how did these changes affect child & family outcomes?

Outcomes Resulting from Organizational and Fiscal Changes
Changes in Caseload Trends Changes in Case Mix Waiver effects on Permanency Thank all the staff that helped

Abuse/Neglect Incidents Fell During Waiver Period for Both Groups

During Waiver Period, Number of Children in Caseloads Increased in Demonstration Counties

Number of Children Entering First Placements Increased in Large Demonstration Counties
Talk about volume,unit cost,duration

Early Findings: Permanency Trends
Both demonstration and comparison counties experienced an increase in the number of children in permanent custody over the first three years of the Waiver. The comparison counties experienced a much larger increase than did the demonstration counties in the number of children in PPLA.

In the first 2 years of the Waiver, both demonstration and comparison county groups increased the percentage of children who experienced no moves in their first placements, to over half of all children in first placements. Nearly 25 percent of children in both groups experienced only one move while in placement, and this percentage increased during the first 2 years of the Waiver. The percentage of children who experienced five or more moves decreased in both groups

During the first 2 years of the Waiver, the number of teens placed in group and residential settings in demonstration counties did not decline. The demonstration counties did not experience greater success than comparison counties at moving children from group and residential settings to less restrictive settings during the first 2 years of the Waiver.

In the first 3 years of the Waiver, both demonstration and comparison county groups experienced an increase in the number of children placed with relatives. However, 9 of 14 demonstration counties experienced an increase, compared with 6 of 14 comparison counties.

Use of Kinship Care Varied by County
Table 5.4: Most Frequent Approaches to the Use of Relative/Kinship Care* Approach Demonstration Counties Comparison Counties Informal placement directly with relative Muskingum Allen, Columbiana, Hancock, Hocking, Mahoning, Miami, Trumbull Custody to relative after short time Belmont, Clark, Crawford, Greene, Medina, Richland, Stark Butler, Clermont, Montgomery, Scioto, Wood Placement with relative, custody to PCSA Ashtabula, Fairfield, Franklin, Hamilton, Portage Summit, Warren Approach not determined Lorain None

The majority of counties in both demonstration and comparison groups had an increase in the number of new children eligible for adoption subsidy. The overall number of adoptions increased 40% in demonstration counties and 32% in comparison counties during the first three years of the Waiver.

Early Findings: Safety Trends
During the first 3 years of the Waiver, the recidivism rate for targeted children (those identified with moderate to high risk) declined slightly in the demonstration counties while decreasing considerably in comparison counties. Nine demonstration counties and seven comparison counties experienced decreases in recidivism rates for targeted children.

Early Findings: Well-being
The balance between in-home and placement services remained stable across the baseline and the Waiver periods, and demonstration counties’ proportions were similar to comparison counties’

Changes in Placement Case Mix During the Waiver
Table 6.2 Changes in Demographic Characteristics of Children in First Placements in Demonstration Counties, from Pre-Waiver to Waiver Period County Sex Age Race Male <1 1-4 5-13 14-17 White Black Other Ashtabula    Belmont Clark Crawford Fairfield Franklin Greene Lorain Medina Muskingum Portage Richland Stark

Percentage of Children Placed From Abuse/Neglect Incidents Changed for Several Counties During Waiver Period Table 6.3 Changes in Abuse/Neglect and Disabilities of Children in First Placements in Demonstration Counties, from Pre-Waiver to Waiver Period County Abuse/Neglect and Disabilities Sexually Abused Alleged Victim of Abuse or Neglect Cognitive Disabilities Physical Disabilities Ashtabula   Belmont  Clark Crawford Fairfield Franklin Greene Lorain Medina Muskingum Portage Richland Stark

Setting of Child’s First Placement
Several Counties Increased First Placements with Relatives During the Waiver Table 6.4 Changes in Settings of First Placements in Demonstration Counties from Pre-Waiver to Waiver Period County Setting of Child’s First Placement Residential Treatment Center Group Home Foster Home Nonlicensed Nonrelative Relative Independent Living Detention Facility or Hospital Adoptive Ashtabula    Belmont Clark Crawford Fairfield Franklin Greene Lorain Medina Muskingum Portage Richland Stark

Waiver Effect on the Exit Type from First Placements Overall
Table 6.5 Waiver Effects on Exit Types from First Placements for All Demonstration Counties First Placements Ending With: Percentage of Cases Waiver Effect Actual Under the Waiver Counterfactual Projection Reunification 45.34 56.74 -11.40* Custody to Relative 18.12 14.46 3.66* Adoption 9.26 0.00 Runaway 1.04 0.54 0.50* Othera 26.22 19.02 7.20* Total 100.00 N/A

The Waiver Shortened the Overall Median Duration of First Placements
Table 6.6 Waiver Effects on Duration of First Placements for All Demonstration Counties First Placements Ending With: Median Placement Duration in Months Waiver Effect Actual Under the Waiver Counterfactual Projection Reunification 3.04 2.90 0.14 Custody to Relative 6.56 6.28 0.28 Adoption 31.78 32.00 -0.22 Runaway 6.96 9.24 -2.28 Any type of exit 4.50 4.90 -0.40*

Significant County-Level Effects of Where Children Exited from Placement
The Waiver significantly increased the percentage of first placements that ended with custody to relatives in six of the 13 demonstration counties and decreased it in one county. The Waiver significantly increased runaways in three counties, although the overall effect was small The Waiver decreased reunifications in the largest county. The Waiver increased adoptions in one county, with no overall effect. The Waiver increased “other” exit types in one county (the largest) and decreased them in two others, for an overall increase

County-Level Waiver Effects on Exit Type
Table 6.7 County-Level Waiver Effects on Exit Types from First Placements County Exit Type Reunification Custody to Relative Adoption Runaway Othera Ashtabula  Belmont  Clark  Crawford Fairfield Franklin Greene Lorain Medina Muskingum Portage Richland Stark Overall

Some County-Level Waiver Effects on Median Duration to Each Exit Type
Table 6.8 County-Level Waiver Effects on Median Duration of First Placements County Exit Type Reunification Custody to Relative Adoption Runaway Any Type of Exit Ashtabula  Belmont Clark Crawford Fairfield  Franklin  Greene Lorain Medina Muskingum Portage Richland Stark Overall Table 6.8 presents the Waiver’s county-level effects on median length of stay in foster care.[1] As with exit types, overall findings on placement duration tend to be driven by results in just a few counties, and the case studies in the next chapter will help explain significant findings (or lack of them) in light of the activities that took place in several individual counties. Table 6.8 specifies whether the Waiver significantly shortened () or significantly lengthened () the median amount of time in placement for each exit type, or whether it had no significant effect (). The effects include: The Waiver significantly lengthened placements ending with reunification in three counties and shortened them in one (the largest county), so that the overall effect was not significant. The Waiver slightly lengthened placements ending with custody to relatives in one county, although the overall effect was not significant. The Waiver shortened placements ending in adoption in two counties, but there was no overall significant effect. The Waiver lengthened placements with any type of exit in two counties and shortened them in one other (the largest), so that the overall effect was to shorten the placements. [1] Table IV-5 in Appendix IV presents county-level data on placement duration, including confidence intervals for county-specific Waiver effects.

No County-Level Waiver Effect on Reentry
Table 6.9 Re-entry After Reunification from First Placement a County Actual Rate Under the Waiver (%) Median Duration Before Re-Entry (months) Ashtabula 26.76 15.32 Belmont 36.10 11.46 Clark 27.30 9.22 Crawford 23.08 18.40 Fairfield 37.44 8.12 Franklin 41.64 9.84 Greene 30.44 14.04 Lorain 23.70 12.88 Medina 38.60 16.22 Muskingum 35.38 10.08 Portage 31.70 11.90 Richland 38.40 13.46 Stark 32.44 15.56 Overall 12.08

Putting it all together: Case Studies of 6 Demonstration sites
Clark, Fairfield, Franklin, Lorain, Muskingum, Stark Major organizational changes adopted and outcomes achieved Offers insights into dynamics of child welfare practice under the Waiver, but cannot be extrapolated to all counties

Case Study Findings (1): Lorain and Muskingum
Commitment to systemic change appears to have fostered a greater degree of success, in terms of fiscal and participant outcomes, than in other evaluation counties. More use of managed care strategies More focus on prevention Increased spending on non-foster care Fewer children in PPLA Quicker adoption, more adoption More exits to relative custody This exercise in examining individual counties brings into focus a crucial underlying dynamic: the substantial variability that exists across counties in both context and actions may serve to obscure the full effects of the Waiver. With only 14 counties in each group, systematic patterns of change are extremely difficult to detect. The Waiver may have a powerful effect on certain demonstration counties, but this effect will likely not be visible in the aggregate.

Case Study Findings (2) Other four counties had less clear-cut results despite initiating a variety of reform efforts. Clark – reformed front-end of the system through expansion of home-based services and collaboration with juvenile court; result was reduced placement days. Franklin – focus on external reform, contracting with private providers; result was reduced LOS in foster care. Fairfield and Stark – less consistent reform agendas during Waiver due in part to major organizational challenges; no significant improvements in outcomes.

Overall Results Mix of evidence of significant impact of the Waiver, areas where systematic impact is lacking, and signs of substantive change in individual counties Confounding circumstances: Funding context (levies, state cutbacks) County-administered system so county-to-county variation Lack of management tools Low financial risk Small sample sizes; differences among data sets Suggests that the flexibility provided by the Waiver… facilitates reform efforts where other factors are already conducive, but may not itself be robust enough to generate fundamental reform of the state’s public child welfare system.

The Future ProtectOhio Good likelihood of Waiver extension
Evaluation focus: More in-depth analysis of foster care utilization More examination of child safety Further analysis of expenditure patterns Targeted investigation of special topics – use of relatives, mental health services, court-referred children, etc.

Summary of Findings from the 5-year study September 2003

Similar presentations

Presentation on theme: "Summary of Findings from the 5-year study September 2003"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Summary of Findings from the 5-year study September 2003

Similar presentations

Presentation on theme: "Summary of Findings from the 5-year study September 2003"— Presentation transcript:

Similar presentations

About project

Feedback