1. Integrating child welfare and Medicaid data to identify and predict superutilization of services for kids in foster care
Strengthening Child Safety and Wellbeing through Integrated Data Solutions: Child Maltreatment Solutions Network Conference
State College, PA
September 27, 2018
Elizabeth Weigensberg, Mathematica Policy Research

2. Agenda Learning objectives Why link child welfare and health data
Superutilization of child welfare and other services
Study overview
Measuring superutilization
Results
States’ research evidence use
Reflections and suggestions

3. Learning Objectives

4. Learning Objectives
Share insights about the potential for linking Medicaid and child welfare data
Learn about innovative ways to assess high levels of service utilization across multiple service systems
Understand how linked services data can be used to predict key outcomes for children in foster care
Discuss how linked Medicaid and child welfare data can inform targeted interventions for children in foster care

5. Why link child welfare and health data

6. Understanding Health Needs
About 437,000 children are in foster care
Children in foster care are eligible for Medicaid
Nearly a billion Medicaid dollars are spent annually on youth in foster care
Children in foster care account for a disproportionate amount of Medicaid expenditures for behavioral health
Data sharing across Medicaid and child welfare is rare despite being primary service providers

7. Data-Informed Decisions
Need for information about how child welfare agencies, Medicaid agencies, and related services come together to support the needs of these children and their families
Linked data needed to consider interventions at key points could prevent children and families from becoming high users of services

8. Superutilization of child welfare and other services

9. Project Collaboration
Work was conducted in collaboration with Casey Family Programs, whose mission is to provide, improve, and, ultimately, prevent the need for foster care

10. Acknowledgements: Project Team and Partners
Mathematica Policy Research
Elizabeth Weigensberg, Matt Stagner, Derekh Cornwell, Lindsey Leininger, Sarah LeBarron, Jonathan Gellar, Sophie MacIntyre, Richard Chapman, Christina Alva, Stephanie Barna, Daniel Kassler, Brenda Natzke, Jessica Nysenbaum, Matt Mleczko, Kelley Monzella, Nora Paxton, Liz Potamites, Dmitriy Poznyak, Christine Ross, Michael Sinclair, and Fei Xing
Casey Family Programs
Erin Maher, Peter J. Pecora, Kirk O’Brien, Linda Jewell Morgan, Fred Simmens, Toni Rozanski, Nadia Sexton
Tennessee site partners
Department of Children’s Services, TennCare
Florida site partners
Department of Children and Families, Agency for Healthcare Administration, Eckerd Kids
Expert reviewers
Richard Thompson (Juvenile Protection Association), Brett Drake (Washington University in St. Louis), Mary Armstrong (University of South Florida), and Laura Nolan (Mathematica Policy Research)

11. What is superutilization of child welfare and other services?
Research Questions
What is superutilization of child welfare and other services?
What are the types of superutilization?
What predicts high levels of placement instability at the time of entry into foster care?

12. Project Sites and Data Partners
Tennessee:
Hillsborough, Pinellas & Pasco Counties, Florida:

13. Data Sources and Linking
Tennessee Department of Children’s Services
Child Welfare
TennCare
Medicaid Claims
Florida Department of Children and Families
Agency for Health Care Administration
Substance Abuse and Mental Health Program Services
For TN:
82% of children with out-of-home placements were linked to Medicaid data
For FL:
Among children with out-of-home placements
89.1% were linked to Medicaid
22.7% were linked to SAMHIS
12.9% were linked to Eckerd
Eckerd Kids/Child Welfare Community-Based Care Purchased Services

14. Analysis Approach Two phases of the analysis:
Descriptive and latent class analyses:
Describe characteristics of high service use
Identify types (latent classes) of superutilization
Describe characteristics related to each type of superutilization
Predictive analysis:
Assess predictive factors for risk of superutilization (placement instability) at time of entry into foster care

15. Study Samples Hillsborough, Pinellas, and Pasco Counties, Florida
Children entering out-of-home custody between:
Sept 1, 2013 – Dec 31, N = 6,695
Jan 1, 2012 – Jan 1, N = 8,290
Tennessee
Children entering out-of- home custody between:
July 1, 2011 – Dec 31, N = 21,672
July 1, 2012 – Jan 1, N = 12,056
Descriptive and latent class analysis sample
Predictive analysis sample

16. Child Welfare, Medicaid and Other Services
Tennessee :
Service receipt among study sample while in child welfare custody:
84.1% received child welfare services
85.6% received Medicaid services
17.0% received inpatient, 85.4% received outpatient, 70.1% received emergency services
Hillsborough, Pinellas, and Pasco Counties (Florida):
19.8% received child welfare community-based care (CBC) purchased services
91.5% received Medicaid services
17.0% received inpatient, 90.8% received outpatient, 64.5% received emergency services
15.8% received other substance abuse or mental health

17. Measuring Superutilization

18. Measurement of Superutilization
Superutilization Dimension
Child welfare custody/ placements
Total number of episodesa
Total number of placement movesa
Total episode length of staya
Average share of time in congregate/ residential/ group homea
Total placement cost per year (TN only)
Child welfare services
Child welfare services per year (TN)
Child welfare CBC purchased services (FL)
Service cost per year
Medicaid services
Emergency services per year
Inpatient services per year
Outpatient services per year
Other substance use / mental health services (SAMH)
Mental health services per year (FL only)
Substance abuse services per year (FL only)
SAMH: Substance Abuse and Mental Health data in Florida.

19. Other Measurement Considerations
Establishing a threshold:
Used a 90th percentile threshold to identify superutilization
Address influence of age on service utilization:
Calculated threshold by age
Address exposure to study window:
Calculated annualized rates using service use among time in child welfare custody

20. Children in Study Sample Identified as Experiencing Superutilization
Measure of superutilization
TN Study Sample
n=21,672
FL Study Sample
n=6,695
Number (Percent)
Total number of episodes
3,190 (14.7%)
894 (13.4%)
Total number of placement moves
3,387 (15.6%)
1,078 (16.1%)
Total placement cost per year
2,552 (11.8%)
N.A.
Total episode length of stay
2,722 (12.6%)
740 (11.1%)
Average proportion of time in congregate care/residential/group home
1,827 (8.4%)
609 (9.1%)
Child welfare/CBC purchased services per year
2,432 (11.2%)
601 (9.0%)
Child welfare/CBC purchased services cost per year
1,789 (8.3%)
567 (2.6%)
Medicaid inpatient services per year
1,257 (5.8%)
380 (5.7%)
Medicaid outpatient services per year
2,261 (10.4%)
762 (5.7%)
Medicaid emergency services per year
2,213 (10.2%)
Mental health services per year
560 (8.5%)
Substance abuse services per year
262 (3.9%)
Total children identified by any measure of superutilization
12,332 (56%)
3,726 (55%)
“Dimensionality” among measures with limited overlap:
- Each measure mostly identified different children, leading to high percentage of sample identified as experiencing superutilization
- Indicates likely distinct types of superutilization among children

21. Types of Superutilization: Latent Class Analysis Results

22. Latent Class Analysis
Conduct analysis to group children into distinct groups or “classes” based on observed superutilization measures
Review analysis results to identify number of classes
Describe and label classes based on measures of service use
Describe characteristics of individuals in each class
- Statistical method for grouping observations (children)
into distinct groups or “classes” based on observed
variables (superutilization measures)
- Allows us to examine distinct combinations of
superutilization patterns that may define a given class
- Ultimately, individual membership in a latent class is
based on probabilistic assignment
- Classes are defined based on how clearly variables are associated with those classes (probabilities close to one or zero) clearly differentiate classes.

23. Types of Superutilization: Latent Class Results for TN
7 classes of superutilization:
Class Name
Description
Percentage of Superutilization population
Class 1
Foster care placement instability
23.0%
Class 2
Multiple foster care episodes
12.2%
Class 3
Child welfare service use
21.5%
Class 4
Duration in foster care
7.3%
Class 5
Medicaid outpatient service use
9.1%
Class 6
Medicaid emergency service use
8.6%
Class 7
Use of group/congregate care placements and high placement costs
18.1%
Total percentage of superutilizers
100.0%

24. Overview of Tennessee Latent Classes

25. TN: Foster Care Placement Instability
Defining characteristics:
High number of foster care placement moves (54.7% had 7 or more placement moves)
Other key findings:
92% received child welfare services and among those receive services:
71.4% received clothing assistance
28.9% received substance abuse testing and treatment
22.8% received family and parenting support services
For reasons for removal:
42.6% had neglect
39.1% had parent drug abuse
14.9% had physical abuse as a reason for removal (highest among all classes)
62.0% exited custody with 48.1% being reunified and % adopted

26. Types of Superutilization: Latent Class Results for FL
8 classes of superutilization:
Class Name
Description
Percentage of Superutilization population
Class 1
Child welfare CBC purchased services use
14.0%
Class 2
Complex child welfare and Medicaid service use
5.4%
Class 3
Medicaid and mental health service use
23.2%
Class 4
Foster care placement instability
10.2%
Class 5
Multiple foster care episodes
19.9%
Class 6
Duration in foster care
5.6%
Class 7
Use of group home/residential treatment placements
8.8%
Class 8
Medicaid emergency services use
12.8%
Total percentage of superutilizers
100.0%

27. Overview of Florida Latent Classes

28. Predicting Superutilization: Predictive Analysis Results

29. Goals of Predictive Analysis
Predict risk of superutilization within 12 months of entering an out-of-home custody episode
Predictive outcome: Superutilization defined as a high number of placement moves within an out-of-home custody episode (also referred to as placement instability)
12-month predictive time period and look-back time period
Assess overall predictive accuracy
Explore what predictive factors are most important
Child characteristics
Reasons for removal, assessments/risk level
Prior child welfare history (investigations, episodes, placements)
Prior child welfare, Medicaid, and substance abuse/mental health service use
Be sure to note there were numerous measure of superutilization but the sites selected “placement moves” as the outcome to focus on for the predictive analysis.

30. Predictive Analytics: A Focus on Placement Instability
Set-up the data: Identify sample, structure the data and measure outcome
Develop the model: Create training/test data sets and identify predictor variables
Compare predictive modeling approaches
Identify most important predictors

31. Structuring the Data: Study Sample and Prediction Period
Qualifying sample:
Florida: Children entering out-of-home custody with a start date between January 1, 2012, and December 1, 2014 (N = 8,290)
Tennessee: Children entering out-of-home custody with start date between July 1, 2012, and December 1, 2014 (N = 12,056)
Need to structure the data in a way that is amenable to predictive modeling. Specifically, we need to have a clean lookback and prediction period. We decided that a 12 month lookback and prediction period was easier to interpret and likely to be most policy-relevant.
To accommodate for 12-month lookback and prediction periods, a qualifying out-of-home custody start had to take place between Jan 1, 2012-Dec 1, 2014
As shorthand, we refer to the first out-of-home custody episode in the 12-month prediction period as the t0 episode.
Note, because we excluded Eckerd data from the analysis, we could use the full study time window available for the other data sources

32. Measuring Superutilization
State
Descriptive statistics
Mean
Standard deviation
P90 superutilization threshold
Florida
3.3
4.3 5
Tennessee
2.6
1.7 4
Number of placement moves (placement instability) during the 12-month prediction period
Superutilization was defined as youth who equaled or exceeded the 90th percentile value for total number of placement moves within the t0 episode
For Florida, 14 percent of the sample met/exceeded the threshold (5 placement moves)
For Tennessee, 20 percent of the sample met/exceed the threshold (4 placement moves)
The proportion of the sample that met or exceeded the 90th percentile threshold value for placement moves (that is, had 4 or more placement moves) is larger that the top 10 percent. The reason for this is due to the skewed distribution of placement moves and a number of ties—that is, the 90th percentile value of 4 placement moves is the same as the 80th percentile value. This results in a higher proportion of sample members meeting the threshold values.

33. Create Training and Testing Data Sets
Sample
(30%)
Training Sample
(70%)
The basic approach was to split the sample into training and test data sets. The idea is that part of the data should be used to develop the model and another part used to validate the model. Separating the training data from the test data helps guard against overfitting the model so that we can assess how well our predictions apply to cases that were not part of model development. This also helps simulate how an actual prediction tool would work in practice—i.e., it would apply to new cases that just entered the child welfare system.
This also helps simulate how an actual prediction tool would work in practice—i.e., it would apply to new cases that just entered the child welfare system.

34. Types of Variables Used for Prediction
Variable domain
Florida
Tennessee
Child demographic characteristics 
Prior investigations
Reasons for removal ─
Foster care placements
Child welfare custodial episodes
Child welfare services
Child welfare assessments
Child welfare investigation average recommended service level
Substance abuse and mental health services/assessments
Medicaid services
Regional composition

35. Compare Predictive Modeling Approaches
Compared 3 predictive modeling approaches:
Logistic regression with elastic net (EN) regularization. Logistic regression with an added constraint intended to maximize flexibility while not overfitting the data.
K-nearest neighbors (KNN). A clustering approach in which the predicted outcome of an observation is based on the outcomes of the most similar observations in the data.
Random forests (RF). An extension of classification and regression trees (CART), in which predictions are made as a sequence of binary partitions, in a flowchart-like structure.
The RF model was selected as the best model based on overall predictive performance.
Additional notes:
Logistic regression with elastic net regularization. The EN is bound by many of the same constraints as logistic regression (for example, linear functional form in the logit and distributional assumptions). The model produces interpretable results (that is, familiar regression coefficients for each predictor) while also providing a more efficient framework for selecting relevant variables compared to stepwise methods. The potential downside is that the model is less flexible in its functional form, which can lower predictive strength.
K-Nearest Neighbors (KNN). (Altman 1992) Similarity between two observations is defined by how close their predictors are to one another, summarized into a single index called the Minkowski distance. Once the most similar training observations (“nearest neighbors”) are identified, their outcomes are combined as a weighted average (weighted by distance to the test observation) in order to generate a prediction. The KNN approach is a simple and often accurate method for classification; however, it comes at expense of the ability to interpret and rank individual variables based on their relative importance. (James et al. 2013).
Random Forest (RF). CART models are well-suited as a prediction algorithm in cases with many predictors because they are robust to the inclusion of irrelevant variables and account for complex interactions between variables. RF is a flexible modeling approach that often delivers predictive performance that is superior to more traditional approaches (such as OLS or logistic regression) due to the ability to account for complicated relationships with between predictors and the outcome.

36. Predictive Performance of Random Forest Model
State
Model performance metrics
Area Under the Curve (AUC)
Sensitivity
Specificity
Accuracy
Positive Predictive Value (PPV)
Negative Predictive Value (NPV)
Florida
0.722
0.589
0.777
0.750
0.304
0.919
Tennessee
0.727
0.682
0.671
0.673
0.342
0.893
Area Under the Curve (AUC). How well the predictive model correctly classifies children who experience superutilization compared to children who do not. Higher values on the AUC indicate better prediction. AUC values greater than 0.7 are typically considered to indicate good predictive performance.
Sensitivity. The ability of the model to correctly identify children who experience superutilization.
Specificity. The ability of the model to correctly identify children who do not experience superutilization.
Accuracy. Agreement between children’s superutilization classification, as predicted by the model, compared to their actual classification (weighted by the prevalence of superutilization).
Positive Predictive Value (PPV). The probability that a child classified as experiencing superutilization actually does experience superutilization.
Negative Predictive Value (NPV). The probability that a child classified as not experiencing superutilization actually does not experience superutilization.
We examined three different statistical models and compared their overall predictive performance based on how well they performed on the training data (i.e., 70% of the full sample). The primary metric we used to compare was the Area Under the Receiver Operator Characteristic Curve (AUC). Higher values of this (typically above 0.70) are indicative of good predictive accuracy. The RF performed best on this criterion. Those results are in the full report. Here, we focus on the RF results for the test sample.
We examined the other components as well as reported in the table. However, these can be manipulated by changing the cut points above which a child is classified as experiencing placement instability. We consciously chose to increase sensitivity at the expense of specificity under the assumption that, from a policy perspective, it was more important to correctly identify children who experience superutilization than to correctly identify children who do not.

37. Tennessee: Top 8 Most Important Predictors
Source: Tennessee DCS; TennCare; American Community Survey 2015; U.S. Census 2010.

38. Tennessee: Age
Older children are at greater risk of placement instability; the largest change in the predicted probability of placement instability occurs between ages 11 and 12.
The first plot shows that the predicted probability of experiencing superutilization based on placement instability is relatively steady for kids at younger ages before increasing notably between ages 11 and 12. Eleven-year-olds have a 0.15 predicted probability of experiencing superutilization, which increases to 0.20 at age 12 (a 33 percent increase). Children ages 15 and 16 have the highest probability of experiencing superutilization. This suggests that child welfare agencies may want to focus on certain age groups, specifically adolescents.

39. Tennessee: Medicaid Outpatient Behavioral Health services
Prior receipt of 1 or 2 Medicaid outpatient behavioral health services is associated with the largest change in the probability of placement instability.
Although the magnitude of change in the Medicaid measures is not particularly high, the fact that these variables seem to have predictive strength suggests that it may be useful for child welfare agencies to know about Medicaid service history at the time of entry into custody.

40. Florida: Top 10 Most Important Predictors
This figure lists the top ten most important predictors of placement instability out of the 53 variables that were used in the final model. The variables are rank ordered based on their relative contribution to overall model performance, as measured by the average decrease in the Gini index—a measure of how much the model performance changes when a variable is excluded.
Age at entry is, by far, the most important predictor of placement instability.
Next, we show the effects of each of the top 10 predictors on the probability of experiencing placement instability.
Source: Florida DCF; Florida AHCA data; Florida SAMH; American Community Survey 2015; Census 2010.

41. Florida: Medicaid Outpatient Behavioral Health Services
The number of outpatient behavioral health services received via Medicaid during the lookback period slightly increases the chances of placement instability.

42. Florida: SAMH Substance Abuse Services
Receiving one substance abuse service from SAMH has a predicted probability of placement instability of 0.15, which is a 50% increase compared to receiving no substance abuse services
If a child receives one substance abuse service, his or her predicted probability of placement instability increases from 0.10 to 0.15—a 50 percent increase.

43. Research Evidence Use

44. Research Evidence Use Action Plan Priorities:
Recruitment and retention of therapeutic foster care providers to reduce placement instability and use of group care
Engage provider community
Action Plan Priorities:
Case record review of young children going to ER for physical health issues
Identify service gaps
Foster more collaboration between state child welfare and Medicaid departments

45. Project Publications Superutilization project web page: Issue Brief
findings/projects/super-utilizers-of-child-welfare-and-other-services
Issue Brief
Executive Summary
Final Report
Interactive web-brief:

46. Reflections and Suggestions

47. Reflections
Linked Medicaid and child welfare data offers a tremendous opportunity to learn about the healthcare needs and service receipt of children (and parents) to develop targeted service interventions
Understand current cross-system service receipt to identify opportunities for improvement
Identify types of service utilization to more efficiently address child welfare and health needs
Identify subpopulations to target limited resources
Learn what information at time of entry into child welfare (from across systems) can be useful predictors
Heightened importance of behavioral health data, especially substance use treatment with the opioid epidemic
Data linking and analysis can be a bridge to better cross- agency collaboration

48. Suggestions
Consider feasibility of linking Medicaid or other health data
Data on eligibility, service claims, prescriptions, diagnosis codes
Medicaid managed care makes this challenging but not impossible
Behavioral health data can come from many sources
Establish teams with cross-disciplinary data expertise
Collaborate with Medicaid data experts
Actively engage child welfare and Medicaid agency partners in using data to understand needs of children and families
Ensure all parties will learn valuable information
Develop data-informed actionable solutions to improve services and outcomes

49. Final Thoughts Context is important!
Variability across states, counties, and communities
Include measures of context and policy differences
Opportunities for state agencies with new Comprehensive Child Welfare Information System (CCWIS)
Focus on data quality
Emphasis on interoperability, linkages with data across systems
Improved reporting and advanced analytics

50. Elizabeth Weigensberg
Senior Researcher & Lead for State/Local Child Welfare
Mathematica Policy Research – Chicago Office