Prepared for the ECO Advisers Meeting February 2, 2011 National Data on Child Outcomes: Are We There Yet?

Computing national numbers 2 Early Childhood Outcomes Center Department of Education needs national numbers for GPRA for Part C and Part B preschool Ideally, we would add up the numbers from each of the states and get the national numbers but….

States are in various stages with regard to building their child outcomes measurement systems Some phrased in Some changed approaches 3Early Childhood Outcomes Center Child Outcomes: What we know

Producing National Numbers: The Issues Missing data? –Which states are reporting data on all the children who stayed 6 months and exited in 08-09?* Data quality –Which states have invested sufficient resources to be producing credible data? –Which states have been producing outcomes data long enough to have credible data? * A very few states are sampling. 4 Early Childhood Outcomes Center

Missing/Incomplete Data: Part C 5 Early Childhood Outcomes Center Aver=35.7%; 31 Sts. > 33%

Missing/Incomplete Data: Part B Preschool 6 Early Childhood Outcomes Center Aver= 21.1%; 22 Sts. > 25%

Data Quality: Outliers 7 Early Childhood Outcomes Center Some data quality issues are obvious States whose data look VERY different from other states or from what would be expected.

What would we expect? 8 Early Childhood Outcomes Center

What would we expect? 9 Early Childhood Outcomes Center

What would we expect? 10 Early Childhood Outcomes Center

What would we expect? 11 Early Childhood Outcomes Center

Producing National Estimates 1.Divide the states into those with large, medium, and small child counts. 2.Identify the states that appear to have the best data. 3.Weight by child count and calculate the overall OSEP percentages using the “best data” states in each category. 4.Use the OSEP percents and total child count for each category to get the national estimates. 12 Early Childhood Outcomes Center

1. Divide the states into those with large, medium, and small child counts. Size of Child CountStates LargeCA, NY, PA….. MediumTN, CO, AZ… SmallMT, ND, DE….. 13 Early Childhood Outcomes Center

How Weighting Works STATES w/ USEABLE DATA a%a% b %c %d %e %Child CountNew # for e A B C D E Total11,3003,300 Re-calculated a-e for this set of states29.2% 14 Early Childhood Outcomes Center

And now the results…. 15 Early Childhood Outcomes Center

16 Note: Analysis based on data from 19 states. Data were weighted by child count to represent the nation.

Early Childhood Outcomes Center17 Note: Analysis based on data from 19 states. Data were weighted by child count to represent the nation.

18 Early Childhood Outcomes Center Note: Analysis based on data from 15 states. Data were weighted by child count to represent the nation.

19 Early Childhood Outcomes Center Note: Analysis based on data from 15 states. Data were weighted by child count to represent the nation.

How trustworthy are these numbers? One possible test: What do we get when we compare this method with other methods for calculating the national numbers? 20 Early Childhood Outcomes Center

3 Approaches to National Numbers 1.Use data from states with the best data (least missing, not an outlier on “a” or “e”). Weight their data to represent states of similar size.* 2.Use data from all states. Average across the percentages. Each state weighted as one. 3.Use data from all the states. Weight each state according to their child count. * We did this twice with a smaller set of states and a slightly larger set. 21 Early Childhood Outcomes Center

Comparing the three methods See handout 22 Early Childhood Outcomes Center

Our Questions for You What do these data say to you about the outcomes of Part C and Part B Preschool? What method makes the most sense for generating the calculations? What would you use as indicators that the national data are valid and meaningful? When would you trust the data? –When all (most?) states can demonstrate valid data? –When the alternate methods converge more closely? 23 Early Childhood Outcomes Center