1. What the data can tell us: Evidence, Inference, Action! 1 Early Childhood Outcomes Center

2. 2 Using data for program improvement E vidence I nference A ction

3. 3 Evidence Evidence refers to the numbers, such as “45% of children in category b” The numbers are not debatable

4. 4 Inference How do you interpret the #s? What can you conclude from the #s? Does evidence mean good news? Bad news? News we can’t interpret? To reach an inference, sometimes we analyze data in other ways (ask for more evidence)

5. 5 Inference Inference is debatable -- even reasonable people can reach different conclusions from the same set of numbers Stakeholder involvement can be helpful in making sense of the evidence

6. 6 Action Given the inference from the numbers, what should be done? Recommendations or action steps Action can be debatable – and often is Another role for stakeholders

7. 7 Data Quality Checks Missing Data Pattern Checking

8. 8 Missing Data - Overall How many children should the state be reporting to OSEP in the SPP/APR table? –i.e., how many children [had entry data,] exited in the year, and stayed in the program 6 months? –Do you have a way to know? What percentage of those children do you have in the table? These questions apply whether or not you are sampling.

9. 9 Are you missing data selectively? By local program By child characteristic –Disability? –Type of exit? (children who exit before 3) By family characteristic –Families who are hard to reach (and may leave unexpectedly) ***Which of these can you check on?***

10. 10 Pattern Checking Checking outcome data for quality: Looking for patterns (see pattern checking document)

11. 11 Looking for Sensible Patterns in the Data Putting together your “validity argument.” You can make a case that your data are valid if …..they show certain patterns. The quality of your data is not established by one or two numbers. The quality of the data is established by a series of analyses that demonstrate the data are showing predictable patterns.

12. 12 Drilling down: Looking at data by local program All analyses that can be run with the state data can be run with the local data The same patterns should hold and the same predictions apply. Need to be careful about the size of N with small programs.

13. 13 If Data are High Quality Assuming: pattern checking and review of individual child records/data show high quality data Now what?

14. Explaining variation Who has good outcomes = Do outcomes vary by Region of the state? Amount of services received? Type of services received? Age at entry to service? Level of functioning at entry? Family outcomes? Education level of parent? 14

15. 15 Data Analysis to identify where differences exist but shouldn’t Data Analysis to identify where program improvements might be made

16. 16 Analysis by Local Area If local areas are serving similar kinds of children, progress data should be similar. Analysis: Distributions by local area Are children making more/less progress in some local programs? Why?

17. 17 Analysis by Outcome Analysis: Distribution across the outcomes Are children making progress in some outcome areas more than others? Why?

18. 18 Analysis by certain characteristics Progress data should not be related to certain characteristics (e.g., race/ ethnicity, primary language, gender). Analyses: Distributions by characteristics Are children with some characteristics making more progress than others? Why?