Integrating Outcomes Learning Community Call February 8, 2012 Improving and Interpreting Child Outcomes Data

Objectives for the call To discuss strategies and resources for improving child outcomes data quality To focus on looking at data through pattern checking as a mechanism for improving data quality Early Childhood Outcomes Center

Quality Assurance: Looking for Quality Data I know it is in here somewhere Early Childhood Outcomes Center

Ongoing checks for data quality Before Good training and assessment Efficient data systems Timely and accurate data entry During Ongoing supervision of implementation Feedback to implementers Refresher training After Review sample of completed COSFs Pattern checking analysis Early Childhood Outcomes Center

Quality checks before data collection begins Good training and assessment Efficient data systems Timely and accurate data entry Early Childhood Outcomes Center

Promoting Quality Data Through training and communication (before and during data collection) related to: Assessment Understanding the COSF process Age expectations Data entry Early Childhood Outcomes Center

Promoting Quality Data Through data systems and verification, such as Data system error checks Good data entry procedures Early Childhood Outcomes Center

Quality checks during data collection Ongoing supervision of implementation Feedback to implementers Refresher training Early Childhood Outcomes Center

e.g. Quality review of COS team discussion Ongoing supervision Review of the process Is the process high quality? Are teams reaching the correct rating? Methods Observation Videos e.g. Quality review of COS team discussion Early Childhood Outcomes Center

Ongoing Supervision Feedback to teams is critical Refresher training Beware of Auto pilot Drift Early Childhood Outcomes Center

Quality review through process checks Provider surveys Self assessment of competence Knowledge checks Process descriptions (who participates?) Identification of barriers Early Childhood Outcomes Center

Quality checks after data collection Review sample of completed COSFs Pattern checking analysis Early Childhood Outcomes Center

Quality Review of Completed COS Forms adequate evidence? match the outcome area? based on functional behaviors? across settings and situations? ratings consistent with evidence? Early Childhood Outcomes Center

Pattern Checking Analysis The quality of the child outcomes data are established by a series of analyses that demonstrate the data are showing predictable patterns. Pattern Checking Table http://www.fpg.unc.edu/~eco/assets/pdfs/Pattern_Checking_Table.pdf

Ongoing checks for data quality - Poll Before Good training and assessment Efficient data systems Timely and accurate data entry During Ongoing supervision of implementation Feedback to implementers Refresher training After Review sample of completed COSFs Pattern checking analysis Early Childhood Outcomes Center

Take Home Message If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else. What do you if the data are not as good as they should be? Answer: Continue to improve data collection through ongoing quality assurance Early Childhood Outcomes Center

Looking at Data Early Childhood Outcomes Center

Using data for program improvement = EIA Evidence Inference Action Early Childhood Outcomes Center

Evidence Evidence refers to the numbers, such as “45% of children in category b” The numbers are not debatable Early Childhood Outcomes Center

Inference How do you interpret the #s? What can you conclude from the #s? Does evidence mean good news? Bad news? News we can’t interpret? To reach an inference, sometimes we analyze data in other ways (ask for more evidence) Early Childhood Outcomes Center

Inference Inference is debatable -- even reasonable people can reach different conclusions Stakeholders can help with putting meaning on the numbers Early on, the inference may be more a question of the quality of the data Early Childhood Outcomes Center 22

Action Given the inference from the numbers, what should be done? Recommendations or action steps Action can be debatable – and often is Another role for stakeholders Again, early on the action might have to do with improving the quality of the data Early Childhood Outcomes Center

Promoting quality data through data analysis 23

We have expectations about how child outcomes data should look Checking to see if ratings accurately reflect child status: Pattern checking We have expectations about how child outcomes data should look Compared to what we expect Compared to other data in the state Compared to similar states/regions/school districts When the data are different than expected ask follow up questions Early Childhood Outcomes Center

Questions to ask Do the data make sense? Am I surprised? Do I believe the data? Believe some of the data? All of the data? If the data are reasonable (or when they become reasonable), what might they tell us? Early Childhood Outcomes Center

Let’s look at some data … Early Childhood Outcomes Center 26

Remember: Child Outcomes (see reference sheet) 1. Positive social-emotional skills (including social relationships); 2. Acquisition and use of knowledge and skills (including early language/communication [and early literacy]); and 3. Use of appropriate behaviors to meet their needs Early Childhood Outcomes Center

Remember: COSF 7-point scale (see reference sheet) 7-Completely- Age appropriate functioning in all or almost all everyday situations; no concerns 6- Age appropriate functioning, some significant concerns 5-Somewhat- Age appropriate functioning some of the time and/or in some settings and situations 4- Occasional age-appropriate functioning across settings and situations; more functioning is not age-appropriate than age appropriate. 3-Nearly- Not yet age appropriate functioning; immediate foundational skills most or all of the time 2- Occasional use of immediate foundational skills 1-Not yet- Not yet age appropriate functioning or immediate foundational skills Early Childhood Outcomes Center

Remember: Reporting Categories (see reference sheet) Percentage of children who: a. Did not improve functioning b. Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers c. Improved functioning to a level nearer to same-aged peers but did not reach it d. Improved functioning to reach a level comparable to same-aged peers e. Maintained functioning at a level comparable to same-aged peers 3 outcomes x 5 “measures” = 15 numbers Early Childhood Outcomes Center

Remember: Summary Statements (see cheat sheet) Of those children who entered the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program. c + d___ a + b + c + d Early Childhood Outcomes Center

Remember: Summary Statements (see cheat sheet) The percent of children who were functioning within age expectations in each Outcome by the time they exited the program. d + e__ a + b + c + d + e Early Childhood Outcomes Center

National Data Early Childhood Outcomes Center

Note: Based on 33 States with highest quality data

Early Childhood Outcomes Center Note: Based on 33 States with highest quality data Early Childhood Outcomes Center

Early Childhood Outcomes Center Note: Based on 29 States with highest quality data Early Childhood Outcomes Center

Early Childhood Outcomes Center Note: Based on 29 States with highest quality data Early Childhood Outcomes Center

Sample State Data Early Childhood Outcomes Center

Have a good outcome measurement Key to Good Data Have a good outcome measurement SYSTEM Early Childhood Outcomes Center

Quality Checks Missing Data Pattern Checking Full ECO Pattern Checking Table: http://www.fpg.unc.edu/~eco/assets/pdfs/Pattern_Checking_Table.pdf Early Childhood Outcomes Center

Missing Data - Overall How many children should the state be reporting to OSEP in the SPP/APR table? i.e., how many children exited in the year, and stayed in the program 6 months? Do you have a way to know? What percentage of those children do you have in the table? Are you missing data selectively? (by local program, by child or family characteristic?) Early Childhood Outcomes Center

Pattern Checking The quality of the child outcomes data are established by a series of analyses that demonstrate the data are showing predictable patterns: Across Outcomes Across time Compared with disability information Early Childhood Outcomes Center

Information available for pattern checking OSEP Progress Categories Entry Data Exit Data Summary Statements Early Childhood Outcomes Center

Predicted Pattern #1 1a. Children will differ from one another in their entry scores in reasonable ways (e.g., fewer scores at the high and low ends of the distribution, more scores in the middle). . 1b. Children will differ from one another in their exit scores in reasonable ways. 1c. Children will differ from one another in their OSEP progress categories in reasonable ways. Early Childhood Outcomes Center

Rationale Evidence suggests EI and ECSE serve more mildly than severely impaired children (e.g., few ratings/scores at lowest end). Few children receiving services would be expected to be considered as functioning typically (few ratings/scores in the typical range). Early Childhood Outcomes Center

Predicted Pattern #1 (cont’d) Analysis Look at the distribution of rating/scores at entry and exit and the data reported to OSEP. Look at the percentage of children who scored as age appropriate (or not) on all three outcomes at entry and at exit. Question: Is the distribution sensible? What do you expect to see? Early Childhood Outcomes Center

Exit Data Early Childhood Outcomes Center

Poll Do you pull child outcomes data by the 1-7 ratings on the scale? What have you learned? Early Childhood Outcomes Center

Hypothetical State Data: Outcome 1 Exit Data Early Childhood Outcomes Center

Hypothetical State Data: Outcome 2 Exit Data Early Childhood Outcomes Center

Hypothetical State Data: Outcome 3 Exit Data Early Childhood Outcomes Center

Poll Do you look at children exiting the program at age expectations across all three outcomes (6 & 7)? What have you learned? Early Childhood Outcomes Center

Question? What would you anticipate as the percentage of children who leave your program and no longer need special education services? Early Childhood Outcomes Center

‘At Age’ across all three outcomes: OSEP progress categories Percent of children that scored a 6 or 7 on all three outcomes at entry %xx (n = xx) Percent of children that scored a 6 or 7 on all three outcomes at exit Early Childhood Outcomes Center

Hypothetical State Data: Across all 3 outcomes Early Childhood Outcomes Center

Poll Do you pull child outcomes data by OSEP progress categories? Are you comparing your data to national or other states’ data? What have you learned? Early Childhood Outcomes Center

OSEP Categories Early Childhood Outcomes Center

Hypothetical State Data: Outcome 1 OSEP Categories Early Childhood Outcomes Center

Hypothetical State Data: Outcome 2 OSEP Categories Early Childhood Outcomes Center

Hypothetical State Data: Outcome 3 OSEP Categories Early Childhood Outcomes Center

Predicted Pattern #4b 4b. Children will not show huge changes between entry and exit. Analyses: Comparison of entry and exit scores (exit score minus entry score) Question: What do we expect to see? Early Childhood Outcomes Center

Poll Are you pulling your data to look at the number of children who increase by 4 or more points on the 7 point scale? What have you learned? Early Childhood Outcomes Center

Outcome 1: Children that increased by 4 or more points from entry to exit Early Childhood Outcomes Center 62

Outcome 2: Children that increased by 4 or more points from entry to exit Early Childhood Outcomes Center 63

Outcome 3: Children that increased by 4 or more points from entry to exit Early Childhood Outcomes Center 64

Wrap-up Early Childhood Outcomes Center

Drilling down: Looking at data by local program All analyses that can be run with the state data can be run with the local data The same patterns should hold and the same predictions apply. Need to be careful about the size of N with small programs. Early Childhood Outcomes Center

Are your data high quality? Are the missing data less than 5%? Do your state’s data support the predicted patterns? If not, where are the problems? What do you know or can you find out about why they are occurring? Early Childhood Outcomes Center

Questions to ask Do the data make sense? Am I surprised? Do I believe the data? Believe some of the data? All of the data? If the data are reasonable (or when they become reasonable), what might they tell us? Early Childhood Outcomes Center 68

Discussion Early Childhood Outcomes Center 69

Find more resources at: http://www. the-eco-center-org Early Childhood Outcomes Center