1. Integrating Outcomes Learning Community Call February 8, 2012
Improving and Interpreting
Child Outcomes Data

2. Objectives for the call
To discuss strategies and resources for improving child outcomes data quality
To focus on looking at data through pattern checking as a mechanism for improving data quality
Early Childhood Outcomes Center

3. Quality Assurance: Looking for Quality Data
I know it is in here
somewhere
Early Childhood Outcomes Center

4. Ongoing checks for data quality
Before
Good training and assessment
Efficient data systems
Timely and accurate data entry
During
Ongoing supervision of implementation
Feedback to implementers
Refresher training
After
Review sample of completed COSFs
Pattern checking analysis
Early Childhood Outcomes Center

5. Quality checks before data collection begins
Good training and assessment
Efficient data systems
Timely and accurate data entry
Early Childhood Outcomes Center

6. Promoting Quality Data
Through training and communication (before and during data collection) related to:
Assessment
Understanding the COSF process
Age expectations
Data entry
Early Childhood Outcomes Center

7. Promoting Quality Data
Through data systems and verification, such as
Data system error checks
Good data entry procedures
Early Childhood Outcomes Center

8. Quality checks during data collection
Ongoing supervision of implementation
Feedback to implementers
Refresher training
Early Childhood Outcomes Center

9. e.g. Quality review of COS team discussion
Ongoing supervision
Review of the process
Is the process high quality?
Are teams reaching the correct rating?
Methods
Observation
Videos
e.g. Quality review of COS team discussion
Early Childhood Outcomes Center

10. Ongoing Supervision Feedback to teams is critical Refresher training
Beware of
Auto pilot
Drift
Early Childhood Outcomes Center

11. Quality review through process checks
Provider surveys
Self assessment of competence
Knowledge checks
Process descriptions (who participates?)
Identification of barriers
Early Childhood Outcomes Center

12. Quality checks after data collection
Review sample of completed COSFs
Pattern checking analysis
Early Childhood Outcomes Center

13. Quality Review of Completed COS Forms
adequate evidence?
match the outcome area?
based on functional behaviors?
across settings and situations?
ratings consistent with evidence?
Early Childhood Outcomes Center

14. Pattern Checking Analysis
The quality of the child outcomes data are established by a series of analyses that demonstrate the data are showing predictable patterns.
Pattern Checking Table

15. Ongoing checks for data quality - Poll
Before
Good training and assessment
Efficient data systems
Timely and accurate data entry
During
Ongoing supervision of implementation
Feedback to implementers
Refresher training
After
Review sample of completed COSFs
Pattern checking analysis
Early Childhood Outcomes Center

16. Take Home Message
If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else.
What do you if the data are not as good as they should be?
Answer: Continue to improve data collection through ongoing quality assurance
Early Childhood Outcomes Center

17. Looking at Data
Early Childhood Outcomes Center

18. Using data for program improvement = EIA
Evidence Inference Action
Early Childhood Outcomes Center

19. Evidence Evidence refers to the numbers, such as
“45% of children in category b”
The numbers are not debatable
Early Childhood Outcomes Center

20. Inference How do you interpret the #s?
What can you conclude from the #s?
Does evidence mean good news? Bad news? News we can’t interpret?
To reach an inference, sometimes we analyze data in other ways (ask for more evidence)
Early Childhood Outcomes Center

21. Inference
Inference is debatable -- even reasonable people can reach different conclusions
Stakeholders can help with putting meaning on the numbers
Early on, the inference may be more a question of the quality of the data
Early Childhood Outcomes Center 22

22. Action Given the inference from the numbers, what should be done?
Recommendations or action steps
Action can be debatable – and often is
Another role for stakeholders
Again, early on the action might have to do with improving the quality of the data
Early Childhood Outcomes Center

23. Promoting quality data through data analysis23

24. We have expectations about how child outcomes data should look
Checking to see if ratings accurately reflect child status: Pattern checking
We have expectations about how child outcomes data should look
Compared to what we expect
Compared to other data in the state
Compared to similar states/regions/school districts
When the data are different than expected ask follow up questions
Early Childhood Outcomes Center

25. Questions to ask Do the data make sense?
Am I surprised? Do I believe the data? Believe some of the data? All of the data?
If the data are reasonable (or when they become reasonable), what might they tell us?
Early Childhood Outcomes Center

26. Let’s look at some data …
Early Childhood Outcomes Center 26

27. Remember: Child Outcomes (see reference sheet)
1. Positive social-emotional skills (including social relationships); 2. Acquisition and use of knowledge and skills (including early language/communication [and early literacy]); and 3. Use of appropriate behaviors to meet their needs
Early Childhood Outcomes Center

28. Remember: COSF 7-point scale (see reference sheet)
7-Completely- Age appropriate functioning in all or almost all everyday situations; no concerns
6- Age appropriate functioning, some significant concerns
5-Somewhat- Age appropriate functioning some of the time and/or in some settings and situations
4- Occasional age-appropriate functioning across settings and situations; more functioning is not age-appropriate than age appropriate.
3-Nearly- Not yet age appropriate functioning; immediate foundational skills most or all of the time
2- Occasional use of immediate foundational skills
1-Not yet- Not yet age appropriate functioning or immediate foundational skills
Early Childhood Outcomes Center

29. Remember: Reporting Categories (see reference sheet)
Percentage of children who: a. Did not improve functioning b. Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers c. Improved functioning to a level nearer to same-aged peers but did not reach it d. Improved functioning to reach a level comparable to same-aged peers e. Maintained functioning at a level comparable to same-aged peers
3 outcomes x 5 “measures” = 15 numbers
Early Childhood Outcomes Center

30. Remember: Summary Statements (see cheat sheet)
Of those children who entered the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program.
c + d___
a + b + c + d
Early Childhood Outcomes Center

31. Remember: Summary Statements (see cheat sheet)
The percent of children who were functioning within age expectations in each Outcome by the time they exited the program.
d + e__
a + b + c + d + e
Early Childhood Outcomes Center

32. National Data
Early Childhood Outcomes Center

33. Note: Based on 33 States with highest quality data

34. Early Childhood Outcomes Center
Note: Based on 33 States with highest quality data
Early Childhood Outcomes Center

35. Early Childhood Outcomes Center
Note: Based on 29 States with highest quality data
Early Childhood Outcomes Center

36. Early Childhood Outcomes Center
Note: Based on 29 States with highest quality data
Early Childhood Outcomes Center

37. Sample State Data
Early Childhood Outcomes Center

38. Have a good outcome measurement
Key to Good Data
Have a good outcome measurement
SYSTEM
Early Childhood Outcomes Center

39. Quality Checks Missing Data Pattern Checking
Full ECO Pattern Checking Table:
Early Childhood Outcomes Center

40. Missing Data - Overall
How many children should the state be reporting to OSEP in the SPP/APR table?
i.e., how many children exited in the year, and stayed in the program 6 months?
Do you have a way to know?
What percentage of those children do you have in the table?
Are you missing data selectively? (by local program, by child or family characteristic?)
Early Childhood Outcomes Center

41. Pattern Checking
The quality of the child outcomes data are established by a series of analyses that demonstrate the data are showing predictable patterns:
Across Outcomes
Across time
Compared with disability information
Early Childhood Outcomes Center

42. Information available for pattern checking
OSEP Progress Categories
Entry Data
Exit Data
Summary Statements
Early Childhood Outcomes Center

43. Predicted Pattern #1
1a. Children will differ from one another in their entry scores in reasonable ways (e.g., fewer scores at the high and low ends of the distribution, more scores in the middle). .
1b. Children will differ from one another in their exit scores in reasonable ways.
1c. Children will differ from one another in their OSEP progress categories in reasonable ways.
Early Childhood Outcomes Center

44. Rationale
Evidence suggests EI and ECSE serve more mildly than severely impaired children (e.g., few ratings/scores at lowest end). Few children receiving services would be expected to be considered as functioning typically (few ratings/scores in the typical range).
Early Childhood Outcomes Center

45. Predicted Pattern #1 (cont’d)
Analysis
Look at the distribution of rating/scores at entry and exit and the data reported to OSEP.
Look at the percentage of children who scored as age appropriate (or not) on all three outcomes at entry and at exit.
Question: Is the distribution sensible? What do you expect to see?
Early Childhood Outcomes Center

46. Exit Data
Early Childhood Outcomes Center

47. Poll Do you pull child outcomes data by the 1-7 ratings on the scale?
What have you learned?
Early Childhood Outcomes Center

48. Hypothetical State Data: Outcome 1 Exit Data
Early Childhood Outcomes Center

49. Hypothetical State Data: Outcome 2 Exit Data
Early Childhood Outcomes Center

50. Hypothetical State Data: Outcome 3 Exit Data
Early Childhood Outcomes Center

51. Poll
Do you look at children exiting the program at age expectations across all three outcomes (6 & 7)?
What have you learned?
Early Childhood Outcomes Center

52. Question?
What would you anticipate as the percentage of children who leave your program and no longer need special education services?
Early Childhood Outcomes Center

53. ‘At Age’ across all three outcomes: OSEP progress categories
Percent of children that scored a 6 or 7 on all three outcomes at entry
%xx (n = xx)
Percent of children that scored a 6 or 7 on all three outcomes at exit
Early Childhood Outcomes Center

54. Hypothetical State Data: Across all 3 outcomes
Early Childhood Outcomes Center

55. Poll Do you pull child outcomes data by OSEP progress categories?
Are you comparing your data to national or other states’ data?
What have you learned?
Early Childhood Outcomes Center

56. OSEP Categories
Early Childhood Outcomes Center

57. Hypothetical State Data: Outcome 1 OSEP Categories
Early Childhood Outcomes Center

58. Hypothetical State Data: Outcome 2 OSEP Categories
Early Childhood Outcomes Center

59. Hypothetical State Data: Outcome 3 OSEP Categories
Early Childhood Outcomes Center

60. Predicted Pattern #4b
4b. Children will not show huge changes between entry and exit.
Analyses:
Comparison of entry and exit scores (exit score minus entry score)
Question: What do we expect to see?
Early Childhood Outcomes Center

61. Poll
Are you pulling your data to look at the number of children who increase by 4 or more points on the 7 point scale?
What have you learned?
Early Childhood Outcomes Center

62. Outcome 1: Children that increased by 4 or more points from entry to exit
Early Childhood Outcomes Center 62

63. Outcome 2: Children that increased by 4 or more points from entry to exit
Early Childhood Outcomes Center 63

64. Outcome 3: Children that increased by 4 or more points from entry to exit
Early Childhood Outcomes Center 64

65. Wrap-up
Early Childhood Outcomes Center

66. Drilling down: Looking at data by local program
All analyses that can be run with the state data can be run with the local data
The same patterns should hold and the same predictions apply.
Need to be careful about the size of N with small programs.
Early Childhood Outcomes Center

67. Are your data high quality?
Are the missing data less than 5%?
Do your state’s data support the predicted patterns?
If not, where are the problems?
What do you know or can you find out about why they are occurring?
Early Childhood Outcomes Center

68. Questions to ask Do the data make sense?
Am I surprised? Do I believe the data? Believe some of the data? All of the data?
If the data are reasonable (or when they become reasonable), what might they tell us?
Early Childhood Outcomes Center 68

69. Discussion
Early Childhood Outcomes Center 69

70. Find more resources at: http://www. the-eco-center-org
Early Childhood Outcomes Center