1. Welcome to the National ECO TA Call Improving the Quality of
Child Outcome Data
Call in number or
Materials at

2. Reminder
ECO looking for states to partner in framework development activities
Call for states interested in the partner state application on March 20, 3 pm EDT/ 2 p.m.CDT/1 p.m. MDT/Noon PDT.
See for application and call in information.

3. Today’s Presenters Christina Kasprzak, ECO at FPG
Lynne Kahn, ECO at FPG
Kathy Hebbeler, ECO at SRI
Lisa Backer, Minnesota

4. To ask a question during the presentation
Use the chat box
If you can’t see the chat box, click on the triangle in front of “Chat” to expand the box
Type your question in the box “Type chat message here”
Send to All Participants.

5. Have a good outcome measurement
Key to Good Data
Have a good outcome measurement
SYSTEM

6. Examples of Components of an Outcomes Measurement System
Data collection procedures
Professional development around data collection --- and data analysis
Ongoing supervision and monitoring of data collection
Ongoing analyses to check on the quality of the data
Etc.
This is the focus of this workshop.

7. Building quality into your outcomes measurement system
Occurs at multiple steps
Requires multiple activities

8. Building quality into your outcomes measurement system
Keep errors from occurring in the first place
Develop mechanisms to identify weaknesses that are lessening the quality of the data
Provide ongoing feedback including reports of the data to programs and providers

9. Different approaches present different kinds of challenges to quality data
For states using COSF
Are all professionals trained in the process?
Are all professionals applying the rating criteria consistently?
For states deriving OSEP data from an assessment
Are all professionals trained in the assessment and administering it properly?
Are the appropriate items/domains being used for each outcome?
Are the appropriate “cut points” or criteria for age appropriate and moved nearer to same age peers being used?

10. Today’s Focus: Using data analysis to check on the quality of your data
Remember this is only weighing the pig
Weighing the pig does not make it fatter
Need to take what you learn from the analysis and do something with it.

11. Child Outcomes Data Quality
So what do you look at to know?
Our game plan
Walk through a series of expected patterns and look at the corresponding analyses
These data are being shared as a teaching tool. Do not cite the data.
Do consider the analyses as a way to examine your own state data.

12. THIS IS A DATA “SAFE ZONE”…

13. Using data for program improvement = EIA
Evidence
Inference
Action

14. Evidence Evidence refers to the numbers, such as
“45% of children in category b”
The numbers are not debatable

15. Inference How do you interpret the #s?
What can you conclude from the #s?
Does evidence mean good news? Bad news? News we can’t interpret?
To reach an inference, sometimes we analyze data in other ways (ask for more evidence)

16. Inference
Inference is debatable -- even reasonable people can reach different conclusions from the same set of numbers
Stakeholder involvement can be helpful in making sense of the evidence

17. Action Given the inference from the numbers, what should be done?
Recommendations or action steps
Action can be debatable – and often is
Another role for stakeholders

18. Quality Checks
Missing Data
Pattern Checking

19. Missing Data - Overall
How many children should the state be reporting to OSEP in the SPP/APR table?
i.e., how many children [had entry data,] exited in the year, and stayed in the program 6 months?
Do you have a way to know?
What percentage of those children do you have in the table?
These questions apply whether or not you are sampling.

20. Are you missing data selectively?
By local program
By child characteristic
Disability?
Type of exit? (children who exit before 3)
By family characteristic
Families who are hard to reach (and may leave unexpectedly)
***Which of these can you check on?***

21. Poll Time!! If you can’t see the poll area:
If you see 3 bars after “polling”, click on the word “polling.”
If you only see the word “polling,” click on the triangle in front of “polling”.
You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).
When you see the poll question, click on your answer.

22. Pattern Checking 3 Possible Sets of Numbers OSEP Progress Categories
Entry Data
Exit Data

23. OSEP Progress Categories
Did not improve functioning.
Improved functioning but not enough to move closer to same-age peers.
Improved functioning to a level nearer to same-age peers but did not reach it.
Improved functioning to reach a level comparable to same-age peers.
Maintained functioning at a level comparable to same-age peers.

24. Looking for Sensible Patterns in the Data
Putting together your “validity argument.”
You can make a case that your data are valid if …..they show certain patterns.
The quality of your data is not established by one or two numbers.
The quality of the data is established by a series of analyses that demonstrate the data are showing predictable patterns.

25. “Invalid Outcomes Data?”

26. Predicted Pattern #1
1a. Children will differ from one another in their entry scores in reasonable ways (e.g., fewer scores at the high and low ends of the distribution, more scores in the middle). .
1b. Children will differ from one another in their exit scores in reasonable ways.
1c. Children will differ from one another in their OSEP progress categories in reasonable ways.

27. Rationale
Evidence suggests EI and ECSE serve more mildly than severely impaired children (e.g., few ratings/scores at lowest end). Few children receiving services would be expected to be considered as functioning typically (few ratings/scores in the typical range).

28. Predicted Pattern #1 (cont’d)
Analysis
Look at the distribution of rating/scores at entry and exit and the data reported to OSEP.
Look at the percentage of children who scored as age appropriate (or not) on all three outcomes at entry and at exit.
Question: Is the distribution sensible? What do you expect to see?

29. Poll Time!! If you can’t see the poll area:
If you see 3 bars after “polling”, click on the word “polling.”
If you only see the word “polling,” click on the triangle in front of “polling”.
You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).
When you see the poll question, click on your answer.

30. Entry &
Exit Data

31. MN: Outcome 1 Entrance: 07-08

32. State with Scores: Distribution of entry scores on Outcome 1

33. MN: Outcome 2 Exit: 07-08

34. OSEP Categories

35. MN: Outcome 3 OSEP Categories: 07-08

36. Fake Data: OSEP progress categories
Possible Problems:
Too many children in “a”
Too many children in “e”

37. Poll Time!! If you can’t see the poll area:
If you see 3 bars after “polling”, click on the word “polling.”
If you only see the word “polling,” click on the triangle in front of “polling”.
You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).
When you see the poll question, click on your answer.

38. Predicted Pattern #2
2. Functioning in one outcome area will be related to functioning in the other outcome areas.
Analyses: Look at the relationship across the outcomes at entry, at exit, across the OSEP progress categories.
1. Crosstabs
2. Correlation coefficient
Question: What do we expect to see?

39. Rationale
For many, but not all, children with disabilities, progress in functioning in the three outcomes proceeds together

40. Outcome 1
Outcome 2 1 2 3 4 5

41. MN: Crosstabulation with Progress Categories: 619: Know/Skills to Soc/EmotD E 14 4 3 1 5
230
116 36 67 6
102
664
172 37 45
227
517
179 53 52
186
440

42. Correlation Coefficient
Useful statistic
Range: 0 to 1
Can be negative
Measure of extent of a relationship between 2 sets of numbers
Closer to 1, stronger the relationship
Negative correlation means as one set of numbers goes up, the other goes down.

43. MN619: Correlation coefficients between exit scores for the 3 outcomes (N=3,160)
Soc Emot_exit
Knowledge_exit
.724
Action_exit
.737
.691

44. MNPart C: Correlation coefficients among entry scores for the 3 outcomes
Soc Emot_entry
Knowledge_entry
.660**
Action_entry
.604**
.594**
**. Correlation is significant at the 0.01 level (2-tailed).

45. Predicted Pattern #3
Functioning at entry within an outcome area will be related to functioning at exit (or – children who have higher functioning at entry in an outcome area will be the ones who are high functioning at exit in that outcome area).
Analyses:
1. Correlation coefficients between entry and exit scores for each outcome
2. Crosstabs between entry and exit scores for each outcome
Question: What do we expect to see?

46. MN 619: O2 Entry X Exit Ratings3 4 5 6 7 11 16 50 33
108 68 15 18
121
151 70 14 21
148
213
233
170 12
138
172
282
217 30 39 47
130
219
257

47. MN Part C: Correlation coefficients between entry and exit scores
O1_exit
O2_exit
O3_exit
O1_entry
.660
.505
.499
O2_entry
.539
.612
.491
O3_entry
.512
.471
.649
**. Correlation is significant at the 0.01 level (2-tailed).
N=1,060

48. “Any Requests?

49. Predicted Pattern #4
4. Most children will either hold their developmental trajectory or improve their trajectory from entry to exit.
Analyses:
1. Comparison of distributions of COSF ratings, standard scores, or some other metric that takes age into account. (Why can’t we use raw scores on an assessment for this?) at entry and exit.
Question: What do we expect to see?

51. Entry & Exit Ratings MN: C-O2

52. Entry & Exit Ratings MN: B-O1

53. Predicted Pattern #4b
4b. Children will not show huge changes in a year (or between entry and exit??).
Analyses:
1. Time 2 scores minus Time 1 scores Crosstabs of scores at each time point
Question: What do we expect to see?

54. Distribution: Exit - Entrance Ratings Minnesota Part B Know/Skills n=3160

55. MN Part C: O3 Entry X Exit 1 2 3 4 5 6 7 18 32 13 23 59 40 14 34 67 42 15 16 49 66 20 17 30 28 80 94 10 27 48 73

56. Predicted Pattern #5
5. Entry, exit, and OSEP progress category distributions from year to year should be similar (assuming the same kinds of children are being served).
Analysis:
1. Frequency distributions of entry data in 2007, , etc.
2. of exit data
3. of OSEP Categories
Question: What do we expect to see?

57. Entry Ratings MN: B-O1

58. Exit Ratings MN: C-O2

59. OSEP Progress Categories MN: B-O2

60. Predicted Pattern #6
6. If local areas are serving similar kinds of children, scores at entry should be similar.
Analysis:
1. Frequency distributions of entry by local areas (Use the big programs.)
2. Means and standard deviations (and Ns!) by local area.
Question: What do we expect to see?

61. Poll Time!! If you can’t see the poll area:
If you see 3 bars after “polling”, click on the word “polling.”
If you only see the word “polling,” click on the triangle in front of “polling”.
You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).
When you see the poll question, click on your answer.

62. B-2 Entry Ratings: MN’s 4 Largest Districts (n’s=275-346)

63. B-2 Entry Ratings: MN’s 4 Largest Districts (n’s=275-346)

64. Predicted Pattern #7
7. Entry and exit scores and OSEP categories should be related to the nature of the child’s disability.
Analyses:
1. Frequency distributions for each disability group
2. Means and standard deviations for each disability group
Question: What do we expect to see?

65. Poll Time!! If you can’t see the poll area:
If you see 3 bars after “polling”, click on the word “polling.”
If you only see the word “polling,” click on the triangle in front of “polling”.
You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).
When you see the poll question, click on your answer.

66. MN: Entry Ratings by Disability: Part B, Soc/Emot Skills

67. MN: Entry Ratings by Disability: Part B, Know/Skills

68. MN: Mean and Standard Deviation by Disability
Soc-Emot. skills
Knowledge/Skills
Action to Meet Needs
ASD
2.88
1.15
3.63
1.48
3.56
1.38 DD
3.43
1.35
3.44
1.27
4.09
1.47
Sp/Lang
5.24
1.44
5.25
1.62
5.84
1.66

69. Predicted Pattern #8
Scores at entry (and exit) should not be related to certain characteristics (e.g., race/ethnicity).
Analyses:
1. Frequency distributions for each group
2. Means and standard deviations for group
Question: What do we expect to see?

70. MN Pt B: Outcome 2 Entry Ratings by Gender

71. MN: Mean & SD by Race/Ethnicity
Soc-Emot
Know/Skills
Action/Need
American Indian n=24
3.00
1.45
4.00
1.38
1.52
Asian/Pac. Islander n=52
4.04
1.55
3.83
1.61
4.37
1.73
Hispanic
n=81
3.60
1.62
3.48
1.53
4.17
1.79
Black
n=95
3.38
1.50
3.95
1.35
3.47
1.54
White
n=907
4.07
1.72
3.96
1.67
4.11

72. Wrap-up

74. Drilling down: Looking at data by local program
All analyses that can be run with the state data can be run with the local data
The same patterns should hold and the same predictions apply.
Need to be careful about the size of N with small programs.

75. Are your data high quality?
Are the missing data less than X% with no systematic biases?
Systematic bias = some LEA/EIS or sub-groups are missing far more data than others (you have non-representative data).
Do your state’s data support the predicted patterns?
If not, where are the problems?
What do you know or can you find out about why they are occurring?

76. Adapting the definition of insanity…
“The definition of insanity is doing the same thing over and over again and expecting different results.”
Einstein

77. Data Insanity
…..is doing nothing over and over again and expecting your data to get better.

78. Achieving high quality data is a process that takes time – and intentional action

79. Reminder
ECO looking for states to partner in framework development activities
Call for states interested in learning about the partner state application on March 20, 3 pm EDT/2 p.m.CDT/1 p.m. MDT/Noon PDT.
See for application and call-in information.