1. Growth in Reading Curriculum – Based Measures and Predicting High Stakes Outcomes

2. ben@measuredeffects.com Questions Are there differences in the growth in Reading (CBM) between students who meet and do not meet standards on high stakes tests? If differences exist can they be predicted from the beginning of the year? What is the cost of setting hard targets for performance on CBM?

3. ben@measuredeffects.com R-CBM for G3 students taking ISAT Median = 97, N m = 388 Median = 61 N nm = 137 Median = 115 Median = 85 Mean 135 (35.6) Median = 129 Mean =91 (27.9) Median = 94 Gain ~.97 WPW Gain ~ 1 WPW

4. ben@measuredeffects.com Distribution of R-CBM Scores by Time (for Gr. 3students who took ISAT) Typical Range Fall 78 – 113; Md 97 Winter 99 – 140; Md 115 Spring 108 – 156; Md 129

5. ben@measuredeffects.com R-CBM for G3 students taking IMAGE Median = 82, N m = 56 Median = 56 N nm = 65 Median = 100 Median = 69 Mean 115 (27.8) Median = 113 Mean = 78.5 (21.0) Median = 80 Gain ~.94 WPW Gain ~.73 WPW

6. ben@measuredeffects.com Distribution of R-CBM Scores by Time (for Gr. 3 students who took IMAGE) Note. N’s less than 100 no whiskers are displayed Typical Range Fall 70 – 101; Md 82 Winter 82 – 119; Md 100 Spring 94 – 131; Md 113

7. What does this relationship look like for Grade 5

8. ben@measuredeffects.com Typical Range Fall 126 – 168; Md 145 Winter 144 – 183; Md 164 Spring 158 – 196; Md 176 Distribution of R-CBM Scores by Time (for Gr. 5 students who took ISAT)

9. ben@measuredeffects.com Median = 145 N m = 375 Median = 110 N nm = 126 Median = 164 Median = 130 Mean 178 (29.3) Median = 176 Mean = 139 (28.5) Median = 142 Gain ~.94 WPW Gain ~.98 WPW R-CBM for G5 students taking ISAT

10. ben@measuredeffects.com Median = 115 N m = 30 Median = 92 N nm = 47 Median = 139 Median = 112 Mean 146 (24) Median = 140 Mean = 114 (34) Median = 118 Gain ~.76 WPW Gain ~.79 WPW R-CBM for G5 students taking IMAGE

11. ben@measuredeffects.com Note. N’s less than 100 no whiskers are displayed Typical Range Fall 96 – 132; Md 115 Winter 112 – 154; Md 139 Spring 128 – 168; Md 140 Distribution of R-CBM Scores by Time (for Gr. 5 students who took IMAGE)

12. Can we use this information for … Educational Decision - Making?

13. ben@measuredeffects.com ISAT Gr. 3 r =.70 R 2 ~ 48%

14. Medical decision - making to Educational decision - making From In medicine indices of diagnostic accuracy help doctors decide who is high-risk and who is not likely at risk for developing a disease Can we borrow this technology for  tracking adequate growth and  educational decision-making

15. ben@measuredeffects.com Diagnostic Indices SensitivitySensitivity –the fraction of those who fail to meet standards who were predicted to fail to meet standards SpecificitySpecificity –the fraction of students who meet standards who were predicted to meet Positive Predictive PowerPositive Predictive Power –the fraction of students who were predicted not to meet who failed to meet standards Negative Predictive PowerNegative Predictive Power –the fraction of students who were predicted to meet who met standards Correct ClassificationCorrect Classification –the fraction of students for whom predictions of meeting or not meeting were correct

16. ben@measuredeffects.com 98%88%84%80%67%54%39%25% ISAT Gr. 3 Sensitivity considers only students who did not meet standards. As WRC increases sensitivity increases

17. ben@measuredeffects.com 98% 93% 74%86% 68% 62% 30% ~100% Specificity considers only students who meet standards. As WRC increases specificity decreases ISAT Gr. 3

18. ben@measuredeffects.com Positive Predictive Power Positive Predictive Power considers the fraction of students who scored Less than a particular cut who did not meet standards. As WRC increases PPV decreases 98%88%78%67%57%53%50%38%

19. ben@measuredeffects.com 78% 75% 91% 90% 86% 98% Negative Predictive Power Negative Predictive Power considers the fraction of students who scored more than a particular cut who met standards. As WRC increases PPV decreases 92% 82%

20. Decisions, decisions How should we determine where to draw the line?

21. ben@measuredeffects.com At 60 we classify 77% of students correctly

22. ben@measuredeffects.com At 80 we classify 80% of students correctly

23. ben@measuredeffects.com At 90 we classify 81% of students correctly

24. ben@measuredeffects.com At 100 we classify 80% of students correctly

25. ben@measuredeffects.com At 110 we classify 76% of students correctly

26. ben@measuredeffects.com At 115 we classify 73% of students correctly

27. ben@measuredeffects.com At 120 we classify 70% of students correctly

28. ben@measuredeffects.com At 150 we classify 51% of students correctly

29. Why do we have to draw just one line? Maximize Correct Classification Admit the limitations of the tool

30. ben@measuredeffects.com Two statistical methods for group determination Logistic RegressionLogistic Regression –Maximum likelihood method for predicting the odds of group membership –Appears to maximize specificity in cut-score selection Linear Discriminant Function AnalysisLinear Discriminant Function Analysis –Least Squares method for predicting the linear relation between variables that best discriminates between groups –Appears to maximize sensitivity in cut score selection

31. ben@measuredeffects.com Cut scores set by LR and LDFA Both LR and LDFA can use multiple predictors to determine group membership, but for this analysis, only R-CBM in the spring was used. Logistic regression ~ 92 WRC LDFA ~ 112 WRC

32. ben@measuredeffects.com NPV = 83% PPV = 77% SENS = 81% Spec = 93%, UNCLASSIFIEDUNCLASSIFIED The LR & LDFA (Cut L92 - H112) classifies 78% of students in the data set. Of these students who were classified, 86% were classified correctly, with a rate of  14% error in classification. Note that the 86% classified correctly in this model is 78% of the total group.  The reduction of error in identification comes at a cost of failing to identify risk status for 127 of 565 students.

33. ben@measuredeffects.com What to do with the “Unclassified” student? R-CBM does not attempt to tell us everything about a student’s reading, it is a strong indicator. Use of convergent data may be able to provide us with a more fine-grained prediction

34. ben@measuredeffects.com At 80 NPV = 83% At 80 SENS = 80% At 59 Spec = 94%, UNCLASSIFIEDUNCLASSIFIED At 59 PPV = 81% The fall r-cbm (LR & LDFA) (Cut L59 - H80) classifies 80% of students in the data set. Of these students who were classified,  87% were classified correctly, with a rate of 13% error in classification.  The 87% classified correctly in this model is 80% of the total group.  The reduction of error in identification comes at a cost of failing to identify risk status for 114 of 565 students.

35. ben@measuredeffects.com At 101 NPV = 89% At 101 SENS = 80% At 79 Spec = 93%, UNCLASSIFIEDUNCLASSIFIED At 79 PPV = 77% The winter r-cbm (LR & LDFA) (Cut L79 - H101) classifies 75% of students in the data set. Of these students who were classified,  86% were classified correctly, with a rate of 14% error in classification.  Note that the 86% classified correctly in this model is 75% of the total group.  The reduction of error in identification comes at a cost of failing to identify risk status for 143 of 565 students.

36. ben@measuredeffects.com

37. The FALL G 5 R-CBM (LR & LDFA) (Cut L95 - H131) serves to classify 71% of students in the data set. Of these students who were classified,  92% were classified correctly, with a rate of 8% error in classification.  The 92% classified correctly in this model is 71% of the total group.  The reduction of error in identification comes at a cost of failing to identify risk status for (166 of 565 students).

38. ben@measuredeffects.com The WINTER G 5 R-CBM (LR & LDFA) (Cut L122 - H144) serves to classify 79% of students in the data set. Of these students who were classified, 87% were classified correctly, with a rate of 13% error in classification. Note that the 87% classified correctly in this model is 79% of the total group. The reduction of error in identification comes at a cost of failing to identify risk status for (118 of 565 students).

39. ben@measuredeffects.com The Spring G 5 R-CBM (LR & LDFA) (Cut L137 - H157) serves to classify 80% of students in the data set. Of these students who were classified,  86% were classified correctly, with a rate of 14% error in classification.  The 86% classified correctly in this model is 80% of the total group.  The reduction of error in identification comes at a cost of failing to identify risk status for 111 of 565 students. r =.62, R 2 =.38

40. ben@measuredeffects.com Adult Readers Examination of aggregated trajectories Annual Targets

41. ben@measuredeffects.com 115 WRC By Spring of Grade 3 The range between the 90 th and 10 th percentile for each sub group is an empirical, non-parametric 80% confidence interval (for the individual of) quartile growth ˘ cross-sectional between grades ˘ longitudinal within grade 160 WRC By Spring of Grade 5 170 WRC By Spring of Grade 8

42. ben@measuredeffects.com The Spring VM (LR & LDFA) (Cut L5 - H9) serves to classify 79% of students in the data set. Of these students who were classified, 89% were classified correctly, with a rate of 11% error in classification. Note that the 89% classified correctly in this model is 79% of the total group. The reduction of error in identification comes at a cost of failing to identify risk status for (117 of 565 students).

43. ben@measuredeffects.com Big Ideas There are reliable quantitative differences in the average performance of Meeters versus Non – Meeters These differences can be predicted at least from Fall if not earlier There are limitations to the “hard and fast” cut score approach with R-CBM that can be partially addressed by admitting some students may require additional assessment to make a determination of status.

44. ben@measuredeffects.com Directions Set up a model for English and Spanish Reading for Students who are ELLs. Expand progress monitoring with Vocabulary Matching

45. Questions? ben@measuredeffects.com