1. Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison

2. Some Unique Features of STAR Assessments Same CAT assessment is administered within and across years (possible to generate look-up tables for the calculation of SGP) Multiple STAR administrations to a student are possible throughout the year, and can occur at different times and/or frequencies for different students

3. Why SGP may be Useful with STAR Assessments Different STAR scores may be associated with different amounts of measurement error (e.g., extremely low or high STAR scores are sometimes of questionable validity) There often exists more/less variability in growth observed across students at different initial STAR score levels

4. Some Practical Issues Related to Administration of STAR Assessments How frequently and at what intervals should STAR be administered to get reliable estimates of end-of-year scores? Does the answer depend on the initial (fall) score of a student? Is there practical value in the use of SGP for answering this question (in contrast to alternative approaches, such as ordinary least squares---OLS--- regression methods)?

5. Goals of the Present Study Examine SGP as a methodology for quantifying growth and for studying the precision of end-of-year growth predictions using STAR Assessments Compare SGP against competing methodologies (OLS regression) in terms of their reported precision of end-of- year predictions

6. GradeMathReading Early Literacy 19928310000096877 2100000 28657 3100000 5758 4100000 554 5100000 6 786975100000 872599100000 92083561919 101381037455 11979626278 12515512763 Sample Sizes (Students with Fall, Winter & Spring Assessments)

7. Evaluating the Precision of Spring Score Predictions How well do winter assessments improve our predictions of end-of-year outcomes? Are the winter assessments more/less useful depending on the fall score obtained by the student? In answering these questions, we find it useful to examine changes in the confidence intervals for spring scores defined by the SGP percentile cuts

8. Example of SGP Percentile Cuts, One Covariate

9. Example of SGP Percentile Cuts, Two Covariates

10. Comparison of SGP & OLS Intervals, Math Grade 1 According to OLS Standard Error of Prediction According to SGP %ile cuts

11. STAR Math, Grades 1-6 OLS SGP

12. STAR Math, Grades 7-12 OLS SGP

13. STAR Reading, Grades 1-6 OLS SGP

14. STAR Reading, Grades 7-12 OLS SGP

15. STAR Early Literacy, Grades K-4 OLS SGP

16. Adding Winter Scores as Covariates with SGP Using 80% interval width curves as a baseline, we can further examine how much the intervals are reduced when adding a winter assessment The decline in the 80% interval can be used as an indicator of the added precision provided by the winter assessment

17. STAR Math Example

18. STAR Reading Example

19. STAR Early Literacy Example

20. Some Examples of STAR Score Patterns SubjectFallWinterSpring Change in 80% Interval MATH50132531(+432) 62591790(+321) 3833852(-208) 2846771(-209) READING392891589(+41) 376915551(+40) 98413461333(-148) 92813451255(-256) LITERACY835315832(+297) 777312840(+282) 328858823(-242) 306829851(-256)

21. STAR Math Example OLS SGP

22. STAR Math, Grade 1-6 OLS SGP

23. STAR Math, Grades 7-12 OLS SGP

24. STAR Reading Example OLS SGP

25. STAR Reading, Grades 1-6 OLS SGP

26. STAR Reading, Grades 7-12 OLS SGP

27. STAR Early Literacy Example OLS SGP

28. STAR Early Literacy, Grades K-4 OLSSGP

29. Evaluating Predicted Spring Scores in Terms of State Proficiency Thresholds The accuracy of SGP and OLS predictions can also be compared against the thresholds associated with state-specific proficiency categories By assuming normally distributed residuals (with constant variance) for OLS, SGP and OLS can each be used to define a probability that the spring score will exceed a predefined threshold

30. Proficiency Threshold Example of SGP Percentile Cuts against State- Defined Proficiency Category

31. MATHREADING Grade Proficiency Cut a PercentileOLS R 2,b SGP R 2,b Proficiency Cut a PercentileOLS R 2,b SGP R 2,b 361748.40.4144548.53.66 470961.40 53152.53.68 575860.42 59249.52.69 681065.46 64742.52.70 782374.48.4982551.55.74 885968.47.4893353.55.73 a Estimated STAR cutscores for the Tennessee Comprehensive Assessment Program (TCAP) b Efron’s Pseudo R 2 Comparing SGP and OLS on Accuracy of Proficiency Predictions

32. Extending SGP to Accommodate Multiple Intermediate Assessments How can additional intermediate assessments be used in SGP to further improve predicted spring scores? Challenge: Handling varying-time point assessment schedules One possible solution: Linear interpolation to fixed node locations

33. Conclusions and Future Directions Our SGP analyses suggest substantial variability in the precision of spring score predictions for STAR Math, Reading and Early Literacy depending on fall scores There is clear value in incorporating winter assessments into SGP---the largest value occurs for students with extreme fall scores in STAR Math, intermediate fall scores in STAR Reading More experimentation needed to determine how best to make use of multiple intermediate assessments within SGP