Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison

Some Unique Features of STAR Assessments Same CAT assessment is administered within and across years (possible to generate look-up tables for the calculation of SGP) Multiple STAR administrations to a student are possible throughout the year, and can occur at different times and/or frequencies for different students

Why SGP may be Useful with STAR Assessments Different STAR scores may be associated with different amounts of measurement error (e.g., extremely low or high STAR scores are sometimes of questionable validity) There often exists more/less variability in growth observed across students at different initial STAR score levels

Some Practical Issues Related to Administration of STAR Assessments How frequently and at what intervals should STAR be administered to get reliable estimates of end-of-year scores? Does the answer depend on the initial (fall) score of a student? Is there practical value in the use of SGP for answering this question (in contrast to alternative approaches, such as ordinary least squares---OLS--- regression methods)?

Goals of the Present Study Examine SGP as a methodology for quantifying growth and for studying the precision of end-of-year growth predictions using STAR Assessments Compare SGP against competing methodologies (OLS regression) in terms of their reported precision of end-of- year predictions

GradeMathReading Early Literacy Sample Sizes (Students with Fall, Winter & Spring Assessments)

Evaluating the Precision of Spring Score Predictions How well do winter assessments improve our predictions of end-of-year outcomes? Are the winter assessments more/less useful depending on the fall score obtained by the student? In answering these questions, we find it useful to examine changes in the confidence intervals for spring scores defined by the SGP percentile cuts

Example of SGP Percentile Cuts, One Covariate

Example of SGP Percentile Cuts, Two Covariates

Comparison of SGP & OLS Intervals, Math Grade 1 According to OLS Standard Error of Prediction According to SGP %ile cuts

STAR Math, Grades 1-6 OLS SGP

STAR Math, Grades 7-12 OLS SGP

STAR Reading, Grades 1-6 OLS SGP

STAR Reading, Grades 7-12 OLS SGP

STAR Early Literacy, Grades K-4 OLS SGP

Adding Winter Scores as Covariates with SGP Using 80% interval width curves as a baseline, we can further examine how much the intervals are reduced when adding a winter assessment The decline in the 80% interval can be used as an indicator of the added precision provided by the winter assessment

STAR Math Example

STAR Reading Example

STAR Early Literacy Example

Some Examples of STAR Score Patterns SubjectFallWinterSpring Change in 80% Interval MATH (+432) (+321) (-208) (-209) READING (+41) (+40) (-148) (-256) LITERACY (+297) (+282) (-242) (-256)

STAR Math Example OLS SGP

STAR Math, Grade 1-6 OLS SGP

STAR Math, Grades 7-12 OLS SGP

STAR Reading Example OLS SGP

STAR Reading, Grades 1-6 OLS SGP

STAR Reading, Grades 7-12 OLS SGP

STAR Early Literacy Example OLS SGP

STAR Early Literacy, Grades K-4 OLSSGP

Evaluating Predicted Spring Scores in Terms of State Proficiency Thresholds The accuracy of SGP and OLS predictions can also be compared against the thresholds associated with state-specific proficiency categories By assuming normally distributed residuals (with constant variance) for OLS, SGP and OLS can each be used to define a probability that the spring score will exceed a predefined threshold

Proficiency Threshold Example of SGP Percentile Cuts against State- Defined Proficiency Category

MATHREADING Grade Proficiency Cut a PercentileOLS R 2,b SGP R 2,b Proficiency Cut a PercentileOLS R 2,b SGP R 2,b a Estimated STAR cutscores for the Tennessee Comprehensive Assessment Program (TCAP) b Efron’s Pseudo R 2 Comparing SGP and OLS on Accuracy of Proficiency Predictions

Extending SGP to Accommodate Multiple Intermediate Assessments How can additional intermediate assessments be used in SGP to further improve predicted spring scores? Challenge: Handling varying-time point assessment schedules One possible solution: Linear interpolation to fixed node locations

Conclusions and Future Directions Our SGP analyses suggest substantial variability in the precision of spring score predictions for STAR Math, Reading and Early Literacy depending on fall scores There is clear value in incorporating winter assessments into SGP---the largest value occurs for students with extreme fall scores in STAR Math, intermediate fall scores in STAR Reading More experimentation needed to determine how best to make use of multiple intermediate assessments within SGP