Supporting Growth Interpretations Using Through-Course Assessments Andrew Ho Harvard Graduate School of Education Innovative Opportunities and Measurement Challenges in Through-Course Summative Assessments National Conference on Student Assessment Monday, June 20, 2011

Through-Course Assessments and Growth Multiple assessments through the school year to support a) instruction and learning and b) inferences about future performance, particularly end-of-year tests and, by extension, career and college readiness. Paper: Google [Andrew Supporting Growth] _Paper_Ho.pdf _Paper_Ho.pdf Webinar: Google [Ho Growth Webinar] In this presentation: Two important distinctions for any multiple-timepoint growth application. Then, additional challenges raised by through-course assessments.

Some important contrasts in growth use and interpretation Growth Description Growth Projection Gain Scores, Trajectories Status Beyond Prediction Trajectory, Gain- Score Model Regression/Pre -diction Model Use a regression model to predict future scores from past scores, statistically, empirically. Where a student was, where a student is, and what has been learned in between. Where a student is, above and beyond where we would have predicted she would be, given past scores. Extend past gains in systematic fashion into the future. Consider whether future performance is adequate

Two Approaches to Growth Description Gain Prediction from previous score (or scores, or scores and demographics) Status beyond prediction Adding two students with equal gains Adding two different students with equal statuses beyond predictions. Gain Scores, Trajectories Status Beyond Prediction

Two Accountability Models Therefrom An “Equal Gain” Requirement: Headline: New Growth Scores Set Unrealistic Expectations on Highest Scoring Students. –Low Standards for Low- Scoring Students. Three students with equal gains Three students with equal statuses beyond predictions. An “Equal Status-Beyond- Prediction” Requirement New “Growth” Scores Expect Less Learning from Highest Scoring Students. –Scores depend on arbitrary decisions

Pros and Cons Pros: Straightforward. Aligns with user intuition about growth. Describes growth and progress along an (ideally) meaningful, relevant scale. Cons: Defensible vertical scales are difficult and costly to support, can be poorly aligned with statistical predictions. Three students with equal gains Three students with equal statuses beyond predictions. Pros: Incorporates statistical predictions. Does not require vertical scales. Cons: Poorly aligned with user intuition about growth. Variables supporting predictions can be atheoretical; variable inclusion/exclusion changes predictions.

Two Approaches to Growth Projection Extends gains over time in straightforward fashion. With more prior years, a best- fit line or curve can be extended similarly. Extended trajectories do not have to be linear. Estimates a prediction equation for the “future” score. Because current students have unknown future scores, estimate the prediction equation from a previous cohort that does have their “future” year’s score. Input current cohort data into past prediction equation. Trajectory, Gain-Score ModelRegression/Prediction Model

Stark Contrasts in Projections Three students with equal projections from a regression model. The same three students’ predictions with a regression model. Three students with equal projections from a trajectory model

Stark Contrasts in Incentives Three students with equal projections from a trajectory model Three students with equal projections from a regression model. Lower initial scores can inflate trajectories: New Model Rewards Low Scores, Encourages “Fail-First Strategy” Very intuitive, requires vertical scales, less accurate in terms of future classifications. Low scorers require huge gains. High scorers can fall comfortably. New Model Labels High and Low Achievers Early, Permanently. Counterintuitive, does not require vertical scales, more accurate classifications.

Growth and Through-Course Assessments For low-stakes, through-course assessments, either incorporate vertical scales with gain-score growth information or don’t overpromise on the usefulness of through-course data to describe growth through a curricular domain. As stakes increase, anticipate and communicate incentives, and adjust weights to discourage both “fail-first” and “inertial status” gaming behaviors, ideally with low but not negative weights on earlier tests. Do not forget that “career and college readiness” does not solve the problem of setting a meaningful and defensible cut score, nor does it predetermine the type of growth model that projects to that target. Paper: Google [Andrew Supporting Growth] Webinar: Google [Ho Growth Webinar]