Educational Analytics JSM 2016 RealVAMS: Getting Real-World Value from Value Added Models Educational Analytics Jennifer Broatch, Ph.D., jennifer.broatch@asu.edu Assistant Professor of Statistics, Arizona State University at the West Campus Jennifer Green, Ph.D., Assistant Professor of Statistics, Montana State University

Overview Introduce the RealVAMS model for educational analytics Interpret the results of an example and discuss implications for the evaluation of other programs Compare/Contrast RealVAMS estimates with a standard VAM

What is a Value Added Model (VAM)? Typically a VAM estimate effects of educational factors (professional development programs, teachers, schools, districts, etc.) on student learning while controlling for prior achievement. VAM extensions- Estimate the effect of offensive and defensive ability on team performance (winning) Estimate the relative contributions of an employee to company performance measures

Value Added VAM Estimates Estimates are measured “indirectly” or “latent” effects “Teacher Effects” -estimate effects of educational factors (professional development programs, teachers, schools, districts, etc.) on student learning while controlling for prior achievement Differences at the classroom level not explained by other terms in the VAM (unexplained classroom-level heterogeneity) Typically estimated relative to other subjects (teachers) in the model

Estimating “Teacher” Effects Value Added Estimating “Teacher” Effects Non-parametric Standardized Gain Method (Reback, 2008) Estimate teacher effect by mean value of a teacher’s students Student Growth Percentiles (Betebenner, 2009) Estimate teacher effect by median value of quantile growth of a teacher’s students Education Value-Added Assessment System (EVAAS; Sanders et al., 1997) Mixed model in which teacher effects persist undiminished over time Generalized Persistence (Mariano, McCaffrey, Lockwood, 2010) Mixed model that allows for non-equated responses Different teacher effect for each year

Potential Limitations Value Added Potential Limitations VAMs typically rely on Continuous - interval scale data Vertically scaled/equated standardized tests - comparable over different forms, ability levels, and time VAMs typically measure one effect per teacher Single grade or time point Individual subject (e.g., Science, Math, Reading) Ignores potential relationship(s) between a teacher’s effects on different outcomes

Motivation for an Improved VAM Value Added Motivation for an Improved VAM High quality data are often hard to obtain Many projects rely on data, which may or may not meet requirements for a typical VAM Educational programs often broadly define “program impact” Desire a “holistic” picture of student success, which cannot be captured by achievement scores alone

What is the RealVAMS model? Multidimensional value-added model that… Accommodates multiple types of student outcomes Continuous, non-equated test scores from multiple subjects and testing instruments (e.g., ACT, SAT) Dichotomous (yes/no) categorical responses (e.g., college entry) Simultaneously estimates multiple effects for a single teacher Separate teacher effect for each response Relationship(s) between a teacher’s effects on different responses

RealVAMS What we want… Teacher “effect” on student achievement as measured by a “typical” standardized test. Teacher “effect” on a real-world outcome, specifically whether a student attended college. Teacher “effect” on student achievement as measured by a “typical” standardized test. Teacher “effect” on a real-world outcome, specifically whether a student attended college.

RealVAMS What we want… Teacher “effect” on student achievement as measured by a “typical” standardized test. Teacher “effect” on a real-world outcome, specifically whether a student attended college. Teacher “effect” on student achievement as measured by a “typical” standardized test. Teacher “effect” on a real-world outcome, specifically whether a student attended college.

RealVAMS Model Illustration…

RealVAMS Model Illustration…

RealVAMS Model Illustration…

RealVAMS Model Illustration…

RealVAMS Model The RealVAMS model for the ith student is 𝒚 𝑖 = 𝑿 𝑖 𝜷 + 𝑺 𝑖 𝜸+ 𝝴 𝑖 where 𝒚 𝑖 is the vector of the t responses on student i 𝜷 represents the coefficients of the fixed effects (teacher and/or student covariates) 𝑺 𝑖 indicates which teachers instruct student i 𝜸 represents the multivariate latent “teacher effect” 𝞮 𝑖 represents the random error term

RealVAMS Program RealVAMS R Package Package found on R CRAN mirror: https://cran.r-project.org/web/packages/RealVAMS/index.html Developed to meet the needs of RealVAMS model Computational issues associated with including the relationship between the teacher effects Computational issues associated with including a dichotomous response Open source and accessible!

Example Data from Large Public School RealVAMS Example Example Data from Large Public School 11th grade students who took the required state assessment Includes 912 students, 86 teachers Responses: Scale score on state assessment College entry (Y/N) Covariate Information: Student gender Student ethnicity Math/Reading PLAN scale scores Gifted status (Yes=1/No=0) Special education status (Yes=1/No=0) ELL status (Yes=1/No=0)

Example RealVAMS Estimates

Example Estimated Covariance Student Covariance: 𝑅 𝑖 = 602.00 3.62 3.62 1.00 Teacher Covariance: 𝐺 𝑖 = 64.75 3.38∗ 3.38∗ 0.07 * The off-diagonal is significantly different from 0 (p=.0007).

Finding Estimated Correlation Example Finding Estimated Correlation Teacher Covariance: 𝐺 𝑖 = 64.75 3.38 3.38 0.07 Teacher Effect Correlation: 𝑟 𝐺 = 3.38 64.75 ∗ 0.07 =0.79

Example Teacher Effects 𝑟 𝐺 =0.79 - Correlation between teacher effects on the score response (student achievement on a standardized test) and the outcome response (college entry)

Example Teacher Effects All Teachers Highlighted Teacher % of College Entry 70.9% 76.0% State Assessment 121.90 55.20

Evaluation Comparison Why not just use a simpler model? Completely ignore/exclude the other “real-world” responses How do the RealVAMS estimates for the standard response compare with a common model such as EVAAS?

Single Response (EVAAS) vs. Multidimensional (RealVAMS) Comparison Single Response (EVAAS) vs. Multidimensional (RealVAMS) *If independence was assumed in RealVAMS model, r=1.

Advantages of the RealVAMS Model Interpretation and Discussion Advantages of the RealVAMS Model

Interpretation and Discussion Comparison of the teacher rank differences shows moderate positive relationship (r=0.50)

Advantages of the RealVAMS Model Interpretation and Discussion Advantages of the RealVAMS Model Multidimensional estimates of “teacher effectiveness” Assessment of outcomes that align with project and/or district goals but are not compatible with other models Use of free open source software for analysis

Evaluation Caution Interpretation and Discussion As with all evaluation, Long-term student trajectories require Quality baseline characterizing starting point Data over multiple years to see how progressing Assessments and outcomes need to Have ability to adequately score all levels of achievement (i.e., free of floor or ceiling effects) Reflect learning objectives Be valid and reliable measures of student progress with respect to these objectives

General Caution Interpretation and Discussion VAMS in general Have standard errors of estimated latent effects that tend to be large May provide information on students’ performance, but not on how to improve teaching Estimate correlation vs. causation in non-randomized trials

Concluding Remarks Interpretation and Discussion “The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data.” -- John Tukey This work is supported by the National Science Foundation under grants DRL-1336027 and DRL-1336265.