1. Educational Analytics
JSM 2016
RealVAMS: Getting Real-World Value from Value Added Models
Educational Analytics
Jennifer Broatch, Ph.D.,
Assistant Professor of Statistics, Arizona State University at the West Campus
Jennifer Green, Ph.D.,
Assistant Professor of Statistics, Montana State University

2. Overview Introduce the RealVAMS model for educational analytics
Interpret the results of an example and discuss implications for the evaluation of other programs
Compare/Contrast RealVAMS estimates with a standard VAM

3. What is a Value Added Model (VAM)?
Typically a VAM estimate effects of educational factors (professional development programs, teachers, schools, districts, etc.) on student learning while controlling for prior achievement.
VAM extensions-
Estimate the effect of offensive and defensive ability on team performance (winning)
Estimate the relative contributions of an employee to company performance measures

4. Value Added VAM Estimates
Estimates are measured “indirectly” or “latent” effects
“Teacher Effects” -estimate effects of educational factors (professional development programs, teachers, schools, districts, etc.) on student learning while controlling for prior achievement
Differences at the classroom level not explained by other terms in the VAM (unexplained classroom-level heterogeneity)
Typically estimated relative to other subjects (teachers) in the model

5. Estimating “Teacher” Effects
Value Added
Estimating “Teacher” Effects
Non-parametric Standardized Gain Method (Reback, 2008)
Estimate teacher effect by mean value of a teacher’s students
Student Growth Percentiles (Betebenner, 2009)
Estimate teacher effect by median value of quantile growth of a teacher’s students
Education Value-Added Assessment System (EVAAS; Sanders et al., 1997)
Mixed model in which teacher effects persist undiminished over time
Generalized Persistence (Mariano, McCaffrey, Lockwood, 2010)
Mixed model that allows for non-equated responses
Different teacher effect for each year

6. Potential Limitations
Value Added
Potential Limitations
VAMs typically rely on
Continuous - interval scale data
Vertically scaled/equated standardized tests - comparable over different forms, ability levels, and time
VAMs typically measure one effect per teacher
Single grade or time point
Individual subject (e.g., Science, Math, Reading)
Ignores potential relationship(s) between a teacher’s effects on different outcomes

7. Motivation for an Improved VAM
Value Added
Motivation for an Improved VAM
High quality data are often hard to obtain
Many projects rely on data, which may or may not meet requirements for a typical VAM
Educational programs often broadly define “program impact”
Desire a “holistic” picture of student success, which cannot be captured by achievement scores alone

8. What is the RealVAMS model?
Multidimensional value-added model that…
Accommodates multiple types of student outcomes
Continuous, non-equated test scores from multiple subjects and testing instruments (e.g., ACT, SAT)
Dichotomous (yes/no) categorical responses (e.g., college entry)
Simultaneously estimates multiple effects for a single teacher
Separate teacher effect for each response
Relationship(s) between a teacher’s effects on different responses

9. RealVAMS
What we want…
Teacher “effect” on student achievement as measured by a “typical” standardized test.
Teacher “effect” on a real-world outcome, specifically whether a student attended college.
Teacher “effect” on student achievement as measured by a “typical” standardized test.
Teacher “effect” on a real-world outcome, specifically whether a student attended college.

10. RealVAMS
What we want…
Teacher “effect” on student achievement as measured by a “typical” standardized test.
Teacher “effect” on a real-world outcome, specifically whether a student attended college.
Teacher “effect” on student achievement as measured by a “typical” standardized test.
Teacher “effect” on a real-world outcome, specifically whether a student attended college.

11. RealVAMS Model
Illustration…

12. RealVAMS Model
Illustration…

13. RealVAMS Model
Illustration…

14. RealVAMS Model
Illustration…

15. RealVAMS Model The RealVAMS model for the ith student is
𝒚 𝑖 = 𝑿 𝑖 𝜷 + 𝑺 𝑖 𝜸+ 𝝴 𝑖
where
𝒚 𝑖 is the vector of the t responses on student i
𝜷 represents the coefficients of the fixed effects (teacher
and/or student covariates)
𝑺 𝑖 indicates which teachers instruct student i
𝜸 represents the multivariate latent “teacher effect”
𝞮 𝑖 represents the random error term

16. RealVAMS Program RealVAMS R Package Package found on R CRAN mirror:
Developed to meet the needs of RealVAMS model
Computational issues associated with including the relationship between the teacher effects
Computational issues associated with including a dichotomous response
Open source and accessible!

17. Example Data from Large Public School
RealVAMS Example
Example Data from Large Public School
11th grade students who took the required state assessment
Includes 912 students, 86 teachers
Responses:
Scale score on state assessment
College entry (Y/N)
Covariate Information:
Student gender
Student ethnicity
Math/Reading PLAN scale scores
Gifted status (Yes=1/No=0)
Special education status (Yes=1/No=0)
ELL status (Yes=1/No=0)

18. Example
RealVAMS Estimates

19. Example Estimated Covariance Student Covariance:
𝑅 𝑖 =
Teacher Covariance:
𝐺 𝑖 = ∗ 3.38∗ 0.07
* The off-diagonal is significantly different from 0 (p=.0007).

20. Finding Estimated Correlation
Example
Finding Estimated Correlation
Teacher Covariance:
𝐺 𝑖 =
Teacher Effect Correlation:
𝑟 𝐺 = ∗ =0.79

21. Example Teacher Effects
𝑟 𝐺 = Correlation between teacher effects on the score response (student achievement on a standardized test) and the outcome response (college entry)

22. Example Teacher Effects All Teachers Highlighted Teacher
% of College Entry
70.9%
76.0%
State Assessment
121.90
55.20

23. Evaluation Comparison
Why not just use a simpler model?
Completely ignore/exclude the other “real-world” responses
How do the RealVAMS estimates for the standard response compare with a common model such as EVAAS?

24. Single Response (EVAAS) vs. Multidimensional (RealVAMS)
Comparison
Single Response (EVAAS) vs. Multidimensional (RealVAMS)
*If independence was assumed in RealVAMS model, r=1.

25. Advantages of the RealVAMS Model
Interpretation and Discussion
Advantages of the RealVAMS Model

26. Interpretation and Discussion
Comparison of the teacher rank differences shows moderate positive relationship (r=0.50)

27. Advantages of the RealVAMS Model
Interpretation and Discussion
Advantages of the RealVAMS Model
Multidimensional estimates of “teacher effectiveness”
Assessment of outcomes that align with project and/or district goals but are not compatible with other models
Use of free open source software for analysis

28. Evaluation Caution Interpretation and Discussion
As with all evaluation,
Long-term student trajectories require
Quality baseline characterizing starting point
Data over multiple years to see how progressing
Assessments and outcomes need to
Have ability to adequately score all levels of achievement (i.e., free of floor or ceiling effects)
Reflect learning objectives
Be valid and reliable measures of student progress with respect to these objectives

29. General Caution Interpretation and Discussion VAMS in general
Have standard errors of estimated latent effects that tend to be large
May provide information on students’ performance, but not on how to improve teaching
Estimate correlation vs. causation in non-randomized trials

30. Concluding Remarks Interpretation and Discussion
“The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data.”
-- John Tukey
This work is supported by the National Science Foundation under grants DRL and DRL