1. Assessments to VAM to VAS to EES Points July 28, 2014 1

2. Which assessments to include? Science Outcome Y2012Y =Xt1 +Xt2 2014201320122011201020092008 1110987 11109876 111098765 76543 432 EOY 2 Science 7 = a + b 1 (Math 6 ) + b 2 (Reading 6 ) + b 3 (Math 5 ) + b 4 (Reading 5 ) + c(Proportion) + e

3. Who’s in each model? Models are developed by course group. A teacher is assigned a course group based on the course code of the courses they teach. – A teacher can be in more than one course group (e.g. 5 th grade math and 5 th grade reading, or Algebra 1 and Geometry). – Course groupings help mitigate against bias that may result from an unequal distribution of assessment difficulty and/or student type. 3

4. Nomenclature Coding – t = the current assessment occasion; – t-1 = the prior assessment occasion; – t-2 = the prior assessment occasion to t-1; – SS = Scale score – M Math, R = Reading, Sci = Science. –. denotes class/teacher mean; –.. Denotes the grand mean (usually by course group E.G. SSMt = the current scale score in Math for an individual student. E.G. SSMt. = the current scale score in Math for an individual student. 4.

5. Preparing the data Step 1: – Normalize the scale scores to a common year (2012); NSSMt = SSMt – SSM 2012../SDM 2012 Where SD = Standard Deviation N = Normalized. 5

6. Preparing the data Step 2: – Link every student’s current score to the Conditional standard error of measurement (CSEM). Step 3: – Use the Structure table to ensure the proper prior scores are linked to each student’s current (outcome) score. 6

7. The Base file (for 2012 7 th grade Biology) 7 Each row is a student Multiple rows will form a teacher’s class.

8. Step 4: – The Base file is aggregated by teacher. 8 Each row is a teacher

9. Step 5: – This step could be carried out by many different statistical software applications, but the PED uses HLM. HLM has a couple of benefits: – It converges quickly (we ran about 120 VAMs) – Output file efficiently provides necessary results for EES. 9

10. The basic Model The outcome variable is NSSSCIT Summary of the model specified – Level-1 Model NSSSCIT ij = β 0j + r ij Level-2 Model β 0j = γ 00 + u 0j Mixed Model NSSSCIT ij = γ 00 + u 0j + r ij -> in English = a student’s 7 th grade Biology score is a function of the grand mean, of all 7 th grade biology scores, a unique contribution of teachers and a random component. – This is a mixed effects model. There are both fixed and random effects. – Teacher VAS are based on random effects. – This is the unconditional model. – It is always the first step in VAM modeling. 10

11. Fixed EffectCoefficient Standard error t-ratio Approx. d.f. p-value For INTRCPT1, β 0 INTRCPT2, γ 00 -0.0046370.042111-0.1101780.912 11 Random Effect Standard Deviation Variance Component d.f.χ2χ2 p-value INTRCPT1, u 0 0.524430.275031782757.06740<0.001 level-1, r0.892560.79667 Final estimation of fixed effects: Final estimation of variance components

12. Note: although a “full” model is used to calculate a teacher’s VAS, we will start with the simple model to demonstrate the steps. Step 6: Use HLM results to calculate a teacher’s unique contribution to student learning (VAS). – Obtain the OLS residual = Observed – expected. 12

13. OLS Resid OLS =.768 – (-.005) =.773. 13 Observed OLS Residual Expected

14. Step 7: Consider the reliability of each teacher’s estimate reliability = variance of true scores variance of observed scores =  00 /(  00 +  2 /nj) Calculate the Empirical Bayes (EB) estimate using the Kelley equation. 14

15. Reliability of Estimates Reliability depends on the degree to which the true underling parameters vary among groups (e.g. schools). Classical test theory notion is that reliability = variance of true scores variance of observed scores =  00 /(  00 +  2 /nj)

16. Step 7 continued – The Kelley equation: b EB = b ols ( ) + Y(1- ) 16

17. Resid EB =.768(.97) + -.005(1-.97) =.751. 17 Reliability EB Residual OLS Residual

18. EB Residual |residual ols | > |residual EB |; |.773| > |.751|, Which is why this is termed a “shrunken estimate.” The EB residual is a teacher’s VAS. 18

19. Step 8: We normalize VAS scores so that results from all course groups (and assessment types, e.g. EoC, Dibels, etc) will be on the same scale. VAS normalized = (VAS –VAS..)/SD VAS VAS.. is calculated for each Course group. And where applicable, by course group by grade. E.g. VAS normalized =.751 – (-.005)/.4896 = 1.54. 19

20. This Teacher’s VAS of 1.54 places him/her in the Highly effective range. 20

21. Step 9: – Converting VAS scores into EES points. – Given the normalization in the previous step, we take the normal CDF of the VAS: – In excel this is =NORMSDIST(VAS). – And results in: 21

22. VAS to Points Conversion 22 VAS = 1.54

23. Notes: – The differences between an actual VAS calculation and the example: Prior achievement (etc) is included in the student level model. Peer effects are included (e.g. class average prior math and reading achievement). – The level 2 (teacher level) model determines what the EB estimates will be shrunk towards (in the previous example this was the grand mean because there were no level 2 predictor variables, but for the EES, it includes peer effects). 23

24. 24

25. Notes continued: – The actual VAM utilizes the CSEM to eliminate potential relationships between the predictors and the VAS, as well as to help guard against the impact of outliers (extreme test scores). A teacher’s VAS in the Summative Report is the weighted Average of all the available VAS scores for a teacher. – The weights are the number of students that contributed to a VAS score (which may not equal enrollment ). – This can consist of multiple VAS scores per year and multiple years. 25

26. 26 E.G. in TotVAS11 =.(58*18+.93*18)/36 =.76

27. 27 TotVAS_all = value in Summative report and used to calculate points =.76*36 + 1.26*48 + 1.04*19 = 107.6/103 = 1.04.

28. VAS score for teacher with unconditional VAS of.751 is –.170 using full model and is – 1.18 when normalized. – This = 61.9 points assuming 70 points possible in STAM 1. 28

29. How about Excel? TeacherStudentCourse ID (4)Year Proportio nGradeSSSCItSSMt-1SSMt-2SSRt-1SSRt-2 111707201210074133293634 121707201210073341396147 131707201210074951435146 1417072012100755434544 1517072012100751 444837 16170720121007414443 47 171707201210072833384543 181707201210072633262920 191707201210072931354243 1101707201210074338 3932 1111707201210074842454743 1121707201210074542323628 18817072012100736313739 Average45.0743.2041.6147.1642.43 Average of prior Averages43.60 Estimated VAS1.63 Estimated Pct of Points Earned0.95 Estimated EES Points (out of 70)66.37 29 There is no guarantee that this method will provide a close approximation of the actual VAS score – however, the sign and magnitude should provide some approximation. A regression for each teacher will result in a VAS of 0.