1. Sensitivity of Teacher Value-Added Estimates to Student and Peer Control Variables October 2013 Matthew Johnson Stephen Lipscomb Brian Gill

2. Value-Added Models (VAMs) Used Today Differ in Their Specifications 2 Value-Added Model Student Characteristics Peer Characteristics Multiple Years of Prior Scores Chicago Public SchoolsYesNo DC IMPACTYes No FloridaYes Pittsburgh Public SchoolsYes No SAS EVAASNo Yes

3.  How sensitive are teacher VAM estimates to choice of control variables? –Are estimates for teachers with more students from disadvantaged backgrounds affected by this choice?  Does the substitution of teacher-year level average student characteristics in place of classroom averages impact teacher VAM estimates?  Does allowing for relationship between current and lagged achievement to vary based on student demographic characteristics matter for teacher VAM estimates? Research Questions 3

4.  Data from a northern state and a medium-sized urban district in that state –District has more minority and low-income students than state average  Estimate separate VAMs using state data and district data – More control variables available in district VAMs –For peer characteristics, use teacher-year level averages in state VAMs, classroom averages in district VAMs  Each VAM uses three years of teaching data from 2008-2009 through 2010-2011 Data 4

5.  Explore sensitivity to several specifications: –Exclude peer average characteristics (X ̅ i,t ) –Exclude student characteristics (X i,t ) and peer characteristics (X ̅ i,t ) –Add scores from two prior years (Y i,t-2 ) –Interact free/reduced lunch status with baseline scores  Estimate all models using the same set of student observations  Control for measurement error in prior test scores using an errors-in-variables approach  Empirical Bayes (shrinkage) adjusted estimates Baseline Model 5

6. Student Controls (State) Student Controls (District) Peer Averages (State) Peer Averages (District) Free or Reduced-Price Mealsxxxx Disabilityxxxx Race/Ethnicityxxxx Genderxx English Language Learnerxxxx Age/Behind Grade Levelxx Gifted Program Participationxx Lagged Rate of Attendancexx Lagged Fraction of Year Suspendedxx Average Lagged Achievementxx SD of Lagged Achievementxx Number of Students in Classx Student and Peer Characteristics 6

7. Correlation of 8th-Grade State Teacher VAM Estimates Relative to Baseline Specification 7 Math (N = 2,778) Reading (N = 3,347) Exclude peer characteristics0.9700.982 Exclude student and peer characteristics0.9640.979 Add scores from t-20.9770.958 Add scores from t-2 and exclude student/peer characteristics 0.946 Baseline: Student characteristics, peer characteristics, and prior scores from t-1 Findings are based on VAM estimates from 2008–2009 to 2010–2011 on the same sample of students.

8. Remove Student/Peer Controls and Add t-2 Scores 1st (Lowest)2nd3rd4th 5th (Highest) Baseline Model 1st (Lowest)8117110 2nd18572330 3rd12353221 4th03225916 5th (Highest)0011683 Percentage of 8th-Grade Reading Teachers in Effectiveness Quintiles, by VAM Specification 8 Findings are based on VAM estimates for 3,347 reading teachers in grade 8 from 2008–2009 to 2010–2011. Correlation with baseline = 0.946.

9. How Are Teachers in One District Affected? 9  District has relatively large fraction poor and minority students Math Grade 8Reading Grade 8 District Percentile Rank 155085155085 State Percentile Rank: Baseline 216286176690 Exclude peer characteristics 235986135987 Exclude student and peer characteristics 205885135789 Percentile Rank of District Teachers in State Distribution

10. Using Additional Controls in District Data 10 Math Grades 6-8 (N = 164) Reading Grades 6-8 (N = 215) Exclude peer characteristics0.9550.963 Exclude student and peer characteristics0.9180.949 Add scores from t-20.9870.967 Add scores from t-2 and exclude student/peer characteristics 0.9270.909 Baseline: Student characteristics, peer characteristics, and prior scores from t-1 Findings are based on VAM estimates from 2008–2009 to 2010–2011 on the same sample of students.

11. Teacher-Year Average Student Characteristics vs. Classroom Average 11 Math Grades 6-8 (N = 164) Reading Grades 6-8 (N = 215) Correlation Between Effect Estimates0.9560.975 Average Standard Error (Classroom)0.0650.073 Average Standard Error (Teacher)0.0690.075

12.  Correlation of teacher effect estimates with baseline model above 0.99 for both subjects Different Relationship Current and Prior Test Scores for FRL Students 12 Math Grade 8 (N = 2,778) Reading Grade 8 (N = 3,347) Non-FRL Student Coefficient on Prior Year Math Score (SE) 0.847 (0.002) 0.199 (0.003) FRL student coefficient on Prior Year Math Score (SE) 0.896 (0.003) 0.154 (0.004) Non-FRL Student Coefficient on Prior Year Reading Score (SE) 0.086 (0.002) 0.641 (0.003) FRL student coefficient on Prior Year Reading Score (SE) 0.031 (0.003) 0.728 (0.004)

13.  Teacher VAM estimates highly correlated across specifications –Choice of control variables –Use of teacher-year level averages in place of classroom averages –Interaction between FRL status and prior scores  Choice of control variables can impact estimates for teachers of disadvantaged students Conclusions 13

14.  Other researchers have examined correlations in teacher effect estimates when different same-subject assessments are used as outcomes for teacher VAMs  The highest correlations these authors found are: –Lockwood et al. (2007): 0.46 –Sass (2008): 0.48 –Concoran et al. (2011): 0.62 –Lipscomb et al. (2010): 0.61 –Papay (2011): 0.54 Context for Results 14

15. Mathematica ® is a registered trademark of Mathematica Policy Research.  Please contact –Matthew Johnson MJohnson@mathematica-mpr.com –Stephen Lipscomb SLipscomb@mathematica-mpr.com –Brian Gill BGill@mathematica-mpr.com For More Information 15

16. Mathematica ® is a registered trademark of Mathematica Policy Research. Remove Student/Peer Controls Add t-2 Scores 1st (Lowest)2nd3rd4th 5th (Highest) Baseline Model 1st (Lowest)7426000 2nd21532320 3rd51451301 4th07215121 5th (Highest)0051679 Percentage of Grade 6-8 Reading Teachers in Effectiveness Quintiles, by VAM Specification 16 Findings are based on VAM estimates for 215 reading teachers in grades 6-8 from 2008–2009 to 2010–2011. Correlation with baseline = 0.909.