Sensitivity of Teacher Value-Added Estimates to Student and Peer Control Variables October 2013 Matthew Johnson Stephen Lipscomb Brian Gill

Value-Added Models (VAMs) Used Today Differ in Their Specifications 2 Value-Added Model Student Characteristics Peer Characteristics Multiple Years of Prior Scores Chicago Public SchoolsYesNo DC IMPACTYes No FloridaYes Pittsburgh Public SchoolsYes No SAS EVAASNo Yes

 How sensitive are teacher VAM estimates to choice of control variables? –Are estimates for teachers with more students from disadvantaged backgrounds affected by this choice?  Does the substitution of teacher-year level average student characteristics in place of classroom averages impact teacher VAM estimates?  Does allowing for relationship between current and lagged achievement to vary based on student demographic characteristics matter for teacher VAM estimates? Research Questions 3

 Data from a northern state and a medium-sized urban district in that state –District has more minority and low-income students than state average  Estimate separate VAMs using state data and district data – More control variables available in district VAMs –For peer characteristics, use teacher-year level averages in state VAMs, classroom averages in district VAMs  Each VAM uses three years of teaching data from through Data 4

 Explore sensitivity to several specifications: –Exclude peer average characteristics (X ̅ i,t ) –Exclude student characteristics (X i,t ) and peer characteristics (X ̅ i,t ) –Add scores from two prior years (Y i,t-2 ) –Interact free/reduced lunch status with baseline scores  Estimate all models using the same set of student observations  Control for measurement error in prior test scores using an errors-in-variables approach  Empirical Bayes (shrinkage) adjusted estimates Baseline Model 5

Student Controls (State) Student Controls (District) Peer Averages (State) Peer Averages (District) Free or Reduced-Price Mealsxxxx Disabilityxxxx Race/Ethnicityxxxx Genderxx English Language Learnerxxxx Age/Behind Grade Levelxx Gifted Program Participationxx Lagged Rate of Attendancexx Lagged Fraction of Year Suspendedxx Average Lagged Achievementxx SD of Lagged Achievementxx Number of Students in Classx Student and Peer Characteristics 6

Correlation of 8th-Grade State Teacher VAM Estimates Relative to Baseline Specification 7 Math (N = 2,778) Reading (N = 3,347) Exclude peer characteristics Exclude student and peer characteristics Add scores from t Add scores from t-2 and exclude student/peer characteristics Baseline: Student characteristics, peer characteristics, and prior scores from t-1 Findings are based on VAM estimates from 2008–2009 to 2010–2011 on the same sample of students.

Remove Student/Peer Controls and Add t-2 Scores 1st (Lowest)2nd3rd4th 5th (Highest) Baseline Model 1st (Lowest) nd rd th th (Highest) Percentage of 8th-Grade Reading Teachers in Effectiveness Quintiles, by VAM Specification 8 Findings are based on VAM estimates for 3,347 reading teachers in grade 8 from 2008–2009 to 2010–2011. Correlation with baseline =

How Are Teachers in One District Affected? 9  District has relatively large fraction poor and minority students Math Grade 8Reading Grade 8 District Percentile Rank State Percentile Rank: Baseline Exclude peer characteristics Exclude student and peer characteristics Percentile Rank of District Teachers in State Distribution

Using Additional Controls in District Data 10 Math Grades 6-8 (N = 164) Reading Grades 6-8 (N = 215) Exclude peer characteristics Exclude student and peer characteristics Add scores from t Add scores from t-2 and exclude student/peer characteristics Baseline: Student characteristics, peer characteristics, and prior scores from t-1 Findings are based on VAM estimates from 2008–2009 to 2010–2011 on the same sample of students.

Teacher-Year Average Student Characteristics vs. Classroom Average 11 Math Grades 6-8 (N = 164) Reading Grades 6-8 (N = 215) Correlation Between Effect Estimates Average Standard Error (Classroom) Average Standard Error (Teacher)

 Correlation of teacher effect estimates with baseline model above 0.99 for both subjects Different Relationship Current and Prior Test Scores for FRL Students 12 Math Grade 8 (N = 2,778) Reading Grade 8 (N = 3,347) Non-FRL Student Coefficient on Prior Year Math Score (SE) (0.002) (0.003) FRL student coefficient on Prior Year Math Score (SE) (0.003) (0.004) Non-FRL Student Coefficient on Prior Year Reading Score (SE) (0.002) (0.003) FRL student coefficient on Prior Year Reading Score (SE) (0.003) (0.004)

 Teacher VAM estimates highly correlated across specifications –Choice of control variables –Use of teacher-year level averages in place of classroom averages –Interaction between FRL status and prior scores  Choice of control variables can impact estimates for teachers of disadvantaged students Conclusions 13

 Other researchers have examined correlations in teacher effect estimates when different same-subject assessments are used as outcomes for teacher VAMs  The highest correlations these authors found are: –Lockwood et al. (2007): 0.46 –Sass (2008): 0.48 –Concoran et al. (2011): 0.62 –Lipscomb et al. (2010): 0.61 –Papay (2011): 0.54 Context for Results 14

Mathematica ® is a registered trademark of Mathematica Policy Research.  Please contact –Matthew Johnson –Stephen Lipscomb –Brian Gill For More Information 15

Mathematica ® is a registered trademark of Mathematica Policy Research. Remove Student/Peer Controls Add t-2 Scores 1st (Lowest)2nd3rd4th 5th (Highest) Baseline Model 1st (Lowest) nd rd th th (Highest) Percentage of Grade 6-8 Reading Teachers in Effectiveness Quintiles, by VAM Specification 16 Findings are based on VAM estimates for 215 reading teachers in grades 6-8 from 2008–2009 to 2010–2011. Correlation with baseline =