1. The Inter-temporal Stability of Teacher Effect Estimates J. R. Lockwood Daniel F. McCaffrey Tim R. Sass The RAND Corporation The RAND Corporation Florida State University National Conference on Value-Added Modeling, April 2008 This presentation has not been formally reviewed and should not be cited or distributed without the authors’ permission.

2. Introduction  Several school districts and states have begun using measures of teachers’ contributions to student achievement to assess and reward teachers Denver, Houston, Florida  For “value added” measures to provide correct incentives and be acceptable to stakeholders they must: be relatively accurate measures of productivity (ie.unbiased) be relatively stable over time  Most observers (implicitly) assume that a given teacher’s productivity doesn’t vary much from year to year

3. Research Questions  How stable are estimated teacher effects?  What factors affect the stability of estimated teacher effects?  Are there methods to enhance the stability of estimated teacher effects?

4. Previous Literature  Ballou (2005) Elementary and middle school teachers in a “moderately large” Tennessee school district in two consecutive years Nearly 50 percent of math teachers in top quartile in one year stay in the top quartile the next year Precision increases with number of student observations per teacher  Aaronson, et al. (2007) High school teachers in Chicago over two years 57 percent of teachers in the top quartile in one year remain there in the next year

5. Previous Literature  Koedel and Betts (2007) Teachers in San Diego Within-school estimates of teacher quality  Models with student and school fixed effects 35 percent of teachers ranked in the top quintile remain there in the next year Omission of student and school fixed effects increases stability of estimated teacher effects

6. Models of Teacher Effects  General Value-added Model Student i, classroom j, teacher k, school m X = time-varying student characteristics P = time-varying classroom peer characteristics T = time-varying teacher characteristics S = time-varying school characteristics  Teacher Classroom Average Effect

7. Data  Seven Large Countywide School Districts in Florida Two among 10 largest in the U.S. (Dade, Broward) Remainder among 25 largest in the U.S. (Hillsborough, Palm Beach, Orange, Duval and Pinellas)  Testing in Grades 3-10 FCAT-NRT (Stanford Achievement Test)  1999/2000-2004/05 (SAT-9 1999/2000, SAT-10 2004/05) FCAT-SSS (Criterion reference exam)  2000/2001-2004/05  Focus on Middle School Math Teachers Teacher effects greater in math More students per teacher in middle school

8. How Stable are Estimated Teacher Effects?  Year-to-Year Correlations  Proportion of Top-Quintile Teachers Remaining in the Top Quintile the Next Year

9. Inter-temporal Correlation in Estimated Teacher Classroom Average Effects Varying Teachers Across 2-Year Periods Correlation Between County 2000/01 and 2001/02 2001/02 and 2002/03 2002/03 and 2003/04 2003/04 and 2004/05 Broward0.480.550.470.35 Dade0.440.420.310.38 Duval0.410.450.340.23 Hillsborough0.350.330.360.23 Orange0.230.180.210.32 Palm Beach0.280.350.360.13 Pinellas0.450.380.490.34

10. Quintile Ranking of Estimated Teacher Classroom Average Effect in 2001/02 by Quintile Ranking in 2000/01 (in Percent) Broward County [Cross-Year Correlation = 0.48] Quintile in 2000/01 Quintile in 2001/02 12345 141.420.725.98.63.5 221.129.828.0714.047.02 325.017.220.3 17.2 420.315.920.326.117.4 54.79.415.6 54.7 Total22.118.321.817.320.5

11. Quintile Ranking of Estimated Teacher Classroom Average Effect in 2001/02 by Quintile Ranking in 2000/01 (in Percent) Orange County [Cross-Year Correlation = 0.23] Quintile in 2000/01 Quintile in 2001/02 12345 127.617.227.617.210.3 232.411.829.411.814.7 313.618.215.925.027.3 49.428.115.625.021.9 56.818.2 15.940.9 Total16.918.620.819.124.6

12. Percentage of Teachers Who Remain in Top Quintile from One Year to the Next County 2000/01 and 2001/02 2001/02 and 2002/03 2002/03 and 2003/04 2003/04 and 2004/05 Broward54.755.248.346.2 Dade45.740.438.939.3 Duval35.941.234.333.3 Hillsborough39.733.337.330.6 Orange40.940.426.039.5 Palm Beach34.040.530.623.1 Pinellas39.441.052.539.4

13. What Factors Affect the Stability of Estimated Teacher Effects?  Changes in the Measurement of Achievement Test Scale  If scaling changes over time, could decrease stability  Norming by grade and year should reduce fluctuations due to scaling changes  Could increase stability if distribution changes over time Test Content  If teacher ability varies across content, changes in test could contribute to instability in measured teacher effectiveness  Compare FCAT-SSS and FCAT-NRT

14. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative Test Score Measures Counties Outcome Student Controls Student Min BrowardDadeDuval Hills- borough Orange Palm Beach Pinellas Correlation Between 2000/01 and 2001/02 Estimates Gain on Normed FCAT- NRT Student Fixed Effects 10 per class 0.480.440.410.350.230.280.45 Gain on FCAT- NRT Scale Score Student Fixed Effects 10 per class 0.410.47 0.320.220.380.45

15. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative Achievement Tests Counties Outcome Student Controls Student Min BrowardDadeDuval Hills- borough Orange Palm Beach Pinellas Correlation Between 2001/02 and 2002/03 Estimates Gain on Normed FCAT- NRT Student Fixed Effects 10 per class 0.550.420.450.330.180.350.38 Gain on Normed FCAT- SSS Student Fixed Effects 10 per class 0.540.350.500.250.340.490.56

16. What Factors Affect the Stability of Estimated Teacher Effects?  Changes in Reference Point (Stratification) Individual Teacher Effectiveness Must be Measured Relative to Some Reference Point  “Holdout” teacher or Average teacher  If reference teacher changes, measured effectiveness changes Comparisons Can Only be Made to Other Teachers Who Are Interconnected by Common Students  Different strata will have different reference points  Within-school vs. between-school measures

17. Number of Teacher-Years in Interconnected Groups Counties Group TypeBrowardDadeDuval Hills- boroughOrange Palm BeachPinellas No Movers 6979335443052145536 Primary Group 5,93916,0346,8379,5838,9288,9645,020 All Others [No. of Groups] 134 [59] 97 [46] 26 [13] 61 [28] 50 [25] 48 [23] 59 [26]

18. What Factors Affect the Stability of Estimated Teacher Effects?  Omitted Variable Bias If Measured Teacher Effects Reflect Omitted Variables, Stability of Measured Teacher Effects Will Depend on Stability of Omitted Variables and Extent of Selection  Past educational inputs (persistence)  Achivement levels vs. achievement gains  Student heterogeneity  No controls vs. student covariates vs. student fixed effects  Peer heterogeneity  No controls vs. controls for peer characteristics

19. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative Persistence Assumptions Counties Outcome Student Controls Student Min BrowardDadeDuval Hills- borough Orange Palm Beach Pinellas Correlation Between 2000/01 and 2001/02 Estimates Gain on Normed FCAT- NRT Student Fixed Effects 10 per class 0.480.440.410.350.230.280.45 Level of Normed FCAT- NRT Student Fixed Effects 10 per class 0.560.550.380.320.500.410.50

20. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative Controls for Time-Invariant Student Heterogeneity Counties Outcome Student Controls Student Min BrowardDadeDuval Hills- borough Orange Palm Beach Pinellas Correlation Between 2000/01 and 2001/02 Estimates Gain on Normed FCAT- NRT Student Fixed Effects 10 per class 0.480.440.410.350.230.280.45 Gain on Normed FCAT- NRT Student Co- variates 10 per class 0.540.420.380.390.30 0.53

21. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative Controls for Time-Varying Factors (Baseline Model – Gain on Normed FCAT-NRT, Student Fixed Effects, Minimum 10 Students per Class) Controls for Time- Varying Co- variates Counties BrowardDadeDuval Hills- borough Orange Palm Beach Pinellas Correlation Between 2000/01 and 2001/02 Estimates None0.480.440.410.350.230.280.45 Student, Peer and Teacher 0.490.460.360.370.190.270.44

22. What Factors Affect the Stability of Estimated Teacher Effects?  Measurement Error in Student Achievement If measurement error is uncorrelated across students within a classroom, then precision should be higher for teachers with larger classes  Minimum class size  Minimum number of “movers” per teacher If measurement error is correlated across students within a classroom, but not across classrooms, precision should increase with the number of classes per teacher  Using middle school teachers who generally teach multiple class per term

23. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative Class Size Restrictions Counties Outcome Student Controls Student Min BrowardDadeDuval Hills- borough Orange Palm Beach Pinellas Correlation Between 2000/01 and 2001/02 Estimates Gain on Normed FCAT- NRT Student Fixed Effects 10 per class 0.480.440.410.350.230.280.45 Gain on Normed FCAT- NRT Student Fixed Effects 2 per class 0.380.330.480.290.330.250.14 Gain on Normed FCAT- NRT Student Fixed Effects 20 per class 0.490.520.410.390.310.270.58

24. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative “Student Mover” Restrictions (Minimum 10 students per class restriction) Counties Outcome Student Controls Student Min BrowardDadeDuval Hills- borough Orange Palm Beach Pinellas Correlation Between 2000/01 and 2001/02 Estimates Gain on Normed FCAT- NRT Student Fixed Effects 1 mover per teacher 0.480.440.410.350.230.280.45 Gain on Normed FCAT- NRT Student Fixed Effects 10 movers per teacher 0.480.440.410.350.230.270.45 Gain on Normed FCAT- NRT Student Fixed Effects 20 movers per teacher 0.460.500.450.360.250.290.50

25. What Factors Affect the Stability of Estimated Teacher Effects?  True Variation in Teacher Quality Over Time Instability in Estimated Effects Could Reflect Changes in True Teacher Quality Over Time  Add time-varying teacher covariates to model  Regress estimated teacher-by-year effects on teacher fixed effect and time-varying teacher covariates

26. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative Controls for Time-Varying Factors (Baseline Model – Gain on Normed FCAT-NRT, Student Fixed Effects, Minimum 10 Students per Class) Controls for Time- Varying Co- variates Counties BrowardDadeDuval Hills- borough Orange Palm Beach Pinellas Correlation Between 2000/01 and 2001/02 Estimates None0.480.440.410.350.230.280.45 Teacher Only 0.460.450.390.350.210.260.44

27. Are There Methods to Enhance the Stability of Estimated Teacher Effects?  Are there methods to enhance the stability of teacher effect estimates? 3-Year Running Averages  Reduces noise by averaging sampling errors  Could add bias if true performance is changing across years Empirical Bayes or “Shrinkage” Estimators  Place greater weight on more reliable estimates and push less reliable estimates toward population mean  If already have a significant minimum class size restriction, “simple” EB adjustments which account for differences in the number of classes per teacher or students per teacher not likely to yield large improvements to stability  Accounting for variability at the individual teacher level requires computation of standard errors on individual teacher effects, which can be problematic

28. Problems in Computing Shrinkage Estimators  Default estimates from most software packages are estimating contrasts between every teacher and a holdout teacher Such estimates support within-year comparisons and cross- year correlations  We cannot shrink these estimates directly and we cannot use the resulting standard errors for shrinkage  We cannot average these estimates without removing yearly means because changes to the holdout create year-to-year fluctuations

29. Summary  Moderate Stability in Teacher Effects Cross-year correlations in range on 0.2-0.5 About 40-50 percent of teachers in top quintile remain in the top quintile the following year  Stability increases with number of students per teacher and when persistence is assumed to equal 0 Nothing else has a consistent appreciable effect on stability Variation across districts appears substantial  Shrinkage estimators or running averages could improve stability of teacher-by-year effects, but need to get appropriate estimates and standard errors  Findings suggest caution in using value-added measures for high-stakes personnel decisions

30. Next Steps  Determine sources of year-to-year variability Improve estimation techniques to obtain comparable estimates across years with accurate measures of within-year standard errors  Current software does not support such estimation Separate noise from “true” short-term variation Model sources of short-term variation  If true year-to-year variations exists more efficient estimation than three year averages might be possible via smoothing or filtering