1. What Does Research Tell Us About Identifying Effective Teachers? Jonah Rockoff Columbia Business School Nonprofit Leadership Forum, May 2010

2. 2 First, Let’s Define “Effective” ● Can be an inputs based concept –Observable actions or characteristics ● Can be outcomes based concept –Measured by student success ● Recent work of economists focuses on outcomes –Use a value-added approach –Outcomes measured are typically standardized exams in math and reading, usually elementary/middle school ● Movement to bring rigorous analysis to teacher evaluations based on in-class observation

3. 3 Basics of Value Added Analysis ● VA is all about comparing actual student outcomes to a counterfactual expectation ● Suppose we knew the “right” counterfactual expectation for each child, call it A* –Expected achievement w/ some basic level of educational quality (e.g., “the average teacher”) 1.Subtract expectation (A*) from actual student achievement (A); call this G 2.To get VA for a teacher, take the average G across all of the students she taught

4. 4 Setting Expectations ● How to set up the counterfactual expectation is the big question in value-added work ● Typically, we estimate expectations with data –Example: set expectation (A*) as the average achievement of students w/ the same prior test scores ● Quality of estimates contingent on quality of the data and the process that generates it –Expectations set too low make teachers look good; expectations set too high make them look bad

5. 5 Potential Statistical Problems ● #1: Systematic sorting of students –Concern here is bias –Unfair treatment of teachers that is systematic Example: P’s friends get “easier” kids ● #2: Instability of VA estimates –Concern here is imprecision –If estimates are very noisy, using them for rewards/consequences means lots of mistakes –Also means it may be a poor motivational tool

6. 6 Basic Findings from VA Research ● Substantial variation in VA across teachers –1 s.d. in VA  0.1 to 0.2 s.d. in achievement –A bit more variation in math than reading/ELA –Much of the variation is within schools ● VA estimates appear to contain real power to predict teacher effectiveness as measured by student achievement –Stability across years is enough to appear useful in teacher evaluation –Bias is not a big deal overall, though it could matter for individual teachers

7. 7 Results on Stability from KS, KRS (1) Group Teachers, Years 1/2 (2) Compare in Years 3/4 (3) Large Persistent Differences

8. 8 Why Get Excited About Value Added? ● Why not just hire good teachers? –Wise selection is the best means of improving the school system, and the greatest lack of economy exists wherever teachers have been poorly chosen. Frank Pierrepont Graves, NYS Commissioner, 1932 ● Because it is, unfortunately, easier said than done –Decades of work on type of certification, graduate education, exam scores, GPA, college selectivity –(Very) small, positive effects on student outcomes ● Rockoff et al. (2008): non-traditional predictors –Personality, content knowledge, cognitive ability, self- efficacy, commercial teacher selection test score –Result: no silver bullets, but moderate power to distinguish when pool measures into an index

9. 9 What You Get is What You See? ● Why not identify individuals likely to be effective teachers through direct observation of teaching? ● There is consistent evidence that subjective evaluations of existing teachers are strongly related to gains in student achievement –Research extends back nearly a century Hill (‘21), Gotham (‘45), Brookover (’45), Anderson (’54) –More recent analysis focuses on rubric-based teaching evaluations and principal opinions Schacter & Thum, Milanowski, Tyler et al., Rockoff & Speroni, Jacob & Lefgren, Harris & Sass, Rockoff et al.

10. 10 Less Math, but No Less Difficult ● One nice aspect of subjective evaluation is it does not rely on complicated formulae ● However, the details of how evaluation is done present issues similar to VA analysis –Context (Does one size fit all?) –Focus (What goes on the evaluation form?) –Bias (Are evaluators fair and impartial?) –Imprecision (A few lessons  a whole year?)

11. 11 A (Modest?) Proposal ● Provide VA estimates to principals –Help them with the problem of estimating A* –Let them combine VA with other information (e.g., observation) to evaluate teachers ● NYC has done this: “Teacher Data Reports” –Piloted in using a randomized control trial “Treatment principals” received reports and training –Rockoff et al. (2009) study this pilot using baseline and follow-up surveys of principals

12. 12 Principals’ Evaluations and VA ● Substantial variation in baseline evaluations ● Strong relationship with VA estimates

13. 13 New and Useful Information? ● Were treatment principals’ evaluations affected by the VA reports? –Are the effects greater for more precise VA?

14. 14 In Conclusion ● Identifying highly effective teachers is near impossible if all you have to go on is a CV ● Value-added and in-class observation offer potential insight into this problem –Both, of course, are imperfect ● Innovative evaluation policies that begin to harness this information can raise teacher quality and improve student outcomes