Presentation on theme: "Policy Applications of Teacher Performance Measures: A Review of the Evidence Brian Jacob Presented at the Policy Forum on Teacher Effectiveness in New."— Presentation transcript:
Policy Applications of Teacher Performance Measures: A Review of the Evidence Brian Jacob Presented at the Policy Forum on Teacher Effectiveness in New Jersey March 22, 2011
My Task Summarize existing evidence on the impact of policy applications of teacher performance measures I will focus on teacher-level programs including: – Teacher value-added measures (VAM) and/or – Classroom observation protocols I will focus on programs designed for evaluation rather than for only diagnostic purposes
Outline 1.The use of VAM to determine teacher compensation, hire/transfer, and tenure/dismissal 2.Classroom Observation Protocols 3.Holistic teacher evaluation “systems” (e.g., TAP, Minnesota Q-Comp) 4.Some thoughts on implementation details
Teacher VAM and Compensation Most common use of VAM to date involves teacher pay, usually bonuses Two goals of these policies: – Increase teacher effort in the short/medium-term – Change the composition of the teaching force in the longer-term Important underlying assumptions: – Teachers are not working on the “efficiency frontier” in the existing system – Teachers have the capacity/support to change behavior – The labor supply of potentially effective teachers is relatively elastic
Evidence on VAM and Teacher Pay Studies in other countries show positive impact on student performance – Probably not very applicable to U.S. context Nashville, TN: individual bonuses for middle school math teachers up to $15,000 – No effect on achievement; few effects on any other intermediate outcomes – Period of tremendous achievement growth in Nashville: student achievement in T and C groups improved substantially New York City: bonuses assigned to schools on the basis of school performance goals, with an average bonus of $3,000/teacher for meeting achievement goals – No effect on student achievement or a host of other student and teacher measures – Some evidence of more positive effects in smaller schools
VAM and Other Teacher Policies No existing programs use VAM for hiring “Talent Transfer” program bases transfers/bonuses on VAM – Identifies high-performing teachers and offers them incentives to stay/move to high-needs schools for up to two years – Large financial incentives $10,000/yr for two years to move – Teachers identified through VA analysis and schools identified on the basis of low performance – Charlotte-Mecklenberg started in 2008; LA and Miami in 2010 No existing policy dismisses and/or denies tenure to teachers based on VAM alone
Classroom Observation Protocols (COPs) Why use classroom observations to measure teacher performance? – Additional measure increases reliability of overall rating – Captures different aspects of effectiveness – By focusing on teacher practices, COPs can (a) mitigate gaming/strategic behavior, and (b) provide useful feedback to teachers How do current COPs differ from traditional teacher evaluations conducted by administrators? – Based on set of pre-specified, observable behaviors – Outside observers; Multiple unannounced observations per year Variation in COPs – Danielson: student-teacher relationships, instructional approaches – CLASS: focus on emotional climate of the classroom – MQI: accuracy and richness of teacher’s mathematical knowledge – PLATO: focus on instructional strategies specific to secondary ELA
Research on COPs Growing research base suggests COP scores are positively associated with student achievement – Danielson: 200+ teachers in Cincinnati, positive relationship with student achievement – PLATO: 24 middle school ELA teachers in NYC – 2 nd vs. 4 th quartile – MQI: 24 middle school math teachers in Southwestern district – correlations of.2 to.5 with VAM Caveats/Limitations – Very small samples; timing of observations and student achievement – COPs are *less* predictive of current student achievement than prior VAM – COPs are*not* generally more predictive than informal/holistic supervisor ratings – Gates-funded MET project (Measures of Effective Teaching) will explore this issue with much larger samples, using much more sophisticated analysis Recent study in Cincinnati shows that participation in evaluation itself increases teacher VA by.1 s.d.
Teacher Evaluation “Systems” No evidence that merit pay alone works in the short-run Some evidence that COPs are associated with teacher VAM, and the evaluation process itself may be beneficial => Comprehensive programs: development and training opportunities, data and feedback, incentives and evaluation Examples include: D.C. IMPACT, Teacher Advancement Program (TAP), Q-Comp in Minnesota
Teacher Advancement Program (TAP) Four components: – (1) performance-based compensation – (2) master/mentor teachers – (3) framework for teacher evaluation – (4) ongoing trainings and collaborative groups Springer, Ballou and Peng (2008) – Uses student-level NWEA data in 2 states to compare achievement gains within schools over time – do gains increase after a school adopts TAP – Some positive effects for gr 1-5, but negative effects for gr 9 and 10 Mathematica Study of Chicago TAP – Yr 2 (2010) – performance pay for principals and other school staff; average teacher bonus was $2,000; mentor (master) teachers received $7,00 ($15,000) – Hybrid design – random assignment of 16 schools + matching – No effects on student achievement, teacher retention
Denver Pro-Comp Consists of 4 components: – Market incentives (hard-to-staff schools/subjects) – Student growth (VAM) – Knowledge and skills (completion of degrees or PD) – Professional evaluations Voluntary for teachers hired before 2006; mandatory for teachers hired on/after Jan 1, 2006 Growth in math and reading since 02-03 districtwide, but hard to attribute this to Pro-Comp alone Some tentative evidence suggesting there might be some very small positive effects – Productivity effects and composition effects Ongoing work looks at retention in hard-to-serve schools
Minnesota’s Q-Comp Voluntary district-level program District plan must include multiple teacher career paths, job- embedded professional development, teacher evaluation, performance pay, and a revised teacher salary schedule Districts get $190 per pupil in state aid + $70 per pupil extra Since inception in 2005, 46 of Minnesota’s 337 traditional districts and several charter schools have opted in Considerable variation in content of district plans Recent evaluation found few positive impacts on student achievement – No effect on math achievement – Potential small effects on reading achievement in some districts that focused on actions/outcomes rather than subjective evaluations
The Devil is in the (Implementation) Details Much of the debate focuses on what measures will be included and how heavily they will weigh in the overall evaluation This misses at least 3 critical implementation choices: – Where to set cutoffs for different levels? – How to average across components (VAM, COP)? – What actions are tied to evaluation outcomes?
Setting Cutoffs to Avoid the Widget Effect Report by TNTP documents that traditional teacher evaluation systems overstate the number of exemplary teachers and understate the number of mediocre or ineffective teachers Distribution of Teacher Ratings Across Districts (Source: Widget Effect) DomainDistinguishedProficientBasicUnsatisfactory Chicago68.7%24.9%6.1%0.4% Cincinnati57.8%34.7%6.9%0.6% Distribution of Teacher Ratings in DC IMPACT 09-10 Highly Effective EffectiveMinimally Effective Ineffective Overall Score16%66%16%2%
The Effect of Averaging Given the 4-point scales commonly used, components such as school performance and “other” outcomes that comprise a small fraction (~10%) of overall rating will generally have no impact on teacher’s overall rating Suppose that a system includes only VAM and COP, classifies teachers on a 4-pt scale, weights both equally, and that VAM and COP are correlated.3 – If 2% of teachers receive ratings of “1” on each, then only.4% will receive a final rating of “1” – If 10% of teachers receive ratings of “1” on each, then only 3.5% will receive lowest rating Possible remedies? – Triage: score of “1” on any measure triggers review
Teacher Non-Renewal Policy in the Chicago Public Schools (CPS) Starting in 2004-05, new collective bargaining agreement in CPS stipulated that principals could dismiss (i.e., non-renew) any probationary teacher (years 1-4) outside the RIF process and without the typical process associated with dismissal for cause Streamlined process: on-line system allow principals to “click” teachers to non-renew Effect of this policy provides some insight on potential impact of COPs and TESs – How frequently will principals to dismiss teachers? – What teacher characteristics do principals value? – What are the effects on teacher and student outcomes?
Teacher Non-Renewal in Chicago Principals do seem to consider (proxies for) teacher productivity in determining which teachers to dismiss. – Principals are more likely to dismiss teachers who are frequently absent, failed the teacher certification exam at least once, and who have received worse evaluations in the past. – Principals are less likely to dismiss teachers who attended a more competitive college and have a MA degree. – Elementary teachers who were dismissed had lower VAM than their peers who were not dismissed. Policy reduced absences among probationary teachers by roughly 10-20 percent
Teacher Non-Renewal in Chicago Roughly 40% of principals did not dismiss any of their probationary teachers over the first 3 years of the policy – Includes many principals in the lowest performing schools in the district Potential explanations – Teacher labor supply – Social norms Implications – Managerial discretion alone will not necessarily change personnel practices – Principal training? – Changing the default?
Cincinnati Teacher Evaluation System Adapted from Danielson framework; started in 2000-01 4 evaluations per year by outside peer experts; 1 additional observation by school administrator 4 domains, only 2 of which assessed by observation: learning environment (D2) and teaching for learning (D3) – Teachers receive scores from 1-4 in each of 32 elements, which are aggregated up to 15 standards and then the 4 domains End-of-year scores in each standard/domain are holistic/subjective determination of outside evaluators – Based on “preponderance of the evidence” and can account for growth over the year and/or extenuating circumstances
Evidence on Cincinnati TES TES scores predict student achievement even after controlling for various student characteristics (including prior achievement) – 1 point increase in TES => 0.10 -.15 sd increase in scores – Top vs. bottom quartile teacher => 3 percentage point diff Conditional on overall score, teachers who score high in classroom environment (Domain 2) relative to teaching practices (Domain 3) appear more effective in math Conditional on overall score, teachers whose instruction focuses on questions/discussion relative to standards/content appear more effective in reading
The Widget Effect Distribution of Teacher Ratings in Cincinnati TES – Since 2001 DomainDistinguishedProficientBasicUnsatisfactory Classroom Environment64.1%31.4%3.7%0.7% Teaching Strategies46.1%47.4%6.4%0.1%