1. Reversing The Widget Effect Teacher Union Reform Network I November 9 -10

2. 2 © The New Teacher Project 2009 Agenda About The New Teacher Project Studying Teacher Evaluation The Current Problem The Current Opportunity Preliminary Data About Peer Review Programs How to Build a Better Evaluation System Appendix

3. 3 © The New Teacher Project 2009 The New Teacher Project (TNTP) helps school districts and states fulfill the promise of public education by ensuring that all students— especially those from high-need communities—get excellent teachers. National nonprofit, founded by teachers in 1997 Partners with school districts, state education agencies, and charter schools Targets acute teacher quality challenges Delivers a range of customized services and solutions on a fee-for-service basis Approx. 200 employees, most embedded in school district offices; majority are former teachers Past and present clients include: Districts: Baltimore, Chicago, Denver, Memphis, New Orleans, New York, Oakland, Philadelphia, San Antonio, Washington, DC States: Alaska, Louisiana, Texas, Virginia

4. 4 © The New Teacher Project 2009 Agenda About The New Teacher Project Studying Teacher Evaluation The Current Problem The Current Opportunity Preliminary Data About Peer Review Programs How to Build a Better Evaluation System Appendix

5. 5 © The New Teacher Project 2009 Study Scope In Fall 2008 through Spring 2009, TNTP partnered with 12 school districts in four states to analyze each district’s evaluation, tenure, remediation and dismissal policies and practices as well as each state’s teacher performance management policies. Represented: Urban, suburban and rural districts Collective bargaining and non-collective bargaining states Peer review and traditional evaluation approaches

6. 6 © The New Teacher Project 2009 Data Collection ArkansasColoradoIllinoisOhioTOTAL Districts Studied 423312 Teachers Surveyed 2,1962,4287,4823,07015,176 Principals Surveyed 1171847941861,281 Former Teachers Surveyed N/A344227 798 Interviews Conducted* 36233641136 Teacher Evaluation Records Collected 1,5885,85528,1477,04042,630 *Interviews conducted with some combination of teachers, principals, local teacher union leaders, local principal professional organization leaders, superintendent, district HR staff, legal counsel, and school board members.

7. 7 © The New Teacher Project 2009 Advisory Panels Membership One panel per state Approximately 80 stakeholders total Advisory Panel members include: oDistrict leaders and HR staff oLocal and state union leadership oSchool board members oState policy makers oEducation advocates oFoundation representatives Three formal meetings, 2-3 hours long. TNTP’s research benefited from the extensive input and feedback of Advisory Panels in each of the four states studied. Role Share information and perspective on state/district context. Suggest issues for additional exploration and sources of information. Comment on study methodology and data collection methods and instruments. React to preliminary findings and recommendations. Craft brief responses to the study for publication in unedited form.

8. 8 © The New Teacher Project 2009 Agenda About The New Teacher Project Studying Teacher Evaluation The Current Problem The Current Opportunity Preliminary Data About Peer Review Programs How to Build a Better Evaluation System Appendix

9. 9 © The New Teacher Project 2009 The Widget Effect “When it comes to measuring instructional performance, current policies and systems overlook significant differences between teachers. There is little or no differentiation of excellent teaching from good, good from fair, or fair from poor. This is the Widget Effect: a tendency to treat all teachers as roughly interchangeable, even when their teaching is quite variable. Consequently, teachers are not developed as professionals with individual strengths and capabilities, and poor performance is rarely identified or addressed.” The New Teacher Project, 2009

10. 10 © The New Teacher Project 2009 Teacher Evaluation: Where We Codify The Widget Effect Treating teachers as interchangeable parts All teachers are rated “good” or “great.” Although teachers and principals report that poor performance is common, less than 1 percent of teachers are identified as “unsatisfactory” on performance evaluations. Excellence goes unrecognized. When excellent ratings are the norm, truly exceptional teachers cannot be formally identified. Nor can they be compensated, promoted or retained. Professional development is inadequate. Almost 3 in 4 teachers did not receive any specific feedback on improving their performance in their last evaluation. Novice teachers are neglected. Low expectations for beginning teachers translate into benign neglect in the classroom and a toothless tenure process. Poor performance goes unaddressed. Half of the 12 districts studied have not dismissed a single non- probationary teacher for poor performance in the past five years. None dismisses more than a few each year.

11. 11 © The New Teacher Project 2009 When multiple ratings are available, teachers tend to be assigned the highest ratings and are very rarely assigned poor ratings.

12. 12 © The New Teacher Project 2009 In districts that use binary “Satisfactory/Unsatisfactory” rating systems, the “Unsatisfactory” rating is almost never used. Evaluation Ratings for Tenured Teachers in Districts with Binary Rating Systems

13. 13 © The New Teacher Project 2009 These data often stand in sharp relief against current levels of student achievement, as this example from Denver illustrates. Frequency of Unsatisfactory Ratings in Denver Public Schools that Did Not Meet AYP In Denver schools that did not make adequate yearly progress (AYP) for three consecutive years (2005-06 to 2007-08), more than 98 percent of tenured teachers received the highest rating —satisfactory.

14. 14 © The New Teacher Project 2009 Teachers say districts aren’t doing enough to recognize and retain top performers as measured by their impact on student learning… 59% of teachers report their district is not doing enough to identify, recognize, compensate, promote and retain the most effective teachers as measured by their impact on student learning. “All the good quality teachers leave the district after just a few years. They need more incentive to stay.” “Some sort of recognition or praise would be nice. Those doing a good or great job are never told so.” “If you pay the shining stars the same as the slackers, you will dim the shining stars and reinforce the sloth of the slackers.” “I, and others, work hard because we have a conscience, but I don't think [the district] sees us as any different than the lower performing teachers. Teachers who work hard receive very little praise or notice.” TNTP survey of 7,318 teachers across four sites conducted May 2008 to April 2009

15. 15 © The New Teacher Project 2009 Percent of teachers who had development areas identified on their most recent evaluation. Tenured/non- probationary teachers Probationary teachers Most likely (Denver) 32%55% Average 22%37% Least likely (Springdale) 2%4% Source 1 : TNTP survey of 15,176 teachers across 12 sites conducted May 2008 to April 2009 Source 2 : TNTP survey of 1,863 Denver Public School teachers conducted November to December 2008 …or to ensure that evaluation practices and systems provide them with meaningful feedback. of Denver teachers who had a development area identified on their most recent evaluation “do not know” which performance standard they failed to meet. 39%

16. 16 © The New Teacher Project 2009 And despite the fact teachers and principals agree that poor instruction is pervasive… Source: TNTP survey of 7,318 teachers and 751 administrators across 4 sites conducted February to April 2009; teacher evaluation data provided by each district. “Are there tenured/non-probationary teachers in your school who deliver poor instruction?” (Percent responding “Yes”) 0%0.4%n/a0% Percent of All Ratings that Indicated “Unsatisfactory” Performance

17. 17 © The New Teacher Project 2009 …dismissal for poor instructional performance almost never occurs. Frequency of Teacher Dismissals for Performance (Non-Probationary Teachers)

18. 18 © The New Teacher Project 2009 Agenda About The New Teacher Project Studying Teacher Evaluation The Current Problem The Current Opportunity Preliminary Data About Peer Review Programs How to Build a Better Evaluation System Appendix

19. 19 © The New Teacher Project 2009 Research has shown that teachers have a greater impact on student learning than any other school-based factor. Dallas students who start 3 rd grade at about the same level of math achievement… After 3 EFFECTIVE Teachers After 3 INEFFECTIVE Teachers …may finish 5 th grade math at dramatically different levels depending on the quality of their teachers. Original analysis by the Education Trust. Source: Heather Jordan, Robert Mendro, and Dash Weerasinghe, The Effects of Teachers on Longitudinal Student Achievement, 1997. 50

20. 20 © The New Teacher Project 2009 Teacher Effectiveness (e.g., Value Add, Growth, Eval. Rating) Boost effectiveness of all teachers through effective evaluation and targeted professional development Improve or exit persistently less effective teachers and replace with more effective teachers. Retain and leverage most effective teachers 5 2 4 1 Optimize new teacher supply by hiring from preparation programs whose teachers consistently achieve higher student outcomes. Increase the concentration of effective teachers in high-need schools. 3 Current Performance Potential Performance Therefore, dramatic improvements in student achievement cannot occur without a sustained focus on maximizing teacher effectiveness. 5 Goals for Optimizing Teacher Effectiveness

21. 21 © The New Teacher Project 2009 TNTP Human Capital Continuum Education leaders must use clear evaluations of teacher effectiveness to inform decisions at each step of the human capital continuum. Supportive policy context that removes barriers and publicizes results / outcomes data Optimizing Supply Managing Effectiveness Recruitment Selection Training / Certification Hiring / Placement On- Boarding School-Level Human Cap. Mngmnt. Evaluation / Prof. Dev. Working Conditions Retention / Dismissal Compensation Evaluation system that differentiates teachers based on student academic growth

22. 22 © The New Teacher Project 2009 Most of the districts we studied considered teacher effectiveness only when making decisions about remediation and dismissal.

23. 23 © The New Teacher Project 2009 Improvement: All teachers should report receiving ongoing feedback that helps them improve, and there should be evidence of that improvement in their observations and in student performance. Higher Standards: Teachers should see a high standard of instruction being upheld for themselves and others, and they should not report instances of poor performance going unaddressed. Credibility: Teachers should feel evaluations are accurate and credible. Differentiation: Each school should know which teachers are currently performing at great, good, fair, and poor levels. Recognition: Examples of outstanding instruction should be recognized and made available to other teachers to help them improve. Support: Professional development should be aligned to the evaluation process. Outcomes of an Effective Teacher Evaluation System

24. 24 © The New Teacher Project 2009 Agenda About The New Teacher Project Studying Teacher Evaluation The Current Problem The Current Opportunity Preliminary Data About Peer Review Programs How to Build a Better Evaluation System Appendix

25. 25 © The New Teacher Project 2009 For many years, peer review has been an innovative approach to teacher evaluation, including in Toledo, a Widget Effect study site. Components of peer review: Early career mentoring and evaluation as a component of the tenure granting process. Joint labor-management decision-making about renewal, support for struggling teachers and teacher dismissal, usually through the use of lead or master teachers. Intervention for poor performance where teachers or principals refer tenured teachers for intervention when necessary. Beliefs guiding peer review: Teachers have a vested interest in the quality of the workforce and will hold their peers to even higher standards than administrators would. The best way to ensure quality is to provide intensive support and feedback to beginning teachers and create avenues for intervention with struggling veterans. Teachers and administrators will make the appropriate referrals for intervention when they observe cases of poor performance.

26. 26 © The New Teacher Project 2009 Compared to teachers in other districts we studied, teachers in Toledo are very satisfied with the supports they receive. *Source: Survey of 14,611 teachers in 11 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.” **Source: Survey of 13,664 teachers in 10 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.” Districts studied in The Widget Effect Toledo

27. 27 © The New Teacher Project 2009 But Toledo ranks in the middle among districts whose teachers believe the evaluation system accurately assesses them and helps them improve their instruction. *Source: Survey of 14,611 teachers in 11 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.” **Source: Survey of 14,611 teachers in 11 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.” Toledo Districts studied in The Widget Effect

28. 28 © The New Teacher Project 2009 About half of Toledo teachers say there is at least one tenured teacher in their school whose instruction is poor enough to warrant dismissal. Source: Survey of 15,176 teachers in 12 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Yes.” Districts studied in The Widget Effect Toledo

29. 29 © The New Teacher Project 2009 According to data from the Toledo Federation of Teachers, the peer review program resulted in 26 departures over a four period. Non-Tenured Teacher DismissalsTenured Teacher Dismissals School YearFormalInformalFormalInformal 2004-2005 4202 2005-2006 4000 2006-2007 2002 2007-2008 4501 TOTAL 14705 6.5 tenured teachers exited annually through formal and informal means, on average. The average number of teachers in Toledo during this time was 2,084.

30. 30 © The New Teacher Project 2009 And less than half of Toledo teachers believe the district maintains high instructional standards—among the lowest percentages of the districts we studied. *Source: Survey of 14,611 teachers in 11 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.” **Source: Survey of 15,176 teachers in 12 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Yes.” Toledo Districts studied in The Widget Effect

31. 31 © The New Teacher Project 2009 Continue visits throughout a teacher’s career, whether they are referred or not. Conduct random observations to audit the quality of instruction, and refer teachers to PAR if performance is poor. Actively canvas teachers about the quality of instruction in their team, grade level and school. Do not wait for teachers to come forward and report problems. Apply multiple ratings rather than only satisfactory and unsatisfactory. Nuanced provides richer feedback, allows progress to be tracked, and recognizes excellence. Use PAR for more than determining which teachers are dismissed. The most comprehensive evaluation systems drive decisions about teacher development, assignment, promotion, compensation, etc. We recommend several strategies to ensure peer review is as effective an evaluation system as possible.

32. 32 © The New Teacher Project 2009 Agenda About The New Teacher Project Studying Teacher Evaluation The Current Problem The Current Opportunity Preliminary Data About Peer Review Programs How to Build a Better Evaluation System Appendix

33. 33 © The New Teacher Project 2009 Evaluator Staffing Staffing structures that allow evaluators to dedicate the time necessary to observe and evaluate teachers. Multiple Evaluation Criteria  Teacher contributions to student learning.  Teacher actions, as compared to clearly articulated criteria that reflect what exemplary performance looks like at a given experience level. District Policies Shift away from centralized PD to tailored school- level development. Requirement that teachers’ evaluations drive renewal, compensation, career path opportunities, and other rewards / consequences. Probationary periods that last long enough to assess effectiveness comprehensively. Reasonable documentation requirements. Efficient processes for recognizing excellence and addressing ineffectiveness. Strong evaluation systems are based on a comprehensive set of strategies. School and District Leader Accountability Accountability for school and district leaders for conducting accurate, useful performance evaluations and improving teacher performance. Processes and checks to ensure accuracy of evaluations (e.g., tenure review board). Evaluator Training and Support Training and ongoing support for administrators and other evaluators to ensure consistency in evaluations. Training on how to use evaluations to inform feedback and individualized developmental interventions.

34. 34 © The New Teacher Project 2009 Trajectory Low High Effectiveness LowHigh Teacher evaluations should assess both effectiveness and trajectory. Place on a rigorous improvement plan or replace Recognize and appreciate, but offer targeted support Renew for another year, but expect more Reward, retain, and provide chances to lead from the classroom Effectiveness : The teacher’s positive impact on student learning during the evaluation period Trajectory : The rate of improvement the teacher has demonstrated during the evaluation period

35. 35 © The New Teacher Project 2009 Expectations should increase with experience, and tenure should be awarded only if a teacher meets those expectations consistently. Effectiveness Year 1Year 2Year 3Year 4Year 5 Minimum expectations for first year teachers to get renewed Minimum expectations for teachers to get tenure Duration of District’s Probationary Period (e.g., 5 years) Expectations The level of effectiveness at which the teacher’s performance could hold steady and still fully meet expectations in the future.

36. 36 © The New Teacher Project 2009 Research suggests that precise language will improve calibration of raters. Less PrecisionMore Precision Criteria that are articulated in such a vague or confusing manner that we are not confident principals are able to minimize subjectivity and make repeatable assessments. Criteria that are oversimplified, too high level, and not actionable, so much so that one can interpret excellence to be very different things (e.g., may allow lower expectations, rather than higher expectations). Criteria that are so poorly worded that it is unclear what is being evaluated. Criteria that evaluate something entirely different from what it can reasonably assumed it was intended to measure. Criteria that bundle multiple, distinct, and elements criteria together and force the principal to make one evaluation. Criteria that are articulated in such a way (e.g., with precise and clear language) that principals can use the criteria to make repeatable evaluations while minimizing false positives. Criteria that are precisely defined for the teacher and the principal. o The criteria help principals know what to look for and minimize interpretation and subjectivity. o The criteria help teachers know what is expected of them, and know what excellence looks like. “One way to increase reliability is to ensure that the evaluation instrument has clearly defined, non-subjective criteria that require minimal interpretation.” (NCCTQ, 2008)

37. 37 © The New Teacher Project 2009 Agenda About The New Teacher Project Studying Teacher Evaluation The Current Problem The Current Opportunity Preliminary Data About Peer Review Programs How to Build a Better Evaluation System Appendix

38. 38 © The New Teacher Project 2009 About The New Teacher Project The New Teacher Project (TNTP) helps school districts and states fulfill the promise of public education by ensuring that all students—especially those from high-need communities—get excellent teachers. A national nonprofit organization founded by teachers, TNTP is driven by the knowledge that although great teachers are the best solution to educational inequality, the nation’s education systems do not sufficiently prioritize the goal of effective teachers for all. In response, TNTP develops customized programs and policy interventions that enable education leaders to find, develop and keep great teachers and achieve reforms that promote effective teaching in every classroom. Since its inception in 1997, TNTP has recruited or trained approximately 33,000 teachers—mainly through its highly selective Teaching Fellows™ programs—benefiting an estimated 4.8 million students. TNTP has also released a series of acclaimed studies of the policies and practices that affect the quality of the nation’s teacher workforce, most recently including The Widget Effect: Our National Failure to Acknowledge and Act on Differences in Teacher Effectiveness (2009). Today TNTP is active in more than 25 cities, including Baltimore, Chicago, Denver, New Orleans, New York, and Oakland, among others. For more information, please visit www.tntp.org.www.tntp.org

39. 39 © The New Teacher Project 2009 For more information: www.tntp.org