1. Measuring Teacher Effectiveness Laura Goe, Ph.D. Research Scientist, ETS Principal Investigator for Research and Dissemination, National Comprehensive Center for Teacher Quality Oklahoma Department of Education Oklahoma City, OK  August 15, 2012

2. 2 Laura Goe, Ph.D. Former teacher in rural & urban schools  Special education (7 th & 8 th grade, Tunica, MS)  Language arts (7 th grade, Memphis, TN) Graduate of UC Berkeley’s Policy, Organizations, Measurement & Evaluation doctoral program Principal Investigator for the National Comprehensive Center for Teacher Quality Research Scientist in the Performance Research Group at ETS 2

3. 3 The goal of teacher evaluation The ultimate goal of all teacher evaluation should be… TO IMPROVE TEACHING AND LEARNING

4. 4 Evaluation for accountability and instructional improvement Effective evaluation relies on:  Clearly defined and communicated standards for performance  Quality tools for measuring and differentiating performance  Quality training on standards and tools - Evaluators should agree on what constitutes evidence of performance on standards - Evaluators should agree on what the evidence means in terms of a score

5. 5 Feedback (Coggshall et al., 2012) “Feedback on whether or not instructional practices are working can come in the form of student learning data, the teachers’ own observations of student engagement, observations from a peer or a coach, a video-taped record of the practice, discussion within a professional learning community, or the results of a formal evaluation.” (p. 6) Coggshall, J. G., Rasmussen, C., Colton, A., Milton, J., & Jacques, C. (2012). Generating teaching effectiveness: The role of job-embedded professional learning in teacher evaluation. Washington, DC: National Comprehensive Center for Teacher Quality. http://www.tqsource.org/publications/GeneratingTeachingEffectiveness.pdf

6. 6 Multiple Standards-based Measures of Teacher Effectiveness Affords many benefits to a comprehensive evaluation system  Ability to triangulate results increases confidence in evaluation outcomes  More complete picture of teacher strengths and weaknesses  Each type of measure provides a different type of evidence All work together to better inform professional development decisions

7. 7 Using surveys for teacher evaluation Surveys are unlikely to show much change within a single year  This limits usefulness in teacher evaluation as a “pre-” and “post-” measure  But they may be useful for revealing trends over time  For example, a new teacher is likely to improve in many aspects of classroom management and discipline in the first few years, which is likely to be reflected in survey results

8. 8 Surveys (2) The survey results will likely confirm what is already known through observations and other measures, but what is most important is what teachers do with the survey results  Discussion with mentor, principal, colleagues  Compare results with school, grade, or team aggregates  Support self-reflection & focus on areas in need of improvement

9. 9 Surveys (3) Almost no research on parent surveys Research on peer surveys is all in higher education Research on student surveys is more advanced, but only on one instrument— Harvard’s Tripod Survey  Tripod was part of the Gates-funded Measures of Effective Teaching study

10. 10 Tripod Survey (1) Harvard’s Tripod Survey – the 7 C’s – Caring about students (nurturing productive relationships); – Controlling behavior (promoting cooperation and peer support); – Clarifying ideas and lessons (making success seem feasible); – Challenging students to work hard and think hard (pressing for effort and rigor); – Captivating students (making learning interesting and relevant); – Conferring (eliciting students’ feedback and respecting their ideas); – Consolidating (connecting and integrating ideas to support learning)

11. 11 Tripod Survey (2) Improved student performance depends on strengthening three legs of teaching practice: content, pedagogy, and relationships There are multiple versions: k-2, 3-5, 6-12 Measures:  student engagement  school climate  home learning conditions  teaching effectiveness  youth culture  family demographics Takes 20-30 min There are English and Spanish versions Comes in paper form or in online version

12. 12 Creating surveys (Over-simplified!) Convene a working group including teachers, principals, parents, students, etc. Determine which aspects of teaching standards can be measured with survey questions Create questions; try to ask some of the same questions in student, peer, parent, and self- assessment surveys for triangulation of responses Field-test the survey and have a psychometrician analyze results (IRT model) Make adjustments and pilot the survey (no stakes); then make final revisions

13. 13 Using service learning for teacher evaluation Service learning has the potential to be useful in teacher evaluation, particularly when trying to measure teachers’ contributions to student learning in non-tested subjects and grades (lower elementary, high school, subjects that are not part of statewide testing) The goal should be comparability in results across students and teachers within a district  As much as possible, teachers within the same subject/grade should do the same service learning projects and use the same assessments

14. 14 Service learning (2) Great value in service learning  Teachers collaborating around student learning across curriculum and grades  Most subject/grade standards can be measures with a service learning project  Learning goals and assessments can be differentiated for special needs students and ELLs This makes it important to select a service learning project that is wide in scope rather than narrow May incorporate “place-based” learning strategies  Students and teachers strengthen relationships with local community and learn about local geography and history

15. 15 Service learning (3) There is considerable research on service learning, but mostly on outcomes related to the projects  Not a lot of research on student growth that could be directly attributed to teaching practices A good service learning project would have multiple measures of student learning over time (pre- and post-) and may include multiple subjects Could take the form of a student portfolio scored with a rubric  Scoring processes, audits, etc. would need to be put into place for validity and comparability

16. 16 Creating a service learning template and guidance (oversimplified!) Convene a working group including teachers, principals, parents, students, & community leaders Determine several broad-scope projects that could take place across a semester or a school year  Using these projects as examples, identify the student learning standards that could be assessed  Identify measures that could show student learning growth on standards (portfolio entries, tests, performances, presentations, group projects, written reports/journals, etc.) Field-test projects and measures and analyze results—use to create guidance docs & templates

17. 17 Service learning for teacher evaluation purposes: A possible approach Teacher teams and principals meet with district staff to select a service learning project that all or most teachers will participate in throughout the district Using state guidance doc and templates, teachers and principals develop plan and timeline for projects  Teachers from grades and subjects work together in teams to select appropriate assessments to measure student learning Using state guidance, assessments are scored and compiled and teachers’ contributions to student learning growth can be determined

18. 18 Resources on Service Learning Bradley, L. R. (2006). Service-Learning: A strategy for school improvement and building effective community partnerships. Columbus, OH: Ohio Department of Education. http://www.servicelearning.org/filemanager/download/Ohio_Department_of_Education_tr aining_manual.pdf Colorado Department of Education. (2004). Creating high-performing schools through service-learning: A service-learning trail guide. Denver, CO: Colorado Department of Education. http://www.ecs.org/clearinghouse/57/13/5713.pdf Davis, K. M., Miller, M. D., & Corbett, W. T. (1998). Methods of evaluating student performance through service learning. Gainesville, FL: College of Education, University of Florida. http://www.fsu.edu/~flserve/resources/methods/MethodsEvaluatingStudentPerformanceT hroughSL.pdf Ernst, J. A. (2004). Prairie Science Class evaluation report. Fergus Falls, MN: Prairie Wetlands Learning Center. http://meera.snre.umich.edu/sites/all/files/Prairie_Science_Class_Evaluation_Report.pdf

19. 19 Teacher portfolios It’s not a “brag book,” it’s evidence of how the teacher sets/pursues meaningful goals related to improving knowledge & instructional practice More like a year-long self-exploration of one or more standards related to aspects of teacher’s practice such as:  Restructuring the classroom learning environment to support diverse learners’ needs  Working with a collaborative team to improve reading outcomes  Developing expertise in classroom technology to support new methods of instruction

20. 20 Portfolios (2) Portfolios do not need to be “physical” binders but can be virtual evidence collections  This is easier for most teachers to create and share and easiest for others to review  Allows teachers to include multiple types of data (videos, spreadsheets, recordings, photographs, work samples, planning documents, self-reflection, peer comments, etc.) Teachers should receive training in  Choosing standards to focus on  Choosing a way to demonstrate growth on the standards  Selecting evidence to include in the portfolio

21. 21 Portfolios (3) State’s role will be to provide guidance docs, templates, and examples of virtual portfolios at various score levels (like “unsatisfactory” and “exemplary”) Principals and others involved in scoring portfolios will need to be trained to score portfolios with accuracy and consistency in order for results to be valid indicators for teacher evaluation purposes Teachers and principals should be involved in the development, and the process should be piloted (no stakes)

22. 22 Some “popular” observation instruments Charlotte Danielson’s Framework for Teaching http://www.danielsongroup.org/theframeteach.htm CLASS http://www.teachstone.org/ Kim Marshall Rubric http://www.marshallmemo.com/articles/Kim%20Marshall %20Teacher%20Eval%20Rubrics%20Jan% Marzano Teacher Evaluation Framework http://www.marzanoevaluation.com/

23. 23 Example: University of Virginia’s CLASS observation tool Emotional Support Classroom Organization Instructional Support Pre-K and K-3 Positive Climate Negative Climate Teacher Sensitivity Regard for Student (Adolescent) Perspectives Behavior Management Productivity Instructional Learning Formats Concept Development Quality of Feedback Language Modeling Upper Elementary/ Secondary Content Understanding Analysis and Problem Solving Quality of Feedback

24. 24 Example: Charlotte Danielson’s Framework for Teaching

25. 25 Domain 1 from “The Four Domains of the Marzano Teacher Evaluation Model”

26. 26 Example: Kim Marshall’s Rubric Planning & Preparation for Learning Highly EffectiveEffectiveImprovement Necessary Does Not Meet Standards a. Knowledge Is expert in the subject area and has a cutting-edge grasp of child development and how students learn. Knows the subject matter well and has a good grasp of child development and how students learn. Is somewhat familiar with the subject and has a few ideas of ways students develop and learn. Has little familiarity with the subject matter and few ideas on how to teach it and how students learn. b. StrategyHas a well-honed game plan for the year that is tightly aligned with state standards and assessments. Plans the year so students will meet state standards and be ready for external assessments. Has done some thinking about how to cover high standards and test requirements this year. Plans lesson by lesson and has little familiarity with state standards and tests.

27. 27 Creating an observation instrument Consider the research: lots of evidence on use of Framework for Teaching and CLASS, little for other instruments  Don’t reinvent the wheel Working group to design instrument should include teachers, principals, other stakeholders Focus on teaching standards  What evidence will you need to be able to score the teacher’s performance on this standard? Field test and pilot instrument Training for scoring is crucial for valid results

28. 28 Questions?

29. 29 Laura Goe, Ph.D. 609-619-1648 lgoe@ets.org www.lauragoe.com https://twitter.com/GoeLaura National Comprehensive Center for Teacher Quality 1000 Thomas Jefferson Street, NW Washington, D.C. 20007 www.tqsource.org