Gathering Feedback for Teaching Combining High-Quality Observations with Student Surveys and Achievement Gains

Multiple Measures of Teaching Effectiveness 2

The MET project is unique …  in the variety of indicators tested, 5 instruments for classroom observations Student surveys (Tripod Survey) Value-added on state tests  in its scale, 3,000 teachers 22,500 observation scores (7,500 lesson videos x 3 scores) trained observers 44,500 students completing surveys and supplemental assessments  and in the variety of student outcomes studied. Gains on state math and ELA tests Gains on supplemental tests (BAM & SAT9 OE) Student-reported outcomes (effort and enjoyment in class) 3

4 Section 1 Reinventing Classroom Observations

5 Four Steps Four Steps to High-Quality Classroom Observations

Actual scores for 7500 lessons. Step 1: Define Expectations Framework for Teaching (Danielson) 6 Four Steps

Step 2: Ensure Accuracy of Observers 7 Four Steps

Step 3: Monitor Reliability 8 Four Steps

9 Multiple Observations Result in Higher Reliability NOTES: The numbers inside each circle are estimates of the percentage of total variance in FFT observation scores attributable to consistent aspects of teachers’ practice when one to four lessons were observed, each by a different observer. The total area of each circle represents the total variance in scores. These estimates are based on trained observers with no prior exposure to the teachers’ students, watching digital videos. Reliabilities will differ in practice. See the research paper, Table 11, for reliabilities of other instruments.

Step 4: Verify Alignment with Outcomes 10 Four Steps Teachers with Higher Observation Scores Had Students Who Learned More

11 Section 2 What do students say?

12 Students Distinguish Between Teachers Percent of Students by Classroom Agreeing

13 Students Distinguish Between Teachers Percent of Students by Classroom Agreeing

14 Students Distinguish Between Teachers Percent of Students by Classroom Agreeing

15 Students Distinguish Between Teachers Percent of Students by Classroom Agreeing

16 Students Distinguish Between Teachers Percent of Students by Classroom Agreeing

17 Students Distinguish Between Teachers Percent of Students by Classroom Agreeing

18 Students Distinguish Between Teachers Percent of Students by Classroom Agreeing

19 Section 3 The “Dynamic Trio”: Classroom observations, student feedback and student achievement gains.

Dynamic Trio 20 Three Criteria: Predictive power: Which measure could most accurately identify teachers likely to have large gains when working with another group of students? Reliability: Which measures were most stable from section to section or year to year for a given teacher? Potential for Diagnostic Insight: Which have the potential to help a teacher see areas of practice needing improvement? (We’ve not tested this yet.)

Dynamic Trio Measures have different strengths …and weaknesses 21

Dynamic Trio Combining Measures Improved Reliability as well as Predictive Power 22 Note: For the equally weighted combination, we assigned a weight of.33 to each of the three measures. The criterion weights were chosen to maximize ability to predict a teacher’s value-added with other students. The next MET report will explore different weighting schemes. Observation alone (FFT) Student survey alone VA alone Combined (Equal Weights) Combined (Criterion Weights) Difference in Math VA (Top 25% vs. Bottom 25%) Reliability Note: Table 16 of the research report. Reliability based on one course section, 2 observations. The Reliability and Predictive Power of Measures of Teaching:

23 Section 4 Better Information?

Compared to What? Compared to MA Degrees and Years of Experience, the Combined Measure Identifies Larger Differences 24 … on state tests

Compared to What? …and on low stakes assessments 25

Compared to What? …as well as on student-reported outcomes. 26

The MET project reporting timeline: 1.Student Perceptions 12/ Classroom Observations 1/ Weighting mid-2012 Rationale for different weighting schemes Consequences for predictive power and reliability 4.Final report using random assignment mid-2012 Do “value-added” estimates control adequately for student characteristics? Do they predict outcomes following random assignment? 27 MET project reports available at

Resources 1.Observation instruments (from developers) 2.Student surveys (MET version of Tripod survey) 3.Rater certification software (coming soon) 4.Reliability check methodology (see Research Report) 28

29 The MET Project is ultimately a research project. Nonetheless, participants frequently tell us they have grown professionally as a result of their involvement. Below is a sampling of comments we received. From Teachers: “The video-taping is what really drew me in, I wanted to see not only what I’m doing but what are my students doing. I thought I had a pretty good grasp of what I was doing as a teacher, but it is eye opening … I honestly felt like this is one of the best things that I have ever done to help me grow professionally. And my kids really benefited from it, so it was very exciting.” "With the videos, you get to see yourself in a different way. Actually you never really get to see yourself until you see a video of yourself. I changed immediately certain things that I did that I didn't like.” “I realized I learned more about who I actually was as a teacher by looking at the video. I learned of the things that I do that I think that I’m great at I was not so great at after all.” “Even the things I did well, I thought, ok that's pretty good, why do I do that, and where could I put that to make it go farther. So it was a two-way road, seeing what you do well, and seeing the things that have become habits that you don't even think about anymore." From Raters: “Being a rater has been a positive experience for me. I find myself ‘watching’ my own teaching more and am more aware of the things I should be doing more of in my classroom.” “I have to say, that as a teacher, even the training has helped me refine my work in the classroom. How wonderful!” “I have loved observing teachers, reflecting on my own teaching and that of the teachers teaching in my school.” What the Participants Said….