1. Evaluation of the Wisconsin Educator Effectiveness System Pilot: Results of the Teacher Practice Rating System Pilot
Curtis Jones, UW Milwaukee Steve Kimball, UW Madison
Presented at the annual meeting of American Education Finance and Policy
2/28/2015

2. Evaluation Questions
How do educators feel about the Framework for Teaching being used to evaluate teachers?
Do educators believe they will have the time and resources necessary to complete teacher practice evaluations?
How do educators perceive the inclusion of the Framework for Teaching will impact the quality of Wisconsin teaching?
How were teachers rated overall?
How were teachers rated on components?
What factors were related to teacher ratings?

3. Pilot Participation
192 principals, and 402 teachers across 195 school districts volunteered and were trained to participate in the pilot.
Ultimately, 385 schools across 123 school districts participated, with ratings data only available for 135 of the 402 teachers who originally volunteered to pilot the process.
Within these districts, far more (449 evaluators and 2,595 teachers) piloted the teacher practice evaluation component of the EE system than were originally planned.
Of the 2,595 teachers involved in the pilot, Announced first observation results were recorded for 2,191 teachers and Unannounced for 1,466, but final ratings were only recorded for teachers across 82 schools and 43 districts.
Invitations to complete surveys were ed to 329 administrators (192 principals and coaches) and the 402 teachers that had originally agreed to pilot the system. Of these, (58%) administrators and 171 (44%) teachers responded.

4. How do educators feel about the Framework for Teaching being used to evaluate teachers?

5. Do educators believe they will have the time and resources necessary to complete practice evaluations?

6. How do educators perceive the inclusion of the Framework for Teaching will impact the quality of Wisconsin teaching?

7. How were teachers rated overall?N
Min
Max
Mean
Std. Dev.
All recorded ratings
Final ratings
497
2.05
3.91
3.13
0.25
1st Announced Observations
2186 1 4
3.03
0.37
1st Unannounced Observations
1460
2.99
0.38
Ratings for teachers with all three ratings
Final Ratings
308
3.11 2
3.9
3.04
0.26
1.89
3.07
0.27

8. How were teacher rated overall?

9. How were teachers rated on components? – Final Ratings

10. How were teachers rated on components? - Announced

11. What factors were related to teacher ratings? – Sample
Only schools with at least 5 teacher ratings were included in analyses.
The resulting sample includes 34 school districts, 129 schools, 2,173 teachers.
The 2,173 teachers represented about half of all the teachers in these schools.

12. What factors were related to teacher ratings? – Descriptive Statistics
Descriptive statistics for schools with at least 5 teachers in the pilot
Schools
Min
Max
Mean
Std. Deviation
Announced observation ratings
129
1.9
3.4
3.00
0.25
Unannounced observation ratings
110 2
3.5
2.95
0.28
Final ratings* 28
2.8
3.14
0.15
Use of EE for high stakes decisions
108 1
0.51
0.50
F/R lunch rate
128
7.8%
100%
42.2
27.5
Teacher to student ratio
126
8.1
41.7
16.5
4.6
Percent of teachers rated 6%
0.49
*Final ratings not modeled due to low n.

13. What factors were related to teacher ratings? - Correlations
Announced ratings
Unannounced ratings
Final ratings
Use of EE for high stakes decisions
F/R lunch rate
Student to teacher ratio
Ratio of teachers rated to teachers
Number of teachers rated
Announced ratings (n= 129) 1
Unannounced ratings (n = 110)
0.544**
Final ratings (n = 28)
0.342
0.347
Use of EE for high stakes decisions (n = 108)
0.227*
0.260*
0.058
F/R lunch rate (n = 128)
-0.578**
-0.321**
-0.07
-0.232*
Student to teacher ratio (n = 126)
-0.242**
-0.220*
-0.195
-0.300**
0.261**
Percent of teachers rated (n = 126)
0.201*
0.135
0.111
-0.064
-0.256**
-0.021
Number of teachers rated (n = 384)
0.13
0.09
0.08
0.013
-0.335**
0.008
0.583**
** Correlation is significant at the 0.01 level (2-tailed).
* Correlation is significant at the 0.05 level (2-tailed).

14. What factors were related to teacher ratings
What factors were related to teacher ratings? (Generalized Linear Mixed Models)
Level 1: Teacher-level model:
Practice ratings ijk = π0jk + eijk
Level 2: School-level model:
π0jk = β00k + β01k F/R lunch ratejk + β02k Student/teacher ratiojk + β03k Percent of teachers ratedjk + r0jk
βpjk = γp0k for p = 1 to 3
Level 3: District-level model:
β00k = γ000 + γ001 High stakes use of results + u00
Practice ratings ijk = γ000 + β01k F/R lunch ratejk + β02k Student/teacher ratiojk + β03k Percent of teachers ratedjk + γ001 High-stakes use of results + r0jk + u00 + eijk

15. What factors were related to teacher ratings?- Model Results
12% of Announced and 8% of Unannounced ratings are attributable to district-level factors, while 16% of Announced and 15% of Unannounced ratings are attributable to school-level factors.
School F/R Lunch Rate was the only factor that independently explained Teacher Ratings.
School F/R Lunch Rate explained 9% (Announced) and 7% (Unannounced) of the school rating variance and 60% (Announced) and 75% (Unannounced) of the district variance.
For Every 10% higher school F/R Lunch Rate, Announced teacher ratings are predicted to go down .048 (0.19 standard deviations) and Unannounced ratings .039 (0.14 standard deviations).
Estimate
Robust Std. Error df t
Sig.
School F/R Lunch (Unannounced)
11.395
-4.36
0.001
School F/R Lunch (Announced)
33.897
-5.298
<.0001

16. Scatter Plot of teacher ratings by School F/R Lunch Rates

17. F/R Lunch Rates as a predictor of component ratings
Modeling the individual components shows that F/R lunch predicts teacher ratings across all of the components.
This may suggest that the relationship with F/R lunch may be partly due to teacher selection.
Averaged Coefficients
Domain 1: Planning and Preparation
Domain 2: Classroom Environment
Domain 3: Instruction
Domain 4: Professional Responsibilities

18. F/R Lunch - Probability of Ratings on 2d (Managing Student Behavior) Compared to Distinguished Rating (Multimodeal Generalized Linear Mixed Model Results)

19. The Relationship of F/R Lunch and Ratings is Difficult to Interpret
In schools and classrooms with higher F/R Lunch rates…
lower achieving students and students with behavior problems may make it more difficult for teachers to demonstrate the higher order skills necessary to be rated as Distinguished.
students are taught by less experienced teachers. More experienced teachers choose to work in more affluent schools.
they are less likely to use ratings for high stakes decisions. Using ratings for high- stakes may put pressure on evaluators to inflate ratings.
teachers have more crowded classrooms.
teachers may be quicker to burn out, be less motivated, and therefore less effective.

20. Summary and Conclusions
Educators reported understanding both the evaluation process and the Framework for Teaching, that it was fair, that the process would likely empower teachers to better understand their instructional skills, and that it will lead to improved teaching in Wisconsin.
The largest concern expressed by educators was that the process is time consuming and that implementing it may leave little time for principals to fulfill all their other duties.
The relationship of teacher ratings with factors exogenous to the quality of teaching suggest that, until we understand these relationships more fully, schools should not use ratings to compare teachers.

21. Limitations and Future Work
The results presented here were for a small sample of the state. It is also not known how teachers were selected to participate in the pilot. It is not known if these results will hold up when more evaluators are certified and the analyses are based on final ratings for the whole state rather than single observations for a selection of schools.
Lack of available teacher data make many of these findings difficult to interpret. This year, we will be able to use more individual teacher data and perhaps classroom information in our analyses.
It is not clear exactly what explains the relationship of F/R lunch with teacher practice ratings. We will gather more qualitative data about how contexts may influence ratings. These case studies can provide narratives to help understand the quantitative findings.

22. Contact information
If you have any questions about this presentation or the evaluation please contact:
Curtis Jones