1. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Correlation Data 07/28/14

2. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Multiple Measures Principal Effectiveness Ratings as Defined in Act 82 For the overall principal effectiveness rating, we know that… –50% will comprise the observation/practice piece (Framework for Leadership). –15% will be derived from building level data (School Performance Profile – SPP). –15% will be determined by the relative strength of conversation regarding the connectedness between average teacher observation/practice ratings and teacher-level measures. –20% will incorporate Elective Data / SLOs for principals. (note: for 2014-15, elective data is optional) 2

3. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Elective Data/SLOs District Designed Measures and Examinations Nationally Recognized Standardized Tests Industry Certification Examinations Student Projects Pursuant to Local Requirements Student Portfolios Pursuant to Local Requirements Principal Effectiveness System in Act 82 of 2012 Correlation Data/Relationship Based on Teacher Level Measures Building Level Data/School Performance Profile Indicators of Academic Achievement Indicators of Closing the Achievement Gap, All Students Indicators of Closing the Achievement Gap, Historically Underperforming Students Academic Growth PVAAS Other Academic Indicators Extra Credit for Advanced Achievement Observation/ Practice Framework for Leadership Domains Strategic/Cultural Leadership Systems Leadership Leadership for Learning Professional and Community Leadership June 26, 2014 3

4. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education It was recommended that we adopt a process that focuses on the conversation between the supervising administrator and the principal / school leader based on the principal’s knowledge, understanding and intended use of the evidence / data presented. The rating for correlation data is based on a qualitative assessment by the supervising administrator of the level of understanding (0, 1, 2, or 3) of the quantitative analysis conducted by the principal / school leader. Correlation Data Based Upon Teacher Level Measures 4

5. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education To aid the supervising administrator in assigning a correlation rating (0, 1, 2, or 3), a Performance Level Descriptor Chart has been developed. This chart will provide guidance for rating the conversation between the supervising administrator and the principal / school leader regarding the connectedness between Teacher Level Measures and Observation and Practice (Framework for Teaching) ratings. Teacher Level Measures “shall include, but not be limited to any combination of one or more of the following data” Building Level Data / SPP Teacher Specific Data (PVAAS, etc.) Elective Data / Teacher SLOs Correlation Data Based Upon Teacher Level Measures 5

6. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Correlation Data Performance Level Descriptor Chart The conversation will be measured based upon the principal /school leader’s understanding of these aspects: –Degree of understanding of evidence presented regarding the relationship between teacher-level measures and teacher observation and practice ratings. –Quality of explanation provided for observed relationships between teacher-level measures and teacher observation and practice ratings. –Plans for how the data will be used to support school and LEA goals. 6

7. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Correlation Data Performance Level Descriptor Chart 7

8. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Correlation Data Looking at “Connectedness” The conversation will be based upon the connectedness between teacher level measures and observation/practice ratings. The following charts (displayed on subsequent slides) may be used to determine the relative strength of the connectedness between the teacher level measures and observation/practice (Framework for Teaching) ratings. Note: The ranges within the charts are based upon the conversion tables within PA 82-1 (Teacher Rating Tool). 8

9. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Possible Levels of Connectedness Between Average Teacher Observation/Practice Ratings and Teacher Level Measures Data (SPP) 9

10. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Possible Levels of Connectedness Between Average Teacher Observation/Practice Ratings and Teacher Level Measures Data (PVAAS) 10

11. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Possible Levels of Connectedness Between Average Teacher Observation/Practice Ratings and Teacher Level Measures Data (Elective Data / SLO) 11

12. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Guidance for Determining Correlation A Process for Exploring Connectedness To assist supervising administrators and principals / school leaders to engage in meaningful conversations, a guidance document has been developed that outlines a process for analyzing Teacher Level Measures Data and Observation and Practice (Framework for Teaching) ratings. Step # Definition of Performance Level Indicator Description 1 Degree of understanding Organize your teacher observation/practice data, demographic data, teacher level measure data, and any other data deemed relevant (i.e. format an Excel Spreadsheet). 2 Degree of understanding Locate the School Performance Profile (SPP) for your school using the School Performance Profile website. 3 Degree of understanding Determine the average observation/practice rating for all your teachers within your building. 12

13. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Guidance for Determining Correlation A Process for Exploring Connectedness Step # Definition of Performance Level Indicator Description 4 Degree of understanding Use the Connectedness with SPP chart to help determine a possible level of connectedness between the average observation/practice rating for all teachers in your building and the teacher-level measure for the School Performance Profile (SPP)**. 5 Degree of understanding Repeat steps #3 and #4 for subsets of teachers in your building. 6 Quality of explanation provided for observed relationships Compare possible levels of connectedness between teacher observation/practice ratings and SPP scores for all the teachers (step #4) and the various subsets (step #5). Is there evidence indicating differences in possible levels of connectedness when comparing the average of all teacher observation/practice ratings and SPP versus the subset(s) of teacher observation/practice ratings and SPP ? 7 Plans for how the data will be used to support school and LEA goals Identify possible opportunities where your analysis may help inform comprehensive planning, principal SLOs, etc. 8 Degree, Quality, and Plan Based upon the conversation, the supervising administrator will assign a rating (0, 1, 2 or 3) relative to the principal/school leader’s knowledge, understanding and intended application of evidence/data presented. 9 Degree, Quality, and Plan The correlation rating is then entered into the Principal Rating Tool. 13

14. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #1) Organize your teacher observation/practice data, demographic data, teacher level measure data, and any other data deemed relevant (i.e. format an Excel Spreadsheet). Provided below is a screen shot of the Building Summary of Teacher Ratings spreadsheet that is provided by PDE to assist in collecting data. 14

15. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #2) Locate the School Performance Profile (SPP) for your school using the School Performance Profile website http://paschoolperformance.org/. –Find the Academic Performance Tab to display the Academic Performance Data for your school. Then click on the View Calculation link located at the bottom of the page for the details. –For ABC High School, we have developed SPP Simulation #1 for use in this training. The SPP score we will use is 54.1. 15

16. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education The SPP Score is 54.1 16

17. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #3) Determine the average observation/practice rating for all your teachers within your building. –Note that if you use the Building Summary of Teacher Ratings spreadsheet, then the average is calculated for you. For our example, average for ABC High School is 2.16. The average teacher observation /practice rating is 2.16 17

18. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #4) Use the Connectedness with SPP chart to help determine a possible level of connectedness between the average observation/practice rating for all teachers in your building and the teacher-level measures data (SPP - School Performance Profile). –On the Connectedness with SPP chart, identify the range where your average teacher observation/practice rating falls. In our example, the 2.16 average teacher observation/practice rating falls within the 1.50 – 2.49 range. –Using the average teacher observation/practice range identified above, identify the range where your SPP score falls on the Connectedness with SPP chart. In our example, the 54.1 SPP score falls within the below 60 range. –Using the SPP range identified above, determine a possible level of connectedness. In our example, we can explore possible sources of evidence that may indicate a limited level of connectedness between the average teacher observation/practice rating and SPP. 18

19. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education 19

20. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #5) Repeat steps #3 and #4 for subsets of teachers in your building. –Locally, the determination is made as to which subset(s) will be meaningful to analyze. This decision should made in such a way that you can conduct a meaningful analysis as to any trends that may be occurring relative to the ratings of subsets of teachers when compared to all teachers. For our ABC High School example, we are going to analyze subsets of those teachers that have associated PVAAS scores. –To do this, we need to conduct three independent sorts of our sample Building Summary of Teacher Ratings spreadsheet (sort by ELA-Literature, Math- Algebra I, and Science-Biology teachers that have associated PVAAS scores). 20

21. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #5) By using Excel and manipulating the data from our sample Building Summary of Teacher Ratings spreadsheet, we can determine the following: –The average teacher observation/practice rating is 2.10 with an average PVAAS score of 62.00 for ELA – Literature teachers. Average teacher rating for ELA- Literature teachers with associated PVAAS reading scores Average PVAAS reading score 21

22. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education The average teacher observation/practice rating is 2.53 with an average PVAAS score of 50.00 for Math – Algebra I. The average teacher observation/practice rating is 1.90 with an average PVAAS score of 62.00 for Science-Biology. Using the Guidance Document Example: ABC High School (Step #5) Average teacher rating for Math – Algebra I teachers with associated PVAAS math scores Average PVAAS math score Average teacher rating for Science-Biology teachers with associated PVAAS science scores Average PVAAS science score 22

23. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #6 - SPP) Compare possible levels of connectedness between teacher observation/practice ratings and SPP scores for all the teachers (step #4) and the various subsets (step #5). –Referring back to the Connectedness with SPP chart and using our SPP score of 54.1, we can determine possible levels of connectedness between teacher observation/practice ratings and SPP. 23

24. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education 24

25. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #6 - Comparison) Is there evidence indicating differences in possible levels of connectedness when comparing the average of all teacher observation/practice ratings and SPP versus the subset(s) of teacher observation/practice ratings and SPP? –In our example, it appears that for our PVAAS Math – Algebra I subset we may have a possible poor level of connectedness with SPP versus a possible limited level of connectedness for the other subsets. –Let’s now consider possible levels of connectedness among teacher observation/practice ratings (all teachers/subsets) and other teacher-level measures (PVAAS, Elective Data) to see if this helps to explain these differences. 25

26. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education 26

27. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #6 – Plausible Explanations) Based upon the results, we see varying levels of possible connectedness within the subsets when considering the possible connectedness with PVAAS. Now we need to look for plausible explanations. Possible guiding questions to assist in this analysis include: What trends are you discovering? Are these the types of relationships you would expect to see? Why or why not? Which trends are the most positive and/or the most problematic? And why? What evidence would define plausible root causes? How does data inform your thinking about how to best support school and/or district goals? 27

28. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #6 – Plausible Explanations) We found a possible poor connectedness level between the Math – Algebra I subset of average teacher observation/practice ratings and SPP, as well as with PVAAS. The ratings of the teachers could in fact be accurate, but they are teaching a poorly aligned curriculum to the standards as demonstrated by the lack of static achievement (Keystone) at 32.1% advanced or proficient and growth (PVAAS) at 50.0. The ratings of the teachers may not be accurate due to various factors including personal relationships, strong personalities, and experience. 28

29. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #6 – Plausible Explanations) We found a possible poor connectedness level between the Math – Algebra I subset of average teacher observation/practice ratings and SPP, as well as with PVAAS (continued)… When looking at the SPP calculations for Other Academic Indicators, we see an attendance rate of 90.12. This indicates that students are regularly attending school; however, academic achievement is lacking, especially in math. Perhaps teachers are rated highly based on observations of classrooms where students are engaged; however, the content being delivered is not aligned to the PA Core Standards and/or inadequate pedagogies are being used? Other plausible explanations … 29

30. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #6 – Plausible Explanations) We found a possible limited connectedness level between the ELA- Literature subset of average teacher observation/practice ratings and SPP, but a possible good connectedness level between the ELA-Literature subset and PVAAS scores. The possible good connectedness level between the ELA – Literature average teacher observation/practice ratings and PVAAS could be due to the curriculum being aligned to the standards, resulting in a stronger link between what is tested and what is taught. The ELA department has two teachers with two or fewer years of experience. These two teachers also have the highest PVAAS scores. This could be due to their more recent exposure to the standards and instructional practices designed to improve student achievement (such as student engagement). Also note that both of these teachers have been rated as quality instructors, which aligns to their student’s growth on PVAAS. 30

31. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #6 – Plausible Explanations) We found a possible limited connectedness level between the ELA- Literature subset of average teacher observation/practice ratings and SPP, but a possible good connectedness level between the ELA-Literature subset and PVAAS scores (continued)… Five of the eight ELA-Literature teachers have PVAAS scores of 61 or lower, which could support the assumption that the curriculum is not aligned and/or not being taught. Or research-based effective pedagogies are not being used in the classroom to improve instruction and thus student achievement. Other plausible explanations… 31

32. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #7) Identify possible opportunities where your analysis may help inform comprehensive planning, principal SLOs, etc. The PVAAS Math – Algebra I subgroup has been identified with a possible poor connectedness level with SPP and PVAAS. Therefore, it might be important to include the PVAAS Math – Algebra I subgroup in the development of strategic interventions to increase student achievement. This could be accomplished through a Principal SLO and/or Teacher SLOs for this subset. The PVAAS ELA-Literature subset was identified with a possible limited connectedness level with SPP and a possible good connectedness level with PVAAS. However, the PVAAS score is close to the lower margin in the “good” range. Therefore, opportunities could exist to work with these teachers in the analysis of the data and develop strategic interventions to increase student achievement. Perhaps reviewing the pedagogies of the two newest teachers would be beneficial since they have the two highest PVAAS scores? 32

33. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #8) Based upon the conversation, the supervising administrator will assign a rating (0, 1, 2 or 3) relative to the principal/school leader’s knowledge, understanding and intended application of evidence/data presented. Referring back to the Performance Level Descriptor Chart, the supervising administrator (in collaboration with the principal), let’s consider the following for our example: –Degree  The principal / school leader demonstrates the ability to disaggregate teacher observation/practice ratings and teacher-level measures, as well as conducts an analysis to determine plausible connections among the data. 33

34. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #8) –Quality  The principal / school leader cites plausible causes for the connections among teacher observation/practice ratings and teacher-level measures. Also the principal / school leader articulates why the plausible connections may have occurred among teacher observation/practice ratings and teacher-level measures. –Plan  General plans for increasing student performance based upon the analysis of teacher observation/practice ratings and teacher-level measures have been provided. Opportunities exist to further define timelines and implementation strategies for the incorporation into a Principal SLO. Therefore, the correlation rating for this principal / school leader is a “2” based upon a general action plan as opposed to a plan that includes specificity. 34

35. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education Using the Guidance Document Example: ABC High School (Step #9) The correlation rating is then entered into the Principal Rating Tool. Enter 0, 1, 2, or 3 here Select the “Correlation Data” tab on the Principal Rating Tool 35

36. www.education.state.pa.us Measuring Principal Effectiveness Tom Corbett, Governor ▪ Carolyn C. Dumaresq, Acting Secretary of Education The mission of the Pennsylvania Department of Education is to lead and serve the educational community, to enable each individual to grow into an inspired, productive, fulfilled lifelong learner. 36