1. March 8, 2011 Student Outcomes and Principal Evaluation: Key Questions for PEAC Principal Evaluation Subcommittee

2. Overview of the Webinar 1.Review of guiding questions for sub-committee consideration 2.Introduction and review of value-added measures and update on value-added models being created in CPS 3.Discussion of guiding questions New Leaders for New Schools 1

3. Overview of the Webinar 1.Review of guiding questions for sub-committee consideration 2.Introduction and review of value-added measures and update on value-added models being created in CPS 3.Discussion of guiding questions New Leaders for New Schools 2

4. Guiding questions on student outcomes What measures should be used in evaluating principals? What is the right balance between value-added growth measures and attainment measures? How, if at all, should be adjust our judgments based on a schools demographics and other characteristics, like student mobility? How many years of data should be used for any single years rating on student growth? What processes and parameters should guide local flexibility and adaptation to the state system over time? For each of these categories, we identify specific questions (noted in bold) and considerations based on research and our experience (noted in italics). New Leaders for New Schools 3

5. Measures of Student Outcomes: K-8 Should we use ISAT data? Better matched to principal evaluation than teacher evaluation: –Larger pool of students in growth analyses allows for less variability in direction of results –Clearer attribution of students to principal (with clear mobility parameters) –Serves as one important element of student outcomes piece, but helpful if balanced with non-test outcomes (in high school) or other assessment data (as long as it is consistent across the LEA) –Important to use multiple years of information to establish trend Can be used to measure attainment (e.g., % of kids meeting proficiency), gain/growth (e.g., increase in % of kids meeting proficiency), and value-added Should we use interim assessments? Technically sound but some cautions -More reliable than summative tests if computer adaptive -Assessments may not cover all content -Students may not take interim assessments seriously -Such assessments not meant for use as accountability tools From 2014-15, the PARCC assessments should provide an integrated solution to interim and summative assessments New Leaders for New Schools 4

6. Measures of Student Outcomes: K-8 (continued) Should we use school-level student outcome goals set by principals and their managers? Common practice, but depends on rigor of principal manager expectations What other measures of student growth, beyond state tests, should we consider? Measures of student aspirations toward college in middle school grades Student attendance New Leaders for New Schools: 5

7. Measures of Student Outcomes: High School Considerations Should we use PSAE data? Can be used to assess subjects beyond reading and math (i.e., writing, science) Can be used as an attainment measure (% of students reaching proficiency) and as a growth (increase in % of students reaching proficiency) Substantial technical issues in converting these data to value-added estimates -Gap between 8 th grade ISAT and 11 th grade PSAE, with data distortion from dropouts and retained students -Anticipate improved ability to make value-add estimates using PARCC assessments in 2014-15 and onward What other measures of student growth, beyond state tests, should we consider? High school student growth measures should expand beyond state tests to include on track to college measures: – Student attendance – Grade to grade progression – Credit accumulation (potentially including quality of credits) – Cohort graduation rates, and quality of diploma earned (if data exists) – Note: These measures can be turned into value added metrics, by looking at predicted values versus actual values at the school level New Leaders for New Schools: 6

8. Balancing attainment, growth, and value-add How should we weight attainment, growth and value-add within an overall rating? Focusing on more on movement measures (i.e., gain/growth, value-add) -Provides a better picture of the impact of the principal -Creates stronger incentives for principals to work in lower performing schools -Pushes schools with higher performing incoming students to keep advancing their performance (past proficiency to college-ready) -Values all students by assessing progress from their starting points -Requires districts to look at same-student comparisons rather than cohort to cohort comparisons whenever possible Where possible, use multiple growth measures Relative weight on attainment (or on maintenance of growth) might increase as performance level of school increases Should we treat low-performing schools and high-performing schools differently or the same? There is a ceiling to growth on proficiency, suggesting two changes for high-performing schools: -Give schools gain/growth points if they exceed a proficiency ceiling (e.g. Chicago approach) -Tie a portion of the gain/growth goal to their success in increasing the percent of students meeting the advanced category on current assessments New Leaders for New Schools: 7

9. Balancing attainment, growth, and value-add: An illustration New Leaders for New Schools: 8 Reward principals for maintaining high levels of achievement Shift to growing the percentage of students reaching advanced Emphasis on measures of growth

10. DCPS Principal Evaluation Components Evaluation Component % allocatedTotal Professional Competencies 30% Leadership Framework Assessments 30% Student Outcomes 50% Value-Added Measure 20% School Specific Goals 10% DC CAS Gains Goals 20% Other 20% Special Education Compliance 10% Teacher Retention 5% Family Engagement 5%

11. New York City Principal Evaluation Components Evaluation Component% allocated Schools Graded Progress Components of the Grade: –Student growth measures make up 60% –Absolute performance 25% –School Climate 15% 32% School Specific Goals31% Compliance with District Mandates 15% School Quality Review*22% *In New York City, a School Quality Review is a two- or three-day visit by experienced educators to a school. The visit typically includes classroom observations, conversations with school leaders and stakeholders, and examinations of student work. New York City has developed a rubric to guide the visits and to determine how well organized a school is to educate its students. 40-50% of the New York Evaluation is made up of Student Outcome data 26% of a schools graded progress is focused on student outcomes 14-24% of the School Specific Goals are focused on student outcomes

12. Chicago Performance Calculators for Principals Elementary SchoolHigh School Attainment43%36% Gain/Growth43%64% Value-Added14%0% Attainment (Status)ISAT targets in reading, math, science, composite and highest grade Attendance target Targets for: ACT average One-year dropout and freshmen on track Attendance PSAE Reading/Math/Science Gain/Growth (Trend)Growth in ISAT in reading, math, science, composite and highest grade Growth in attendance Growth in: ACT average One-year dropout and freshmen on track Attendance PSAE Reading/Math/Science AP enrollment and success Reading and Math scores from Explore/Plan/ACT sequence Value-Added (Growth)Reading and math (ISAT) New Leaders for New Schools: 11

13. Adjusting for student characteristics Should we include controls in the value-added growth models to account for student characteristics? Increases the accuracy of value-added estimates -Controls can be changed from year to year to alter the approach to a given population (e.g., special education, English language proficiency, homelessness) There may be some value in excluding some controls – at the sake of maximal accuracy of estimates – in order to signal heightened responsibility for schools to accelerate achievement for low income students of color. Should we give extra weight for improving results for students who start out further behind? Set targets that expect faster growth for lower performing students in the district/state How should we address the question of student mobility? VARC and others use methods that assign portions of value-added growth to a school based on the percentage of the school year a student has been enrolled at the school. New Leaders for New Schools: 12

14. Years of data used for judgments of principals How many years of data should be used for any single years rating on student growth? Given the variation in single-year results, evaluate student outcomes based on multi-year trends -Note: Value-added estimates are more reliable at the school-level than at the classroom level, since higher student numbers reduce the impact of year-to-year fluctuations. BUT, we want to create incentives for long-term improvement, not quick fixes. Provide additional time or use more years of data for early tenure principals Plan for the availability of sufficient data before any significant consequences (e.g. ensuring most recent test data is available before making spring retention decisions) New Leaders for New Schools: 13

15. Processes for adaptation What guidelines do we put in place for all districts to follow if they want to design their own systems? The balance of growth and attainment should be fixed. Measuring success in other academic subjects depends on the presence of reliable local assessments. The technical capability to develop and implement value-added models is not present in most districts. What should be the ongoing process for evaluating the system and adapting it? Among other things, the state will need to adjust its test measures when the PARCC assessments are rolled out in 2014-15 New Leaders for New Schools: 14

16. Overview of the Webinar 1.Review of guiding questions for sub-committee consideration 2.Introduction and review of value-added measures and update on value-added models being created in CPS 3.Discussion of guiding questions New Leaders for New Schools: 15

17. Common Approaches to Measuring Student Success New Leaders for New Schools: 16 Source: VARC (http://varc.wceruw.org/tutorials/Oak/index.htm) Our overall goal is to measure the performance of a principal based on student performance. How is this accomplished?

18. Understanding Value-Added Measures Stephen Ponisciak Value-Added Research Center School of Education, University of Wisconsin- Madison New Leaders for New Schools: 17

19. Overview of the Webinar 1.Review of guiding questions for sub-committee consideration 2.Introduction and review of value-added measures and update on value-added models being created in CPS 3.Discussion of guiding questions New Leaders for New Schools: 18

20. Guiding questions on student outcomes What measures should be used in evaluating principals? What is the right balance between value-added growth measures and attainment measures? How, if at all, should be adjust our judgments based on a schools demographics and other characteristics, like student mobility? How many years of data should be used for any single years rating on student growth? What processes and parameters should guide local flexibility and adaptation to the state system over time? New Leaders for New Schools: 19