MNPS Academic Performance Framework 2011 - 2013 Paul Changas, Alan Coverstone and Christine Stenson Metropolitan Nashville Public Schools (MNPS) October 29, 2013

The MNPS Framework Presentation Overview Why the need for a Framework and how it will be used? Our process in building a Framework Components of our Framework – indicators and measures Next steps Lessons learned that might be helpful to other districts A framework template for other districts/schools

The MNPS Framework Possible Framework Uses To support efforts to raise student achievement To support the district’s accountability status To offer standardized accountability metrics to complement increased school-level autonomy To inform — but not determine — decisions regarding rewards, supports, and resource allocation for schools To provide school communities with a transparent set of indicators to understand school performance

Why the Need for a Framework Why the Need for a Framework? (Why Not Rely on State Accountability Results?) State absolute accountability system is primarily focused on districts rather than schools Key school relative accountability results reported every three years rather than annually State accountability is determined entirely by test scores – there are no school culture measures There is value in an overall performance index While a single number cannot tell the whole story, a “bottom line” based upon multiple measures is needed for making decisions Transparency – outline exactly what measures will be included and how they will be weighted Manageability – quantity of data can be overwhelming 3 M’s of data use: make it Manageable make it Meaningful make it Matter

State Accountability Flowchart

The MNPS Framework Indicators and Weighting

What Factors Should be Included in School Evaluation What Factors Should be Included in School Evaluation? Audience Participation Participants pair up Identify 3-5 key factors or indicators that you would include in a school performance system (academic, non- academic, school culture, etc.) How would you prioritize these factors (or should they be equally weighted)? Discuss for 5 minutes and then we will report out

The MNPS Framework K-8 Measures and Weighting

The MNPS Framework High School Measures and Weighting

The MNPS Framework The Academic Performance Scale Identify key performance measures Determine the weight of each measure Determine the performance scale of each measure Assign performance points to each school based upon position on performance scale

The MNPS Framework The Academic Performance Scale Determine the performance scale of each measure

The MNPS Framework School Ratings for Growth Two measures (50% of Total Framework): Value Added and Mean Achievement Level Increase K-8 subjects: Math, Reading/LA, & Science HS subjects: Algebra I and II, English I, II, II, & Biology I Not highly correlated to socioeconomic status

The MNPS Framework Mean Achievement Level Increase School A School B

The MNPS Framework Mean Achievement Level Increase Goal Example Establish Proficiency Increase Goals Based Upon Prior-Year Results

The MNPS Framework K-8 Ratings for Achievement & College Readiness Two measures (30% of Total): Percent Proficient/Advanced and percent of 4th/8th grade students projected to score 21 or higher on the ACT Percent Proficient/Advanced includes Math (or Algebra I), Reading/LA, & Science These measures are correlated with socioeconomic status (SES), but schools often break the pattern

The MNPS Framework High School Ratings for Achievement & College Readiness Percent Proficient/Advanced includes Algebra I and II, English I, II, and III, and Biology I ACT Composite score of 21 or higher is required for the Hope Scholarship and is the average of the subject area college readiness benchmarks. These measures are correlated with Socioeconomic status, but schools often break the pattern.

The MNPS Framework Achievement Gap and Survey Data Gap Closure (5% of framework) K-8 subjects: Math, Reading/LA, & Science HS subjects: Algebra I and II, English I, II, II, & Biology I TELL and TRIPOD surveys each count as 5% of framework. Parent survey to be added

The MNPS Framework School Rankings Across Measures

The MNPS Framework Report Format

Charter School Accountability MNPS Commitments Set and hold charter schools accountable to clear, measurable, and attainable academic, financial, and operational performance standards and targets; Close schools that fail to meet performance standards and targets; and Work proactively to identify and establish new, high quality charter schools to serve students who attend schools identified for closure.

Outcome-based Performance Management Performance Contracts (Predictable, enforceable) Accountability-based Interventions (Transparent, balanced, comprehensive) Consistent Communication (Face validity, engagement) District-Charter Collaboration Compact Focus Groups Scorecard Performance Management, Replication, and Closure (PMRC) Grant Performance Frameworks Policies and Contracts MNPS Research and Assessment Technical Development Balanced Measures Broad Applicability

Outcome-based Performance Management How will we use the APF? Publish Annual School Report Cards Shape Renewal Process Shape Recommendations each October

Overall Performance (APF) Academic Performance Mean Achievement Increase TVAAS TCAP (%PA) Achievement Gap School Culture Measures ACT (21+) Overall Performance (APF) Year by year

Renewal Information Projection and Review Level Renewal Application Deadline 5-year Review Year

Outcome-based Performance Management How will we use the APF? Publish Annual School Report Cards Shape Renewal Process Shape Recommendations each October

3-year status Action Timeline Renewal Review Level   NA Simple Renewal (Updated budget, plans, targets) Watch 2-years Full Renewal Review (Full Renewal Application: Renewal Based on Best Interests Standard) 1-year Conditional Renewal Review (Renewal possible but unlikely; significant changes required; may recommend revocation depending on end of year results) Current Cycle No Renewal (May recommend revocation to take affect end of year)

Outcome-based Performance Management How will we use the APF? Publish Annual School Report Cards Shape Renewal Process Shape Recommendations each October

School 2013 Status 3-Year Status Recommended Action KIPP Academy Excelling Simple Renewal Review Lead Academy MS Satisfactory None LEAD Academy HS New Vision Academy Liberty Collegiate Acad 2-years STEM Prep Academy Drexel Prep School Target Notice: Revocation in 2014 Likely Nashville Prep School East End Prep Insuffic Data Cameron College Prep Review Knowledge Academy Achieving 1-year Boy's Prep Notice: Revocation in 2014 Possible Smithson-Craigh Acad Smithson-Craigh MS Closed May 2013

What Decisions Would You Be Comfortable With? Audience Participation Participants pair up What types of decisions would you be comfortable making with a performance framework for your school(s)? How many years of data would you need to make these decisions? What additional information would you want in making decisions about school performance? Discuss for 5 minutes and then we will report out

The MNPS Framework Next Steps Show expanded data by individual year Break out mean achievement level increase, TVAAS, and gap calculations by subject Break out key results by subgroup Provide professional development on the Framework Expand documentation Utilize results in evaluation of initiatives and in identifying schools needing support and resources

The MNPS Framework Lessons Learned Include stakeholders in development Transparency is critical Flexibility is essential as standards, assessments and policies change, but the basic principles we value and include in the Framework should hold up over time Construct the Framework in such a way that the top performance category is within reach of any school, regardless of socioeconomic factors Performance measures can vary significantly from year to year, so multiple years of data should be utilized in critical decisions Presentation format is important

The MNPS Framework Performance Framework Template

The MNPS Framework Decision-making Process for Each Indicator Each Category of Data Requires the Following Decisions: Which Measure Subject Areas Years of Data Unit (e.g. School) Performance Thresholds Weighting/ Points For Example: Value Added • Should Math, Reading/LA, Science be included? • Should Math, Reading/LA, Science be weighted equally and be combined into a single score? • Should there be discrete score categories (e.g. 1-5), or a linear transformation of the scores such that schools earn from 0 to 100 percent of possible points, or should we use a method that takes into account the fact that most schools’ scores are going to be clustered around the average? • Should the categories be discrete or continuous, do we base them on average growth from 2012 to 2013, or take other years of growth into consideration? If we use two years of data, we get a bigger range of possible scores. • Do we set our evaluation based on what has been average in the school district and the state or based on the growth standard (i.e. zero growth is average)? If we use the growth standard, we may fall behind the state. If we use actual average NCE gains, we will be rating schools on a very different basis than the state does in its report card and on the TVAAS site (for grades 3-8 only).

The MNPS Framework Questions?