Understanding How Evaluations are Calculated Professional Practices, Measures of Student Learning/ Outcomes- Calculating Scores & Translating SLOs/SOOs into an Overall Effectiveness Rating Erin Brophy-Lindo, Director of Educator Effectiveness, HR

What Makes Up the Entire Evaluation? State Model Evaluation System adopted by APS: Part 1: Professional Practice Standards & Elements (Rubric= Standards 1-5) Part 2: Measures of Student Learning (Student Growth= Standard 6) Part 2 A: Individually Attributed Measure Part 2 B: Collectively Attributed Measure Parts 1 + 2 (2a and 2b) = Overall Educator Effectiveness Rating for the year 50% 50% Must include: State Summative Assessment Data, Colorado Growth Model Data (When Available & Other Local Measures

How is the Professional Practices Rating Calculated? Standard APS Weights Standard 1: Content Knowledge 15.00% (30 % of the overall 100% of PP) Standard 2: Learning Environment 10.00% (20% of the overall 100% of the PP) Standard 3: Facilitation of Learning Standard 4: Reflective Practice 7.50% (15% of the overall 100% of the PP) Standard 5: Leadership Total Weight 50% (100% of the PP) In the Model System, an educator earns a professional practice rating based on the accumulation of points on the 27 elements of the model system rubric. The rubric scoring process is designed so that school districts and BOCES have the option of weighting the Standards equally or differentially as allowed by S.B. 10-191. The weights of the Standards or elements by the district emphasizes the initiatives or skills of importance to the district. The weighting of the Standards impacts the overall professional practices rating (Basic, Partially Proficient, Proficient, Accomplished, Exemplary), which in turn impacts the educator's final effectiveness rating (Ineffective, Partially Effective, Effective or Highly Effective). APS weights were determined through joint collaboration between the District and AEA.

How is the Professional Practices Rating Calculated? Each Standard is scored by rating the individual elements associated with that Standard. The score for an element is determined by marking the professional practices observed/ demonstrated by the educator under the rating categories (Basic, Partially Proficient, Proficient, Accomplished and Exemplary.) An overall rating for an element that is rated as Basic= 0 pts, Partially Proficient= 1 pts, Proficient= 2 pts, Accomplished= 3 pts, Exemplary= 4 pts. The overall rating for the Standard is then determined by the number of points earned by the element ratings. The scale for the overall rating is aslo impacted by the weight assigned to the Standard. The formula for calculating an individual standard’s contribution to the overall professional practices rating is: Total Pts Earned for Std. x (Std. Weight x No. of Stds.) divided by No. of Elements in the Std.* *All calculations involved in determining professional practices and effectiveness ratings are carried to three (3) decimal places and rounded to two (2).

How is the Professional Practices Rating Calculated? In this teacher example, the educator was rated proficient in 2 elements, partially proficient in 2 elements, and accomplished in 2 elements. 2pts + 2pts + 1pt + 3pts +3pts + 1pt = 12 pts for this standard. The calculation for determining the weight of this standard in the overall professional practices rating is: 12 x (.30 x 5)/6= 3.00 The 3.00 weighted score is added to the weighted score of all the standards to get the overall professional practices rating (Standards 1-5)

How is the Professional Practices Rating Calculated? The total professional practices weighted points total is determined by applying the weighted calculation formal to the total points earned per Standard and then adding those weighted points together as seen here. Teacher Example Specialized Service Professional Example Quality Standard Total Points Earned by Standard Formula Applied Total Weighted Points Earned Standard 1 12 12 x (.30 x 5)/6 3.00 12 x (.30 x 5)/5 3.60 Standard 2 14 14 x (.20 x 5)6 2.33 14 x (.20 x 5)/5 2.80 Standard 3 17 17 x (.20 x 5)/8 2.13 17 x (.20 x 5)/7 2.43 Standard 4 6 6 x (.15 x 5)/3 1.50 Standard 5 10 10 x (.15 x 5)/4 1.88 10 x (.15 x 5)/5 Total Weighted Points for all Standards 10.84 11.83

How is the Professional Practices Rating Calculated? The overall professional practices rating is determined by applying the total weighted points to the following scoring guide. Teacher Example SSP (Counselor Rubric) Example Total # Of Weighted Pts Earned Rating for # of Weighted Pts Earned Total # of Weighted Pts Earned for this Evaluation 0-2.00 pts Basic 10.84 11.83 2.01-7.00 pts Partially Proficient Overall Professional Practices Rating 7.01-12.00 pts Proficient 12.01-17.00 pts Accomplished 17.01-20.00 pts Exemplary

How is the Professional Practices Rating Calculated? The overall Professional Practices Rating is converted to a point scale using the following formula: Teacher Formula (Weighted Score x Total No. of Elements)= Total Points for the Professional Practices (Standards 1-5) Teacher Example: 10.84 x 27= 293 pts Specialized Service Professionals Formula (Weighted Score x Total No. of Elements)= Total Point SSP (Counselor rubric) Example: 11.83 x 25 = 296 pts Professional Practices Ratings Point Value Per Rating Level Professional Practices Rubric Weighted Point Scale (Rounded to the nearest hundredth) Point Scale When Converted to 0-540 (Rounded to the nearest whole number) Basic 0 to 2 0 to 54 Partially Proficient 1 2.01 to 7 55 to 189 Proficient 2 7.01 to 12 190 to 324 Accomplished 3 12.01 to 17 325 to 459 Exemplary 4 17.01 to 20 460 to 540

How is the Measures of Student Learning Rating Calculated? Local school districts identify the different Measures of Student Learning comprising an educator’s body of evidence for the 50 percent Measures of Student Learning portion of the evaluation. Districts determine the best approach for combining these measures. By assigning weights to each score associated with the multiple measures in educator evaluations, districts are signaling which results or measures in the system are deemed to have more value than others, are better aligned with learning goals, are more appropriate for measuring educator impact or may signal that all results should be weighted equally. The Measures of Student Learning are scaled (e.g., on a zero-three scale), the next step entails assigning weights to each measure and applying an approach to calculate a total score earned by educators on the Measures of Student Learning. Although districts can decide how to weight the scores from each of the multiple measures, districts may want to keep things simple by selecting weighting percentages that sum up to 100 percent.

How is the Measures of Student Learning Rating Calculated? In APS, the chosen measures and weights in 2016-17 for Teachers /TOSAs are: School Performance Framework (SPF) 10% or (20% of the whole MSL)= Collective Measure (The SPF represents the collective attribution, the statewide summative assessment results and the Colorado Growth Model Results) Student Learning Objective (SLO) 40% (or 80% of the whole MSL)= Individual Attribution Total Weight of 50% (or 100% of the whole MSL) Teacher evaluations must include the following measures: A measure of individually attributed student learning outcomes (SLO) A measure of collectively attributed student learning outcomes (SPF) When available, statewide summative assessment results (SPF) For subjects with statewide summative assessment results available in two consecutive grades, results from the Colorado Growth Model (SPF)

How is the Measures of Student Learning Rating Calculated? In APS, the chosen measures and weights in 2016-17 for Specialized Service Professionals are: Two Student Outcome Objectives (SLO) 25% and 25% (or 100% of the whole MSL) Total Weight of 50% (or 100% of the whole MSL) Specialized Service Professional (SSP) evaluations must include the following measures: At least two measures of student outcomes aligned with the role and duties and the individual SSP being evaluated (SOO)

How is the Measures of Student Learning Rating Calculated? In the state model, each measure is awarded points that range from 0 to 3. The Measures of Student Learning are weighted and combined, then converted to a score between 0 and 540. Measures of Student Learning Rating Point Value Per Measure Cut Point Scores Much Less Than Expected 0 to 134 Less Than Expected 1 135 to 269 Expected 2 270 to 404 More Than Expected 3 405 to 540

How is the Measures of Student Learning Rating Calculated? Collective Attribution Rating for Teachers/TOSAs The School Performance Framework rating is the Collective rating for all teachers in the Measures of Student Learning (MSL) section of the Evaluation for APS. This does not apply to SSPs who do not have a collective attribution. SSPs have two Student Outcome Objectives (SOOs) equally weighted to make up the MSL. The SPF rating scale is determined by building level performance according to state expectations. Much Less Than Expected Growth (0 pts) Less Than Expected Growth (1 pts) Expected Growth (2 pts) More Than Expected Growth (3pts) Turnaround Priority Improvement Improvement Performance

How is the Measures of Student Learning Rating Calculated? Individual Attribution Rating for Teachers/ TOSAs The Student Learning Objective (SLO) for APS is rated based on the percent of students successful in meeting the performance targets set by the teacher for students. When developing the SLO, the teacher sets performance targets for 1 or more subsets of students along with the expected performance (success criteria). This(ese) subsets of students’ data is examined to determine the overall rating for the SLO and the end of the instructional interval. Much Less Than Expected Growth (0 pts) Less Than Expected Growth (1 pts) Expected Growth (2 pts) More Than Expected Growth (3pts) % of students meeting the performance target is below 60% % of students meeting the performance target is at or above 60% but less than 75%. % of students meeting the performance target is at or above 75% but less than 91%. % of students meeting the performance target is at or above 91%

How is the Measures of Student Learning Rating Calculated? Combining the Collective and Individual Measures: The district can allocate smaller or higher percentages to each measure but must ensure that the weights assigned across all measures sum up to 1 or 100% as shown in the third column. Each measure is rated and the weighted score calculated. The weighted scores are combined to get an overall weighted score. The calculation for the weighted scored is: (No. of pts for the measure x weight of the measure= weighted score) Example: SPF rated Less Than Expected = 1pts x .20= .20 SLO rated Expected= 2pts x .80= 1.60 Total Weighted Score= 1.80 Measures/ Results from Collective and Individual Attributions Score Earned Weight Assigned Weighted Score School Performance Framework (SPF) 1pts (Less Than Expected Growth) .10 (or .20 of the overall 100% of Standard 6) .10 (.20) Student Learning Objective (SLO) 2 pts (Expected Growth) .40 (or .80 of the overall 100% of Standard 6) .80 (1.60) Sum of Weights .50 (or 1.00 of the overall 100% of Standard 6) .90 (1.80)

How is the Measures of Student Learning Rating Calculated? Combining the Two SSP Measures: The district can allocate smaller or higher percentages to each measure but must ensure that the weights assigned across all measures sum up to 1 or 100% as shown in the third column. Each measure is rated and the weighted score calculated. The weighted scores are combined to get an overall weighted score. The calculation for the weighted scored is: (No. of pts for the measure x weight of the measure= weighted score) Example: SOO #1 rated Less Than Expected = 1pts x .50= .50 SOO #2 rated Expected= 2pts x .50= 1.00 Total Weighted Score= 1.50 Measures/ Results from Collective and Individual Attributions Score Earned Weight Assigned Weighted Score Student Outcome Objective (SOO) #1 1pts (Less Than Expected Growth) .25 (or .50 of the overall 100% of Standard 6) .25 (.50) Student Outcome Objective (SOO) #2 2 pts (Expected Growth) .50 (1.00) Sum of Weights .50 (or 1.00 of the overall 100% of Standard 6) .75 (1.50)

How is the Measures of Student Learning Rating Calculated? The sum of all weighted scores from the teacher example was (1.80) and represents the composite Measures of Student Learning Weighted points earned by the teacher. This table translates the composite point ranges into Measures of Student Learning ratings for a given teacher. (This works the same for Specialized Service Professionals) The weighted point score of (1.80) in our teacher example would be Expected Growth. The weighted point score of (1.50) in our SSP example would also be Expected Growth. Composite Weighted Point Rating Scale Much Less Than Expected Less Than Expected Expected More Than Expected Total Composite Weighted Points (0-3) 0.0 to 0.49 0.50 to 1.49 1.50 to 2.49 2.50 to 3.00

How is the Measures of Student Learning Rating Calculated? How is the weighted score for the Measures of Student Learning translated into a point scale for combining with the Professional Practices points? The weighted score of 1.80 in the teacher example (or 1.50 in the SSP example) is converted to a Measures of Student Learning score between 0 and 540. The Measure of Student Learning score will be added to an educator’s Professional Practices score in order to determine an overall effectiveness rating. Using the teacher example of 1.80 as the weighted average of two measure ratings, convert 1.80 to the 540 scale with the Expected Growth formula: (1.80 – 1.5) x 135 + 270 = 311, which would be the final Measures of Student Learning point for this teacher. For the SSP (1.50 - 1.50) x 135 + 270= 270 Measures of Student Learning Rating Levels Measures of Student Learned Weighted Score Computing a Measures of Student Learning Point Score Much Less Than Expected (0 < score < .5) (score-- .0) x 270 Less Than Expected (.5 < score < 1.5) (score--.5) x 135 + 135 Expected (1.5 < score < 2.5) (score – 1.5) x 135 + 270 More Than Expected (2.5 <= score <= 3.0) (score –2.5) x 270 + 404

How are the Professional Practices (Standards 1-5) combined with the Measures of Student Learning (Standard 6) to get an Overall Effectiveness Rating? Professional Practices Cut Scores Basic 0 to 54 Partially Proficient 55 to 189 Proficient 190 to 324 Accomplished 325 to 459 Exemplary 460 to 540 Measures of Student Learning Cut Scores Much Less Than Expected 0 to 134 Less Than Expected 135 to 269 Expected 270 to 404 More Than Expected 405 to 540 To arrive at the Final Effectiveness score, the Professional Practice score is simply added to the Measures of Student Learning score. Final Effectiveness Rating Cut Scores Ineffective 0 to 188 Partially Effective 189 to 458 Effective 459 to 728 Highly Effective 729 to 1080

How are the Professional Practices (Standards 1-5) combined with the Measures of Student Learning (Standard 6) to get an Overall Effectiveness Rating? In our teacher and SSP examples, the calculation of the final ratings are: Overall Final Effectiveness Rating Calculation Teacher Example SSP Example Professional Practices Points 293 296 Measures of Student Learning Points 311 270 Total Final Points 604 566 Final Effectiveness Rating Effective Final Effectiveness Rating Cut Scores Ineffective 0 to 188 Partially Effective 189 to 458 Effective 459 to 728 Highly Effective 729 to 1080