Using Student Growth Percentiles in Educator Evaluations Oregon Department of Education Educator Effectiveness Summer Assessment Institute 2015

Outcomes for Today Common understanding of SLG goal requirements for 2015-16 Explain the role of Student Growth Percentiles (SGPs) in relation to Category 1 Assessments Answer questions TANYA Because there will be lots of new information… Please jot down questions as they arise Q & A opportunities will be provided throughout today’s presentation PUT UP PARKING LOT

SLG Requirements for 2015-16 8 components including Rationale REQUIRED 8 components including Rationale Minimum of 2 SLG goals each year *Statewide assessments (Category 1) required for teachers and principals in tested grades and subjects Oregon Matrix used for summative evaluation *Pending review of HB 2680 RECOMMENDED Content is focused, not everything that is taught Context can help ascertain instructional needs Tier goals/targets where appropriate Include the support needed to attain the goal Even though ESEA waiver has been approved, we are now waiting on determinations from state level leadership on whether HB suspends the use of statewide assessments for the upcoming school year.

ESEA Waiver Requirement Teachers in tested grades and subjects (Grades 4-8 ELA/Math) and principals must use state assessments as one measure of student learning and growth (SLG) in their evaluation Oregon educators set two SLG goals annually; Category 1 goals are measured by state assessment; Category 2 goals are measured by other assessments Teachers in tested grades and subjects & Principals Category 1 SLG goal Category 2 SLG goal Teachers in non-tested grades and subjects ELA/Math Grades 4-8 As you know, the waiver requires the use of statewide assessments as a measure of SLG goal attainment for teachers and principals in tested grades and subjects for one of the two goals set. In October of 2014, Oregon’s waiver was conditionally approved with the question remaining of how statewide assessments would be used consistently in evaluations across the state.

ESEA Waiver Update Oregon’s waiver approved for 3 years with conditions removed For evaluation purposes statewide assessments will only be used as a measure of SLG goal attainment for those grades that have baseline data Tested grades and subjects are now ELA and Math 4-8 Oregon’s waiver was approved on July 23rd with all conditions removed. In that waiver Oregon indicated that it will only be using the statewide assessment for those grades in which baseline data is available. This means that tested grades and subjects are now grades 4-8. Grades 3 and 11 no longer required.

Two Options Using SGPs ODE co-developed with the OEA, Chalkboard, and COSA two options that meet USED’s waiver requirements. Districts will choose one. In both options, districts will use median Student Growth Percentiles (SGPs) based on the Oregon Growth Model SGPs measure growth for an individual student The median SGP represents the exact middle of students’ SGP scores *Both options apply only to grade 4-8 ELA and math teachers and principals The development of Oregon’s plan for incorporating statewide assessments was a collaborative effort and offers districts two choices. Oregon’s approved waiver involves the use of median Student Growth Percentiles (SPGs) to fulfill the requirement of using state assessments in tested grades and subjects. SGPs measure growth for an individual student by comparing the change in his or her achievement on state assessments (Smarter Balanced) to that of his or her “academic peers” (those having similar historical assessment results) The next section of the presentation will go into greater detail about how SGPs and median SGPs are determined.

Questions? Field questions around SLG requirements and waiver. SGP info coming up next.

Student Growth Percentiles (SGPs) Normative growth measure One year of growth SARAH P SGPs are a normative measure of student growth. SGPs quantify the growth an individual student made in one year relative to all other students in Oregon with similar score history in state assessments. The slides we are going share in this next section are from a presentation created by OSPI in Washington Growth is relative to students with a similar score history

What do we mean by normative growth? What do we mean by normative growth? We are referring to the amount of growth a child makes in comparison to his or her peers. For instance, when a child goes to the doctor’s office, his weight and height are measured. The doctor can tell us if the child’s growth in the last year is “typical”, that is: is he growing the same, less, or more than other children his age?  

Percentiles 35% 65% 35th Often this is expressed in percentile terms. The child’s height may be at the 35th percentile which means that the child is taller than 35 percent of other children his age and shorter than 65 percent of children his age. Normative growth is the basis for student growth percentiles.

Anthony 3rd Grade 2262 Level 1 4th Grade 2398 Level 2 Let’s look at an example. Anthony was a 4th grade student last year. On his Smarter Balanced assessment in the spring, he scored a 2398. In the previous year, his 3rd grade score was 2262. There appears to be “growth” in that his score improved…but relative to what? How do we place Anthony’s growth in context?

Oregon Score Distribution 3rd Grade Reading Level 4 Level 3 Level 2 Let’s start by first examining the test score distribution for all students across the state in Anthony’s cohort. We can see there is a spread of scores, from the “Level 1” range up to “Level 4”. Anthony’s score of 2262 is at the lower end of the distribution, in “Level 1”. 2432 2367 2490 ELA/Literacy Scale Score Thresholds

Grade 4 Grade 3 Level 1 Level 2 Level 3 Level 4 3rd Grade 4th Grade 2262 4th Grade 2398 Now, let’s look at Anthony’s 3rd and 4th grade reading scores in this context. For simplicity, we’ll show a standard curve. We can see that in 3rd grade he scored 2262 (or “Level 1”) and in the following year in 4th grade he scored 2398 (moving to the performance level of “Level 2”).  Although Anthony did not meet standard in 4th grade, his score does show an improvement in performance. However, it remains unclear if this is an average or below average amount of growth.

2398 - 2262 136 points We want to understand how Anthony has improved. To do this we cannot simply subtract his 3rd grade score from his 4th grade score because Oregon state assessments is not vertically scaled, that is they do not combine to form one long yard stick from year to year so we cannot say he grew by XX points.

Anthony’s Comparison Group BRIAN Instead we attempt to define “above” and “below” average growth by placing Anthony’s growth in the context of a comparison group.

All 3rd grade test takers in Oregon Level 1 Level 2 Level 3 Level 4 All 3rd grade test takers in Oregon When Anthony was in 3rd grade, he scored a 2262. Many other 3rd graders in Oregon took the same state test in reading. Anthony was just one student among them. Of all these 3rd graders in the state many others scored a 2262…. …with Anthony’s Prior Score ~2262 Anthony

Anthony’s Comparison Group Level 1 Lev el 2 Level 3 Level 4 Let’s call this “Anthony’s Comparison Group” because these students had a similar level of academic proficiency as Anthony. Anthony’s comparison group is defined solely on their state assessment scores, not on any other characteristics. What we want to know is: relative to his comparison group, how well did Anthony perform in the next year - the 4th grade? Anthony’s Comparison Group Anthony’s prior score ~2262 Anthony

4th Grade Score Distribution Level 1 Level 2 Level 3 Level 4 4th Grade Score Distribution Middle score High score Here we can see the distribution of 4th grade test scores for Anthony’s comparison group. You can see that this is an overall lower performing subgroup of students relative to the statewide distribution *** The highest scoring student met standard but the middle score of Anthony’s Comparison Group is on the border of “Level 1” and “Level 2”. Anthony’s Comparison Group Anthony’s prior score ~2262

4th Grade Score Distribution Level 1 Level 2 Level 3 Level 4 4th Grade Score Distribution 80% of students in Anthony’s comparison group scored below him 2398 Recall that this year Anthony scored 2398. Although Anthony is “Below Standard” in the state distribution, he is definitely above the middle score within his comparison group. Analyzing the data more closely, we can see that Anthony scored higher than 80 percent of students in his comparison group. Anthony’s Comparison Group

4th Grade Score Distribution Level 1 Level 2 Level 3 Level 4 4th Grade Score Distribution SGP = 80 This translates to a Student Growth Percentile (or SGP) of 80. - 2 slow clicks – Anthony’s Comparison Group

Processing Time With the colleagues at your table: Talk about what you’ve heard so far Identify a question your group still has about SGPs (Up Next: Determining Median SGPs)

SARAH M Next, let’s move from an individual child to the teacher level. How can we calculate the median SGP?

Median Student Growth Percentiles Students in Mr. Waters’ class Student SGP Sheryl 3 Hector 22 Robert 36 Miranda 38 Tre 51 Anna 54 Eric 71 Anthony 80 Amina 93 To find the median, we first need a valid class roster. Using that roster, the growth percentiles for all the students in Mr. Waters’ class are listed in numerical order from lowest to highest. In this chart, assuming this was the total enrollment in Mr. Waters class, the median student growth percentile is 51. The median SGP shows that half of the students had growth above that level, and half had growth below. This gives us a median number for that teacher, which we refer to as the “median student growth percentile”. Median is used rather than the average, because an average can be influenced greatly by a few very high or very low scores. The median can give a better overall picture for a teacher.

Making a Determination of Student Growth The Median Student Growth Percentile (SGP) determines the educator’s Category 1 SLG Rating Much Less than Typical Less than Typical More than Category 1 Rating 1 2 3 4 Median SGP 1 to 34 percentile 35 to 49 50 to 64 65 to 99 Interpretation Majority of your students have “low” growth Majority of your students have below “average” growth Majority of your students have “above average” growth Majority of your students have “high” growth With a median SGP of 51, Mr. Waters would receive a score of 3 using the SGP rubric. This number is then used as one of the 2 SLG goal scores that determine his rating on the X axis of the Oregon Matrix. This means that Mr. Waters would only write and score one SLG goal.

X-Axis = SLG Rating Mr. Waters SGP was rated 3 Second SLG was rated 3 SLG performance level based on two goals Two-year cycle select two of four goals Score SLG goals Get a rating between 1 and 4; Use X-Axis thresholds to determine SLG level: 4 = both goals 4s 3 = both goals 3s; one goal 3 & one goal 4; one goal 2 & one 4 2 = both goals 2s; one goal 2 & one 3; one goal 1 & one 3; one goal 4 & one 1 1= both goals 1s; one goal 1 & one 2 Mr. Waters SGP was rated 3 Second SLG was rated 3 X-Axis Rating = Level 3 SLG Rating For the X axis, the determination of overall performance level is based on 2 goals. For those educators on a two year cycle, two of the four goals would be selected With a score of 3 on the SGP rubric, Mr. Waters has one goal rated a 3. The goal he wrote that used a Category 2 measure was also rated a 3. Using the threshold this means his overall SLG rating is a 3.

Mr. Waters: Y-axis = 3 & X-axis = 2 The matrix is a decision-making tool. It is the instrument that bring two inputs (PP/PR and SLG) together to help you make a decision for the final performance level you need to report. Districts use the thresholds established by ODE to determine the rating for each axis. The Y axis represents overall performance related to Professional practice and responsibility and the X axis represents overall performance related to student learning and growth. It is important to note that the matrix is not a uniform percent model, and that by design the Y axis has more influence on the overall score (summative rating) than the X axis. For example, if you score a 4 on the Y-axis (PP/PR) and a 1 on the X-axis (SLG), the matrix shows that you’re still a 3 overall. In a 50/50 percentage model, you would be a 2.5. Going back to our example of Mr. Waters, recall that he had an SGP score of 3, and an SLG score of 3 which made his overall rating on the X axis a 3. Mr. Water’s professional practice and responsibility rating on the Y axis was a 3. You can see that with a 3 on the Y axis and a 3 on the X axis the overall rating is a 3 which would mean a Collegial path. This means that paths for professional growth are developed jointly by the educator and evaluator. Even if Mr. Waters had scored a 2 overall on the X axis he would still be a 3 overall and on the Collegial path. *Inquiry Process

Two Options; Districts Choose Option A: State Assigned SGPs Option B: Verify SLG Goal This option does not require educators to set a Category 1 SLG goal Student Growth Percentiles (SGPs) are used exclusively to determine the Category 1 SLG rating This option requires educators to set Category 1 SLG goals using Smarter Balanced and rate their goals using the SLG scoring rubric The SLG goal rating is compared with the educator’s median SGP rating to determine the Category 1 SLG rating In Option A educators are provided with a median SGP which is used as a the SLG score for that goal. Best practice would include discussion of context, strategies and professional supports needed to attain your Category 1 goal. In Option B, educators set and score a goal using Smarter Balanced and the state scoring rubric and then compare that score to their median SGP to determine the final score for the SLG.

What can districts do to prepare? Study options with collaborative team and develop district policies related to SGPs Create rosters this fall to verify in the spring Practice calculating Median SGPs using growth data from ODE We know that SGPs will be the method for incorporating Category 1 measures, it is just a question of when. All districts will need to select either Option A or Option B and have policies in place around teacher of record, procedure for roster verification and attendance. Because districts will get student growth data from ODE later this month they can begin the process of creating class rosters and calcuating median SGPs. This will be a valuable exercise even if SGPs are not used until 2016-17.

Remaining questions?

Feedback Please! On an index card: Do you have any feedback that could improve this presentation?

ODE Contacts Educator Effectiveness Team: Tanya Frisendahl tanya.frisendahl@state.or.us Sarah Martin sarah.martin@state.or.us Sarah Phillips sarah.phillips@state.or.us Brian Putnam brian.putnam@state.or.us