Pennsylvania Value-Added Assessment System Overview: PVAAS Hatboro-Horsham School District February 15, 2013

Session Essential Questions: During this session, you will begin to form answers to these questions: What is PVAAS? What is a growth model? How is PVAAS data different from PSSA or Keystone data? How will this impact my yearly evaluation? What resources are available if I’d like to learn more?

Measuring Growth, or Progress Students don’t all start the year at the same place academically If we are going to measure the academic progress that students make, we have to consider more than proficiency level. In addition, it’s critical to recognize that students don’t all start the year at the same place academically. PVAAS + Other Data Consultation 2012 PVAAS Statewide Team for PDE June 2012

Measuring Growth or Progress 4/15/2017 Measuring Growth or Progress Standard for PA Academic Growth Regardless of their entering achievement level, students should not lose ground academically Is it not a reasonable goal to maintain the achievement level of a group of students? Not necessarily increase their achievement level, but maintain their achievement level. In Pennsylvania, growth is assessed against the Standard for PA Academic Growth. It is based on the philosophy that regardless of the entering achievement level of a group of students, they should not lose ground academically. So we ask, is it not a reasonable goal to maintain the achievement level of a group of students? Is that not a reasonable expectation? In other words, if students are fairly high achieving, is it not reasonable to expect that we, as educators, keep them fairly high achieving? And if students are low achieving, is it not reasonable to expect that we would at least keep those students at that level of achievement (and not have them slip further behind)? We would hope that everyone would agree that this is a reasonable expectation. PVAAS + Other Data Consultation 2012

PVAAS Growth Measure & Color Indicator 4/15/2017 PVAAS Growth Measure & Color Indicator PVAAS yields a Growth Measure AND a Growth Color Indicator. The Growth Measure represents the amount of growth a group of students has made. The Growth Color Indicator represents the level of evidence we have in the data to say whether or not the group of students increased, decreased, or maintained their achievement levels on average. PVAAS Value Added reports yield a Growth Measure and a Growth Color Indicator. [click] The Growth Measure represents the amount of growth a group of students has made. [click] The Growth Color Indicator represents the level of evidence we have in the data to say whether or not the group of students increased, decreased, or maintained their achievement levels on average. In other words, the color tells us how sure we are that the group of students increased, decreased, or maintained their achievement level on average. PVAAS + Other Data Consultation 2012

Concept of Measuring Growth in PVAAS – Grades 4-8 Math & Reading 4/15/2017 Concept of Measuring Growth in PVAAS – Grades 4-8 Math & Reading Prior Achievement for Group Most Recent School Year for Group Higher Achievement When the PSSA is administered in consecutive grade levels as it if in reading and math for grades 3-8, PVAAS uses a growth standard method to measure the growth or progress of a group of students. Let’s consider this group of students. Each year, these students are tested with the PSSA. Each dot represents a student in the group. Those students are going to score along the achievement spectrum from low to high achievement. To estimate the achievement level of this group of students, [click] PVAAS uses all of the student level data from all years, from all grades, in both Reading and Math from each year this group of students was tested: [click] to estimate the average achievement level of the group. [click] After the next round of testing, PVAAS uses all the prior data for the group [click] and adds the data from the most recent round of testing [click] to get a new estimate of the average achievement level of the group. Now we can compare the prior achievement level of the group to the new achievement level of the group [CLICK If the evidence indicates the achievement is roughly at the same point within the distribution of scores as the previous year, this would indicate that the group of students maintained their achievement level which in PVAAS would be indicated as Green – evidence that the student group met the standard for PA Academic Growth. [click] If there is moderate evidence that the average achievement level of this group of students was at a higher point within the distribution, this would indicate that the group of students increased their average achievement level indicated with a Light Blue in PVAAS – moderate evidence that the student group exceeded the standard for PA Academic Growth. [click] Similarly if there is significant evidence that the average achievement level of this group of students was at a higher point within the distribution, this would indicate that the group of students increased their average achievement level indicated with a Dark Blue in PVAAS – significant evidence that the student group exceeded the standard for PA Academic Growth. [click] Now let’s consider now what it would look like if the there was moderate evidence that the average achievement level of this group of students was at a lower point within the distribution, this would indicate that the average achievement level of the group of students slipped indicated with a Yellow – moderate evidence that the student group did NOT meet the standard for PA Academic Growth. [click] Similarly if there was significant evidence that the average achievement level of this group of students was at a lower point within the distribution, this would indicate that the average achievement level of the group of students slipped indicated with a Red – significant evidence that the student group did NOT meet the standard for PA Academic Growth. Higher Achievement Lower Achievement Lower Achievement - Represents the mean achievement level of the group of students PVAAS + Other Data Consultation 2012

PVAAS + Other Data Consultation 2012 4/15/2017 Concept of Measuring Growth in PVAAS Science, Writing, Grades 9-11 Math & Reading Prior Achievement When the PSSA is not administered in consecutive grade levels as in science, writing, or grades 9-11 math and reading, PVAAS uses a predictive method to measure the growth or progress of a group of students. Let’s consider this group of students who were last tested in 8th grade. Each dot represents a student in the group. Those students are going to score along the achievement spectrum from low to high achievement. [click] PVAAS uses all of the student’s prior data to predict the scaled score or achievement level for a student on the 11th grade PSSA. [click] Then the predicted scaled scores for all students are used [click] to calculate the mean predicted achievement level for the group. [click] Next, these students were tested again with the PSSA in grade 11 again the students will score along the achievement spectrum from low to high achievement. [click] The mean scaled score or achievement level is calculated for the group. [CLICK If the evidence indicates the achievement is roughly at the level that what was predicted, this would indicate that the group of students maintained their achievement level which in PVAAS would be indicated as Green – evidence that the student group met the standard for PA Academic Growth. [click] If there is moderate evidence that the average achievement level of this group of students was higher than predicted, this would indicate that the group of students increased their average achievement level indicated with a Light Blue in PVAAS – moderate evidence that the student group exceeded the standard for PA Academic Growth. [click] Similarly if there is significant evidence that the average achievement level of this group of students was higher than predicted, this would indicate that the group of students increased their average achievement level indicated with a Dark Blue in PVAAS – significant evidence that the student group exceeded the standard for PA Academic Growth. [click] Now let’s consider now what it would look like if the there was moderate evidence that the average achievement level of this group of students was lower than predicted, this would indicate that the average achievement level of the group of students slipped indicated with a Yellow – moderate evidence that the student group did NOT meet the standard for PA Academic Growth. [click] Similarly if there was significant evidence that the average achievement level of this group of students was lower than predicted, this would indicate that the average achievement level of the group of students slipped indicated with a Red – significant evidence that the student group did NOT meet the standard for PA Academic Growth. Higher Achievement Mean Predicted Achievement Predicted Achievement Lower Achievement - Represents the mean predicted achievement level of the group of students - Represents the mean achievement level of the group of students PVAAS + Other Data Consultation 2012

Schools with Lower Achievement 4/15/2017 Schools with Lower Achievement Is a “green” in PVAAS sufficient for moving students to higher levels of achievement? Prior Achievement for Group Most Recent School Year for Group Higher Achievement Higher Achievement Let’s now consider how this concept applies for a school where the students are lower achieving. [click] Remember, to estimate the achievement level of this group of students, PVAAS uses all of the student level data from all years, from all grades, in both Reading and Math from each year this group of students was tested: [click] to estimate the average achievement level of the group. [click] After the next round of testing, PVAAS uses all the prior data for the group [click] and adds the data from the most recent round of testing for the group [click] to get a new estimate of the average achievement level of the group. Now we can compare the prior achievement level of the group to the new achievement level of the group [CLICK If the achievement is roughly at the same point within the distribution of scores as the previous year, this would indicate that there is evidence that the group of students maintained their achievement level which in PVAAS would be indicated as Green. [CLICK] So we ask……is a “green” in PVAAS sufficient for moving students to higher levels of achievement? The good news is that the average achievement level of the group did not slip lower and they met the standard for PA Academic Growth. However, our goal for this group would be to raise their achievement to a higher level. To do that [click] we would want see a Light Blue or Dark Blue reported in the Value Added Report indicating there is moderate or significant evidence that the average achievement level of this group of students was at a higher point within the distribution or that the group of students increased their average achievement level. Again, let’s connect this example to a Value Added Report. Lower Achievement Lower Achievement - Represents the mean achievement level of the group of students PVAAS + Other Data Consultation 2012

Schools with High Achievement 4/15/2017 Schools with High Achievement Can this group of high achieving students still meet the standard for PA Academic Growth? Prior Achievement for Group Most Recent School Year for Group (100% Advanced) (100% Advanced) Higher Achievement Higher Achievement Let’s consider how the concept applies for a school where the students are very high achieving. [click] Perhaps 100% of their students have scored in the Advanced range each year that students have taken the PSSA. To estimate the achievement level for this group of students, [click] PVAAS uses all of the student level data from all years, from all grades, in both Reading and Math from each year this group of students was tested: [click] to estimate the average achievement level of the group. [click] The question is…Can this group of high achieving students still meet the standard for PA Academic Growth? [click] After the next round of testing, PVAAS uses all the prior data for the group [click] and the new testing data [click] to get a new estimate of the average achievement level of the group. Now we can compare the prior achievement level of the group to the new achievement level of the group [CLICK If the achievement is roughly at the same point within the distribution of scores as the previous year, this would indicate that there is evidence that the group of students maintained their achievement level which in PVAAS would be indicated as Green. Would you be satisfied that a group of students who were high achieving, on average maintained that high achievement level? Some people might say “yes”. [click] However, if the achievement goal for the school or this group would be to raise their achievement to a higher level, then [click] we would want see a Light Blue or Dark Blue reported in the Value Added Report indicating there is moderate or significant evidence that the average achievement level of this group of students was at a higher point within the distribution or that the group of students increased their average achievement level. Let’s go back to the growth chart. Lower Achievement Lower Achievement - Represents the mean achievement level of the group of students PVAAS + Other Data Consultation 2012

Measuring Growth at a Deeper Level 4/15/2017 Measuring Growth at a Deeper Level PVAAS does NOT use the percentages of students at various PSSA performance levels to measure growth. Instead, estimated growth measures for groups of students in PVAAS are in terms of NCE units (grades 4-8 Math & Reading) or scaled score points (Science, Writing, & grades 9-11 Math & Reading). So hopefully through this conceptual model, it is clear that: [click] PVAAS does NOT use the percentages of students at various PSSA performance levels to measure growth. [click] Instead, estimated growth measures for groups of students in PVAAS are in terms of NCE units (grades 4-8 Math & Reading) or scaled score points (Science, Writing, & grades 9-11 Math & Reading). PVAAS + Other Data Consultation 2012

Regardless of the entering achievement level of the group… 4/15/2017 Regardless of the entering achievement level of the group… Students should not lose ground academically A reasonable goal would be to at least maintain the achievement level of the group So….regardless of the entering achievement level of the group. Whether that achievement level is: [click] Higher [click] Middle [click] or Lower [click] Students should not lose ground academically or receive a yellow or red color indicator. [click] A reasonable goal for all groups of students would be to at least maintain the achievement level of the group or receive a green color indicator [click] And…all groups of students, even higher achieving students can make progress or growth or receive a light blue or dark blue color indicator. All groups of students can make progress or growth PVAAS + Other Data Consultation 2012

PVAAS + Other Data Consultation 2012 May 2012 C. G. – pg. 2 Practice! Turn to a partner Explain the concept of growth Previous School Year Current School Year Higher Achievement Higher Achievement [Note to facilitator: Have folks turn to a partner and practice explaining the concept of growth] Lower Achievement Lower Achievement PVAAS + Other Data Consultation 2012 PVAAS Statewide Team for PDE June 2012

How is PVAAS data different from the PSSA data I am used to seeing? Achievement (PSSA and Keystones) vs. Growth (PVAAS) Achievement – student is measured against a static level of proficiency based on their grade level and rated advanced, proficient, etc. For many students, this may not reflect changes year to year (ex – many students rate “advanced” every year). Growth – measures student performance relative to that student’s previous performances. Did a student move from below basic to basic? From advanced to proficient? From “low advanced” to “high advanced?” Growth models are more sensitive to individual student progress over time, and compare that progress to an expected standard for growth. DMW

How will this impact my yearly evaluation?

Additional Resources Overview of today’s session handouts The best way to learn more about PVAAS is through exploring the PVAAS site: https://pvaas.sas.com/ There are a number of video learning modules available for viewing. After logging in to the PVAAS site, select the e-learning tab in the top right of the screen. Other materials – common district document location