PVAAS Overview: Evaluating Growth, Projecting Performance

PVAAS Overview: Evaluating Growth, Projecting Performance
SY PVAAS Overview

Partners & Teams for Session Activities
Some activities throughout the session will be completed in pairs or small groups, while others will be done independently. Take a few minutes now to divide yourselves into: Partners (groups of 2), AND Small groups (groups of 4-6). Facilitator says: Some activities throughout the session will be completed in pairs or small groups, while others will be done independently. Please take a few minutes now to divide yourselves into: Partners (groups of 2), and Small groups (groups of 4-6). These will be your partners and small groups for the remainder of the session. For each activity, we will indicate if you are to do the activity independently, with a partner, or as a small group. [CLICK] F2F Overview SY14-15 PVAAS Overview

Goal of Session By the end of the session, participants will be able to answer the following questions: What is PVAAS? How does PVAAS measure growth? What does PVAAS tell us about growth in our district and school(s)? What does PVAAS tell us about how students are projected to perform on a future state assessment? Which reports are most meaningful for school level planning, course level planning, grade level planning, and student level planning? Facilitator says: Note the workbook page reference in the top left corner of each slide so you can follow in your participant workbook. STILL TRUE??????? By the end of the session, participants will be able to answer the following questions: What is PVAAS? How does PVAAS measure growth? What does PVAAS tell us about growth in our district and school(s)? What does PVAAS tell us about how students are projected to perform on a future state assessment? Which reports are most meaningful for school level planning, course level planning, grade level planning, and student level planning? [CLICK] F2F Overview SY14-15 PVAAS Overview

What Is PVAAS? F2F Overview SY14-15 PVAAS Overview Facilitator says:
We will begin by describing what PVAAS is and clarify what PVAAS is not. [CLICK] What Is PVAAS? F2F Overview SY14-15 PVAAS Overview

What is PVAAS? Longitudinal analysis of existing state assessment data
Reliable estimates of academic growth & projections of future performance Effectiveness of the educational system Facilitator says: In this high-stakes assessment and accountability climate, we need accurate information to guide us in making appropriate decisions for students…decisions about instruction, curriculum, and assessment. PVAAS is one tool to assist schools in this process. PVAAS is a very robust statistical analysis of existing assessment data (PSSA & Keystone assessments) – no additional testing is necessary for PVAAS analyses. PVAAS utilizes longitudinal test scores to produce reliable estimates of academic growth and projections of performance on a future assessment [CLICK] PVAAS is a tool to provide a measure of the effectiveness of a school’s or district’s educational system from year to year. [CLICK] It provides growth data to add to achievement data as schools are analyzing their educational programs for students. PVAAS offers powerful information about school practice by leveraging student achievement data with robust statistical tools. It provides a tool to identify strengths and weaknesses, allowing educators to focus their efforts on root causes and solutions where needed. [CLICK] F2F Overview SY14-15 PVAAS Overview

What is PVAAS? Growth of a group of students over time
Grade 5 Grade 6 Grade 3 Grade 4 Growth of a group of students over time NOT a simple comparison of test scores Facilitator says: PVAAS uses existing assessment data to measure the growth of each group of students as they progress through grade levels over time. It compares the group’s most recent academic performance to their own prior academic performance to monitor and estimate their academic growth. PVAAS measures growth by the comparison of estimates based on multiple test scores. Single test scores, including scale scores, are only one measure and are simply “snapshots in time.” Comparison of these single test scores would NOT be an effective way to measure growth. [CLICK] F2F Overview SY14-15 PVAAS Overview

PVAAS Addresses Complex Issues!
Minimizes the effects of measurement error associated with any one test score Measurement Error Accounts for mobility of students Mobility Compensates for missing data Missing Data Facilitator says: The analytic approach (the methodologies) used in the PVAAS analyses to generate PVAAS reports specifically addresses the complexity of measuring growth by addressing issues such as: measurement error, mobility, and missing data. That is why it is not simple to measure growth. The analyses used by the SAS EVAAS team are sophisticated and robust, and are able to accommodate for missing data; they minimize the impact of measurement error; and they account for the mobility of students. This is critically important as more simplistic growth models are not able to accommodate for all 3 of these factors. [CLICK] PVAAS addresses these issues through research-based statistical modeling. This sophisticated approach used for PVAAS results in more reliable information for you – the educator. Research-based Statistical Modeling F2F Overview SY14-15 PVAAS Overview

Demographics and Growth
As long as you can sufficiently account for the prior achievement of the student,… …there is LITTLE TO NO significant relationship between demographics and student progress, or growth! Facilitator says: This is another Key issue when thinking about growth measures and PVAAS. Due to the relationship of demographics and achievement results, many assume there is the same relationship between growth and demographics. They think that students from different demographic groups cannot make a year of growth. However, demographic variables have LITTLE TO NO significant relationship with the growth measures for groups of students…as long as you can sufficiently account for the prior achievement of students. Value-added analysis in PVAAS measures the change in students’ academic achievement levels from one point in time to another using all prior data so as to sufficiently account for the prior achievement of students. Factors that remain relatively constant over time, such as socioeconomic status, have shown to have little or no impact on students’ progress or growth. Evidence from PVAAS reporting in Pennsylvania since 2002 has yielded results to show that there are many schools in Pennsylvania making significant growth with ELL students, minority students, economically disadvantaged students, and students with IEPs. [CLICK] F2F Overview SY14-15 PVAAS Overview

Prior Achievement How can we adequately account for prior achievement?
The SAS EVAAS team has found that the only way to adequately account for prior achievement is to use all available test data for each student to dampen the effects of measurement error. PVAAS does NOT need to adjust for the demographic status of students directly since it is accounting for students’ prior achievement. In other words, demographics are inherently controlled for in PVAAS by the use of all longitudinal data on all students. Facilitator says: The SAS EVAAS team has found that the only way to adequately account for prior achievement is to use all available test data for each student to dampen the effects of measurement error. PVAAS does not need to adjust for the demographic status of students directly since it is accounting for the students’ prior achievement. In other words, demographics are inherently controlled for in PVAAS by the use of all longitudinal data on all students. [CLICK] F2F Overview SY14-15 PVAAS Overview

Work with a partner to decide if each statement is true or false.
Activity Work with a partner to decide if each statement is true or false. Facilitator says: Please work with a partner to draw lines between each statement and the TRUE and FALSE boxes. We will give you about 3 minutes to complete the task. [Note to Facilitator: Have participants complete the activity and then share their findings. The animations to this slide includes the “answers” to these statements.] F2F Overview SY14-15 PVAAS Overview

Work with a partner to decide if each statement is true or false.
Activity Work with a partner to decide if each statement is true or false. Facilitator says: [CLICK] PVAAS adds 2 days of testing to the PSSA schedule. – False, no additional testing is needed. Achievement and demographic variables such as poverty and race have been shown to be related. – True, achievement and demographics are correlated; however, growth and demographics are not correlated as long as students’ prior achievement is sufficiently accounted for. Growth data are more important than achievement data. – False, they are equally important. Comparing two PSSA scores is another effective way to get a measure of growth. – False, two PSSA scores are simply two snapshots in time (two test scores); due to measurement error, it is important to consider all longitudinal data in order to effectively measure growth. Combining growth and achievement data provides better insight for decision-making than looking individually at either growth or achievement data. – True, both pieces of data provide distinct information with two purposes. Achievement tells us where students end up without taking into account where they started. Growth tells us how much students have progressed over time taking into account students’ starting and ending achievement. There is little to no relationship between growth and demographic variables, as long as you can sufficiently account for the prior achievement of the student. – True F2F Overview SY14-15 PVAAS Overview

PVAAS Misconceptions:
For More Information Virtual Learning Module PVAAS Misconceptions: Clarifying Questions and Misunderstandings For more in-depth information on these topics, please view the virtual learning module entitled: PVAAS Misconceptions. This module was designed to clarify questions and misunderstandings about PVAAS. This module in particular provides more information on the topics of measurement error, student mobility, missing data, and student demographics. Simply use the e-Learning link that is available in the blue menu bar on either the public PVAAS site or the password protected PVAAS site. F2F Overview SY14-15 PVAAS Overview

Two Types of PVAAS Information
Facilitator says: We now move into considering the different types of PVAAS information. [CLICK] Two Types of PVAAS Information F2F Overview SY14-15 PVAAS Overview

PVAAS What is PVAAS? LOOKING BACK LOOKING AHEAD District
Keystones PSSA Science Writing PSSA Math Reading LOOKING AHEAD Planning for Students’ Needs: Student Projections to Future Tests LOOKING BACK Measuring Academic Growth of Groups of Students: Value-Added & Diagnostic Reports LOOKING BACK Measuring Academic Growth of Groups of Students: Value-Added & Diagnostic Reports LOOKING AHEAD Planning for Students’ Needs: Student Projections to Future Tests District School Teacher PVAAS is a reporting system based on Value-Added analysis, which is a way of quantifying the growth students make, or the “value” that educators add to student achievement. PVAAS includes data from Pennsylvania’s state assessments and uses the information from student testing histories to help educators identify how effective they have been in helping students to make academic growth. So how does PVAAS work? All of the standardized test results from PSSA and Keystones exams go through rigorous analyses by SAS EVAAS. From this information, we get two key pieces of information. [CLICK] First, we can look back to see how our programs are supporting students in making academic progress. This type of information measuring academic growth of students using Value-added and Diagnostic Reports. This Looking Back component of PVAAS provides District, School, and Teacher Specific Reporting. Second, we can look ahead by seeing student projections to future state assessments and plan for their needs. Student *Writing used only in District & School Reporting T **Note: All Prior Data are Used PVAAS Overview

LOOKING AHEAD LOOKING BACK
Facilitator says: Let’s start with the looking back side The Value-added, or growth, information is what “looks back” This information helps educators to investigate the effectiveness of their schools and academic programs and the influence they have on the academic growth of their students. How much growth did groups of students make in the past school year? This data is available on several different types of reports in PVAAS. F2F Overview SY14-15 PVAAS Overview

PVAAS Value-Added Reporting How is growth measured in PVAAS?
Facilitator says: Let’s start with the PVAAS Value-added Reporting which yields growth measures of groups of students at a district, school, or teacher specific level. [CLICK] F2F Overview SY14-15 PVAAS Overview

PVAAS Value-Added Reporting
Testing Subjects – Grades Methodologies Assessments in consecutive years Mathematics, 4-8 Reading, 4-8 Growth Standard Assessments not in consecutive years Writing – 5 & 8 Science – 4 & 8 Keystones – Algebra I, Biology, & Literature Predictive Facilitator says: It is important to know that PVAAS utilizes two different methodologies for estimating growth measures for groups of students. The Growth Standard and Predictive methodologies are used based on whether the test data are administered in consecutive grades, or not. The Growth Standard Methodology is employed for consecutive grades of assessment data. In Pennsylvania, this means we use this for reporting in grades 4-8 in the subjects of Reading and Math. The Predictive Methodology is used for non-consecutive grades of assessment data. In Pennsylvania, this means we use this for all grade levels of Science and Writing reporting, as well as for all reporting of Keystone content areas (specifically Algebra I, Biology, and Literature). *Writing only in District & School Reporting F2F Overview SY14-15 PVAAS Overview

Measuring Academic Growth
Students don’t all start the year at the same place academically Facilitator says: If we are going to measure the academic growth that students make, we have to consider more than whether or not they reached proficiency. In addition, it’s critical to recognize that students don’t all start the year or a course at the same place academically. [CLICK] F2F Overview SY14-15 PVAAS Overview

Measuring Academic Growth
Standard for PA Academic Growth Regardless of their entering achievement level, students should not lose ground academically Is it not a reasonable goal to maintain the achievement level of a group of students? Not necessarily increase their achievement level, but maintain their achievement level. Facilitator says: In Pennsylvania, growth is assessed against the standard for PA Academic Growth. It is based on the philosophy that regardless of the entering achievement level of a group of students, they should not lose ground academically. [CLICK] So we ask, is it not a reasonable goal to maintain the achievement level of a group of students? Is that not a reasonable expectation? In other words, if students are fairly high achieving, is it not reasonable to expect that we, as educators, keep them fairly high achieving? And if students are low achieving, is it not reasonable to expect that we would at least keep those students at that level of achievement (and not have them slip further behind)? We would hope that everyone would agree that this is a reasonable expectation. F2F Overview SY14-15 PVAAS Overview

PVAAS Value-Added Growth Indicators
Math and Reading Grades 4-8 Science, Writing, Keystones Facilitator says: PVAAS Value Added reports yield a Growth Measure and a Growth Color Indicator. The Growth Measure represents the amount of growth a group of students has made. [CLICK] The Color Indicators represent the level of evidence we have in the data to say whether or not the group of students: (1) did NOT meet the standard for PA Academic Growth, (2) met the standard for PA Academic Growth, or (3) exceeded the standard for PA Academic Growth. In other words, the color tells us how sure we are that the group of students increased, decreased, or maintained their achievement level on average. F2F Overview SY14-15 PVAAS Overview

Philosophy Behind PVAAS
All students can learn. All students deserve to make appropriate academic progress each year. PVAAS Overview

Concept of Measuring Growth Prior (or Entering) Achievement of the
Higher Achievement Higher Achievement Facilitator says: The actual analyses used for calculating growth contain robust statistical models that are much more complex than what we are illustrating here. However, this provides a big picture concept of measuring growth. When the state assessment is administered in consecutive grade levels as it is in reading and math for grades 3-8, PVAAS uses a growth standard method to measure the growth or progress of a group of students. When the state assessment is not administered in consecutive grade levels as in science, writing, or the Keystone content areas, PVAAS uses a predictive method to measure the growth or progress of a group of students. However, while the reports are slightly different and the behind the scenes methodology is different, we will think about these conceptually in the same way for our purposes today. Suppose that the star estimates the estimated achievement of the group based on all of their prior test scores. After the next round of testing, PVAAS uses all the prior data for the group and adds the data from the most recent round of testing… [CLICK] To calculate an updated estimate of the current achievement of the group. If the evidence indicates the achievement is roughly at the same point within the distribution of scores as the previous year (or is roughly the same as predicted in the case of Science, Writing, and the Keystone content areas), this would indicate that the group of students maintained their achievement level which in PVAAS would be indicated as Green – evidence that the student group met the standard for PA Academic Growth. If there is moderate evidence or significant evidence that the group of students exceeded the standard for PA Academic Growth or increased their average achievement level or performed higher than predicted, this would be indicated with a Light Blue or Dark Blue on the school or district value-added report. If there is moderate evidence or significant evidence that the group of students did not met the standard for PA Academic Growth or their average achievement level slipped or they performed lower than predicted, this would be indicated with a Yellow or Red on the school or district value-added report. Lower Achievement Lower Achievement Prior (or Entering) Achievement of the Group of Students Current Achievement of the Group of Students F2F Overview SY14-15 PVAAS Overview

Regardless of the entering achievement level of the group…
A reasonable goal would be to at least maintain the achievement level of the group Students should not lose ground academically Facilitator says: So….regardless of the entering achievement level of the group. Whether the group of students is: Higher achieving, Middle achieving, or Lower achieving [CLICK] A reasonable goal for all groups of students would be to at least maintain their achievement level or meet their predicted achievement level, or in other words, receive a green color indicator Students should not lose ground academically or receive a yellow or red color indicator. And…all groups of students, even higher achieving students can receive a light blue or dark blue color indicator. All groups of students can make growth F2F Overview SY14-15 PVAAS Overview

Practice! Turn to a partner Explain the concept of growth
Previous School Year Current School Year Higher Achievement Higher Achievement Facilitator says: Please work with a partner and practice explaining the concept of growth using the graphics. You will have about 5 minutes to practice. [CLICK] [NOTE TO FACILITATOR: Ask several participants to share the key phrases or points they used as they explained the concept of growth.] Lower Achievement Lower Achievement F2F Overview SY14-15 PVAAS Overview

For More Information TWO Virtual Learning Modules
Introduction to Measuring Academic Growth Concept of Growth: Connections to School Value-Added Reporting For more in-depth information on the concept of growth, please view the two virtual learning modules entitled Introduction to Measuring Academic Growth and Concept of Growth: Connections to School Value-Added Reporting. Simply use the e-Learning link that is available in the blue menu bar on either the public PVAAS site or the password protected PVAAS site. F2F Overview SY14-15 PVAAS Overview

Questions Facilitator says:
Before continuing, let’s pause and see if there are any questions. F2F Overview SY14-15 PVAAS Overview

Math and Reading Grades 4-8 Science, Writing, Keystones Facilitator says: Remember earlier that we discussed how the PVAAS reports have both a growth measure and a growth color indicator. The colors we just discussed (dark blue, light blue, green, yellow, and red) are the color indicators you will find on PVAAS value-added reports – whether at the district, school [CLICK] Or teacher level. F2F Overview SY14-15 PVAAS Overview

Measuring Growth in PVAAS
Remember, PVAAS does NOT compare the achievement of one group of students in a particular subject and grade with the previous year’s students in the same subject and grade. PVAAS does NOT use the percentages of students at various academic performance levels to measure growth. PVAAS uses students’ scale scores within those performance levels to better estimate growth. Facilitator says: So hopefully through this conceptual model, it is clear that: [CLICK] PVAAS does NOT compare the achievement of one group of students in a particular subject and grade level with the previous year’s students in that same subject and grade level. For example, the achievement of the most recent group of students in 8th grade Math is NOT compared to the achievement of the previous year’s group of 8th grade students in Math. Instead, PVAAS measures the growth of each distinct group of students as they progress through grade levels over time. It compares the group’s most recent academic performance to their own prior academic performance to track their growth. In addition, PVAAS does NOT use the percentages of students at various academic performance levels to measure growth. Each performance level contains a range of scores, and students move around within these ranges, as well as between ranges. PVAAS uses students’ scale scores within those performance levels to better estimate growth. F2F Overview SY14-15 PVAAS Overview

District and School Facilitator says: PVAAS growth indicators for groups of students are displayed at the bottom of all Value-added reports. [CLICK] PVAAS growth indicators for groups of students are based on how much evidence we have in the data that students exceeded or fell short of the standard for PA Academic Growth. These colors can be found on all PVAAS Value Added reports no matter the type of analysis (growth standard or predictive) is used or what level or reporting is provided. Teacher F2F Overview SY14-15 PVAAS Overview

Growth Standard VAA Report (Reading and Math – grades 4 though 8)
Here is a demo Growth Standard VAA Growth Report at the district level. The report has two sections: [CLICK] The table at the bottom of the report displays the estimated district Average Achievement scores for each grade in the district for the current year and prior years. These scores should be interpreted relative to the State (NCE) Average Achievement value of 50. In addition, the bottom of the report displays the color-coded interpretation guide for the growth values. The top of the table details the growth values for each grade in the district/school for the current and past years as well as the 3-year average growth value for each displayed grade. We will pause for any questions… F2F Overview SY14-15 PVAAS Overview

Trends in Growth Standard Value-Added Report (Reading and Math – grades 4 though 8)
The Growth Standard VAA report provides trend data in two ways… [CLICK] By focusing on the data displayed in each column, it is possible to see the history of NCE Achievement estimates and PVAAS Growth values over the past three years for each grade. This provides insight into growth values for the same grade with three different groups of students. Please keep in mind that any indications should be tempered by the fact that these data reflect the growth of three different groups of students. In addition, the inspection of the diagonal values for both the NCE Achievement estimates and the PVAAS Growth values provides insights as to how a group of students displayed growth in three consecutive years. While the student group may not be identical each year in the progression, patterns across the three consecutive years will provide deeper insights. F2F Overview SY14-15 PVAAS Overview

PVAAS Value-Added Growth Measures
For Reading and Math in grades 4-8, Growth is measured in Normal Curve Equivalent (NCE) units as scale scores cannot be compared from grade to grade and year to year. The Growth Measure is NOT reported in years and cannot be referenced in terms of number of year’s growth. The Growth Measure is NOT reported in percentages and cannot be referenced in terms of an increase or decrease in percentage points. Facilitator says: For Reading and Math in grades 4-8, Growth is measured in Normal Curve Equivalent (NCE) units as scale scores cannot be compared from grade to grade and year to year. Remember, the Growth Measure is NOT reported in years and cannot be referenced in terms of number of year’s growth. It is also NOT reported in percentages and cannot be referenced in terms of an increase or decrease in percentage points. F2F Overview SY14-15 PVAAS Overview

Rescaling Grades 4-8: Math & Reading
Scale Score Points NCE Units Facilitator says: The use of NCEs is simply a way of rescaling scale score points so that scores can be compared across different years. To give you a reference point, 50 on an NCE is always the state average. In this way, we can compare scores across different grades, subjects, and years. [CLICK] State Average F2F Overview SY14-15 PVAAS Overview

Predictive Value-Added Report (Science, Writing and Keystone exams)
Here is a demo Predictive VAA report at the district level. All of the critical information on the calculation of the growth value is included: [CLICK] The first set of information details the subject, grades, year(s), numbers of students in each grade’s report, and the average scaled score and percentile of the included students. The middle of the table details the average predicted score and percentile. And most important, the color-coded Growth Measures for each grade are displayed with their respective standard errors. Notice also that the three year trend data and 3-year average can be seen easily as well in this report. This is the Predictive VAA Report for Science, Writing and Keystone exams. We will pause for any questions… F2F Overview SY14-15 PVAAS Overview

School Value-Added Reports Science, Writing, & Keystone content areas
School A School B Facilitator says: These are examples of the school value added reports that available for science, writing, and the Keystone content areas. On school Value-Added Reports for Science and Writing, educators can view a Growth Measure and a Growth Color Indicator for each group of students in each tested grade level in the school as we see here for School A for science. In School A, we see a green color indicator for grade 4 indicating that students met the standard for PA Academic Growth, while there is a light blue indicator for grade 8 indicating that there is moderate evidence that students exceeded the standard for PA Academic Growth. On school Value-Added Reports for the Keystone content areas of Algebra I, Biology, and Literature, educators can view a Growth Measure and a Growth Color Indicator for each group of students as we see in School B for Literature. So in School B, there is a dark blue color indicator for that group of students indicating there is significant evidence that the group exceeded the standard for PA Academic Growth. It is important to remember that this is not disaggregated or reported separately by grade level as the Keystone exams are NOT grade-specific assessments. Let’s walk through the columns of data you can find in these reports. The column next to the year displays the Number of Students that are included in the report. Students must have a minimum of three prior test scores across subjects to be included. The only exception to this rule is for 4th grade science, where only two prior test scores are needed. This data requirement may account for any differences between the number of students tested and the number of students represented in your Value-Added Report. The next column displays the Average Scale Score for the group of students who had the minimum number of prior test scores. The associated average percentile rank is displayed in the next column. The average percentile rank indicates where the average scale score for these students falls in the distribution of scores for all students in Pennsylvania. This gives you an idea of where the achievement level of your students relative to all students in Pennsylvania who took the indicated assessment in the same year. The next two columns contain the Average Predicted Scale Score for these students and its associated percentile rank. The Average Predicted Scale Score is simply the average of all of the students’ individual predicted scores. A predicted score is what we would have expected a student to score given the student’s testing history across subjects, if the student had an average schooling experience. The associated predicted average percentile rank is displayed in the next column. The average predicted percentile rank indicates where the average predicted scale score for these students falls in the distribution of scores for all students in Pennsylvania. This gives you an idea of where the predicted achievement level of your students relative to all students in Pennsylvania who took the indicated assessment in the same year. [CLICK] F2F Overview SY14-15 PVAAS Overview

PVAAS Value-Added Growth Measures
For Science, Writing and Keystone content areas, Growth is measured in scale score points. The Growth Measure is NOT reported in years and cannot be referenced in terms of number of year’s growth. The Growth Measure is NOT reported in percentages and cannot be referenced in terms of an increase or decrease in percentage points. Facilitator says: For Science, Writing and Keystone content areas, Growth is measured in scale score points. We can compare scale scores in this case because we are comparing a predicted score to an observed score which is different than how growth is measured for reading and math in grades 4-8. Remember, the Growth Measure is NOT reported in years and cannot be referenced in terms of number of year’s growth. It is also NOT reported in percentages and cannot be referenced in terms of an increase or decrease in percentage points. F2F Overview SY14-15 PVAAS Overview

Activity: How would you interpret?
Facilitator says: Now it’s your turn! Interpret this report for a K-8 school. You may work with a partner and you will have 5 minutes to complete this activity. We will discuss this report and ask for volunteers to share their insights. [Note to Facilitator: Leave this on the screen so participants can SEE the color codes. Also point out how the letters DB, LB, G, Y, and R indicate the color code even if the report is printed in black/white.] [Note to Facilitator: After 5 minutes have volunteers share their insights.] [CLICK] F2F Overview SY14-15 PVAAS Overview

Facilitator says: Now let’s practice interpreting the Value-Added reports based on the Predictive Methodology. Work with a partner to interpret these reports, one for 5th grade writing and the other, high school Keystone Literature. You will have 5 minutes to complete this activity. Again, we will ask for volunteers to share their interpretations. Keep this report on the screen as participants DO the activity so they can SEE the colors in case the handout is printed in black/white. [Note to Facilitator: After 5 minutes, have participants share some of their insights within small groups. Ask for questions/clarifications needed from the group.] [CLICK] F2F Overview SY14-15 PVAAS Overview

Teacher Value Added Summary
Facilitator says: Here is a sample Teacher Value Added Summary Report. The Teacher Value Added Summary displays the most recent growth index values and growth measures for all assessed grades, subjects, and courses in which the teacher had full or partial instructional responsibility for a group of students. Let’s examine the key differences between district/school reporting and teacher specific reporting. [click] PVAAS Overview

Key Differences Between District/School and Teacher Specific Reporting
PVAAS School Reporting uses full academic year as a requirement for including students PVAAS Teacher Specific Reporting requires 10% Total Instructional Responsibility for including students PVAAS Teacher Specific Reporting uses % Student +Teacher Enrollment and % Full/Partial Instruction to determine the weighting of each student on each teacher’s PVAAS Teacher Value Added Report PVAAS School Reporting uses full academic year as a requirement for including students. PVAAS Teacher Value-added reporting uses % Student +Teacher Enrollment and % Full/Partial Instruction to determine the weighting of each student on each teacher’s PVAAS Teacher Specific Reporting PVAAS Teacher Specific Reporting requires 10% Total Instructional Responsibility for including students. PVAAS Teacher Specific Reporting uses % Student +Teacher Enrollment and % Full/Partial Instruction to determine the weighting of each student on each teacher’s PVAAS Teacher Value Added Report. Students are weighted equally for district and school reporting. PVAAS Overview

Composite v. Teacher Value-Added
The growth color indicators are categories or ranges of indicators of growth. We cannot “average” categories. In other words, green + red ≠ yellow; likewise green + dark blue ≠ light blue. For PVAAS, the assumption is that the achievement level of the teacher's group of students is maintained (green) UNLESS there is enough evidence in the assessment data to say otherwise. It is all about the amount of evidence in the student assessment results –the assumption is that the achievement level of the teacher's group of students is maintained (green) UNLESS there is enough evidence in the assessment data to say otherwise. The more data available, the more evidence we have to see if the group of students exceeded the growth standard OR not. There is more evidence when all data was combined for a composite score. With data from multiple subjects (or multiple years when that’s available) included in the Composite, there’s more evidence/more data. It is all about the amount of evidence in the student assessment results –the assumption is that the achievement level of the teacher's group of students is maintained (green) UNLESS there is enough evidence in the assessment data to say otherwise. The more data available, the more evidence we have to see if the group of students exceeded the growth standard OR not. There is more evidence when all data was combined for a composite score. With data from multiple subjects (or multiple years when that’s available) included in the Composite, there’s more evidence/more data. An analogy- we can estimate how many blue M&M’s are in bags of M&M’s by looking at one bag, BUT the error would be a little larger than if new looked at more bags. We will have a better estimate/more evidence/less error on our measure if we look at 100 bags of M&M's. Remember, for PVAAS the assumption is that the achievement level of the teacher's group of students is maintained (green) UNLESS there is enough evidence in the assessment data to say otherwise. More details: To calculate the composite, a simple average is taken of all of the teacher's individual index values for up to three years. Then the average is multiplied by the square root of the number of individual index values that went into the average. This step is a necessary step that accounts for the fact that more data was used to generate the average than was used to generate each individual index, which affects the standard error for the composite. PVAAS Overview

For More Information Value-Added Report TWO Virtual Learning Modules
For more in-depth information on the value-added reports, please view the two virtual learning modules entitled Value-Added Report – one for the Growth Standard methodology and one of the Predictive methodology. Simply use the e-Learning link that is available in the blue menu bar on either the public PVAAS site or the password protected PVAAS site. F2F Overview SY14-15 PVAAS Overview

Teacher Specific Reporting
For More Information Virtual Learning Module Teacher Specific Reporting For more in-depth information on the value-added reports at the teacher level, please view the virtual learning modules entitled Teacher Specific Reporting. Simply use the e-Learning link that is available in the blue menu bar on either the public PVAAS site or the password protected PVAAS site. F2F Overview SY14-15 PVAAS Overview

District/School Performance Diagnostic Report
Missing Bar – Insufficient Number of Students Blue Bar – Current Year Gold Bar – Previous Years Whisker – Margin of Error on Growth Value Facilitator says: The Performance Diagnostic Reports are a report that allows educators to dig deeper into the growth data to help answer the question: In all grades, subjects, and Keystone content areas, are learners at all achievement levels making adequate growth? This report can be viewed for an entire grade level or Keystone content area or for different demographic groups of students. Educators can use this report to identify patterns or trends of academic growth among groups of students at various entering achievement levels. Let’s do a quick walk through of this report. On the graph the blue bars represent the growth of students in the selected Keystone content area, or grade and subject, in the most recent year. Please remember that the color of the bar does not reflect anything about growth! You’ll see a bar for each of the state’s assessment performance levels – Below Basic, Basic, Proficient, and Advanced. Students are placed into these groups based on their entering achievement level in the Keystone content area, or grade and subject. Just like the value-added reports, note that the reports for Keystone content areas are not limited to a grade level of students. This is because the Keystone exams are NOT grade-specific assessments. In this example, you’ll notice that there is no blue bar for students who entered this grade in the Basic range. This is due to the fact that PVAAS will provide an estimate of growth only for a group with 5 or more students. Estimates of growth based on fewer students would not yield reliable information. The green line represents the standard for PA Academic Growth. This line marks the growth that students in each group must make to maintain their achievement level, or perform as predicted, from one year to the next. The standard error is represented by the red whisker at the end of each blue bar. This is a margin of error, meaning that the true value of growth for the group is likely to be within the whisker. The whisker plays a critical part in the interpretation of the growth estimate. [click] F2F Overview SY14-15 PVAAS Overview

PVAAS Performance Diagnostic Growth Descriptors - Interpretation
What the whiskers tell us… Exceeded the standard for PA Academic Growth Met the standard for PA Academic Growth Zero (0) Line Growth Facilitator says: The position of the whisker relative to the green zero (0) reference line is the key to the interpretation of the Performance Diagnostic Report. On this slide, we display the interpretation of the whiskers for this report. The green line represents the standard for PA Academic Growth. The key is where the red whiskers or I-bars are located relative to the green line. [CLICK] If the whisker or I-bar intersects the green line at all, we conclude that the group of students has met the standard for PA Academic Growth. On a pie chart version of this report or on the Performance Diagnostic Summary report, this would be reflected as green. If the whisker or I-bar is completely below the green line, we conclude that the group of students did not meet the standard for PA Academic Growth. On a pie chart version of this report or on the Performance Diagnostic Summary report, this would be reflected as pink. If the whisker or I-bar is completely above the green line, we conclude that the group of students exceeded the standard for PA Academic Growth. On a pie chart version of this report or on the Performance Diagnostic Summary report, this would be reflected as blue. Did not meet the standard for PA Academic Growth F2F Overview SY14-15 PVAAS Overview

Performance Diagnostic Patterns
Shed Pattern Reverse Shed Pattern Facilitator says: In this example here on the left, the lowest achieving students are making great growth. However, the highest achieving students appear to be losing ground. What happens to the achievement gap if you have a pattern like this? It closes, it narrows, but not in the right way. It narrows at the expense of students in the Proficient and Advanced ranges of achievement. Teachers and administrators will want to find ways to create more opportunities for progress for higher-achieving students. [CLICK] We see the opposite in the pattern in this example to the right; it is less common but it does exist. In this example, high achieving students are making excellent growth. Students entering in the Basic and Proficient ranges are also making good growth. In contrast, the lowest achieving students are not making as much growth as they should. A pattern like this one may widen the achievement gap, and low achieving students may fall even farther behind their peers. Teacher and administrators will want to find ways to create more growth for lower-achieving students. System benefits lower-achieving students System benefits higher-achieving students F2F Overview SY14-15 PVAAS Overview

Tent Pattern V Pattern Facilitator says: In this example on the left, the students entering in the Basic and Proficient ranges are making great growth. However, both lower achieving and higher achieving students are losing ground academically. This kind of pattern suggests a narrow curricular focus that only meets the needs of a narrow range of students. Low achieving students will need more support, and high achieving students will need a more challenging experience to continue making growth at a high achievement level. This is the tent pattern – we’re teaching to the middle. [CLICK] In this example on the right, the V pattern (the opposite of the Tent pattern), only the lowest and the highest achieving students are making good growth. Students entering in the Basic and Proficient ranges have not had enough opportunities for academic growth. This pattern may occur in cases where regular classroom instruction is not effective and not meeting the needs of students so students are falling further behind. However, the lowest achieving students may be receiving additional services and the highest achieving students may be challenged perhaps through a strong gifted program or honors level courses. System benefits middle level learners System benefits higher and lower achieving students F2F Overview SY14-15 PVAAS Overview

Uniform I Pattern Uniform 2 Pattern Facilitator says: In this example on the left, all groups of students are making excellent growth. With this pattern, the educational program benefits all students in all achievement levels similarly and adequately. This pattern illustrates that all students are exceeding the standard for PA Academic Growth. [CLICK] In this example on the right, none of the groups of students are making good growth. This pattern is of greatest concern as all student groups are not meeting the standard for PA Academic Growth. In other words, all blue bars including the red whiskers are well below the green reference line, or standard for PA Academic Growth. System is not supporting growth of any students System supports growth of all students similarly and adequately F2F Overview SY14-15 PVAAS Overview

Opportunity Gap Pattern Facilitator says: In this example, students at all achievement levels are making good growth in the most recent year, except for students entering in the Basic group. This type of pattern sometimes occurs when classroom instruction is aimed at students entering at the Proficient and Advanced levels, while the lowest achieving students are benefitting from additional services and interventions. If students entering in the Basic range do not qualify for additional services, they can lose ground. [CLICK] System benefits all types of learners except those right below proficiency F2F Overview SY14-15 PVAAS Overview

Desirable Pattern Facilitator says: In this example, all bars extend above the green line, indicating the school/district was highly effective with students at all achievement levels. Additionally, students in the lowest achievement group made more growth than students in the other achievement groups. Effectively, these students are starting to catch up with their peers. In other words, the students who need to gain the most ground are in fact gaining the most. The achievement gap may be closing without any groups of students losing ground. [CLICK] System benefits all types of learners with more growth occurring with the lower achieving students F2F Overview SY14-15 PVAAS Overview

Activity: Check for Understanding
Sketch a sample performance diagnostic bar graph that fits each description. Facilitator says: Work individually for this activity. Sketch a sample Performance Diagnostic bar graph that fits each description. [Note to Facilitator: After 5 minutes, have participants share their results. Ask participants what was the most important piece to draw (the bar or the whisker?).] It is important for participants to recognize the importance of the whisker and its relationship to the green reference line. [CLICK] F2F Overview SY14-15 PVAAS Overview

District/School Quintile Diagnostic Report
There are 4 groups of students on a Performance Diagnostic Report. Students are placed into a predicted performance groups based on where they are predicted to perform on their next assessment. There are 5 groups of students on a Quintile Diagnostic Report. Students are placed into an achievement group based on the rank of their achievement compared to other PA students. Facilitator says: There is another type of diagnostic report in PVAAS called a Quintile Diagnostic Report. This report is very similar to a Performance Diagnostic Report in that growth can be interpreted in the exact same way. The only difference between the reports is the groups into which the students are placed. [CLICK] In the Quintile Diagnostic Report, students are placed into 5 groups rather than 4 like on a Performance Diagnostic Report. As you will recall, on a Performance Diagnostic Report, students are disaggregated by predicted academic performance levels based on students’ prior test performances across grades and subjects. PVAAS looks at each record for each student in the group and places them into the academic performance level where they are predicted to perform on their next state assessment based on their prior test performances. Hence, we can think of this placement into one of the 4 groups on a Performance Diagnostic Report as students’ entering achievement level. On a Quintile Diagnostic Report however, students are assigned to one of five prior-achievement level (or quintile) groups based on where they profile within the state distribution, without regard to the state’s academic performance level ranges. This means that placement into one of the 5 prior-achievement level groups (or quintile groups) is not based on students’ academic performance level of Below Basic, Basic, Proficient, or Advanced. Rather, all students are assigned to a prior-achievement level (or quintile) group according to the rank of their achievement compared to other students in the same grade and subject in Pennsylvania. Placement in the 1 (or Lowest) group indicates those students are in the lowest achievement group (bottom 20% of students in the state), while placement in the 5 (or Highest) group indicates those students are in the highest achievement group (top 20% of students in the state). Similarly, placement in the 3 (or Middle) group indicates those students are in the middle 20% of students in the state in terms of achievement. So in summary, there are 4 groups of students on a Performance Diagnostic Report. Students are placed into a predicted performance groups based on where they are predicted to perform on their next assessment. However, there are 5 groups of students on a Quintile Diagnostic Report. Students are placed into an achievement group based on the rank of their achievement compared to other PA students. F2F Overview SY14-15 PVAAS Overview

Diagnostic Summary Report at the Teacher Level
Facilitator says: A Diagnostic Report is also available at the teacher level. These reports are more similar to a District or School Quintile Diagnostic Report, rather than a Performance Diagnostic Report, as students are placed into 3 groups based on their achievement compared to other students in PA in the same grade/subject/course. On a teacher’s Diagnostic Report, the information is displayed in pie charts by default rather than bars and whiskers like on a district or school report. The pie charts visually show the proportion of students that fall into each group, and the color indicator tells about the growth for that group of students. So in other words, it is looking at the growth of students who are in the lowest third of the state in terms of achievement, then those who are in the middle third of the student in terms of achievement, and then finally those who are in the highest third of the state in terms of achievement. [CLICK] A legend is found at the bottom of the report that provides an explanation of the color in terms of the growth of that particular group of students. Blue indicates moderate evidence that the group exceeded the standard for PA Academic Growth; Green indicates evidence that the group met the standard for PA Academic Growth; Pink indicates moderate evidence that the group did not meet the standard for PA Academic Growth; and White indicates there were not enough students in the group to measure growth. PVAAS Overview

Performance Diagnostic Subgroup Reports
Facilitator says: PVAAS also provides Diagnostic Reports (at the district, school, or teacher level) for demographic subgroups. [CLICK] This is a display of the subgroup selection pop-up window used to select the demographic subgroup(s) you wish to investigate. The categories on this screen reflect the categories on the statewide student level file. F2F Overview SY14-15 PVAAS Overview

Performance Diagnostic Subgroup Report
Facilitator says: This is an example of a Performance Diagnostic Report for the subgroup selected: Economically Disadvantaged students only. [CLICK] F2F Overview SY14-15 PVAAS Overview

Best to Compare Subgroup to Entire Grade or Course Level
Same District/School – Same Grade/Course Facilitator says: The real power of the Performance Diagnostic for subgroups becomes clear when you compare the growth of the whole grade level to the growth of the subgroup. You should be aware that the report on the whole group DOES also include those in the subgroup. [CLICK] Entire grade or course Subgroup F2F Overview SY14-15 PVAAS Overview

Economically Disadvantaged
Activity What conclusions can you make about the growth of different groups of students in 6th grade Reading? Entire Grade or Course Economically Disadvantaged Special Education Facilitator says: Let’s take a look at the 3 Performance Diagnostic Reports relating to the same grade and subject displayed on this slide. The first report reflects all students in 6th grade Reading, while the other two reports reflect only the Economically Disadvantaged students and the students with IEPs. What conclusions can you make about the growth of different groups of students in 6th grade Reading? [CLICK] F2F Overview SY14-15 PVAAS Overview

Purpose of Value-Added & Performance Diagnostic Reports
Assessing curricular needs across grades, subjects, and Keystone content areas Evaluating instructional needs of students at various achievement levels and in different subgroups Ensuring instructional coherence/alignment Curriculum Instruction Assessment Supports, including scheduling and staffing needs Monitoring implementation and effectiveness of grade-level or course-specific programs Facilitator says: In summary and from what was discussed, growth reports in PVAAS can be used in a number of ways for a variety of purposes. Assessing curricular needs across grades, subjects, and Keystone content areas Evaluating instructional needs of students at various achievement levels and in different subgroups Ensuring instructional coherence/alignment Curriculum to Instruction to Assessment Supports, including scheduling and staffing needs Monitoring implementation and effectiveness of grade-level or course-specific programs [CLICK] F2F Overview SY14-15 PVAAS Overview

Performance Diagnostic Report Quintile Diagnostic Report
For More Information Two Virtual Learning Modules Performance Diagnostic Report Quintile Diagnostic Report For more in-depth information on the Performance Diagnostic and Quintile Diagnostic reports, please view the virtual learning modules entitled Performance Diagnostic Report and Quintile Diagnostic Report. Simply use the e-Learning link that is available in the blue menu bar on either the public PVAAS site or the password protected PVAAS site. F2F Overview SY14-15 PVAAS Overview

Teacher Specific Reporting
For More Information Virtual Learning Module Teacher Specific Reporting For more in-depth information on the diagnostic reports at the teacher level, please view the virtual learning module entitled Teacher Specific Reporting. Simply use the e-Learning link that is available in the blue menu bar on either the public PVAAS site or the password protected PVAAS site. F2F Overview SY14-15 PVAAS Overview

Many Reports Facilitator says:
For a typical middle school, a school could have sixteen Value-Added and Performance Diagnostic Reports to get a view of academic growth across all assessed subjects and grades in their school. We are pleased to announce a new report that will allow schools to get a consolidated view of these reports. [CLICK] F2F Overview SY14-15 PVAAS Overview

PVAAS Launchpad See multiple DISTRICT/SCHOOL reports in a consolidated format Provides a user-friendly visual representation of district/school level data Facilitator says: The PVAAS Launchpad is a user-friendly report which gives schools a consolidated view of District/School Value-Added and District/School Performance Diagnostic Reports. F2F Overview SY14-15 PVAAS Overview

Provides the Big Picture
PVAAS Launchpad Provides the Big Picture Across PSSA grades and Keystone content areas Across subject areas Facilitator says: The Launchpad is designed to allow educators to view growth patterns across grade levels and Keystone content areas, or across tested subject areas. F2F Overview SY14-15 PVAAS Overview

PVAAS Launchpad Facilitator says:
Here are the growth color indicators we discussed earlier for value added reports. F2F Overview SY14-15 PVAAS Overview

PVAAS Launchpad Facilitator says:
Here are the growth color indicators we discussed earlier for Performance Diagnostic reports. F2F Overview SY14-15 PVAAS Overview

Remember…the Launchpad
Provides a comprehensive view showing multiple reports on a new single report Allows you to see patterns by grade and subject Provides you with growth patterns that speak to the strength of your overall core program deeper analysis is needed, such as looking at subgroups Facilitator says: The Launchpad is a comprehensive view showing multiple reports on a new single report Which allows you to see patterns by grade and subject. And, provides you with growth patterns that speak to the strength of your overall program – deeper analysis is needed such as looking at subgroups [CLICK] F2F Overview SY14-15 PVAAS Overview

For More Information Launchpad Virtual Learning Module
For more in-depth information on the Launchpad, please view the virtual learning module entitled PVAAS Launchpad. Simply use the e-Learning link that is available in the blue menu bar on either the public PVAAS site or the password protected PVAAS site. F2F Overview SY14-15 PVAAS Overview

Achievement and Growth Together!
Facilitator says: We now consider the value of looking at Achievement and Growth together. [CLICK] Achievement PVAAS Overview

Percent Proficient or Advanced Standard for PA Academic Growth
PVAAS Scatterplots Vertical Axis Percent Proficient or Advanced Standard for PA Academic Growth Facilitator says: On this screen you should see an example of a scatterplot. The Horizontal Axis is for displaying the Average Growth Index (AGI) . We have highlighted the AGI boundary of 0 which represents the standard for PA Academic Growth. A value of 0 means that the group of students has met the standard for PA Academic Growth. In other words, the value of 0 means the group of students maintained their achievement level or performed as expected given their own academic histories. [CLICK] If the AGI is positive, that is to the right of 0, then we have an indication that the group of students exceeded the standard for PA Academic Growth (i.e., the group of students increased their achievement level or performed higher than expected given their own academic histories). Similarly, if the AGI is negative, then we have an indication that the growth of the group of students did NOT meet the standard for PA Academic Growth (i.e., the group of students did not maintain their achievement level or performed lower than expected given their own academic histories). The vertical axis displays the percent Proficient or Advanced for each group of students. Horizontal Axis Average Growth Index F2F Overview SY14-15 PVAAS Overview

Scatterplot Quadrants and Their Meanings
Quad 2 - Slipping Adequate Achievement, Negative Growth Quad 1 - Excelling Adequate Achievement, Positive Growth Facilitator says: We will now discuss the interpretations of the locations of schools in each quadrant of the scatterplot. If a school’s dot is in the first quadrant, it means that they have accomplished adequate achievement and positive growth. The farther the school’s dot is away from the intersection of the boundaries, the higher the achievement and/or the greater the evidence of growth. Some apply the term “Excelling” to schools in this quadrant. If a school’s dot is in the second quadrant, this indicates that they have adequate achievement but negative growth. This negative growth should be of great concern since continued adequate achievement may be in jeopardy. Some apply the term “Slipping” to schools in this quadrant. If a school’s dot is in the third quadrant, this indicates that the school has both below adequate achievement and negative growth. This situation suggests that substantial investigation and remediation is necessary since the students are not achieving and are headed in the wrong direction. Some apply the term “Underperforming” to schools in this quadrant. If a school’s dot is in the fourth quadrant, this indicates the school does not have adequate achievement; however, their growth is positive. This situation may indicate that interventions implemented are producing positive results. Some apply the term “Improving” to schools in this quadrant. [CLICK] Quad 3 - Underperforming Below Adequate Achievement, Negative Growth Quad 4 - Improving Below Adequate Achievement, Positive Growth F2F Overview SY14-15 PVAAS Overview

Interpreting Scatterplots
Work with your small group to review BOTH scatterplots on the next 2 slides, one for Math and one for Reading. Discuss strengths and weaknesses of both scatterplots. Remember, each “dot” represents a school! We will discuss as a group…“What can you conclude by looking at both?” Facilitator says: Next let’s look at two scatterplots from the same district are displayed (scatterplot A and scatterplot B). Each “dot” represents a school - it is located/placed on the scatterplot based on the achievement (% Proficient+ Advanced) and growth (PVAAS Average Growth Index). Work with your small group to list strengths and weaknesses in each scatterplot. Be prepared to discuss as a group “What can you conclude by looking at both?” [CLICK] F2F Overview SY14-15 PVAAS Overview

Activity: District Scatterplot
MATH Note: Each dot on the scatter plot represents a school from this district. F2F Overview SY14-15 PVAAS Overview

Activity: District Scatterplot Same Schools, Different Subject
READING Note: Each dot on the scatterplot represents a school from this district. F2F Overview SY14-15 PVAAS Overview

Restricted-View Scatterplots
For More Information TWO Virtual Learning Modules Public Scatterplots Restricted-View Scatterplots For more in-depth information on the scatterplots, please view the two virtual learning modules entitled Public Scatterplots and Restricted-View Scatterplots. Simply use the e-Learning link that is available in the blue menu bar on either the public PVAAS site or the password protected PVAAS site. F2F Overview SY14-15 PVAAS Overview

Before continuing, let’s pause and see if there are any questions. F2F Overview SY14-15 PVAAS Overview

LOOKING AHEAD LOOKING BACK
Facilitator says: Now let’s move to the looking ahead or projection side of PVAAS reporting. PVAAS Projections give us insights into how students may perform on future state assessments. [CLICK] F2F Overview SY14-15 PVAAS Overview

What Are Projections? Given a specific set of circumstances…
…what’s the most likely outcome? Facilitator says: So what are projections? They are not a prediction of the future. Rather, projections are a way to express what is most likely to occur under a certain set of conditions. They inform us about what to expect in the future. [CLICK] F2F Overview SY14-15 PVAAS Overview

Setting Expectations How do teachers set expectations for their students? Information What do they know about this student? Experience How have students like this one performed in the past? Facilitator says: It can be useful to think about the PVAAS projections as being very similar to the expectations we set in our everyday lives. Each day, whether we realize it or not, we generate expectations – about our work, our personal lives, and the world in general. These expectations are based on two things: information and experience. [CLICK] Teachers also generate expectations about how students are likely to perform on classroom assignments, tests, and end-of-grade or end-of-course assessments. These expectations are also based on information and experience. Teachers have information about how the student has performed on other assignments and tests. Teachers who have been in the classroom for a number of years also have experience with similar students, or in other words, they know how similar students have performed in previous years. This combination of information about the student and experience with similar students empowers the teacher to generate reasonable expectations. F2F Overview SY14-15 PVAAS Overview

PVAAS Projections What are they based on?
Expectations based on what we know: About this student: Prior test scores Across subjects Facilitator says: The individual student projections that are provided in PVAAS also rely on information and experience. In this case, the information is a student’s testing history, across grades and subjects. [CLICK] F2F Overview SY14-15 PVAAS Overview

PVAAS Projections What are they based on?
About other students: Prior test scores across subjects Scores on the test to which we are projecting Facilitator says: The experience is represented by how similar students have scored on the test we’re projecting to. [CLICK] Using this data, the projection model quantifies the projection in a precise and reliable way. F2F Overview SY14-15 PVAAS Overview

What Are PVAAS Projections?
[CLICK] Given this student’s testing history, across subjects and The results of all similar students who have already taken the test of interest, PVAAS calculates the student’s projection, that is, the most likely score the student will earn on the future exam. Individual Student Projection F2F Overview SY14-15 PVAAS Overview

Student Projection Methodology
What data are used in calculating projections? Projection TO…. Data Used to Calculate Projection PSSA Math PSSA Math and Reading PSSA Reading PSSA Science PSSA Math, Reading, and Science (in grades available) PSSA Writing PSSA Math, Reading, and Writing (in grades available) Keystone content areas (Algebra I, Literature, Biology) Facilitator says: Most important to consider is that the projection methodology uses ALL of the available data in both Reading and Math in its’ projection calculations. This can be done because students’ performance in Reading is highly correlated to their performance in Math (and the reverse is also true). Science data are also used in projection calculations for Science and the Keystone content areas. Likewise, Writing data are also used in projection calculations for Writing. [CLICK] F2F Overview SY14-15 PVAAS Overview

Student Projection Methodology
Which students have projections? In what subjects? Projections are available for all students who have a minimum of 3 data points. The exception to this are projections to grade 4 where only 2 data points from grade 3 are needed. The next two slides detail the grade levels, subjects, and Keystone content areas in which projections are reported. Facilitator says: Projections are available for individual students who have a minimum of 3 data points. The exception to this are projections to grade 4 where only 2 data points from grade 3 are needed. [CLICK] F2F Overview SY14-15 PVAAS Overview

PVAAS Projection Reporting: Projections to PSSA (Basic, Proficient, Advanced)
Math and Reading: 3  4 or 5 4  5 or 6 5  6 or 7 6  7, 8 7  8 Writing: 3  5 4  5 5  8 6  8 Science: 3  4 4  NA Facilitator says: This slide depicts the scheme of projections to PSSA assessments available for students with sufficient data. Note that the projections available are dependent upon the grade level in which the PSSA was last taken, meaning students last taking the PSSA in grades This is important to remember for middle and high schools where they will need to access projections for students last tested in grade levels not in their building. [CLICK] Grade PSSA Last Taken Grade PSSA Last Taken Grade PSSA Last Taken F2F Overview SY14-15 PVAAS Overview

PVAAS Projection Reporting: Projections to Keystones (Basic, Proficient, Advanced)
Algebra I Keystone 5  Algebra I 6  Algebra I 7  Algebra I 8  Algebra I Literature Keystone 5  Literature 6  Literature 7  Literature 8  Literature Biology Keystone 5  Biology 6  Biology 7  Biology 8  Biology The projections that are available are dependent upon the grade level in which students had last taken the PSSA. It is anticipated that if there is sufficient historical PSSA data for a student, projections to Keystones will be available beginning after the student has taken the 5th grade assessment. This is important to remember for high schools where they will need to access projections for students last tested in grade levels not in their building. Note for facilitator: One questions that may come up when discussing projections to Keystones - Do projections for a non-proficient student on a Keystone Exam take into account their previous attempts on that same Keystone Exam? Answer: no, they do not Grade PSSA Last Taken Grade PSSA Last Taken Grade PSSA Last Taken Available for all students who have not yet taken the indicated Keystone exam, OR did not reach proficiency on the Keystone exam F2F Overview SY14-15 PVAAS Overview

PVAAS Student Projection Report
Facilitator says: This is what a Student Projection Report looks like. [CLICK] The red dots show the student’s observed performance in the indicated subject area at grade levels 3, 4, 5, 6 and 7. The yellow square to which the red arrow points indicates the projection for 8th grade Math. This says that based on this student’s history and his/her achievement pattern (taking both Reading and Math into account) the student is projected to be in the 41st state percentile (and marginally likely score in the proficient range) on the 8th grade Math assessment. The calculation of this projection assumes too that the conditions that were in place for the data that are used to create the model continue to be in force. The red, blue and green lines on the graph indicate the percentile boundaries for the Basic, Proficient and Advanced categories for the specified subject area based on the most recent state testing. These values are utilized in the calculation of the probability of a student score being in an indicated category. F2F Overview SY14-15 PVAAS Overview

Facilitator says: The circle highlights the probabilities of this student scoring in each of the following ranges: the probability of scoring in the Basic level or above (89.3%), in the Proficient level or above (64.1%), or in the Advanced level (19.7%). [CLICK] F2F Overview SY14-15 PVAAS Overview

Facilitator says: A table of interpretations and recommendations based on the probabilities are also provided. Please note that some districts are sharing these reports with parents and are using them during teacher/student data meetings. [CLICK] F2F Overview SY14-15 PVAAS Overview

Facilitator says: Please complete analyses of these reports on the next three slides. Work with a partner to: Analyze the report; and Consider possible actions that the report is suggesting. You will have 5 minutes after which we will ask for you to share your insights. F2F Overview SY14-15 PVAAS Overview

[Note to Facilitator: After 5 minutes, have the participants share. Remind them these data should be used with other data such as CDTs, 4Sight, DIBELS, GRADE/GMADE, etc.] F2F Overview SY14-15 PVAAS Overview

PVAAS Projections for Groups of Students
Student Search Facilitator says: If you would like projections for groups of students, you can get them using the Student Search feature. [CLICK] Currently the default setting for all schools is to access students in the grade last tested. Schools also have the option of accessing students by grades currently enrolled if their LEA/district submits currently enrolled data into PIMS. Participants may also want to access projections for students by specific demographic information or projections to specific academic performance levels. PVAAS will search for students meeting those criteria. The Search for Students also allows teachers and administrators to generate a list of students whose probabilities of achieving a selected performance level fall within a specified range of probabilities. You can have as many as four search criteria for each search. F2F Overview SY14-15 PVAAS Overview

Meeting Search Requirements
Identified Students Meeting Search Requirements Facilitator says: When the parameters of the search have been submitted, a report is generated that lists the students that meet the specified criteria outlined in the student search. [CLICK] All demographics for the included students are displayed. Each of the columns are ‘sort-able’ by clicking on the column header. If you have forgotten the criteria used for your selection (the information for each column Prob1 and Prob2) use the key at the bottom of the page or the information in the title of the heading of the report. Each student is listed by name and PASecureID along with their probability of reaching the selected academic performance level on the specified state assessment. F2F Overview SY14-15 PVAAS Overview

How often did students with PVAAS probabilities of 70% or higher score at proficient or advanced?
A document containing more information can be found on the PVAAS Login Page. Facilitator says: We are often asked, “How reliable are PVAAS projections?” The SAS EVAAS team analyzed the data to answer the following question: How often did students with PVAAS probabilities of 70% or higher score at proficient or advanced? Students’ test scores from the school year were compared to their probabilities for achieving proficiency at the beginning of the school year. Students were divided into two groups based on their probabilities at the end of the previous school year. Comparisons were made for students with probabilities less than 70% and for students with probabilities equal to or greater than 70%. The table shown on this slide helps to establish the validity of the individual PVAAS probabilities as an important indicator in developing individualized student academic plans. How accurate are the PVAAS projections? A student’s score from the end of a grade/subject reflects the student’s achievement as well as the educational influence of the district, school and teaching that he or she experienced. The PVAAS probabilities assume students will have the average schooling experience. They do not account for the effectiveness of the teacher in the current year. Rather, they reflect what the student is likely to score based on their own academic history. Highly effective teaching at the school can increase the likelihood that students will reach the proficiency cut. Highly effective schools have value-added results shaded dark blue. When students experience very ineffective teaching at their school, the students are less likely to score as predicted. Students with 40-70% probabilities may not reach proficiency in these learning environments. Very ineffective schools have value-added results shaded red. [CLICK] A full document containing this information can be found on the PVAAS login page under the link What should educators know about PVAAS student probabilities? F2F Overview SY14-15 PVAAS Overview

Projection Summary Reports
What are they? Summarizes the numbers and percentages of student probabilities to a selected performance level on a future state assessment Summary available for projections to a Proficient level or higher, or to an Advanced level Cautions ONE indicator Additional data should be used Facilitator says: PVAAS includes Projection Summary Reports – a report that summarizes the numbers and percentages of students in likelihood ranges of performing at a selected performance level on a future state assessment. The summaries are available for projections to a Proficient level or higher, or to an Advanced level. [CLICK] Users should remember that this report is only ONE indicator about the likelihood of future performance of students. Additional data available on these same students should be used with the PVAAS data for better decision-making. F2F Overview SY14-15 PVAAS Overview

District/School Projection Summary Report
Facilitator says: The District/School Projection Summary Report is displayed on this slide and summarizes the projections of all students at that grade level whose records contain sufficient data to calculate projections. Remember, a Student Projection Report can be viewed for any student who has at least three data points and was last tested in grade 4, 5, 6, 7 or 8. This report can also be viewed for any student last tested in grade 3 who has at least two data points. Note that you may select projections to the next tested grade or in some cases to the grade two years in the future. The number and percentage of students who are likely to be Proficient or higher (or Advanced if that performance level is selected) are summarized as Likely – probabilities between 70% and 100%; Marginally likely – probabilities between 40% and 70%; and Unlikely – probabilities between 0% and 40% Notice also that the number of students in each category is hyperlinked, or is a “hot-key”. This will lead to a complete list of students comprising that probability range. [CLICK] If you choose a Keystone for a Projection Summary, you will see an additional row in the table and an additional slice of the pie chart, colored orange. These data represent the number and percent of student who have already scored in the proficient or advanced performance category for this Keystone content area. In this way the green, yellow and red data represent the projections only for those who need to pass the exam. F2F Overview SY14-15 PVAAS Overview

Facilitator says: Please complete analyses of these reports on the next two slides. Work with a partner to: Analyze the report; and Consider possible actions that the report is suggesting. You will have 5 minutes after which we will ask for you to share your insights. [CLICK] F2F Overview SY14-15 PVAAS Overview

[Note to Facilitator: After 5 minutes, have the participants share. Remind them these data should be used with other data 7th and 8th grade PSSAs, PSAT, SAT, ACT, etc.] F2F Overview SY14-15 PVAAS Overview

Projection Summary  Student List
If you click on any of the hyperlinked number of students, PVAAS will return a list of the student who are projected to be in the selected probability range. [CLICK] The data reporting in this listing is displayed in this table. Data reported for all students in the selected probability range Student Name PAsecureID School Demographic classifications Educational classifications Achievement Probability F2F Overview SY14-15 PVAAS Overview

How Might Educators Use Projection Summary Reports?
Administrators can use for program planning: Intervention Planning Resource Allocation Course Availability & Placement Decisions Grade-level and Course-level Planning School-level Planning Comprehensive Planning Teachers can use to facilitate instructional planning: Proportion of students on a path to success (proficiency or higher) Proportion of students at risk for reaching proficient or advanced levels of performance Plan for instructional strategies and classroom practices based on students’ needs Facilitator says: Projection summary reports can be used for planning purposes by both administrators and teachers, just at a slightly different level. Administrators can use these reports to facilitate program planning: Intervention Planning Resource Allocation Course Availability & Placement Decisions Grade-level and Course-level Planning School-level Planning Comprehensive Planning Teachers can use these reports to facilitate instructional planning: Proportion of students on a path to success (proficiency or higher) Proportion of students at risk for reaching proficient or advanced levels of performance Plan for instructional strategies and classroom practices based on students’ needs [CLICK] F2F Overview SY14-15 PVAAS Overview

Student Projection Report Projection Summary Report
For More Information Virtual Learning Modules Student Projection Report Projection Summary Report For more in-depth information on projections, please view the two virtual learning modules entitled Student Projection Report and Projection Summary Report. Simply use the e-Learning link that is available in the blue menu bar on either the public PVAAS site or the password protected PVAAS site. F2F Overview SY14-15 PVAAS Overview

Another Powerful Projection Report Option…Custom Student Report
A report that provides … Selection of students Controlled access Can be saved so is retrievable and editable F2F Overview SY14-15 PVAAS Overview

Custom Student Report Create a new report and name it
Search for and select students of interest Save the report in PVAAS Add students, remove students, print, and delete options F2F Overview SY14-15 PVAAS Overview

Custom Student Report: Student Projections
F2F Overview SY14-15 PVAAS Overview

Student Search and Custom Student Report
For More Information Virtual Learning Module Student Search and Custom Student Report For more in-depth information on this last feature, please view the virtual learning module entitled Student Search and Custom Student Report. Simply use the e-Learning link that is available in the blue menu bar on either the public PVAAS site or the password protected PVAAS site. F2F Overview SY14-15 PVAAS Overview

Before we wrap up, let’s pause and see if there are any questions. F2F Overview SY14-15 PVAAS Overview

PVAAS Resources PVAAS Overview

PVAAS Virtual Learning Modules
Online, interactive resources Provides real-time professional development Designed in sections – pick & choose based on your needs Various types of VLMs Concept & foundations of measuring growth Understanding and using reports Integrating reports for decision-making Facilitator says: There are many Virtual Learning Modules now available for your own self-guided professional development and learning. These are online, interactive modules that provide users with real-time professional development when it is needed. Each module is divided into various sections so that the user can pick and choose the sections to view based on their needs, or the user can choose to view the module in its entirety. There are several types of Virtual Learning Modules available to increase users understanding and use of PVAAS including: Concept and Foundations of Measuring Growth Understanding and Using Reports Integrating Reports for Decision-Making [CLICK] F2F Overview SY14-15 PVAAS Overview

PVAAS Help Menus F2F Overview SY14-15 PVAAS Overview Facilitator says:
Please be aware that you can access PVAAS Help Menus if you need assistance with any report within the system. The Help Menus have been designed around Frequently Asked Questions (FAQs) that the Core Team and IU staff often receive. They can be printed and used during a team meeting or when you are individually reviewing a report(s). [CLICK] F2F Overview SY14-15 PVAAS Overview

PVAAS Data Packets F2F Overview SY14-15 PVAAS Overview
Facilitator says: PVAAS contains a Data Packet feature that enables users to create packets of reports in PDF format. These packets are designed with the purpose of assisting educators in efficiently retrieving reports needed for district, school, grade, and student level planning. Users are guided through a series of steps to customize the packet, including choosing the packet type, schools to be included, and the grade levels and subject areas for each report included in the packet. [CLICK] F2F Overview SY14-15 PVAAS Overview

PVAAS Data Packets Packet choices include: comprehensive planning
school level planning grade and course level planning district diagnostic school projection summaries student projections Facilitator says: There are many packets to choose from including comprehensive planning, school level planning, grade and course level planning, district diagnostic, school projection summaries, and student projections. [CLICK] F2F Overview SY14-15 PVAAS Overview

Additional PVAAS Supports
Facilitator says: There are a variety of additional supports and resources available on the PVAAS login page…as well as… F2F Overview SY14-15 PVAAS Overview

Additional PVAAS Supports
Facilitator says: …the redesigned PDE PVAAS website. F2F Overview SY14-15 PVAAS Overview

Questions. PVAAS Materials or Statewide Implementation pdepvaas@iu13
Questions? PVAAS Materials or Statewide Implementation PVAAS Report Web Site Facilitator says: Questions related to the statewide implementation of PVAAS or PVAAS materials can be directed to or Remember to go to to access your district and school PVAAS reports. [CLICK] F2F Overview SY14-15 PVAAS Overview

333 Market Street Harrisburg, PA 17126
PENNSYLVANIA Value Added Assessment System ______________________________________________________________________ PVAAS 333 Market Street Harrisburg, PA 17126 PVAAS Overview

PVAAS Overview: Evaluating Growth, Projecting Performance

Similar presentations

Presentation on theme: "PVAAS Overview: Evaluating Growth, Projecting Performance"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

PVAAS Overview: Evaluating Growth, Projecting Performance

Similar presentations

Presentation on theme: "PVAAS Overview: Evaluating Growth, Projecting Performance"— Presentation transcript:

Similar presentations

About project

Feedback