PVAAS Overview: Evaluating Growth, Projecting Performance PVAAS Statewide Core Team Fall 2008

–What do you think you already know about PVAAS? –What do you want to find out about PVAAS today? Goal: PVAAS will be used by teachers and administrators for annual school evaluation and planning, grade/subject level evaluation and planning, and for student level planning.

PDE Provides The Data Tools… PA AYP Public website that reports all AYP-related data on district, school, grade and subgroup. -- Subject area information is also available. PSSA Data Interactive by eMetric Interactive web-based tool for analyzing PSSA data by district, school, grade, subgroup and student. -- Reporting category information is also available. PVAAS Web-based value-added tool that reports growth of cohorts of students and projections for individual students for performance on future PSSA assessments. 4Sight Member Center Interactive web-based tool for benchmark assessment data by grade, class, subgroup and student. -- Intended to be formative and used often throughout the school year.

PDE Provides The Framework… “Getting Results!™” A Framework for Continuous School Improvement Planning The framework provides guiding questions for root cause analyses in the following data environments: –PSSA and AYP inquiry –PVAAS –4Sight Benchmark Data –Local Assessment Data Sets

Review of Literature While there is a relationship between achievement and demographic variables, there is NO relationship between growth and demographic factors, such as socioeconomic status and racial/ethnic background. --VAAS can remove the effects of factors not under the control of the school. (McCaffrey, Lockwood, Koretz & Hamilton, 2003; Ross, Wang, Sanders, Wright & Stringfield, 1999a; Wright, Horn & Sanders, 1997).

What Is PVAAS?

PVAAS IS : A statistical analysis of existing assessment data. A realistic measurement of a school’s progress/growth over time. Progress data to add to achievement data as schools are analyzing their work with students and making improvement plans.

PVAAS: Is not another test. Is not being used for teacher- level accountability in Pennsylvania. Does not use teacher-level data elements. Does not yield a metric on the effectiveness of a teacher.

High Achievement Average Achievement Low Achievement Current Data Collection Approaches Achievement is based on each student’s ending point.

PVAAS Adds Another Variable To Achievement…Progress Below Growth Standard At or Above Growth Standard Progress is based on each student’s starting point.

The Scatter Plot Coordinate System Vertical Axis PSSA Percent Proficient or Advanced Horizontal Axis PVAAS Growth Measure AYP Percent Proficient Target Growth Standard Boundary

Scatter Plot Quadrants and Their Meanings Quad 1 Adequate Achievement Positive Growth Quad 4 Below Adequate Achievement Positive Growth Quad 2 Adequate Achievement Negative Growth Quad 3 Below Adequate Achievement Negative Growth

Example: Regional Scatter Plot

Example: Regional Scatter Plot: Same Schools

The TWO PVAAS Methodologies Today Looking Back/Evaluation… Value-added Growth Reports For Cohorts of Students Looking Forward/Planning… PVAAS Projection Reports For Individual Students and Cohorts of Students NO methodology to consider teacher effectiveness.

PVAAS Fall 2008 Reporting Value-added growth reports in –Mathematics (grades 4 through 8 and 11) –Reading (grades 4 through 8 and 11) –Science (only 4, 8, and 11) –Writing (only 5, 8, and 11) Projections to basic, proficient, advanced  3  4 or 5  4  5 or 6  5  6 or 7  6  7 or 8  7  8  8  11 Grade Last Tested

Student Projections Wouldn’t it be great to know the likelihood that a student will be proficient on a future PSSA?

Student Projection Methodology A separate longitudinal modeling process focused on the performance scores of individual students. The model uses all of the available data (both math and reading) to project future PSSA performances in Math, Reading, Science and Writing. Projections can be calculated for an individual student or groups of students.

PVAAS Student Projection Report

PVAAS Projection Reports for Groups of Students by Subgroup and/or Projected Proficiency Level

Identified Students Meeting Search Requirements

Projection Summary Reports What is it? What is it? Report that summarizes the numbers and percents of students in likelihood ranges of performing at the proficient level on a future PSSA exam.Report that summarizes the numbers and percents of students in likelihood ranges of performing at the proficient level on a future PSSA exam. How might a school use this report? How might a school use this report? Intervention PlanningIntervention Planning Resource AllocationResource Allocation Strategic PlanningStrategic Planning School Improvement PlanningSchool Improvement Planning Cautions Cautions This report provides another indicator about likelihood of future performance.This report provides another indicator about likelihood of future performance.

Grade Projection Summary Report

School Projection Summary Report

District Projection Summary Report

Turn to someone next to you… 1.What is a PVAAS Projection? 2.What are some examples (2 or 3) of how PVAAS Projections can be used in your school?

Wouldn’t it be great to know if you made at least one year’s growth with students last tested in grades 4-8? Value-Added

Value-added Measure of Cohort Growth Grades 4 through 8 in Reading and Math Comparison to a State Growth Standard The Growth Standard specifies the minimal acceptable academic gain from grade to grade for a cohort of students. The use of a Growth Standard creates the possibility that ALL schools can demonstrate appropriate growth.

The Assessment Dilemma True Student Achievement Any test is just a snapshot in time!

Four Important Issues 1. Measuring Gain 2. Growth Standard 3. Comparing Scores from year to year 4. Estimating Achievement Difference in scores in consecutive years. Combine current observed scores with all previous scores in students’ histories. Convert all scores to same scale – NCE scale Student NameQuiz 1Test 1Quiz 2Project 1Test 2Test 3 Cum Possible Percent Joe Betty Elizabeth

PVAAS Value-added Cohort Growth Descriptors – Grade 4 – 8 Favorable Indicator Estimated gain at or above growth standard. Students in this cohort have made at least one year growth. All schools can achieve this rating. Caution Indicator Estimated gain below growth standard but by less than one standard error. Students in this cohort have grown less than the standard. Stronger Caution Estimated gain below growth standard by more than one but less than two standard errors. Students in this cohort have fallen behind their peers. Strongest Warning Estimated gain well below growth standard by more than two standard errors. Students in this cohort have made little progress.

PVAAS School Report Grades 4 through 8 Every grade in the school has met or exceeded a year’s worth of growth? (“Getting Results!”™, p. 12)

Your Turn! Interpret this report!

PVAAS School Report High School Based on a predictive methodology A predicted score is calculated or each student; A predicted score is calculated or each student; Mean of predicted scores is calculated for students in a cohort; Mean of predicted scores is calculated for students in a cohort; Mean of observed scores of students in the cohort; Mean of observed scores of students in the cohort; School Effect is the difference between the Observed and the Predicted Scores. School Effect is the difference between the Observed and the Predicted Scores.

11 th Grade Growth Descriptors Below Observed performance is significantly BELOW the expected performance. NDD Observed performance is NOT DETECTABLY DIFFERENT than the expected performance. Above Observed performance is significantly ABOVE the expected performance.

School Report

PVAAS Performance Diagnostic Report The Key is the RED Whisker!

What the whiskers tell us… Growth Standard (One Year’s Growth) Met the Growth Standard; Made One Year’s Growth Yellow Below Growth Standard; Less than One Year’s Growth Rose Exceeded Growth Standard; More than One Year’s Growth Green

Every performance levels (below basic, basic, proficient, advance) in each grade has met or exceeded a year’s worth of growth? (“Getting Results!”™, p. 12) Performance Diagnostic Report

Greater Detail – based on % of students

Click on Hot Keys for lists of students.

Student Lists from Performance Diagnostic Reports

Custom Diagnostic Report What is it? What is it? Procedure to examine subgroup growth patterns based on user-defined educational criteria for groups of 15 or more students.Procedure to examine subgroup growth patterns based on user-defined educational criteria for groups of 15 or more students. How might a school use this report? How might a school use this report? Explore effects of intervention programs, etc.Explore effects of intervention programs, etc. Explore effects of varied curricular and/or instructional experiences.Explore effects of varied curricular and/or instructional experiences. Cautions Cautions This report does not infer any causal relationships between any educational variables and student growth.This report does not infer any causal relationships between any educational variables and student growth. This report should NOT be used to estimate teacher effect on learning.This report should NOT be used to estimate teacher effect on learning.

Custom Diagnostic Report Selection Screen

Custom Diagnostic Report

Turn to someone next to you… What are some examples (2 or 3) of how PVAAS Value-added and Performance Diagnostic Reports can be used in your school?

PA’s Intent for PVAAS Another Tool for Continuous Improvement Measure of Growth to Add to Achievement/Status Measures Not to Be Used in Isolation No Teacher-Level Analyses

Additional Support Account Management for any assistance needed regarding your PVAAS account or Professional Development For assistance with professional development, interpretation and use of PVAAS reports, contact your local Intermediate Unit PVAAS contact. If additional support is needed, your local IU will contact the PVAAS Statewide Core Team and/or the supporting PaTTAN PVAAS team.

Additional Professional Development Each IU has a PVAAS contact Fall Professional Development –21 statewide webinars –38+ days of hands-on professional development at IUs –Follow-up support Professional Development Materials –PDE website

Questions: PVAAS Materials or Statewide Implementation PVAAS Report Web Site

Questions: Professional Development, Support & Technical Assistance IU ## PVAAS Team Contact: XXXXXXX PHONE

Gerald L. Zahorchak, D.Ed. Secretary of Education Commonwealth of Pennsylvania Market Street Harrisburg, PA 17126