1. PVAAS Overview: Evaluating Growth, Projecting Performance PVAAS Statewide Core Team for PDE Fall 2012 1

2. Partners & Teams for Session Activities Some activities throughout the session will be completed in pairs or small groups, while others will be done independently. Take a few minutes now to divide yourselves into: Partners (groups of 2), AND Small groups (groups of 4-6). Fall 2012 2

3. Session Workbook Workbook contains: Key notes from PowerPoint Session activities Yellow boxes on PPT slides indicate the page number(s) you should be referencing in your workbook. Fall 2012 3

4. Goal of Session By the end of the session, participants will be able to answer the following questions: What is PVAAS? What does PVAAS tell us about how students are projected to perform on a future PSSA? How does PVAAS measure growth? What does PVAAS tell us about growth in our district and school(s)? Which reports are most meaningful for school level planning, grade level planning, and student level planning? WB P. 1 Fall 2012 4

5. What Is PVAAS? Fall 20125

6. PVAAS: Is an analysis of existing assessment data Longitudinal PSSA scores in available content areas of Math, Reading, Science, and Writing Provides reliable projections and estimates of growth Is a measure of district’s and school’s effectiveness and growth over time Provides information on the critical relationship of… Growth vs. Achievement WB P. 2 Fall 2012 6

7. PVAAS: Is not another test Is not a comparison of single test scores! WB P. 2 Fall 2012 7

8. In the Perfect World… Calculating Growth Measures would be easy The 3 M’s would not exist Measurement Error Mobility Missing Data We live in the real world! Fall 2012 8 WB P. 3

9. PVAAS Analyses Address the Issues – the 3 M’s! How? Research-based Statistical Modeling Minimizes the effects of measurement error associated with any one test score Measurement Error Accounts for mobility of students Mobility Compensates for missing data Missing Data Fall 2012 9 WB P. 3

10. PVAAS uses advanced statistical methodologies   Multivariate Response Modeling (also known as the Growth Standard Methodology)   Univariate Response Modeling (also known as the Predictive Methodology) Research-based Statistical Modeling Regularly used in other fields, such as psychology, pharmaceutical sciences, health care, and medicine Proven to be precise and reliable Fall 2012 10 WB P. 3

11. Achievement vs. Progress Student Achievement: Where are we? Highly correlated with demographic factors Student Progress: How far have we come? Highly dependent on what happens as a result of schooling rather than on demographic factors By concentrating on the progress students make, PVAAS puts the emphasis on something educators can influence! Fall 2012 11 WB P. 4

12. Demographics and Growth As long as you can sufficiently account for the prior achievement of the student,… WB P. 4 Fall 2012 12 …there is NO significant relationship between demographics and student progress, or growth!

13. Prior Achievement How can we adequately account for prior achievement? What about measurement error in test scores? The SAS, Inc. EVAAS team has found, the only way to adequately account for prior achievement is to use all available test data for each student to dampen the effects of measurement error. WB P. 4 Fall 2012 13

14. In Summary… Is there a relationship between: Demographics vs. Achievement? YES Demographics vs. Growth? NO WB P. 4 Fall 2012 14 PVAAS does not need to adjust for the demographic status of students directly since it is accounting for the students’ prior achievement. In other words, demographics are inherently controlled for in PVAAS by the use of all longitudinal data on all students.

15. Summary and Reflections Work with a partner to complete the activity on page 5 of your workbook. WB P. 5 Fall 2012 15

16. Summary and Reflections Answers WB P. 5 Fall 2012 16

17. Two Types of PVAAS Information Fall 201217

18. PVAAS Analyses PSSA Writing PSSA Science PSSA Reading PSSA Math LOOKING AHEAD Planning for Students’ Needs: Student Projections to Future Tests LOOKING BACK Evaluating Schooling Effectiveness: Value Added & Diagnostic Reports 18 Keystones WB P. 6 TWO Types of PVAAS Information Fall 2012

19. PVAAS Projections Individual Students and Groups of Students Fall 201219

20. How Are Projections Used? 20 Weather Forecasts Population Growth Demand for Electricity Inventory Management Investment Strategies Fall 2012

21. What Are Projections? Given a specific set of circumstances… …what’s the most likely outcome? WB P. 6 Fall 2012 21

22. Setting Expectations How do teachers set expectations for their students? Information o o What do they know about this student? Experience o o How have students like this one performed in the past? 22 Fall 2012 WB P. 6

23. PVAAS Projections What are they based on? Expectations based on what we know: o o About this student » » Prior test scores (PSSA), across subjects o o About other students who have already taken this test (PSSA) » » Their prior test scores (PSSA), across subjects » » Their scores on the test (PSSA) to which we are projecting o o About how much progress students at this school have made in the past in this subject, on average » » Growth Measure from School Value-Added Report 23 Fall 2012 WB P. 6

24. Given this student’s testing history, across subjects… …what is the student likely to score on an upcoming test, assuming the student has the kind of schooling experience this school has provided in the past? What Are PVAAS Projections? WB P. 6 Fall 2012 24

25. Student Projection Methodology What data are used in this methodology? ALL available longitudinal data in both Reading and Math are used in projection calculations for all subject areas. Science data are also used in projection calculations for Science. Likewise, Writing data are also used in projection calculations for Writing. What students have projections? In what subjects? Projections are available for all students who have a minimum of 2 years of historical data. The next slide details the grade levels and subjects in which projections are reported. Fall 2012 25 WB P. 7

26. PVAAS Reporting: Projections to PSSA (Basic, Proficient, Advanced) Grade PSSA Last Taken Math and Reading:  3  4 or 5  4  5 or 6  5  6 or 7  6  7 or 8  7  8  8  11 Writing:  3  5  4  5  5  8  6  8  7  8  8  11 Science:  3  4  4  NA  5  8  6  8  7  8  8  11 Grade PSSA Last Taken Grade PSSA Last Taken WB P. 7 Fall 201226 Projections to Keystone Exams available in fall 2013

27. Information to complete boxes on page 8 in your workbook will be discussed on the next slide. WB P. 8 Fall 2012 27

28. PVAAS Student Projection Report WB P. 8 Fall 2012 28

29. How would you interpret? WB PP. 9-10 Fall 2012 Report A Report B 29

30. PVAAS Projections for Groups of Students Student Search WB P. 11 Fall 2012 30

31. Identified Students Meeting Search Requirements WB P. 11 Fall 2012 31

32. Quality of Projections for Decision Making PVAAS projections have been reviewed and approved by four different peer review panels and the US Government Accountability Office prior to being used as one of Pennsylvania’s Provisions for making AYP. Recent studies have confirmed that the PVAAS projections (even as far as 3 years into the future) are more reliable at looking at the future performance of a student than the most recent PSSA score. Fall 2012 32 WB P. 12

33. How often did students with PVAAS probabilities of 70% or higher score at proficient or advanced? 33 WB P. 12 Fall 2012 A document containing more information can be found on the PVAAS Login Page.

34. Setting Goals and Raising Expectations 34 Using PVAAS projections (along with other data) to set the appropriate goals for all students Fall 2012

35. Projection Summary Reports What are they? This is a report that summarizes the numbers and percentages of students in various likelihood ranges of performing at a proficient level or higher on a future PSSA exam. Cautions This report provides ONE indicator about likelihood of future performance. Additional data should be used with the projection for better decision-making. WB P. 13 Fall 2012 35

36. District/School (Single Grade) Projection Summary Report WB P. 13 Fall 2012 36 Use the Projection tab in the blue menu bar to select projections to different grade levels and subjects.

37. How would you interpret? WB PP. 14-15 Fall 2012 Report B 37 Report A

38. Using PVAAS Projection Reporting Fall 201238

39. How Might Educators Use Projection Summary Reports? Take 3 minutes with your small group to discuss how you, as educators, might use PVAAS Projection Summary Reports. Capture your ideas on page 16 of your workbook. 1. 1.Forecasting the future needs of groups of students for: Instructional Decisions Intervention Enrichment Course Placement Decisions 2. 2.Anticipating staffing/support needs for students WB P. 16 Fall 2012 39

40. Purpose of Projection Summary Reports Administrators can use for program planning: Intervention Planning Resource Allocation Course Availability & Placement Decisions School Improvement Planning Strategic Planning Teachers can use for instructional planning: Proportion of students on a path to success (proficiency or higher) Proportion of students at risk for reaching proficient or advanced levels of performance Plan for instructional strategies and classroom practices based on students’ needs WB P. 16 Fall 2012 40

41. PVAAS Analyses PSSA Writing PSSA Science PSSA Reading PSSA Math LOOKING AHEAD Planning for Students’ Needs: Student Projections to Future Tests LOOKING BACK Evaluating Schooling Effectiveness: Value-Added & Diagnostic Reports 41 Keystones WB P. 17 TWO Types of PVAAS Information Fall 2012

42. PVAAS Value-Added Reporting How is growth measured in PVAAS? Fall 201242

43. PVAAS Value-Added Reporting TestingSubjects – GradesMethodologies PSSA in consecutive years Mathematics, 4-8 Reading, 4-8 Growth Standard PSSA not in consecutive years Writing – 5, 8, 11 Science – 4, 8, 11 Math & Reading – 11 Predictive Keystone Exams (beginning 2013) Algebra I English Lit Biology Predictive Fall 201243 WB P. 17

44. Measuring Growth or Progress Students don’t all start the year at the same place academically Fall 2012 44

45. Measuring Growth or Progress Standard for PA Academic Growth Regardless of their entering achievement level, students should not lose ground academically Is it not a reasonable goal to maintain the achievement level of a group of students? Not necessarily increase their achievement level, but maintain their achievement level Fall 2012 45 WB P. 17

46. PVAAS Growth Measure & Color Descriptors The Growth Measure represents the amount of growth a group of students has made. The Growth Color Descriptors represent the level of evidence we have in the data to say whether or not the group of students increased, decreased, or maintained their achievement levels on average. PVAAS yields a Growth Measure AND a Growth Color Descriptor. Fall 2012 46 WB P. 18

47. PVAAS Value-Added Growth Descriptors Fall 2012 47 WB P. 18 DB 2 SEs or More Above 0 Significant evidence that the district/school exceeded the standard for PA Academic Growth LB 1 to 2 SEs Above 0 Moderate evidence that the district/school exceeded the standard for PA Academic Growth G Between 1 SE Above 0 & 1 SE Below 0 Evidence that the district/school met the standard for PA Academic Growth Y 1 to 2 SEs Below 0 Moderate evidence that the district/school did not meet the standard for PA Academic Growth R 2 SEs or More Below 0 Significant evidence that the district/school did not meet the standard for PA Academic Growth

48. An Analogy for Measuring Growth – Grades 4-8 Math & Reading Fall 2012 48

49. Growth Charts Doctors plot a child’s length, or height, over time. How would we describe Destiny’s growth or progress? Destiny’s Growth Fall 2012 49

50. Prior Achievement for Group Most Recent School Year for Group Higher Achievement Concept of Measuring Growth in PVAAS – Grades 4-8 Math & Reading - Represents the mean achievement level of the group of students Lower Achievement Higher Achievement Lower Achievement Fall 2012 50 WB P. 19

51. Value-Added Reports Fall 2012 51 WB P. 19

52. Measuring Growth at a Deeper Level PVAAS does NOT use the percentages of students at various PSSA performance levels to measure growth. Instead, the use of NCE units are used to estimate growth measures for groups of students. Fall 2012 52 WB P. 20

53. Example of Rescaling Grades 4-8: Math & Reading 1000 1100 1200 1300 1400 1500 Scaled Score Points Fall 2012 53 0 10 20 30 40 50 60 70 80 90 100 NCE Units WB P. 20

54. Value-Added Growth Measure Grades 4-8, Reading and Math Fall 2012 54 0 10 20 30 40 50 60 70 80 90 100 NCE Scale State Mean = 50 Achievement 5 th grade 47.6 Achievement 6 th grade 52.7 PVAAS Growth ≈ Change in position from 5 th to 6 th grade ≈ 52.7 – 47.6 ≈ 5.1 NCE units WB P. 20

55. Complete the boxes on page 21 of your workbook. Fall 2012 55 WB P. 21

56. PVAAS District Value-Added Report Grades 4 through 8 Fall 2012 56 WB P. 21

57. How would you interpret? Fall 2012 57 WB P. 22

58. PVAAS Growth Methodologies TestingSubjects – GradesMethodologies PSSA in consecutive years Mathematics, 4-8 Reading, 4-8 Growth Standard PSSA not in consecutive years Writing – 5, 8, 11 Science – 4, 8, 11 Math & Reading – 11 Predictive Keystone Exams (beginning 2013) Algebra I English Lit Biology Predictive Fall 201258

59. An Analogy for Measuring Growth – Science, Writing, Grades 9-11 Math & Reading Using past performance to predict future performance Fall 2012 59

60. Prior Achievement Higher Achievement Concept of Measuring Growth in PVAAS Science, Writing, Grades 9-11 Math & Reading Lower Achievement Predicted Achievement Mean Predicted Achievement - Represents the mean achievement level of the group of students - Represents the mean predicted achievement level of the group of students Fall 2012 60 WB P. 23

61. Value-Added Reports Fall 2012 61 WB P. 23

62. Measuring Growth at a Deeper Level PVAAS does NOT use the percentages of students at various PSSA performance levels to measure growth. Instead, the use of scaled score points are used to estimate growth measures for groups of students. Fall 2012 62 WB P. 24

63. Answer the questions in the boxes on page 24 in your workbook using the information on the next slide. Fall 201263 WB P. 24

64. Science Value-Added Report WB P. 24 Fall 201264

65. How would you interpret? Fall 2012 65 WB P. 25

66. Can both high achieving and low achieving schools make growth? Fall 201266

67. Schools with Lower Achievement - Represents the mean achievement level of the group of students Is a “green” in PVAAS sufficient for moving students to higher levels of achievement? Higher Achievement Lower Achievement Prior Achievement for Group Most Recent School Year for Group Higher Achievement Lower Achievement Fall 201267 WB P. 26

68. Is a “green” in PVAAS sufficient for moving students to higher levels of achievement? Fall 201268

69. Schools with Lower Achievement Is a “green” in PVAAS sufficient for moving students to higher levels of achievement? Higher Achievement Lower Achievement Prior Achievement - Represents the mean achievement level of the group of students - Represents the mean predicted achievement level of the group of students Predicted Achievement 8 th Grade Science Mean Predicted Achievement for the Group Fall 201269 WB P. 26

70. Is a “green” in PVAAS sufficient for moving students to higher levels of achievement? B A Fall 201270

71. (100% Advanced) Schools with High Achievement - Represents the mean achievement level of the group of students Can this group of high achieving students still meet the standard for PA Academic Growth? Higher Achievement Lower Achievement Prior Achievement for Group Most Recent School Year for Group (100% Advanced) Fall 201271 WB P. 27

72. Value-Added Reports Fall 201272

73. (100% Advanced) Schools with High Achievement Can this group of high achieving students still meet the standard for PA Academic Growth? Higher Achievement Lower Achievement Prior Achievement 11 th Grade Reading (100% Advanced) - Represents the mean achievement level of the group of students - Represents the mean predicted achievement level of the group of students Mean Predicted Achievement for the Group Predicted Achievement 11 th Grade Reading Fall 201273 WB P. 27

74. Value Added Reports Fall 201274

75. Regardless of the entering achievement level of the group… Students should not lose ground academically A reasonable goal would be to at least maintain the achievement level of the group All groups of students can make progress or growth Fall 2012 75 WB P. 28

76. Practice! Turn to a partner Explain the concept of growth 76 Previous School Year Current School Year Higher Achievement Lower Achievement Higher Achievement Lower Achievement Fall 2012 WB P. 28

77. Performance Diagnostic Report Looking Deeper into Growth Fall 201277

78. Performance Diagnostic Report Math, Reading, Science, & Writing Grades 4-8 & Grades 9-11 Fall 201278 WB P. 29 Fill-in the appropriate information on page 29 using the information on the next two slides.

79. Performance Diagnostic Report WB P. 29 Fall 2012 Blue Bar – Current Year Gold Bar – Previous Years Missing Bar – Insufficient Number of Students Whisker – Margin of Error on Growth Value 79

80. PVAAS Performance Diagnostic Growth Descriptors - Interpretation Fall 2012 Zero (0) Line Growth Met the standard for PA Academic Growth Green Did not meet the standard for PA Academic Growth Pink Exceeded the standard for PA Academic Growth Blue What the whiskers tell us… WB P. 30 80

81. Performance Diagnostic Report Fall 2012 81

82. Check for Understanding WB P. 30 Fall 2012 82

83. Performance Diagnostic Report Patterns of Growth Fall 201283

84. WB P. 31 Fall 201284 Patterns of Growth The patterns of growth serve as an indicator of where you are headed in terms of growing students, and where you may have areas of improvement. The meaning of the patterns of growth should be viewed through the context of your achievement goals. What are your achievement goals? As a result, where do you need to make more than a year’s worth of growth?

85. Performance Diagnostic Patterns 85 System benefits lower-achieving students Fall 2012 WB P. 31 Shed Pattern Reverse Shed Pattern System benefits higher-achieving students

86. Performance Diagnostic Patterns 86 Fall 2012 WB P. 31 Tent PatternV Pattern System benefits middle level learners System benefits higher and lower achieving students

87. Performance Diagnostic Patterns 87 System benefits all types of learners except those right below proficiency Fall 2012 WB P. 32 Opportunity Gap Pattern

88. Performance Diagnostic Patterns 88 System benefits all types of learners with more progress occurring with the lower achieving students Fall 2012 WB P. 32 Desirable Pattern

89. Performance Diagnostic Report Growth of Subgroups Fall 201289

90. Performance Diagnostic Subgroup Reports WB P. 33 Fall 201290

91. WB P. 33 Fall 2012 Performance Diagnostic Subgroup Report 91

92. Best to Compare Subgroup to Entire Grade Level Same District/School – Same Grade – Same Subject WB P. 33 Entire grade levelSubgroup Fall 201292

93. Let’s discuss these reports. What conclusions can you make about the growth of different groups of students in 8 th grade Reading? WB P. 34 Entire Grade Level Economically DisadvantagedSpecial Education Fall 2012 93

94. Using PVAAS Growth Reporting Fall 201294

95. How Might Educators Use Value-Added & Diagnostic Reports? Take 5 minutes with your small group to discuss how you, as educators, might use the PVAAS Value-Added and Performance Diagnostic Reports. Fall 2012 95 WB P. 35

96. Purpose of Value-Added & Performance Diagnostic Reports   Assessing curricular needs across grades and subjects   Evaluating instructional needs of students at various achievement levels and in different subgroups   Ensuring instructional coherence/alignment Curriculum Instruction Assessment Supports, including scheduling and staffing needs   Monitoring implementation and effectiveness of grade-level or course-specific programs Fall 2012 96 WB P. 35

97. Achievement and Growth Together! Fall 201297

98. High Achievement Average Achievement Low Achievement Traditional Data Collection Approaches Achievement is based on each student’s ending point. WB P. 36 Fall 2012 98

99. PVAAS Adds Another Variable To Achievement…GROWTH! Below Standard for PA Academic Growth Growth, or progress, is based on each student’s starting point. WB P. 36 Fall 2012 Above Standard for PA Academic Growth 99

100. Take notes on page 36 of your workbook using the information on the next slides WB P. 36 Fall 2012100

101. The Scatterplot Coordinate System Vertical Axis PSSA Percent Proficient or Advanced Horizontal Axis PVAAS Growth Index AYP Percent Proficient Target Standard for PA Academic Growth Boundary WB P. 36 Fall 2012 101

102. Scatterplot Quadrants and Their Meanings Quad 1 - Excelling Adequate Achievement Positive Growth Quad 4 - Improving Below Adequate Achievement Positive Growth Quad 2 - Slipping Adequate Achievement Negative Growth Quad 3 - Underperforming Below Adequate Achievement Negative Growth WB P. 36 Fall 2012 102

103. Web-Based Scatterplots Fall 2012103

104. Web-based Scatterplots Fall 2012 104

105. Web-based Scatterplots Fall 2012 Growth versus Entering Achievement Level Growth versus Percentage of Students Proficient or Advanced Entering Achievement Level versus Percentage of Students Proficient or Advanced Customized based on Demographic Population 105 Users may choose from several different scatterplots!

106. Web-based Scatterplots Fall 2012 Customized Axis Selections include: % Tested Student Population Economically Disadvantaged % Tested Student Population Limited English Proficiency % Tested Student Population Special Education % Tested Student Population Minority 106

107. Interpreting Scatterplots Work with your partner to review BOTH scatterplot A and scatterplot B. Fill-in strengths and weaknesses of both scatterplots (A or B) on page 37 of your workbook. Remember, each “dot” represents a school! We will discuss as a group…“What can you conclude by looking at both?” WB P. 37 Fall 2012 107

108. Example: District Scatterplot WB P. 37 Fall 2012 Note: Each dot on the scatter plot represents a school from this district. 108 A Math

109. Example: District Scatterplot: Same Schools, Different Subject WB P. 37 Fall 2012 Note: Each dot on the scatterplot represents a school from this district. 109 B Reading

110. Let’s Look at the Live Site! https://pvaas.sas.com Fall 2012110

111. PA’s Intent for PVAAS Measure of Growth to Add to Existing Achievement Measures Indicator of Likelihood of Future Student Performance to Add to Current Achievement Data Another Tool for Continuous Improvement Not to Be Used in Isolation WB P. 38 Fall 2012 111

112. PVAAS Resources Fall 2012112

113. NEW! PVAAS Virtual Learning Modules Fall 2012 113 Various types of VLMs Concept & foundations of measuring growth Understanding and using reports Integrating reports for decision-making New online, interactive resource Provides real-time professional development Designed in sections – pick & choose based on your needs WB P. 38

114. PVAAS Help Menus WB P. 39 Fall 2012 114

115. PVAAS Predefined Packets WB P. 40 Fall 2012 115 Packet choices include: comprehensive planning school improvement grade level planning district diagnostic school projection summaries student projections

116. PVAAS Podcasts on iTunesU Fall 2012 WB P. 41 116

117. Additional PVAAS Supports Fall 2012 WB P. 41 117

118. Questions? PVAAS Materials or Statewide Implementation pdepvaas@iu13.org 717-606-1911 PVAAS Report Web Site https://pvaas.sas.com Fall 2012 118

119. www.pde.state.pa.us 333 Market Street Harrisburg, PA 17126 Fall 2012 119