1. 1 PDE Benchmark Initiative An Introduction to the 4Sight Benchmarks September 9, 2005 PaTTAN Pittsburgh

2. © 2005 Success for All Foundation2 Today’s Objectives: Understand the purpose and design of the 4Sight Reading and Math Benchmark Assessments Prepare to administer 4Sight Benchmarks Administration and Scoring Using the SFA Member Center to Analyze Student Data Use 4Sight Data to Drive Instructional Decision Making

3. © 2005 Success for All Foundation3 Why do we monitor student progress? What are some of the ways you monitor student progress and achievement in your classroom or school? How do you use the information/data you obtain? Reflect With a Colleague Reflect With a Colleague

4. © 2005 Success for All Foundation4 Pennsylvania’sAccountabilit Plan Pennsylvania’s Accountability Plan Performance indicators in PA include: Achievement in reading and mathematics. For 2005-2007, the goal is: _____% proficient in reading _____% proficient in math. Also: _____% test participation. _____% graduation rate or improvement

5. A DEQUATE DEQUATE - 45% 54% 63% 72% 81% 91% 100% 200420032002 2004 2003 2002 Y EARLY P ROGRESS READING – % Proficient by Year 200720062005 2007 2006 2005 201020092008 2010 2009 2008 201120122013 2014

6. A DEQUATEDEQUATE - 35% 45% 56% 67% 78% 89% 100 % 200420032002200420032002 Y EARLY P ROGRESS MATH – % Proficient by Year 200720062005200720062005201020092008201020092008 201120122013 2014

7. © 2005 Success for All Foundation7 No Child Left Behind puts special emphasis on closing the A________________ G_______. Targets for achievement apply to ALL s______________. An “Achievement Gap” represents a large d____________ between subgroups. Pennsylvania has one of the largest achievement gaps in the country Pennsylvania has one of the largest achievement gaps in the country. Our Tasks as Instructional Leaders

8. © 2005 Success for All Foundation8 Using Data to Guide Educational Decision-Making Instruction Professional Development Allocation of Resources Policies and Procedures

9. © 2005 Success for All Foundation9 Overview of 4Sight Reading and Math Benchmark Assessments What are they? Who takes them? Why are we doing this? What does it mean for me?

10. © 2005 Success for All Foundation10 What do each of these items have in common? The dip-stick under the hood of a car A blood pressure cuff A weather thermometer 4Sight Benchmarks A Formative Data Tool

11. © 2005 Success for All Foundation11 4Sight Benchmarks A Formative Data Tool Another tool that can help direct our actions is 4Sight Benchmarks. 1 01 0

12. © 2005 Success for All Foundation12 4Sight Benchmarks What are they? Reading and Math Assessments Mirror the look and feel of the PSSA Multiple choice items Open response items Aligned to PA Academic Standards and Assessment Anchors

13. © 2005 Success for All Foundation13 Alignment of 4Sight with State Expectations Take a Closer Look How does the 4Sight Reading and Mathematics Benchmark Assessments mirror the look and feel of the PSSA assessments?

14. © 2005 Success for All Foundation14 Alignment of 4Sight with State Expectations Particularly note: Lengths of passages Types of passages Types of questions (what skill or strategy is being addressed) Types of question stems/or the format of the question

15. © 2005 Success for All Foundation15 PSSA is the Reading Measure in Pennsylvania Student reports include: a) Scale score b) Performance level: advanced, proficient, basic, & below basic c) Reporting categories: Comprehension and Reading Skills (standards 1.1 and 1.2) Interpretation and Analysis of Fiction and Nonfiction Text (standards 1.1, 1.2, and 1.3)

16. © 2005 Success for All Foundation16 PSSA Reading Blueprint beginning 2005 Reporting CategoryGrade 3Grade 5Grade 8 Comprehension and Reading Skills: A.1Fiction A.2Nonfiction 75%70%50% Interpretation and Analysis of Fiction and Nonfiction Text: B.1Components within text or components within and across text B.2Literary Devices B.3Concepts and Organization of Nonfiction Text 25%30%50% Source: Pennsylvania Department of Education

17. © 2005 Success for All Foundation17 Open-Response Reading Items Open-Response Reading Items 1. The 3 point rubric used for PSSA reading is also used with 4Sight. 2. Each open-response item has item specific examples to assist scoring, just as the PSSA. inter-rater reliability 3. Plan for establishing inter-rater reliability. 4. Transfer open-response item scores to answer sheet.

18. © 2005 Success for All Foundation18 Alignment of 4Sight Math Benchmarks with PSSA The Design: Type and variety of questions from all five PSSA Math Reporting Categories Numbers and Operations Measurement Geometry Algebraic Concepts Data Analysis & Probability Multiple choice items Open-ended items

19. © 2005 Success for All Foundation19 Alignment of 4Sight Math Assessments with PSSA Aligned to the PSSA Test Blueprint Each item is matched to a PA Assessment Anchor and its corresponding PA Academic Standard Math Administration and Scoring Guide gives helpful information about skills needed for each item.

20. © 2005 Success for All Foundation20 PSSA Math Blueprint Reporting Category Grade 3Grade 5Grade 8 Number and Operations 42 – 48%41 – 45%18 – 22% Measurement13 – 16%12 – 15% Geometry13 – 16%12 – 15%15 – 20% Algebraic Concepts 13 – 16%13 – 17%25 – 30% Data Analysis and Probability 13 – 16%12 – 15%15 – 20%

21. © 2005 Success for All Foundation21 Open-Response Math Items 1. The 4 point rubric used for PSSA Math is also used with 4Sight. 2. Each open-response item has an item specific rubric to assist scoring, just as the PSSA. inter-rater reliability 3. Plan for establishing inter-rater reliability. 4. Transfer open-response item scores to answer sheet.

22. © 2005 Success for All Foundation22 4Sight Math Math Tools For Each Student Math Tools For Each Student: A ruler that shows sixteenth inches - use same rulers for consistency. (PSSA punchout rulers are usable, if available) Grade-level appropriate calculators. Calculators are not permitted on the first five questions of the test - they are permitted on rest of the test. Word-to-word dictionaries for LEP students.

23. © 2005 Success for All Foundation23 Data Provided by 4Sight 1.Estimate of student performance on the PSSA 2.Diagnostic data for each Subscale - by Assessment Anchors and PA Academic Standards. What will they tell us?

24. © 2005 Success for All Foundation24

25. © 2005 Success for All Foundation25 Subscale Data Subscale Information Provided on 4Sight: Comprehension and Reading Skills Interpretation and Analysis of Fiction and Nonfiction Text 1.1Reading Independently 1.2Reading Critically in All Content Areas 1.3Reading Analyzing and Interpreting Literature Open-Response Items

26. © 2005 Success for All Foundation26 Subscale Data - Math Subscale Information Provided on 4Sight: Numbers and Operations (2.1 and 2.2) Measurement (2.3) Geometry (2.9 and 2.10) Algebraic Concepts (2.8) Data Analysis & Probability (2.6 and 2.7)) Open-Response Items

27. © 2005 Success for All Foundation27

28. © 2005 Success for All Foundation28 Administering 4Sight Benchmarks Who takes 4Sight Benchmarks? Students in grades 3–8 take an on-grade-level assessment. Use the same modifications/exemptions that are used for the PSSA.

29. © 2005 Success for All Foundation29 Administering 4Sight Benchmarks When, Where, How Long Baseline and then quarterly 60-minute testing period Homeroom or other grade-level groups School sets common date and time(s) Directions in Administration and Scoring Guide 12 3 6 9

30. © 2005 Success for All Foundation30 Administering 4Sight Benchmarks Preparing Staff to Administer 4Sight 1.Overview: What is it? Why are we doing it? Who takes it? What does it mean for me? 2.Schoolwide testing and scoring plan 3.Materials: test booklets, answer sheets/teacher score sheets, scoring masks, administration and scoring guide

31. © 2005 Success for All Foundation31 Administering 4Sight Benchmarks Developing Your School’s 4Sight Administration Plan 1.Staff 4Sight orientation: set date and draft agenda. 2.Select dates for first benchmark administration. 3.Decide who will administer the test such as homeroom teacher, etc. 4.Decide who will distribute 4Sight materials.

32. © 2005 Success for All Foundation32 Scoring the 4Sight Benchmarks Multiple-Choice Items 1.Grade 3: transfer student answers from the test book to a Teacher Score Sheet. 2.Align scoring mask over the answer sheet. 3.Mark items correct or incorrect in each of the item’s subscales that are highlighted in that row.

33. © 2005 Success for All Foundation33 Scoring the 4Sight Benchmarks Open-Response Items 1.The 3-point rubric used for PSSA reading is also used with 4Sight. 2.Each open-response item has item-specific examples to assist scoring, just as the PSSA. 3.Plan for establishing inter-rater reliability. 4.Transfer open-response item scores to answer sheet.

34. © 2005 Success for All Foundation34 Inter-Rater Reliability Inter-rater reliability is established when different judges/teachers rate the similar performance or open-ended response items the same way.

35. © 2005 Success for All Foundation35 Establishing Inter-Rater Reliability Inter-rater reliability increases when all raters have been trained in the use of a common rubric. Inter-rater reliability increases when two or more people score each response. At a minimum, teachers should not score their own students if only one rater is used per student test.

36. © 2005 Success for All Foundation36 Scoring the 4Sight Benchmarks Completing the Scoring Total the items correct in each column on the answer sheet for an overall total and the number of items correct in each subscale. Return answer sheets/data to the school’s 4Sight point person. Enter scores into Member Center OR scan completed answer sheets into scanner and upload into the Member Center.

37. © 2005 Success for All Foundation37 Developing a Scoring Plan Who will score the test items? -Multiple choice/multiple mark -Open-response (Is training needed? Will scoring be done in teams? Etc.) -Timeline Who will collect, organize, enter, or upload the data?

38. © 2005 Success for All Foundation38 Member Center Allows schools to: sort and analyze data by: -whole school -grade level -homeroom -reading group -individual student track data over time to assess progress.

39. © 2005 Success for All Foundation39 Member Center web address: https://members.successforall.net Help Line: (800) 548-4998, ext. 2563 Setting Up Your School in the Member Center

40. © 2005 Success for All Foundation40

41. © 2005 Success for All Foundation41 Purpose of Benchmarks “Benchmarks allow you to take a pulse and not an autopsy.” Robert Slavin

42. © 2005 Success for All Foundation42 1. Proficiency Reports 2. Subscale Reports 3. Scaled Scores Let’s Look… Sample Reports Available on the Member Center

43. © 2005 Success for All Foundation43 Sample Proficiency Graph for a 5 th Grade Class

44. © 2005 Success for All Foundation44 Sample Proficiency Table for a 5 th Grade Class

45. © 2005 Success for All Foundation45 Finding the students behind the numbers

46. © 2005 Success for All Foundation46 Sample Subscale of a 5 th Grade Class

47. © 2005 Success for All Foundation47 Sample Subscale Table for a 5 th Grade Class

48. © 2005 Success for All Foundation48 Sample Subscale Graph for a 5 th Grade Class

49. © 2005 Success for All Foundation49 Who are the students behind the numbers?

50. © 2005 Success for All Foundation50 Finding the students behind the numbers

51. © 2005 Success for All Foundation51 Locating individual student data

52. © 2005 Success for All Foundation52 Individual student table

53. © 2005 Success for All Foundation53 Individual student graph

54. © 2005 Success for All Foundation54

55. © 2005 Success for All Foundation55

56. © 2005 Success for All Foundation56 Individual student scaled scores

57. © 2005 Success for All Foundation57 Analyzing Proficiency Reports How many students in this subgroup are proficient? How many students do you need in that subgroup to make AYP? How many more quarters until the PSSA is given? Who are the students just below proficiency? How many points do these students need to move up to be proficient? How many points per quarter?

58. © 2005 Success for All Foundation58 Estimates of Proficiency Correlation data allows you to put students in groups. The group in which you place students allows you to decide if they are secure in hitting the target or if interventions are needed and, if so, at what level of intensity.

59. © 2005 Success for All Foundation59 Analyzing Subscale Data Using the subscales on 4Sight allows us to identify strengths and areas for improvement. +∆+∆

60. © 2005 Success for All Foundation60 Analyzing Subscale Data What are the strengths identified on the graph? What are the areas for concern? If the graph is flat, are the students proficient readers or close to nonreaders?

61. © 2005 Success for All Foundation61 Analyzing 4Sight Data to Inform Instruction Identify what students already know Identify what students need to learn Identify the effectiveness of instruction Identify differentiated student needs

62. © 2005 Success for All Foundation62 Analyzing Subscale Data to Inform Instruction What does the data tell you about: Instruction provided to date? Areas for maintenance-only status? What do students need to learn? How to prioritize areas of need? Other areas for concern?

63. © 2005 Success for All Foundation63 Analyzing and Interpreting 4Sight Data Who owns the 4Sight data? Who owns the challenge of addressing the areas for improvement?

64. © 2005 Success for All Foundation64 Analyzing and Interpreting 4Sight Data How do we involve the entire staff in the data analysis? How do we develop mutual accountability? Where do we find the time? Who makes it happen?

65. © 2005 Success for All Foundation65 Pause and Reflect : Using 4Sight Benchmarks… May help me to… May help my students by… Will give me information on… Will help my grade level team to… Will benefit our building by…