1. 1 New England Common Assessment Program (NECAP) Setting Performance Standards

2. 2 Purpose Provide data to establish the following cut scores for Math, Grades 3-8: –Proficient/Proficient with Distinction –Partially Proficient/Proficient –Substantially Below Proficient/Partially Proficient

3. 3 What is Standard Setting? Set of activities that result in the determination of threshold or cut scores on an assessment We are trying to answer the question: –How much is enough?

4. 4 What is Standard Setting Data collection phase Policy/Decision making phase

5. 5 Many Standard Setting Methods Angoff Body of Work Bookmark

6. 6 Choice of Method is Based on Many Factors Prior usage/history Recommendation/requirement by some policy making authority Type of assessment

7. 7 Choice of Method is Based on Many Factors Weighing all these factors, it was determined that the Bookmark Method would be used for NECAP Math

8. 8 The Bookmark Procedure Well established procedure that has been successfully used on many assessments Has produced defensible results Appropriate for assessments that consist primarily of multiple-choice items but also include some constructed-response items

9. 9 Standard Setting vs. Standards Validation Standard setting –Generally three rounds: Round 1: individual ratings Round 2: table group discussion of Round 1 results, followed by second round ratings Round 3: whole room discussion of Round 2 results, followed by final ratings

10. 10 Standard Setting vs. Standards Validation Standards validation –Starting cut points are provided to panelists at the beginning of the process –Two rounds: Round 1: table group discussion of starting cut points, followed by first round ratings Round 2: whole group discussion of Round 1 results, followed by second round ratings The process you will be following will be more of a standards validation process

11. 11 Details for Standards Validation using the Bookmark Procedure

12. 12 What is the bookmark procedure? A standard setting procedure that uses a book of items (ordered from easiest to hardest) Panelists place bookmarks in that book of items

13. 13 W hat is the bookmark procedure?

14. 14 What is the bookmark procedure? For purposes of NECAP standard setting, you will be provided with starting cut points and will either validate those cuts, or recommend modifications to them

15. 15 Starting Cut Points Starting cuts were determined using teacher judgment data: –At the time of testing, teachers were asked to categorize each student into an achievement level category, based on classroom performance (not test performance) –Using these categorizations as well as the students’ test scores, Measured Progress staff calculated starting cut points

16. 16 Starting Cut Points Analyses of teacher judgment data indicate: –Good participation rates –Teachers appear to have been conscientious in completing the task In other words, available evidence supports the validity of the starting cuts; therefore…

17. 17 Starting Cut Points You are free to recommend changing the starting cuts, but we expect that changes will be relatively minor You should have justification for any changes you recommend –rationale for changes should be based on the Achievement Level Descriptions and the test items

18. 18 Starting Cut Points Note that, in some cases, the lowest starting cut appears very early in the Ordered Item Booklet What this means is that the test was difficult for these students: –for several of the math tests, borderline Partially Proficient students (as identified by the teacher judgment data) have less than a two-thirds chance of answering even the easiest items correctly

19. 19 Starting Cut Points What you need to know is that the ordered item cut point for a given cut does not equal the raw score a student must obtain to be categorized into the higher achievement level For example, the Substantially Below Proficient/ Partially Proficient starting cut for grade 7 math is between ordered items 3 and 4; however, a student must obtain at least 20 points on the test in order to be classified into the Partially Proficient level

20. 20 How to Place a Bookmark A few concepts you will need to know: –The starting cut points –The achievement level descriptions –‘Borderline’ students –What knowledge, skills, and abilities (KSAs) are needed to answer each question

21. 21 How to Place a Bookmark Start at the beginning of the ordered item book Evaluate whether at least 2 out of 3 students demonstrating skills at the ‘borderline’ of Partially Proficient would correctly answer item 1 Moving through the book, make this evaluation of each item The bookmark should go where you no longer think 2 out of 3 Partially Proficient ‘borderline’ students would correctly answer the question. You may decide that the starting cuts are in the right place, or you might recommend moving them.

22. 22 How to Place a Bookmark Item Number Would at least 2 out of 3 students who demonstrate skills at the Partially Proficient 'borderline' correctly answer this question? 1Yes 2 3 4 5 6 7 8 9No 10No 11No 12No 13No 14No 15No …

23. 23 How to Place a Bookmark In the example, the bookmark would go between items 8 and 9 However, it won’t be that easy; there will be gray areas You will have the opportunity to discuss your bookmark placements and change them if desired Place one bookmark for each cut score

24. 24 How to Place a Bookmark To place your bookmarks you will need to be familiar with the achievement level descriptions and the assessment items

25. 25 How to Place a Bookmark Don’t worry, we have procedures, materials and staff to assist you in this process.

26. 26 Any questions about the Bookmark Procedure?

27. 27 What Next? After this session, you will break into grade-level groups, where you will: –take the assessment to familiarize yourself with the test items –discuss the Achievement Level Descriptions and develop definitions of “borderline” Partially Proficient, Proficient, and Proficient with Distinction students

28. 28 What Next? You will: –complete the Item Map, which is a document that will help you with the bookmark placement process –discuss the starting cut points in table groups and do the first round of bookmark placement –discuss the first round bookmark placements as a whole room then do the second round of bookmark placement

29. 29 What Next? After the second round of bookmark placement, you will have an opportunity to provide feedback about the Achievement Level Descriptions As the final step, we will ask you to complete an evaluation of the standard setting process

30. 30 Good Luck!