1. Next-Generation MCAS: Update and review of standard setting
Deputy Commissioner Jeff Wulfson
Associate Commissioner Michol Stapel
MCAS Chief Analyst Robert Lee
May 23, 2017

2. Next-Generation MCAS Updates

3. Standard Setting for Next-Generation MCAS

4. Next-Generation MCAS Milestones
November 2015 Board establishes
next-generation MCAS
March Board approves new achievement levels
April-May First administration of next generation MCAS in grades 3-8
August 14-18, 2017 Standard setting panels convened
Massachusetts Department of Elementary and Secondary Education

5. What is standard setting?
Cut score recommendations
(how many points needed to reach each achievement level)
Standard Setting
Massachusetts Department of Elementary and Secondary Education

6. Student Expectations Content Standards
Massachusetts Curriculum Frameworks in English Language Arts (ELA) and Mathematics
Grade- and Subject-Specific Achievement Level Descriptors
Descriptions of the knowledge, skills, and abilities associated with the top three achievement levels
Massachusetts Department of Elementary and Secondary Education

7. Example of Achievement Level Descriptors (Draft) Grade 4 ELA - Writing
Massachusetts Department of Elementary and Secondary Education

8. Logistics Dates: August 14–18, 2017 Location: Danvers, MA
Standard setting meetings: 4 days
Vertical articulation meeting: ½ day
Location: Danvers, MA
Meetings: 6 committees
ELA: Grades 3–4, Grades 5–6, Grades 7–8
Mathematics: Grades 3–4, Grades 5–6, Grades 7–8
Massachusetts Department of Elementary and Secondary Education

9. Committee Composition
Committee Participants
20–24 participants per committee (4-5 table leaders)
Qualifications
Be subject-matter experts
Understand the examinee population and the instructional environment
Be knowledgeable about the curriculum frameworks
Be representative of key stakeholder groups
Approximately equal representation of panelists at each grade level, including several panelists with experience at one grade higher than upper grade
Massachusetts Department of Elementary and Secondary Education

10. Overview of Standard Setting Activities
Plenary session that describes the purpose of standard setting and provides background to panelists and observers
Panelists divide into grade- and subject-specific groups to complete standard setting tasks and make judgments, lead by facilitators
Experience the test; review rubrics, scoring guides, and samples of student work
Develop draft borderline achievement level descriptors
Item judgment and feedback rounds
Table leaders from different grade panels in the same subject come together to look at results across grades and make adjustments as needed to provide a coherent system (“vertical articulation”)
Massachusetts Department of Elementary and Secondary Education

11. Achievement Levels and Cut Scores
Massachusetts Department of Elementary and Secondary Education

12. Panelist Judgment Progress (Extended Modified Yes/No Angoff Method)
Imagine a student with just enough ability to match the achievement level descriptor (“borderline student”)
For each test item, answer the following question:
For each item, the question is answered three times:
Partially Meets Expectations
Meets Expectations
Exceeds Expectations
The number of points the “borderline student” earns on the whole test will represent the cuts at each achievement level
“How many points would a borderline student at the achievement level likely earn if he or she answered the question?”
Massachusetts Department of Elementary and Secondary Education

13. Example of how ratings produce proposed cuts
Massachusetts Department of Elementary and Secondary Education

14. Judgment Rounds Round 1: Panelists make individual judgments
Rounds 2 and 3: Panels refine individual judgments, based on feedback and discussion
Feedback data
Recommended cuts that result from their item judgments (individual and group recommendations)
Impact data (e.g., What percent of students would be Meeting Expectations?)
Rounds of judgments conducted for the upper grade, then the lower grade
Massachusetts Department of Elementary and Secondary Education

15. Vertical Articulation
Examples of considerations during vertical articulation
Looking across grades at the proportion of students at each achievement level
Evaluating areas of disagreement among panelists or between panels
Process
Day 1-Day 4
Panelists and policymakers review impact data after each round
Informs judgments and facilitation during the rounds
Day 5
Table leaders convene to discuss the proposed cuts among all grades in a subject
Table leaders recommend any needed adjustments to cuts, or provide an explanation for remaining differences
Massachusetts Department of Elementary and Secondary Education

16. After Standard Setting
Results are summarized by the contractor and presented to ESE staff and the Commissioner for approval
Each student receives a scaled score and achievement level
Reports printed and sent to parents/guardians (target date: end of October)
Massachusetts Department of Elementary and Secondary Education