1. Standard Setting: Grade 3 Mathematics Sheraton Four Points Hotel Norwood, MA August 15-16, 2007 Massachusetts Comprehensive Assessment System (MCAS)

2. Wednesday, August 15 Overview of Plenary Session  Welcome/Introductions  Overview of MCAS Program  Purpose of 2007 Standard Setting  Body of Work Method and Procedures  Ground Rules for Standard Setting  Agenda (Wednesday-Thursday)

3. Department of Education  Bob Bickerton, Associate Commissioner  Wayne Fernald, MCAS Mathematics Lead Developer  Haley Freeman, MCAS Mathematics Development Specialist  Mark Johnson, Director of MCAS Test Development  Bob Lee, MCAS Chief Analyst  Matt O’Connor, Administrator for Administration, Analysis and Reporting  Kit Viator, Director of Student Assessment

4. Measured Progress  Sally Blake, MCAS Lead Developer, Mathematics  Lee Butler, Administrative Assistant  Lisa Ehrlich, Assistant Vice President  Kevin Haley, Manager of Data Analysis  Renee Jordan, Service Center Representative  Mark Peters, Program Assistant  Miechelle Poulin, Program Assistant  Michael J. Richards, Program Manager  Kevin Sweeney, Assistant Vice President, Research & Analysis  David Tong, Assistant Director, MCAS Program Management  Eric Wigode, Director of MCAS Test Development

5. Standard Setting Facilitator  Sally Blake

6. Welcome Grade 3 Mathematics Panelists Karen AndersonAssociate Professor & Chair, Education Dept.Stonehill College Nancy BuellElementary Mathematics SpecialistWilliam H. Lincoln School Bruce CarterCase ManagerUrban League of Eastern Mass. Robert Cote3 rd Grade Classroom TeacherJordan/Jackson Elementary Linda Gauthier **Curriculum CoordinatorSaugus Public Schools Cheryl Goguen **Grade 4 General EducatorMiriam F. McCarthy School Rebecca Gutierrez4 th Grade TeacherNewton Elementary School Steven KaczmarczykSpecial Education TeacherEllen Bigelow School Kristine Klumpp **Grade 3 TeacherAlden Elementary School Carol LaPolice **Math Instructional Leadership Specialist-ElementaryDaniel B. Brunton School Marlena McCoyGrade 4 TeacherMittineague Elementary School Elaine McNamaraTitle I Director and TeacherParker Avenue School Lyudmila Moiseyeva **ELL TeacherBaker Elementary School Judy Moore **Grade 3 TeacherHarvard Elementary School Stephanie Morris **Grade 4 TeacherCraneville School Judith RichardsMathematics TeacherGraham & Parks School Jennifer Rubera **Grade 4 TeacherPentucket Lake Elementary Michael Stanton **PrincipalBoyden Elementary School Deborah StewartCommunity RepresentativeUrban League Elizabeth Sweeney **Assistant Program DirectorBoston Public School Denise Young **Grade 3 TeacherBrown School **Served on 2006 panel

7. Historical Background of the MCAS Tests First MCAS operational tests introduced (ELA, Math, and Science & Technology, grades 4, 8, and 10) 1998 NCLB requires states to annually test reading & math in grades 3-8 Grade 3 Math test administered 2006 Grade 3 Reading, grade 6 Math, and grade 7 ELA tests introduced 2001 Massachusetts Education Reform Law passed 1993 Class of 2003 first graduating class required to earn a CD (ELA and Math) 2003 Grade 3 Math standard setting 2006 Grade 3 Math standard setting revisited 2007

8. Purpose of MCAS Program   Inform/improve curriculum and instruction   Evaluate student, school, and districtperformance according to Curriculum Framework content standards and MCAS performance standards   Certify eligibility for high school Competency Determination (CD)

9. Selected Features of MCAS  Custom developed based on Massachusetts Curriculum Framework content standards and MCAS performance standards  100% of questions used to determine student scores released annually  Measures performance of ALL students educated with public funds  Results reported according to raw scores and performance levels

10. Overview of 2006 Standards Setting Event and Outcomes Cut scores successfully established at Warning/Needs Improvement and at Needs Improvement/Proficient Some panelists expressed concern about whether any test questions existed at the Above Proficient level; cut score at Proficient/Above Proficient set at 40 (out of 40) 2007 test designed to have sufficient questions at Above Proficient level

11. Purpose: 2007 Grade 3 Mathematics Standard Setting Primary purpose: Establish a cut score at Proficient/Above Proficient Secondary purpose Validate cut scores at Warning/Needs Improvement and Needs Improvement/Proficient

12. Standard Setting vs. Standards Validation  Standard setting (top cut point) –Process of establishing original cut scores –Panelists are not provided initial cut points  Standards validation (bottom two cut points) –Process of validating cut scores –Panelists are provided initial cut points

13. 2007 Standard Setting/Validation WarningNeeds Improvement Proficient Cut score to be validated Cut score needed Cut score to be validated Above Proficient

14. Development of Content Standards Supplement to the CF was created, pulling out specific content standards for grades 3, 5, and 7; no “brand-new” standards were written 2000Mathematics Curriculum Framework content standards written for grade spans (e.g., grades 5-6 and grades 7-8) 2004

15. Content Standards vs. Performance Standards  Content standards = “What” Describe the knowledge and skills students should acquire in a particular content and grade  Performance Standards = “How well” Describe student work on MCAS tests at the Needs Improvement, Proficient, and Above Proficient levels

16. General MCAS Performance Level Descriptors Needs Improvement Students at this level demonstrate partial understanding of subject matter and solve simple problems Proficient Students at this level demonstrate a solid understanding of challenging subject matter and solve a wide variety of problems Above Proficient Students at this level demonstrate a comprehensive and in-depth understanding of rigorous subject matter, and provide sophisticated solutions to complex problems

17. Linking Performance Standards with Student Work  What is standard setting? Establishment of cut scores to distinguish between performance levels  What is your job? Use the PLDs to evaluate student work and make recommendation for Proficient/Above Proficient cut score

18. Purpose of Standard Setting   Determine cut scores for reporting assessment results   Answer the question: – –How much is enough?

19. General Phases of Standard Setting/ Standards Validation  Data-collection phase  Policy-making/decision-making phase

20. Standard-Setting Methods  Angoff  Bookmark  Body of Work

21. Choosing a Standard-Setting Method  Prior usage/history  Recommendation/requirement by policy-making authority  Type of assessment Body of Work method chosen for MCAS test in Grade 3 Mathematics

22. What is the Body of Work Procedure? Panelists examine student work (actual responses to test questions) and make a judgment regarding the performance level to which the student work most closely corresponds. Top cut Standard Setting: Panelists examine student work that has not been previously classified and determine how that work should be classified. Lower cuts Standards Validation: Panelists examine student work that has been initially classified into a performance level based on starting cut points and determine if they agree with these classifications or recommend changes to them.

23. Initial Classification of Student Work Initial classification of student work in grade 3 mathematics based on 2006 test results. Step 1: Step 1: Equate the 2007 grade 3 mathematics test to the 2006 test. Step 2: Step 2: Find the raw score cuts on the 2007 form that are equivalent to the cut points established in August 2006. Step 3: Step 3: Select student work with scores ranging from very low to very high; classify them into performance levels based on preliminary cut points found in Step 2.

24. Selected Student Work Example Distribution of Selected Student Work: Grade 3 Math WarningNeeds ImprovementProficient Above Proficient XXXX XXXXXX XXXXXXXXXXXXXXXXXX XXXXXX XXXXXXXXXXXXXXXXXXXXX

25. How to Classify Student Work  Performance Level Definitions General Grade and content specific Materials you will need:   Bodies of Student Work Responses to constructed-response questions Multiple-choice summary sheet   Rating Forms

26. How to Classify Student Work  Examine the student’s responses to multiple-choice questions  Examine the student’s responses to open-response questions  Judge the student’s knowledge and skills demonstrated relative to the PLDs  Panelists do not need to reach consensus on the classifications

27. How to Classify Student Work  Grade 3 mathematics test  General MCAS and grade 3 math Performance Level Descriptors  Bodies of student work Responses to multiple-choice items AND constructed-response items To help prepare you to do these ratings, you will spend time becoming familiar with the following:

28. How to Classify Student Work  You will have the opportunity to discuss your classifications and change them if desired.  Don’t worry! We have procedures, materials, and staff to assist you in this process.

29. What Next?  Take the assessment  Complete the Item Map  Discuss the Performance Level Definitions  Complete training round  Complete individual ratings  Receive feedback from first round of ratings  Discuss feedback and provide final ratings  Complete an evaluation form

30. Top 8 Most Misunderstood Things about Standard Setting 5.We should use this time to rework Math performance level definitions. 8.Standard setting is a great opportunity to rewrite Curriculum Framework standards. 7.The process is rigged. 6.This is a good time to vent about all the things you hate about MCAS.

31. Top 8 Most Misunderstood Things about Standard Setting 1.Disagreement is bad. 4.Standard setting is scoring. 3.Only Mathematics scholars should be doing this work. 2.Only teachers should be doing this work.

32. Ground Rules  Role of facilitator is to “facilitate” and keep process on track  Process solely focused on recommending performance standards (cut scores) for MCAS  MCAS performance level definitions are integral to process but are not up for debate  Panelists’ recommendations are vital; however, final cut scores determined by the MDOE  Each panelist must be in attendance for the duration of the process for his/her judgments to be considered  Each panelist must complete evaluation form at the end of the event  Cell phones off, please!

33. Agenda Wednesday, August 15 Breakfast 8:00 am – 9:00 am Work session 9:00 am – 12:00 pm Lunch12:00 pm – 1:00 pm Work session 1:00 pm – 4:00 pm Thursday, August 16 Breakfast8:00 am – 9:00 am Work session9:00 am –12:00 pm Lunch12:00 pm – 12:45 pm Work session 12:45 pm – Until completion

34. Room Assignment Grade 3 Math – 105/106

35. Questions?