1. MCAS Standards Validation: High School Introductory Physics Sheraton Hotel Braintree, MA September 17-18, 2007

2. Overview of August Standards Setting  Independent standard-setting for: Biology Chemistry Introductory Physics Technology/Engineering  Cut scores recommended for: Failing/Needs Improvement Needs Improvement/Proficient Proficient/Advanced  Each panel composed of: high school science teachers and administrators university representatives community representatives

3. Post Standard-Setting Analysis Analyzed each of the recommended cut scores (n=12) across all four content areas Validated 10 of 12 recommended cut scores Issues related to 2 of the recommended cuts: –Raw score ranges associated with two cuts for F/NI and NI/P for Introductory Physics were extreme outliers –Some Introductory Physics panelists expressed concern about standard-setting process

4. Plenary Session  Welcome  Overview of August Standard Setting Meeting  Post Standard Setting Analysis  Standard Setting versus Standards Validation  STE tests: efforts to establish “equivalence” and uses of test results  Recap of Body of Work (BOW) method and role of performance level descriptors  Questions and Answers

5. Additional Analyses Conducted  Relationship of projected cut scores to psychometric properties of test (TCCs and TIFs)  Relationship of previous student performance in science with that of IP test based on recommended cut scores  Relationship of student work classified as Needs Improvement and Proficient to: Introductory Physics performance level descriptors student work classified as Needs Improvement and Proficient for Biology, Chemistry, and T/E

6. Conclusions Post Standard-Setting Interpretation of Needs Improvement and Proficient diverged significantly from interpretation made in the other content areas Considered statistical adjustment as remedy but determined Standards Validation preferable given high stakes use of results Delay in posting of scaled score/raw score conversion tables for all four content areas until standards validation completed

7. Remember, we knew this was going to be challenging…  Standard setting for multiple tests–-where roughly “comparable” standards across all tests is the goal— is technically challenging  Other approaches to standard setting considered:  Overlapping content panels  One panel composed of all representatives for all four content areas Independent panels

8. General Phases of Standard Setting  Data-collection phase  Policy-making/decision-making phase

9. Standard Setting vs. Standards Validation  Standard setting –Process of establishing original cut scores –Panelists are not provided initial cut points or focused cut point range  Standards validation –Process of validating cut scores –Panelists are provided initial cut points or a narrowed cut point range

10. Introductory Physics Standards Validation WarningNeeds Improvement Proficient Cut score needed Cut score established Cut score needed Advanced

11. Standards Validation  Bodies of work to be classified represent a narrowed range of student work based upon:  Standard setting panelists’ recommendations (August)  Psychometric and statistical analyses  Analysis of classification of student work for each of the 12 recommended cut scores (conducted by DOE and Measured Progress content experts)  Pinpointing cut score for Failing/Needs Improvement Needs Improvement/Proficient

12. Massachusetts High School Competency Determination (CD) Requirements: Class of 2010 Meet local graduation requirements Attain scaled score of 220 or higher on high school MCAS tests in ELA, mathematics, and STE (or equivalent on MCAS Alternate Assessment) To earn diploma, a student must:

13. Efforts to Establish “Equivalence” of High School STE Tests The High School STE tests:  are based on parallel test design and development processes  are based on performance level descriptors that are comparable in scope and rigor  have been designed to have similar psychometric properties

14. External Validation of Efforts to Establish “Equivalence” of STE Tests  Performance level descriptors used in standard setting have been externally validated by Massachusetts teachers  Technical/psychometric properties analyzed by Dr. Ronald Hambleton, University of Massachusetts, Amherst  Test design and plans for standard setting endorsed by the MCAS National Technical Advisory Committee (TAC)

15. What is the Body of Work Procedure? Standard Setting method where panelists:  examine student work (actual responses to test questions) and performance level descriptors  make a judgment regarding the performance level to which the student work most closely corresponds.

16. Body of Work Fundamentals  Examine the student’s responses to multiple-choice questions  Examine the student’s responses to open-response questions  Judge the student’s knowledge and skills demonstrated relative to the PLDs  Panelists do not need to reach consensus on the classifications

17. Materials Used During Standards Validation  Performance Level Descriptors General Content specific   Bodies of Student Work Responses to constructed-response questions Multiple-choice summary sheet   Rating Forms

18. General MCAS Performance Level Descriptors Needs Improvement Students at this level demonstrate partial understanding of subject matter and solve simple problems. Proficient Students at this level demonstrate a solid understanding of challenging subject matter and solve a wide variety of problems. Advanced Students at this level demonstrate a comprehensive and in-depth understanding of rigorous subject matter, and provide sophisticated solutions to complex problems.

19. Needs Improvement On MCAS, a student at this level Proficient On MCAS, a student at this level Advanced On MCAS, a student at this level Conceptual Understanding and Factual Knowledge  Demonstrates a partial understanding of some facts, concepts, principles, and theories  Uses basic scientific terms  Demonstrates a solid understanding of many facts, concepts, principles, and theories  Uses appropriate scientific terms  Demonstrates a comprehensive, in-depth understanding of many facts, concepts, principles, and theories  Applies scientific terms in an appropriate context Scientific Process and Skills  Interprets simple data and creates generalized questions without necessarily using the scientific method  Identifies a problem to be solved  Uses most of the steps of the scientific method to design and interpret experiments  Finds solutions to a variety of problems  Designs and evaluates scientific experiments and generates full interpretations of data  Finds solutions to complex problems Application/ Synthesis  Makes simple predictions about a specific topic  Makes predictions based on information given  Justifies predictions in a general sense  Makes sophisticated predictions  Synthesizes a wide array of information from multiple sources  Applies knowledge to abstract or novel situations General STE Performance Level Descriptors

20. What Next?  Take the Introductory Physics test  Discuss the Performance Level Descriptors  Complete the Item Map  Complete training round  Complete an evaluation form  Complete individual ratings  Receive feedback from first round of ratings  Discuss feedback and provide final ratings  Complete final evaluation form

21. Ground Rules  Role of facilitator is to “facilitate” and keep process on track  Process solely focused on recommending performance standards (cut scores) for MCAS  MCAS performance level descriptors are integral to process but are not up for debate  Panelists’ recommendations are vital; however, final cut scores determined by the MDOE  Each panelist must be in attendance for the duration of the process for his/her judgments to be considered  Each panelist must complete evaluation form at the end of the event  Cell phones off, please!

22. Agenda Monday, September 17 Breakfast8:00 am – 9:00 am Plenary9:00 am – 10:30 am Break10:30 am – 10:45 am Work session10:45 am –12:00 pm Lunch12:00 pm – 1:00 pm Work session 1:00 pm – 5:00 pm Tuesday, September 18 Breakfast 8:00 am – 9:00 am Work session 9:00 am – 12:00 pm Lunch12:00 pm – 12:45 pm Work session 12:45 pm – until completion

23. Questions?