OSPI Science Assessment Update WERA, December 2010 The State of Material on 2011 Science MSP

Science Assessment Office Elaine Wood—Secondary Specialist Kara Monroe—Elementary Specialist Troi Williams—Administrative Assistant

Agenda 2011 Measurements of Student Progress Standard Setting Resources available Lessons learned from scoring

2011 Measurements of Student Progress Grades 5 and 8 First assessment of the K-12 Science Learning Standards (adopted June 2009)

Big Ideas in 2009 Science Standards 5 EALRs 1-3 Crosscutting Concepts and Abilities EALR 4 Domains of Science EALR 1 SystemsPhysical Science …is a way of thinking that makes it possible to analyze and understand complex phenomena. Force and Motion Matter: Properties and Change Energy: Transfer, Transformation and Conservation EALR 2 InquiryEarth and Space Science …is a process of asking and answering questions about the natural world that forms the bedrock of science. Earth and Space Earth Systems, Structures and Processes Earth History EALR 3 ApplicationLife Science …is about the interaction between science and technology, and how both can help solve real-world problems. Structures & Functions of Living Systems Ecosystems Biological Evolution

Operational Scoring Operational Range Finding Committee Operational Range Finding Committee Items available for Operational Testing Data Review Committee Data Review Committee Pilot Scoring Pilot Range Finding Committee Pilot Range Finding Committee Scenario & Item Pilot Testing Scenario & Item Pilot Testing Science Assessment Development Cycle Content Review & Bias/Sensitivity Committees Scenario & Items reviewed for content by OSPI/contractor Scenario & Item Writing Committee Develop Test and Item Specifications Develop Test and Item Specifications

Scenarios and Items Scenarios provide context for sets of items. Systems, Inquiry, and Application Physical, Earth/Space, and Life Science The items associated with a scenario cover a range of standards & range of cognitive complexity. Items not connected to a scenario are called “Stand Alone” items, which is clearly marked for students.

From WASL to Measurements of Student Progress Grade WASL 2010 MSP 2011 MSP Multiple Choice CompletionNA pilot only 1-4 Short Answer844 Extended Response000 % Points from MC/CP57%76% Embedded pilot items makes the total number of items 35.

From WASL to Measurements of Student Progress Grade WASL 2010 MSP 2011 MSP Multiple Choice CompletionNA pilot only 1-4 Short Answer755 Extended Response200 % Points from MC/CP56%75% Embedded pilot items makes the total number of items 40.

Test Specifications Grade 5 & 8 The test map designates the following proportions: At least 20% Systems (with connected EALR 4 content) 30% Inquiry 20% Application (which will apply EALR 4 content) At most 30% EALR 4 domain knowledge that is not associated with a cross-cutting concepts or abilities* *EALR 4 content may be assessed in questions that only assess EALR 4 or in questions that cut across both EALR 4 and a cross-cutting concept or ability (Systems, Inquiry, Application).

More on Science MSP Test Length Grade 5: Single session minutes* Grade 8: Single session minutes* *additional time is needed for passing out materials and reading directions New Item TypeCompletion: worth 1 score point Test Window May 2 to May 19 for paper/pencil tests May 2 to June 3 for online tests Online Testing Voluntary in grades 5 and 8 in 2011 Goal: 25%

Standard Setting: Summer 2011 Following the process used by Mathematics in grades 3-8 last year.

Science Standard Setting Plan DateEvent June 2009New Science standards approved May 2010Spring 2010 MSP; New items piloted Aug 2010Test Map meeting Sept 2010Performance Level Descriptors (PLDs) developed Oct 2010Spring 2011 tests built

Science Standard Setting Plan DateEvent June 2009New Science standards approved May 2010Spring 2010 MSP; New items piloted Aug 2010Test Map meeting Sept 2010Performance Level Descriptors (PLDs) developed Oct 2010Spring 2011 tests built Feb/Mar 2011 Teachers from across state trained on how to use PLDs to participate in Contrasting Groups Study (CGS) Mar/Apr 2011NTAC and SBE formally approve standard setting plan April 2011Teachers participate in CGS by providing ratings of students May 2011Spring 2011 MSP administered

Science Standard Setting Plan DateEvent June 2009New Science standards approved May 2010Spring 2010 MSP; New items piloted Aug 2010Test Map meeting Sept 2010Performance Level Descriptors (PLDs) developed Oct 2010Spring 2011 tests built Feb/Mar 2011 Teachers from across state trained on how to use PLDs to participate in Contrasting Groups Study (CGS) Mar/Apr 2011NTAC and SBE formally approve standard setting plan April 2011Teachers participate in CGS by providing ratings of students May 2011Spring 2011 MSP administered Aug 2011 Standard Setting events: --Grade level panels (use “Bookmark process”, start with teacher ratings) --“Policy panel” recommendations --NTAC certifies the planned process was followed Aug 2011SBE Sets Achievement Standards (Reviews all recommendations)

“Contrasting Groups” method affords broader involvement in standard setting Everyone in the state who teaches science in grades 5 and 8 is invited to help set the performance standards. Performance Level Descriptors (PLDs) are developed for “Basic” and “Proficient” Web-based training on PLDs is provided to participating teachers In April, participating teachers rate each of their students as being “Basic or below” or “At or above Proficient” Two distributions of scores are produced: Scores obtained by students rated as “Basic”, and scores for “Proficient” Where the two distributions merge represents the separation point between “Basic” and “Proficient.”

Students rated as “Basic or below” using criteria in PLD for Basic Raw score points Number of students

Students rated as “Proficient or above” using criteria in PLD for Proficient Raw score points Number of students

Intersection indicates a region for where “Basic” separates from “Proficient” Raw score points Number of students

PLD training Available via Moodle, February 1, 2011 Learn to use evidence to evaluate 5 th or 8 th grade students’ proficiency (other grade levels can participate) Using Moodle website Short videos of presentations All documents available for download Can accomplish all at once, or in multiple sittings Free clock hours (3) Encourage groups of teachers to participate together To sign up, go to link and create an account

Contrasting Groups Study Available via EDS, April 11-29, 2011 After completing the PLD Training, 5 th or 8 th grade teachers can predict their students’ performance on the Science MSP 5 th and 8 th grade science teachers can gain access to the EDS application by contacting their District Assessment Coordinator

Resources Available On the OSPI website:

OSPI Science Assessment Webpage Select Science Assessment from Assessment Drop-down Select Assessments

Science Assessment Resources Test and Item Specifications—more information about: Types of scenarios Test map Item specifications Vocabulary lists Cognitive complexity

Test and Item Specs Select Test and Item Specifications Select a Grade

Test and Item Specs Pages 5-14 Page 22

Science Assessment Resources Updates for 2011—each grade level contains: General information on the 2011 test Sample items for classroom use Scoring information, including rubrics and sample student responses

Updates for Select a Grade Select Educator Resources

Science Assessment Resources Online Testing Website: Demo Flash player Tutorial CAL testing program must be installed Practice items, same as Update for 2011 documents Both tools help students become familiar with testing platform, and tests the computer system

CAL online testing Select Downloads or Demos Stand-Alone item from 8 th grade tutorial

CAL online testing Short answer item from 5 th grade tutorial

CAL online testing Short answer item from 5 th grade tutorial Same item scrolled down to show symbols button and symbols tool pop-up.

Science Assessment Resources Teacher Resource tool—tests based on previous standards: Types of scenarios Generic questions Item specifications (previous standards) Cognitive complexity State-wide statistics

Teacher Resource Tool Select a Grade and a Year Select Science Select Teacher Resource Tool

Type of scenario Generic question, rolls over to item specification Percent correct or mean score

Science Assessment Resources Powerful Classroom Assessments (PCAs)—currently linked to WASL released scenarios and items: Sample items Scoring information, including rubrics New PCAs will be developed from future released items

Powerful Classroom Assessments Select PCA Select Educator Resources Select grade level

Lessons Learned from Scoring

Transition to new standards Grade 5Grade 8 Terms : “system” and “subsystem” “inputs” and “outputs” Human needs leading to inventions Weight as a measurement of the force of gravity Terms “closed” and “open” systems “balanced” and “unbalanced” forces Roles of engineers and scientists in technological advances

Common misconceptions (gr. 5) When the results don’t match the prediction, students struggle with how to report honestly Students think the mass of an substance changes when the substance breaks or melts Students think that plants receive food from soil Students have misconceptions about what makes something a producer in an ecosystem. Students think genetic characteristics are acquired from the environment

Common misconceptions (gr. 8) Students have trouble comparing a model to the real-world system Students think the mass of an substance changes when the substance changes state or temperature Students have trouble calculating speed Students don’t understand photosynthesis Students struggle to see the how natural selection leads to changes in a population over time

Going well (gr. 5) Variables questions 65% to 90% of the time Students are including data in conclusion items 28-37% get full points Between 30% and 40% give each supporting data point Students are attempting energy transfer and new procedure items 3.1% to 5.0% blanks

Going well (gr. 8) Variables questions 66% to 84% of the time Students are including data in conclusion items 54% get full points Between 55% and 58% give each supporting data point Students are attempting new procedure items 7.9% - 10% blanks

Constructed responses (both) Students confuse different forces (gravity, push, pull, magnetic) with forms of energy (5 th : light, heat, sound, motion, electricity; 8 th : light, thermal (heat), chemical, sound, kinetic, electrical) Students need more help writing their own procedures: Changing the manipulated variable indicated in the question Measuring the responding variable indicated in the question Students need to be more detailed when answering application items; more specific than “it will work” or “it must be broken, so fix it.”

Upcoming Events Performance Level Descriptors training Via Moodle website (February-March 2011) Scenario Writing Workshops 5 th and 8 th grade (February 2011) High School (February 2011) Contrasting Groups Study (April 2011)

Upcoming Events Operational Rangefinding Committees High School (Olympia, April 2011) 5 th and 8 th grade (Cincinnati, Ohio, May 2011) Pilot Rangefinding Committees (July 2011) Standard Setting (August 2011)

Opportunities for Involvement PEPPER list PreSALTers Enthusiastically Promoting Powerful Educational Resources Get updates about science happenings Sign up today with your address, or us and ask to be a PEPPER SALT list Science Assessment Leadership Team Anyone can apply Respond to s to sit on Item Development committees Committees are chosen to balance state representation and content expertise

Science Assessment Office Elaine Wood—Secondary Specialist Kara Monroe—Elementary Specialist Troi Williams—Administrative Assistant