1. Washington State Science Education and Assessment
Allison Greenberg, K-5 STEM Specialist, Woodside Elementary, Everett
Dawn Cope, Science Assessment Lead, OSPI
Office of Superintendent of Public Instruction

2. Science Learning Standards
Washington State adopted the NGSS as science learning standards in October of 2013.
Washington State 2013 K-12 Science Learning Standards
Washington adopted the Next Generation Science Standards in October 2013.
Over 4,000 Washington educators participated in reviews of the NGSS during various stages of their development.
November 2016

3. NGSS Performance Expectations per Grade Band
Each Performance Expectation (PE) includes a Science and Engineering Practice,
a Disciplinary Core Idea, and a Cross-cutting Concept

4. NGSS Performance Expectation

5. Transition Timeline http://www.k12.wa.us/Science/WSSLS.aspx
Since adoption, the Learning and Teaching team and the Science Assessment team have been facilitating a 4-year transition to three-dimensional science instruction and a three dimensional state summative assessment.
Our two teams have worked closely during the transition–attending meetings together, presenting at each other’s event, and coordinating our messages to ensure quality, consistency, and adherence to the intent the NGSS.
The implementation of the new standards required the collaboration of various critical stakeholder groups as well as colleagues and organizations throughout the United States. We are very fortunate to have access to and the support of these individuals and organizations. We brainstormed a few to include here as examples, but this is definitely not a complete list.

6. Transition Planning Document

7. Critical Stakeholders

8. WCAS--Goals
Design an assessment that reflects how science content is taught and tested in the classroom.
Phenomenon-based, reflecting student interest and relevance
Use WA educators in assessment development.
Test Design recommendations
Assessment development workgroups
Researchers Involvement
One of our main goals in developing our state science assessment is to reflect how science is taught and tested in the classroom. We want to be sure that the WCAS are phenomenon-based and reflect student interest and relevance.
To that end, we include state educators at several steps in test design and item development. Our educator work groups are carefully chosen to represent the demographic of our state. We include a variety of science educators including classroom teachers, principals, higher education professionals, science coaches, and curriculum specialists. We look for educators with experience in special education, career and technical education, informal education, etc.

9. Tasks that must be accomplished to develop a new state assessment.
Develop item specifications from standards
Determine claims and test map/blueprint
Develop new items
Field test new items
Develop Achievement Level Descriptors (ALDs) based on the science standards
Administer operational test
Alignment Study
Achievement Level Setting
What was the assessment team doing during those 4.5 years?

10. Science Assessment Item Collaborative (SAIC)
Led by CCSSO, Achieve, WestEd
Membership-14 states and the Virgin Islands
Products
Assessment Framework
Item Specifications Guidelines
Prototype Items
Initial example for measuring three-dimensional science learning
Can support states in guiding their test vendors with the design and development of NGSS- aligned assessments.
Can promote dialogue about a large-scale assessment for measuring the NGSS
How did we get started?

11. Cluster Map

12. WCAS Features First operational administration Spring 2018 Design
Grades 5, 8, 11
Design
Comprehensive at each grade
5-6 performance tasks (clusters) per test plus standalones
Each cluster is 3-dimensional and assesses 1-2 PE’s
Phenomenon
Stimuli items
The clusters included on each test are chosen to mirror the representation of science domains in the NGSS.
Wide range of SEPs, DCI, CCCs are represented
Yellow stars indicate tasks that normally occur one time in development of a new state assessment.
Its been about 4.5 years since Washington adopted the NGSS, but we are very happy that the WCAS was being administered in grades 5, 8, and 11 this spring.
Whether a student is in grade 5, 8, or 11, when they take the WCAS, the student will experience comprehensive test, composed mainly of item clusters (performance tasks) , that require them to use practices, core ideas, and crosscutting concepts to explain scientific phenomena.
An item cluster is a set of related stimuli and items. Each cluster is based on science phenomenon and as student works their way through a cluster, they have the opportunity to demonstrate their knowledge and understanding of practices, core ideas, and crosscutting concepts in several ways. Each cluster includes 1-4 stimuli and 3-6 items. The stimuli can include text, diagrams, graphs, or animations that provide relevant information at the appropriate place in a cluster. The items could be one of many types like drag and drop, short answer, multiple choice, multi-select, constructed response, etc. are used. Each item is aligned to two or three dimensions of a Performance Expectation. The cluster, as a whole, must be “three dimensional” or align to all of the dimensions from the PE or PEs the cluster assess.
The clusters included for a test are carefully chosen to mirror the representation of science domains in the NGSS. Careful planning ensures that a wide range of practices, DCIs, and crosscutting concepts are represented also.
There are other design and delivery features of the test that allow for a fair and accurate assessment of all students and ensure that valid and reliable inferences to be drawn from the results. We have included a link to WCAS resources for anyone interested in more information.

13. Science Assessment Development Cycle
Item Cluster Writing Work Group, Teams of 2-3 educators write stimuli, items, and rubrics designed to validly measure student understanding of the state standards. (3-5 days)
Content Review, Educators review the products of the item cluster writing work group to ensure every stimulus, item, and rubric is scientifically accurate and gathers appropriate evidence about student mastery of the NGSS. (4 days)
Field Test Rangefinding Educators look at a range of student responses to each short answer item and decide how to score each response. Scoring rubrics are refined and the materials used to score the field test items are produced. (3-5 days)
Content Review with Data. Educators review student performance data for every field tested item and make recommendations for the operational item bank based on criteria for acceptable stimuli and items. (1-2 days)
Leading up to the first administration of the WCAS we held a few meetings to elicit input for decisions like test blueprints and reporting claims. But, the main avenue for educator involvement on an ongoing basis are our assessment development work groups. Every item and rubric is written and reviewed by teachers. Teachers help prepare training materials for the scoring of constructed response items. Teachers review field test data for every item and determine whether or not the item is approved of our operational bank. To facilitate this work, we hold multi-day work groups four times a year. During these work groups we provide professional development that includes training on the standards and on test design. We brought in a researcher in the early stages of item development to help us elevate our understanding of three dimensional standards. Most participants tell us that this is some of the best professional development they receive (time, knowledge, networking), but the science team gets the benefit of the teacher’s student-centered perspective and their content expertise to help guide our work.

14. Development Work Groups
Pre-Meeting Training
6 hour Moodle Course
Current state science assessments
Washington Comprehensive Assessment of Science (WCAS) development events 
NGSS 101  
Item cluster expectations—PE bundling, Phenomena, Stimuli, Alignment 
Item Specifications
WCAS Training Tests.
Face-to-Face Meeting
Follow-up training
Work products

15. WCAS PD comments from teachers
I have a much clearer idea of how the DCIs, CCCs, and SEPs intertwine. I feel more confident about developing lessons around phenomena that relate to the standards.
I know the standards MUCH better. Impacts: lesson design and alignment. Impacts: identifying holes in units. Impacts: assessment design and depth of knowledge asked from students.
I am much more familiar w/ the NGSS now. I know how to parse out the info I need to plan for course alignment at my school.

16. Training Test

17. District Application of Assessment Development Process
present:
Participated in OSPI Assessment Development work groups for the prior state assessment (MSP) –
Performance Level Descriptors Development
Standard Setting
Scenario and Item Writing
Participated in OSPI Assessment Development work groups for the new state assessment (WCAS) –
2014, 2015 & Item Cluster Development
State Test Map committee
Achievement Level Descriptor Development
Achievement Level Setting, Standard Setting, and Pilot Range Finding/Rubric Validation committees
2019 – Content Review with Data committee

18. District Application of Assessment Development Process cont.
present:
Using the state’s assessment development process, facilitated Science Teacher Leaders through the creation of K-5 NGSS-aligned Unit Assessments
Led WCAS Professional Development with STEM colleagues, whole-school staff and/or 5th grade teams at multiple sites
Test Format, Test and Item Specifications and WCAS Training Tests
Supported Science Teacher Leaders through planning and delivering WCAS PD to their building staff
Leading building-wide support for WCAS-aligned unit assessment implementation and analysis K-5

19. K-5 NGSS-aligned Unit Assessment Development
Assessment Development PD:
18 Science Teacher Leaders with prior knowledge/training on NGSS
2-3 days –
Day 1: 3D Assessment Training & Item Writing
Day 2: Item Writing, Peer Feedback, Assessment Revision
Day 3: Peer Feedback, Assessment Revision
Goals:
Bundle 2 NGSS PEs that are addressed by the unit
Describe a phenomenon for the bundle
Create a stimulus for the phenomenon
Develop 3 – 4 test items connected to the stimulus that are 2D or 3D and use a variety of item types

20. Kindergarten – Balls and Ramps Unit Assessment
K-PS2-1 (1.a) With guidance, students collaboratively identify the phenomenon under investigation, which includes the following idea: the effect caused by different strengths and directions of pushes and pulls on the motion of an object. (DCI – Force and Motion; CCC – Cause and Effect)
K-PS2-1 (3.a.i-iii) Planning the investigation (DCI – Force and Motion; SEP – Planning and Carrying-out Investigations)
In the collaboratively developed investigation plan, students describe:
The object whose motion will be investigated.
What will be in contact with the object to cause the push or pull.
The relative strengths of the push or pull that will be applied to the object to start or stop its motion or change its speed.
K-PS2-2 (1.a.i-iii) Organizing data (DCI – Force and Motion; SEP – Analyzing and Interpreting Data)
With guidance, students organize given information using graphical or visual displays (e.g., pictures, pictographs, drawings, written observations, tables, charts). The given information students organize includes:
The relative speed or direction of the object before a push or pull is applied (i.e., qualitative measures and expressions of speed and direction; e.g., faster, slower, descriptions of “which way”).
The relative speed or direction of the object after a push or pull is applied.
How the relative strength of a push or pull affects the speed or direction of an object (i.e., qualitative measures or expressions of strength; e.g., harder, softer).
Challenges:
Developing an NGSS-aligned 3D assessment for students with limited reading ability.
Credits: EPS Elementary Science Leaders

21. Grade 2 – EIE: Best of Bugs Unit Assessment
Table Matching
Graphic Response assessing:
SEP – Dev. Models
CCC – Structure/Function
Credits: EPS Elementary Science Leaders

22. Grade 2 - Air and Weather Unit Assessment
Graphic Response assessing:
SEP – Analyzing & Interpreting Data
DCI – ESS2.1 Weather and Climate
Note: Using data from the curriculum.
Multi-part: Multiple Choice & Extended Response assessing:
SEP – Engaging in Argument from Evidence
CCC – Patterns
Note: Using data from the curriculum.
Credits: EPS Elementary Science Leaders

23. Grade 5 - Ecosystems Unit Assessment
Graphic Response assessing:
SEP – Developing and Using Models
CCC – Energy and Matter
DCI – PS3.D – Energy in Chemical Processes and Everyday Life
More detailed stimulus
Multi-select assessing:
CCC – System and System Models
DCI – LS2.A – Interdependent Relationships in an Ecosystem
Credits: EPS Elementary Science Leaders

24. Grade 5 – EIE: Designing Maglev Systems Assessment
More detailed stimulus integrating magnetism concepts and engineering design
Multi-part item:
SEP – Constructing Explanations and Designing Solutions
DCI – PS2.B: Types of Interactions & ETS1.B – Developing Possible Solutions
Credits: EPS Elementary Science Leaders

25. Grade 5 – Motion and Design Assessment
*Multi-part item:
Multi-part item:
SEP: Constructing Explanations and Designing Solutions
DCI: PS3.B – Conservation of Energy and Energy Transfer; DCI: ETS1.A: Defining Engineering Problems
CCC: Energy and Matter)
More detailed stimulus integrating force and energy concepts and engineering design
Credits: EPS Elementary Science Leaders

26. Feedback from Teachers
Impact of 3D Assessments on Student Learning:
Students “use a lot more evidence in their thinking.”
“Before hand they would provide a basic explanation and now they want to really try to dig deeper and apply their learning.”
Students “are more likely after one assessment to use the S&E practices in their daily student habits.”
Impacts on Teaching:
“Helping to create the assessments and using them in my classroom has helped me know what types of formative assessment tasks or instructional activities would help students get to the point where they could successfully take the assessment independently.”

27. Test Planning Meeting November 2016 Washington state educators
15 educators at each grade level (5, 8, 11)
Approval of Test Blueprints
PEs Ranked
1-Should be assessed every year
2-Should be assessed every other year
3-Should be assessed every 3rd or 4th year
PE rankling outcomes not made public
All PEs at each grade should be assessed after 3 to 4 years.

28. Test Blueprints 5 8 11 SEPs and CCCs in the context of
Grade
Total Points
Percent of Exam
SEPs and CCCs in the context of
Physical Sciences
Life Sciences
Earth and Space Sciences 5 35
40 %
~14 pts
29 %
~10 pts
31 %
~11 pts 8 40
35 %
38 %
~15 pts
27 % 11 45
36 %
~16 pts
28 %
~13 pts
ETS PEs assessed but not included here.

29. Additional Blueprint Criteria
One PE from each DCI domain (ESS, PS, LS, and ETS) is included in at least one item cluster
A minimum of three different Science and Engineering Practices are included across the clusters
A minimum of three different CCCs are included across the clusters
Stand-alone items increase DCI, SEP, and CCC coverage to achieve overall expectations.
Distribution of 2 and 3 dimensional items matches item bank
SEP-DCI-CCC
SEP-DCI
DCI-CCC
Test Design and Item Specifications documents
Recorded Webinar 1/24

30. Achievement Level Descriptors (ALDs)
November 2017
Washington state educators
15 educators at each grade level (5, 8, 11)
Make up half of each Achievement Level Setting panel
Group process
Drafted Level 3 descriptors in grade level group
Aligned Level 3 with other two grade levels
Drafted Level 2 and Level 4 descriptors in grade level group
Aligned Levels 2 and 4 with other grade levels

31. Alignment Study July 31-August 1, Olympia
5 Washington State Educators per grade level panel (5, 8, 11)
New to state science assessment development
Designed to answer two key questions:
How well does the test design/blueprint represent the NGSS?
Sampling plan or strategy used to represent the range of the standards on a test form of the assessment.
The extent to which the blueprint represents an appropriate sample of the NGSS over time.
How well do the set of items on each form match the design/blueprint?
The extent to which the test forms meet the expectations in the test design/blueprint.

32. WCAS Achievement Level Setting
August 6-8, Bellevue
Purpose-Develop cut score recommendations corresponding to each performance level (e.g., Level 2, Level 3) for grades 5, 8, and 11
30 Washington State educators per grade level panel
Chosen to carefully represent the state demographic
Classroom teachers, science coaches, curriculum specialists, etc.
Process
Orientation to test development and achievement level setting process
Taking the online WCAS test
Examining the Achievement Level Descriptors (ALDs)
Ratings using an Ordered Item Booklet (OIB)
Round 1
Round 2
Round 3
Articulation Panel
Recommendations were accepted by the State Board of Education on August 9, 2018

33. Test Design and Item Specifications

35. WCAS Scores Communication Timeline
Scores are available in the Online Reporting System (ORS) on the WCAP Portal in mid-August.
Statewide test scores are publically released on Report Card in September.
Paper WCAS Individual Score Reports (ISRs) arrive in districts in early October
Webinar: Understanding WCAS Score Reports
The purpose of the webinar was to introduce the WCAS test score reports, and explain how to access and understand student test score data. Resources for school and district staff to communicate test scores to families were shared. Audience: teachers, principals, district test coordinators, district administrators, and family-school partnership staff.

36. Online Reporting System (ORS)

37. Washington State Report Card

41. Sample Individual Student Score Report Front page

42. Sample Individual Student Score Report Back Page

43. Classroom Considerations
How do we assess and grade for these new standards?

44. Next Steps OSPI Classroom Formative assessment development
Resources
Item Specifications
Training tests
Reporting
Peer Review
Assessment Development
Classroom
PD Time with Teachers to create/revise assessments
Developing 3D Assessments for curriculum not fully-aligned to NGSS
Adapting state item types for local paper/pencil unit assessments
Developing on-line assessments in Canvas to mirror state format
Developing Elementary Teacher capacity to understand NGSS, shift to 3D instruction and assess/report grades in an integrated 3D way

45. Washington Comprehensive Assessment of Science (WCAS) Events
Date
Event
Oct 2013
NGSS Adopted
May 2015
National Technical Advisory Committee (NTAC) Initial Review of NGSS and Assessment Challenges
Sept 2015
NTAC Review of Science Assessment Item Collaborative Assessment Framework
Oct 2015
Item development begins
Jan 2016
NTAC Review of Proposed Test Structure, Measurement Model, & Reporting
Apr 2016
Limited pilot, grades 5 and 8
Sept 2016
NTAC Review of Reporting Claims & Test Design Meeting Plan
Nov 2016
Test Design Meeting: Analysis of assessable standards and prioritization
Apr 2017
Field test embedded, grades 5 and 8
May 2017
Voluntary, online field test for high school
NTAC Review of Paper/Pencil Form Considerations & Future Events

46. Washington Comprehensive Assessment of Science (WCAS) Events
Date
Event
Sept 2017
NTAC Review of Achievement Level Setting Plan—Achievement Level Descriptors (ALDs)
Oct 2017
Training tests and draft item specifications available
Nov 2017
State Board of Education (SBE) Briefed on WCAS & Achievement Level Setting Plan
Draft Achievement Level Descriptors (ALDs)Developed
Jan 2018
NTAC & SBE Approval of Achievement Level Setting Plan
Winter ‘17/’18
Teachers from across state trained on ALDs
Feb-Apr 2018
Contrasting Groups Study - Teachers use ALDs to provide ratings of students
Mar-Jun 2018
NGSS Operational Exam
July 2018
Alignment Study
Aug 2018
Achievement Level Setting--Grade-level panels, Articulation panel, NTAC certifies process was followed
SBE sets the cut scores