Office of Assessment October 22, 2014

Smarter ELA/Literacy Smarter Mathematics Smarter Interim Comp Assessments Smarter Digital Library DCAS Science DCAS Social Studies DCAS Social Studies NAEP SAT PSAT DCAS-Alt1 End of Course Alg II and IM III Smarter Interim Assessment Blocks End of Course US History ACCESS (ELL) Grade 9/10 Option 1

Summative Assessments for Accountability 2

Smarter ELA/Literacy and Mathematics Assessments  Replaces DCAS reading and mathematics in the 2014–2015 school year  These assessments will be available for one (1) opportunity and Administered in grades 3–8 and 11 in spring of 2015 Aligned with the Common Core State Standards Comprised of 2 components  Computer Adaptive Test (CAT) – contains multiple choice, grid, and interactive items  Performance Tasks (PT) – requires students to complete extended projects with real-world skills (research, writing, application) 3

Summative Assessment (Computer Adaptive)  Administered within the last 12 weeks (grades 3–8) and 7 weeks (grade 11) of the instructional year  Includes a variety of questions types Selected response Short constructed response Extended constructed response Technology enhanced  A small number of items will require hand-scoring by professionally trained scorers 4

Summative Assessment (Computer Adaptive)  Assesses the full range of Common Core in ELA and mathematics for students in grades 3–8 and 11—interim comprehensive assessments can be used in grades 9 and 10  Measures current student achievement and growth across time, showing progress toward college and career readiness 5

Summative Assessment (Performance Task)  A PT is made up of a Classroom Activity and an individually administered, computer-generated task  The Classroom Activity takes place before the student engages in the performance task, is not scored, and does not have special technology requirements  The Classroom Activity is a teacher-led group activity designed to provide needed context for a particular PT topic 6

Summative Assessment (Performance Task)  Each student will complete one PT in each content area  There are multiple performance tasks associated with a particular Classroom Activity and PT topic  PT items require hand-scoring—for summative by professionally trained scorers 7

Assignment of Performance Task Topics and Tasks States will be responsible for ensuring that:  No performance task topic is overassigned within the state, and  No performance task is overassigned within the state. 1. States should randomly assign performance task topics to schools by grade. 2. The test engine should randomly assign performance tasks to students within a topic, except for students requiring a fully accessible performance task. 3. States should periodically monitor assignments to verify that the process is functioning correctly and neither task topics nor tasks are over-assigned. 8

Flexible Item Pool Design Principles  The item selection algorithm will be designed to administer out-of-grade-level items.  All tests will adhere to on-grade blueprint content requirements.  Testing out of grade level will not increase test length. 10

On-grade content criteria minimum met Performance is clearly in Level 4 or Level 1. Off grade level item better meets measurement and content requirements than on-grade items. Conditions for Students Receiving Items from Adjacent Grade Levels 11

1. On-Grade Content Criteria Minimum Met  A decision point is reached when a substantial proportion of content has been covered.  ELA-CAT (no human-scored) about 60%  Mathematics-CAT (no human-scored) about 60% 12

2. Performance is clearly above/below Level 3  Decision – Is there a high degree of certainty that the student current score is at Level 1 or Level 4?  Action: If yes, add appropriate items from adjacent grades to the pool. (Content experts have screened items for appropriateness.) If no, continue with the on-grade-level pool. 13

Off-grade level item better meets measurement and content requirements than on-grade items  For all students, continue with the on-grade blueprint.  Item selection: For students with expanded item pools, an off- grade level item can be chosen if it satisfies content and information constraints better than an on-grade item. 14

Final Scores and Achievement Categories  The final score is based on responses to items from both the PT and CAT components.  All PTs are delivered on grade level.  Smarter Balanced will report final overall scores, achievement levels, and claim information from the whole test.  Score reports will display student performance relative to enrolled grade; there will not be off-grade score reports. 15

What Is a Test Blueprint?  Test blueprints (specifications) should clearly describe Assessment purpose(s) What assessment is designed to measure Sufficient details  Blueprint specifies intended test construct and desired psychometric properties  Sample blueprints presented today focus on content specifications for summative assessments 17

Content Weight at Claim Level – ELA Claims Grades 3–5 Grades 6–8Grade 11 Claim 1. Reading40%41%40% Claim 2. Writing26%25%26% Claim 3. Listening18% Claim 4. Research16% ELA/Literacy Content Weighting 18

ELA Preliminary Blueprint – Grade 3 19

ELA Blueprint Notes – Grade 3  The number of items is not necessarily equal to the weight by claim  Each student will receive an overall ELA/Literacy score and 4 claim-level scores  The CAT component includes machine-scored and short-text items. The 2 short-text items in Reading (associated with the long reading stimuli) and 1 short-text item in Writing will be hand-scored. 20

ELA Blueprint Notes – Grade 3  Each student will receive 1 PT, which includes a set of stimuli on a given topic. A classroom activity will be conducted for each PT to help orient students to the PT context. The same set of stimuli is used for the Writing and Research items. The short-text items and the essay are hand-scored.  The Research PT includes 2 short-text items and 1 machine-scored item. 21

ELA Blueprint Notes – Grade 3  Total items in Claim 2 – Writing includes 10 CAT items and 3 items from the PT  Writing PTs are noted as 3 separate “items”; however, the Writing PT score is derived from a single student response scored on 3 distinct traits  For more information on content categories, see the Content Specifications document at assessments/ assessments/ 22

Content Weight at Claim Level – Mathematics Claims Grades 3–5 Grades 6–8Grade 11 Claim 1. Concepts and Procedures54% 56% Claim 2. Problem Solving Claim 4. Modeling and Data Analysis 24% 23% Claim 3. Communicating Reasoning22% 21% Mathematics Content Weighting 23

Mathematics Preliminary Blueprint – Grade 6 24

Mathematics Blueprint Notes – Grade 6  The number of items is not necessarily equal to the weight by claim  Claim 2 (Problem Solving) and Claim 4 (Modeling and Data Analysis) are combined because of content similarity for the flexibility in item development. Although there are 4 claims, only 3 claim-level scores will be reported. 25

Mathematics Blueprint Notes – Grade 6  In the CAT component, each student at grades 6–8 will receive one item from either Claim 3 Target B or Claim 4 Target B that needs Hand-scoring  For more information on content categories, see the Content Specifications document at assessments/ assessments/ 26

Additional Information About Blueprints  The blueprints for CAT component are specified at the target level to Control item type and task specifications Determine item proportion in the pool  Control of the proportion of item DOK (depth of knowledge) at the claim level  In ELA, there are also genre considerations  In mathematics, the content specifications further delineates detail 27

Estimated Testing Times 28

Three Phases in Process  Four achievement levels are reported for student performance  Over 10,000 people registered to virtually participate in the online setting of 1 cut score for level participants were from Delaware (exceeded Smarter’s expectations for Delaware)  In-person workshops with panels of educators and other stakeholders set cut scores of the 4 achievement levels. 25 Delawareans participated.  Vertical articulation committee, a subset of the in- person workshops, examined recommended cut scores across grades for adjustment 30

Reviewing Items in the OIB for Bookmarking 31

Level 3 Bookmark Placement  When you place your Level 3 bookmark, think about a student at the threshold of Level 3 based on the achievement level descriptors Place your Level 3 bookmark at the point in the OIB such that A threshold Level 3 student would be have at least a 50% likelihood of success on each of the items before the bookmark (and about a 50% chance of success on the item just before the bookmark). A threshold Level 3 student would have less than a 50% likelihood of success on the items from the bookmark on. It may be easiest for you to first identify a range of items that begins where threshold Level 3 students begin to be challenged and end where they would definitely be challenged. Then pick the best page in that range. 32

Vertical Articulation 33

Follow-Up Activities  Technical Advisory Committee review  Smarter Balanced review  State Chiefs review and approval  Documentation – technical report 34

wp-content/uploads/2014/06/Reporting- System-OverviewV2.pdf 36

Smarter Score Reporting  Things we know: Scores will not be available until after testing is completed Score reports to families a little later than for DCAS Score reports to schools – date to be determined in summer Full score reporting presentation at December DTC meeting 37

Smarter Accessibility 38

Smarter Assessments …  … provide more accommodations/supports than any one state could develop individually  … are some of the most accessible tests available American Sign Language (ASL) (video) Refreshable Braille Text to speech (TTS) Various language glossaries available External devices/software Supports available to general education students 39

Universal Tools, Designated Supports, and Accommodations 40

Smarter ELA/Literacy Smarter Mathematics Smarter Interim Comp Assessments Smarter Digital Library DCAS Science DCAS Social Studies DCAS Social Studies NAEP SAT PSAT DCAS-Alt1 End of Course Alg II and IM III Smarter Interim Assessment Blocks End of Course US History ACCESS (ELL) Grade 9/10 Option 41

42