Regional Smarter Balanced Assessment Workshops January/February 2014

Introductions Jennifer Judkins Anton Jackson Julie Wagner OSPI Mathematics Assessment, EOC Jennifer.Judkins@k12.wa.us Anton Jackson OSPI Mathematics Assessment, Grades 6 – 8 Anton.Jackson@k12.wa.us Julie Wagner OSPI Mathematics Assessment, Grades 3 – 5 Julie.Wagner@k12.wa.us

Housekeeping Be sure to sign-in. Be on time when returning from breaks. Use cell phones and personal electronic devices at breaks. Keep side conversations to a minimum. Use the Parking Lot for non-item-writing questions. Provide evaluation and clock-hour forms at end of day.

Workshop Program Smarter Balanced Assessment System Overview Shifts in the CCSS Smarter Balanced Summative Assessment Claims and the CCSS Progressions Smarter Balanced Resources Claim 1 Item Writing Claim 2-4 Item Writing

Purposes and Goals of Workshop Increase understanding of the Smarter Balanced assessment system Increase understanding of Smarter Balanced claims, item changes, and how the shifts in the CCSS manifest in items Observe how the progression of learning is assessed through the years Bring awareness to the Smarter Balanced resources that are available to educators Provide participants with knowledge of the item-writing process

Smarter-Balanced Assessment System Formative Resources Interim Assessments Summative Assessments

A Balanced Assessment System Summative: College and career readiness assessments for accountability Teachers and schools have information and tools they need to improve teaching and learning Common Core State Standards specify K-12 expectations for college and career readiness All students leave high school college and career ready Formative resources: Digital Library with instructional and professional learning resources for educators to improve instruction Interim: Flexible and open assessments, used for actionable feedback

Formative Resources Digital Library with formative assessment instructional and professional learning resources for educators

Digital Library Leadership Structures National Advisory Panel (NAP) 11-20 experts Began December 2012 Provided policies and criteria for resources State Leadership Team (SLT) 5- 8 members Recruiting began November 2012 Provides training for State Network of Educators State Network of Educators (SNE) 70 to 100 members Representation from LEAs, AEAs, content leaders, ELL, IHE Began recruiting in January 2013 Provides reviews and resources Structure of Supports to develop the digital Library

Resources in the Digital Library Commissioned Professional Learning Modules Resources for students and families Frame the formative assessment process within a Balanced Assessment System Articulate the formative assessment process Highlight formative assessment practices and strategies Assessment Literacy Modules Commissioned Professional Learning Modules Instructional materials for educators Instructional materials for students Demonstrate/support effective implementation of the formative assessment process Focus on key content and practice from the Common Core State Standards for Mathematics and English Language Arts Exemplar Instructional Modules As part of the Consortium's balanced assessment system, Smarter Balanced will provide resources, tools, and professional learning opportunities to educators so that they can advance their use of formative assessment practices and tailor their CCSS instruction to student needs based on evidence they gather from a coherent, balanced assessment approach. Specifically, Smarter Balanced is: Commissioning innovative, interactive professional learning and exemplar instructional modules Aggregating relevant resources that may already be available in public sources but are difficult for teachers to locate These resources will have the potential to help educators transform classroom practice and improve student learning. Specifically, educators will be able to use these resources throughout the school year and to better understand where students are in their learning of CCSS, identify any misconceptions, and make quick adjustments to instruction. High quality vetted resources and tools for students and families Reflect and support the formative assessment process Reflect and support the Common Core State Standards for English Language Arts/Literacy and Mathematics Create Professional Learning Communities Education Resources * Resources include the following file types: Video, HTML5, Audio, PPT, Excel, Word, and PDF.

Digital Library Functionality Enable trained educators (SNEs and SLTs) to submit, review, and publish resources Allow educators to view, download, and rate resources Use state-of-the-art tagging and search to quickly find resources by CCSS and other topics Enable educators from across the Consortium to collaborate and share their knowledge The Digital Library is the tool that the Consortium will use to disseminate resources to teachers. The Library will present resources in a in a graphically compelling way and will provide state-of-the-art functionality for educators, including the ability for educators from across the Consortium to collaborate and share their knowledge.

Resource Submission Process SNE1 SNE SNE SNE1 Review of Resource with Quality Criteria Rubric Reviewed by 3 SNEs Resource Passes Gate-Keeping SNE SNE Submit Resource with Cover Profile Choose Resource to Review in SNE’s Queue Initial Review of Resource with Gate-Keeping Criteria SNE 1: The SNE who reviews the resource with the Gate-Keeping Criteria will also be one of the 3 SNE reviewers. Resource Fails Gate-Keeping—Return to SNE to Revise SNE Session 3: Submitting Quality Resources

Interim Assessments ICAs and IABs

Interim Assessment Interim Comprehensive Assessments (ICA) Two Types: Interim Comprehensive Assessments (ICA) Fixed form in fall 2014 for all grades Adaptive (as pool allows) Will be taken online in same form as summative Some level of hand-scoring Interim Assessment Blocks (IAB) Fixed form for most blocks in all grades Adaptive for certain blocks (as pool allows) Adaptive for all blocks – later

Summative Assessment Assessment Claims Computer Adaptive Testing (CAT) Item Types Performance Tasks

Smarter Balanced Claims Claims are the broad statements of the assessment system‘s learning outcomes, each of which requires evidence that articulates the types of data/observations that will support interpretations of competence towards achievement of the claims.

Overall Assessment Claims Claim for Grades 3-8: “Students can demonstrate progress toward college and career readiness in mathematics.” Claim for Grade 11: “Students can demonstrate college and career readiness in mathematics.”

Assessment Claims 1- 4 Claim 1 – Concepts & Procedures Claim 2 – Problem Solving Claim 3 – Communicating Reasoning Claim 4 – Modeling and Data Analysis The claims are supported by the Standards for Mathematical Practice.

Computer Adaptive Testing “Based on student responses, the computer program adjusts the difficulty of questions throughout the assessment. … By adapting to the student as the assessment is taking place, these assessments present an individually tailored set of questions to each student and can quickly identify which skills students have mastered.” Smarter Balanced

Computer Adaptive Testing (CAT) New Item Types Multiple response multiple choice Matching Tables, Drag and Drop, Hot Spot Performance Tasks Automatically Scored Machine Scored Artificial-Intelligence (AI) scored I am thinking we can be brief here since this section is about the overview and we will be spending most of the rest of the day on the summative assessment.

Key CCSS Shifts Focus Coherence Rigor

Shift One: Focus strongly where the Standards focus Significantly narrow the scope of content and deepen how time and energy is spent in the math classroom Focus deeply only on what is emphasized in the standards, so that students gain strong foundations

The Shape of Math in A+ Countries Mathematics topics intended at each grade by at least two-thirds of A+ countries Mathematics topics intended at each grade by at least two-thirds of 21 U.S. states 1 Schmidt, Houang, & Cogan, “A Coherent Curriculum: The Case of Mathematics.” (2002).

Traditional U.S. Approach K 12 Number and Operations Measurement and Geometry Algebra and Functions Statistics and Probability

Focusing attention within ‘Number and Operations’ Operations and Algebraic Thinking Expressions and Equations Algebra  Number and Operations— Base Ten The Number System Number and Operations—Fractions K 1 2 3 4 5 6 7 8 High School

Shift Two: Coherence Think across grades, and link to major topics within grades Carefully connect the learning within and across grades so that students can build new understanding onto foundations built in previous years. Begin to count on solid conceptual understanding of core content and build on it. Each standard is not a new event, but an extension of previous learning.

4.NF.4. Apply and extend previous understandings of multiplication to multiply a fraction by a whole number. 5.NF.4. Apply and extend previous understandings of multiplication to multiply a fraction or whole number by a fraction. 5.NF.7. Apply and extend previous understandings of division to divide unit fractions by whole numbers and whole numbers by unit fractions. 6.NS. Apply and extend previous understandings of multiplication and division to divide fractions by fractions. 6.NS.1. Interpret and compute quotients of fractions, and solve word problems involving division of fractions by fractions, e.g., by using visual fraction models and equations to represent the problem. Grade 4 Grade 5 Grade 6 CCSS Informing Grades 1-6 Mathematics Standards Development: What Can Be Learned from High-Performing Hong Kong, Singapore, and Korea? American Institutes for Research (2009, p. 13)

Coherence Example: Grade 3 The standards make explicit connections at a single grade Multiplication and Division 3.OA.5 Properties of Operations 3.MD.7a 3.MD.7c Area

Shift Three: Rigor Equal intensity in conceptual understanding, procedural skill/fluency, and application The CCSSM require a balance of: Solid conceptual understanding Procedural skill and fluency Application of skills in problem solving situations This requires equal intensity in time, activities, and resources in pursuit of all three

(a) Conceptual Understanding... or is it?

Place Value Concepts 206 = ____ hundreds ____ tens ____ones Is the comparison true or false? 12 tens + 11 ones > 1 hundred + 2 tens

(b) Fluency The standards require speed and accuracy in calculation. Teachers structure class time and/or homework time for students to practice core functions such as solving equations so that they are more able to understand and manipulate more complex concepts.

Fluency Beyond the Facts Select all of the equations that have integer solutions: A) 81 = x - 4 B) x - 7 = 28 C) 40 - 4 = x D) 5𝑥 = 25 E) 2x = 121

(c) Application Students can use appropriate concepts and procedures for application even when not prompted to do so Provide opportunities at all grade levels for students to apply math concepts in “real world” situations, recognizing this means different things in K-5, 6-8, and HS Teachers in content areas outside of math, particularly science, ensure that students are using grade-level-appropriate math to make meaning of and access science content

Application

Operationalizing the CCSS Shifts Do: Read the standards including critical areas of focus. Know the structure of the standards. Know the major, supporting and additional clusters for your grade. Study the progression documents with colleagues. Think about unit design first, then lesson, then task. It is not about what you are going to ‘do’ but what the students will learn.

Activity Turn to your elbow partner: Explain in your own words what the three shifts are in the CCSS. Explain to your partner how these shifts will change your instructional practice.

The Smarter Balanced Summative Assessment Considerations Claims Item Types Performance Tasks

Considerations

Evidence-Centered Design (ECD) Item development approach that begins with claims about what test scores will tell us about students' learning Determine evidence needed to support claims made about students based on their test scores Determine types of items and tasks needed to collect evidence A key concept that guides each of these steps is Evidence-Centered Design. As is described in greater detail in a separate module, Evidence-Centered Design {+} provides a framework that helps clearly define the interpretations or claims Smarter Balanced wants to make about students and their cognition, {+} the observations that provide evidence needed to support those claims, and {+} the types of items and tasks needed to collect that evidence. The use of Evidence-Centered Design is a unique and important feature of the Smarter Balanced assessment system.

Common Core Standards Math 4 Math Claims Content Specifications Six Key Concepts of ECD 1. Define the domain Common Core Standards Math 2. Define claims to be made 4 Math Claims Content Specifications 3. Define assessment targets Knowledge, Skills, & Abilities 4. Define evidence required Evidence to be Elicited from Student The final step {+} is to develop one or more items based on the task model. A task model might be thought of as a parent from which many items are developed, with common characteristics across these items. 5. Develop Task Models Methods for Eliciting Evidence 6. Develop Items or Performance Tasks

Universally Designed Items and Tasks – Some Guidelines Simplify vocabulary and sentence structure Place problems in familiar contexts with familiar items and names Use graphics only when necessary Simplify graphics and tables Avoid the use of color and maximize contrast Avoid content that spans multi-pages or that contains multiple-elements Use highly interactive manipulations only when necessary Identify terms and visuals that should not be altered As mentioned earlier, the General Accessibility Guidelines provide a larger number of strategies to employ and issues to consider when designing and reviewing items and tasks. While item writers and reviewers should be familiar with all of the concepts explored in the General Accessibility Guidelines, here are a sub-set of the guidelines that should be kept in mind when writing and reviewing items and performance tasks. These include: {+} Simplify vocabulary and sentence structure; {+} Place problems in familiar contexts with familiar items and names; {+} Use graphics only when necessary; {+} Simplify graphics and tables; {+} Avoid the use of color and maximize contrast; {+} Avoid content that spans multi-pages or that contains multiple-elements; {+} Use highly interactive manipulations only when necessary; {+} and Identify terms and visuals that should not be altered. By following these and the other strategies presented in the General Accessibility Guidelines, the items and performance tasks educators write and review will better access the knowledge, skills and abilities that are the targets of assessment and support the claims that form the heart of the Smarter Balanced assessment system.

Bias Concerns Bias occurs when content contained in an item or task creates an unfair disadvantage for a sub-group of students Unfamiliar contexts or examples Unusual names of people or places References to local events or issues One aspect of an item or task that affects its ability to accurately access the knowledge, skill, or ability that is the target of assessment is the extent to which information in the item creates bias for a specific sub-group of students. {+} Bias occurs when an aspect of an item that is not specific to the knowledge, skill, or ability that is the target of assessment places a sub-group of students at an inappropriate disadvantage or advantage. Some characteristics of an item that can create bias include: {+} Contexts or examples that are unfamiliar to sub-groups of students, such as sports, hobbies, or activities. {+} The use of names of people or places that may be unfamiliar to or cause confusion for sub-groups of students. For example, using Jacquim, Sarasota, or Paris, as a person’s name in a word problem may be confusing to some students. {+} Other examples include references to events or issues that may be unfamiliar to sub-groups of students, such as a snow day, a state-specific holiday such as Bunker Hill Day, or a localized issue such as the development of a pipeline through a specific region. To reduce bias, a good practice is to refer to names, school-based activities, events, and issues that are likely to be familiar to students across the country.

What is this question asking me? Jimmy is the star on his football team. This season he scored ten touchdowns, kicked twenty extra points, and had eight field goals. What is the total number of points Jimmy has scored this season? What is a field goal? What is this question asking me? 104 As an example, consider this mathematics item that is designed to elicit evidence about a student’s ability to use arithmetic operations with whole numbers to solve problems. {+} Now consider three students who all perform arithmetic equally well. The first student is a very strong reader and is a native English speaker who is familiar with American football. The second student is still developing familiarity with English, and although he is able to decode English text well, he is unfamiliar with many pastimes common in the United States. The third student is also a native English speaker, but has dyslexia and experiences difficulty decoding text. {+} While the item elicits evidence from the first student that accurately reflects her ability to solve problems using arithmetic operations with whole numbers, {+} the second student may struggle with this item only because he is unfamiliar with the game of football and does not know the number of points awarded for a touchdown, field goal, or an extra point. {+} The third student has difficulty understanding what is being asked because of his challenge decoding text. The way in which this item is designed and presented to the second and third students makes it difficult for them to access the item’s content. In turn, this item does not appropriately measure the assessment target for each student, which in this example is their ability to solve problems using whole number computation, rather than their ability to decode word problems about the sport of football. For this reason, this item raises accessibility issues. Discussed in greater detail later in this module, one remedy for this problem is to place the problem in a school-based context that is more likely to be familiar to all students.

Sensitivity Sensitivity focuses on content that creates unease, provokes negative feelings, or challenges beliefs or values Religions, religious practices, and religious figures Political topics Issues of gender, race, and ethnicity Sensitivity is another issue that can adversely affect the performance of some students. {+} Like bias, sensitivity focuses on content that is unrelated to the assessment target itself, but is used to create a context in which the assessment target is measured. Sensitivity also focuses on content that may create unease in a student, may provoke negative feelings, or may appear to introduce beliefs or values that are counter to the student’s own beliefs and values. Topics that may raise issues of sensitivity include: {+} religions, religious practices, and religious figures; {+} political topics such as party affiliations, abortion, school-prayer, and the teaching of evolution; {+} and issues of gender, race, and ethnicity, such as single-sex marriage, immigration, and cultural customs. Avoiding content that may raise issues of bias and sensitivity is an important part of Universal Design, and helps to expand the number of students for whom an item or performance task is able to access the knowledge, skill, or ability that is the target of assessment. Specific guidelines that focus on issues of bias and sensitivity have been developed for the Smarter Balanced assessment program and should be familiar to all people involved in developing and reviewing items and performance tasks.

Acceptable or Unacceptable? Two people who were conversing at a street corner parted and moved away from the corner in straight lines that are perpendicular to each other. If one person walked at 3 miles per hour and the second person jogged at 4 miles per hour, how far apart would they be after one hour? Unacceptable. The linguistic load is high for a math question. The sentences are long, and the syntax is complicated. “Conversing” is a difficult synonym for “talking,” and the people’s actions before they started to move are not relevant in any case. (“Perpendicular” is acceptable as a valid mathematical term.)

Acceptable or Unacceptable? Two people stood next to each other. They started walking in straight lines that are perpendicular to each other. One person walked at 3 miles per hour. The other person walked at 4 miles per hour. How far apart are they after one hour? a. Acceptable. Unnecessary information has been deleted. Two long sentences have been replaced by five shorter sentences. The conditional syntax (“If one person. . .”) has been replaced by brief statements of fact.

Acceptable or Unacceptable? It takes Sarah an average of 30 minutes to clean her bedroom. She cleans her bedroom once a week. How many hours would Sarah spend cleaning her bedroom in one year? Unacceptable if many questions in the test had girls cleaning rooms or doing what was traditionally considered “woman’s work,” because the test would reinforce a stereotype and be unfair. b. Acceptable if combined with questions showing women doing nontraditional work. Not all children have their own bedrooms, but the concept that some children have individual bedrooms should be neither strange nor upsetting. Whether or not the required knowledge of the number of minutes in an hour and the number of weeks in a year is fair depends on the grade level of the test takers. Acceptability depends on the mix of items.

The Claims

Assessment Claims Claim 1 – Concepts & Procedures “Students can explain and apply mathematical concepts and interpret and carry out mathematical procedures with precision and fluency.” Claim 2 – Problem Solving “Students can solve a range of complex well-posed problems in pure and applied mathematics, making productive use of knowledge and problem solving strategies.” Claim 3 – Communicating Reasoning “Students can clearly and precisely construct viable arguments to support their own reasoning and to critique the reasoning of others.” Claim 4 – Modeling and Data Analysis “Students can analyze complex, real-world scenarios and can construct and use mathematical models to interpret and solve problems.”

Claim 1 Concepts and Procedures Grade 8 Detailed: Rapid Development: The graph shown compares the height of Tree A and the height of Tree B over time (in years). A system of two linear equations has no solution. One equation is 3𝑥 + 𝑦 = –2. Select the equation that would make this system have no solution.   A. 2x + y = 4 B. 2x + y = 5 C. 3x + y = 4 D. 4x + y = 5 How many years after Tree B was planted did Tree A and Tree B have the same height?

Claim 2 Problem Solving Grade 7 (Source: Adapted from Illustrative Mathematics) The students in Mr. Sanchez's class are converting distances measured in miles (m) to kilometers (km). Abby and Renato use the following methods to convert miles to kilometers. Abby takes the number of miles, doubles it, then subtracts 20% of the result. Renato first divides the number of miles by 5, then multiplies the result by 8. Which equation correctly shows why both their methods produce the same result? A. 2𝑚−0.20= 𝑚 5  8 B. 2𝑚−0.20(2𝑚)= 𝑚 5  8 C. 2𝑚−2.20𝑚= 𝑚 5 + 8 𝑚 5 D. 0.20 2𝑚 −2𝑚= 𝑚 5 + 8 𝑚 5

Claim 3 Communicating Reasoning Grade 6 Also note that this is a drag and drop example and its format.

Claim 4 Modeling and Data Analysis Grade 5

Contrasting the Claims - Activity Individually review the items on the ‘Contrasting the Claims’ Worksheet. Review with a partner the items by claim number. What are the differences in the way the problems are written by claim? How does the focus on four claims change the way you instruct and assess students in the classroom?

Item Types

CAT Item Types Multiple Choice Equation/Numeric Matching Table Single correct answer Multiple correct answers Equation/Numeric Matching Table Fill-in Table Grid Items Drag and Drop Graphing Hot Spot

Multiple Choice – single correct response Grade 3

Multiple Choice – multiple correct responses A) B) C) D) Grade 8 Select all the graphs that show a proportional relationship.

Equation/Numeric Grade 5 A triangular-shaped garden is shown. Enter the area of the garden, in square meters.

Equation/Numeric

Matching Table Grade 11

Fill- in Table

Drag and Drop Grade 8 Kayla asked 10 students in her class whether they owned a dog or a cat or both. Drag one number into each box to complete the table, given this information:   40% of the students own a dog. 30% of the students own a cat. 10% of the students own both a dog and a cat.

Drag and Drop

Graphing Grade 8

Hot Spot Grade 11

Item Type Implications Review the ‘Item Type’ worksheet and talk to a partner about these questions. Be prepared to share your discussion: What implications do these item types have, if any, on your instruction and assessment of students? What resources do you have that might allow you to create similar items and imbed them in instruction and assessment? N-rich Virtual manipulatives

Performance Tasks

Performance Tasks 6 questions Distributed among Claim 2, 3, and 4 Item types on CAT may be used in a PT Also included short, constructed response: “Show your work.” “Explain your reasoning.” Classroom activity to precede work on PTs 2 items automatically scored, rest hand-scored

Classroom Activity

Sample Task Grade 3 https://sat2.sbacpt.tds.airast.org/Student/Pages/TestShellModern.aspx

Culminating Question

Performance Task Implications Talk to a partner about these questions and be prepared to share your discussion: How would you prepare students for sustained, connected activities like the performance tasks? What resources do you have for authentic, real-world applications of the mathematics students are learning? Illustrative Mathematics Engage NY

Progression of Learning

Progression Documents http://ime.math.arizona.edu/progressions/

Progression of Learning and Items through the Grades Individually review the items in the progressions document. Find an example where what is learned in a prior year is fundamental to what is learned later. With a partner, determine the new learning that is added from year to year. Be prepared to share.

Smarter Balanced Resources

Smarter Balanced Website http://www.smarterbalanced.org/

For Educators Field Test Practice and Pilot Tests Sample Items and Performance Tasks Item Writing and Review Content Specifications Item Specifications Considerations, Style Guide, etc. Achievement Level Descriptors and College Content Readiness

Claim 1 Item Writing Depth of Knowledge Item Specification Documents

Depth of Knowledge

Cognitive Rigor and Depth of Knowledge Level of complexity of cognitive demand Level 1: Recall and Reproduction Requires eliciting information such as a fact, definition, term, or a simple procedure, as well as performing a simple algorithm or applying a formula. Level 2: Basic Skills and Concepts Requires the engagement of some mental processing beyond a recall of information. Level 3: Strategic Thinking and Reasoning Requires reasoning, planning, using evidence, and explanations of thinking. Level 4: Extended Thinking Requires complex reasoning, planning, developing, and thinking most likely over an extended period of time. Smarter Balanced items and tasks will elicit evidence that students have the ability to integrate knowledge and skills across multiple assessment targets and are ready to meet the challenges of college and careers. {+} Items and tasks must be constructed at various levels of cognitive rigor. Smarter Balanced has defined four levels of depth of knowledge. The first level focuses on recall and reproduction of facts and other types of information. The second level focuses on basic skills and concepts that require cognitive processes that extend beyond the recall of information. The third level focuses on strategic thinking and reasoning. The fourth and final level requires extended thinking that includes complex reasoning, planning, development, and cognition that occurs over an extended period of time. Let’s take a look at a sample item for each of the four levels of depth of knowledge.

Level 1 – Recall and Reproduction · 87 8–12 74 7–3 1 3 2 1 3 9 · This is a grade 8 item that is coded to depth of knowledge level one. This item requires students to recall the rules for exponents to evaluate each expression and select the expression or expressions with a value between zero and one. (–5)6 (–5)10

Level 2 – Application of Skills/ Concepts A cylindrical tank has a height of 10 feet and a radius of 4 feet. Jane fills this tank with water at a rate of 8 cubic feet per minute. How many minutes will it take Jane to completely fill the tank without overflowing at this rate?   Round your answer to the nearest minute. This is a grade 8 item that is coded to depth of knowledge level two. This item requires students to use the formula for the volume of a cylinder, as well as a basic understanding of rate, to calculate the number of minutes Jane will take to fill the water tank.

Level 3 – Strategic Thinking and Reasoning The total cost for an order of shirts from a company consists of the cost for each shirt plus a one-time design fee. The cost for each shirt is the same no matter how many shirts are ordered. The company provides the following examples to customers to help them estimate the total cost for an order of shirts. 50 shirts cost $349.50 500 shirts cost $2370   Part A: Using the examples provided, what is the cost for each shirt, not including the one-time design fee? Explain how you found your answer. Part B: What is the cost of the one-time design fee? Explain how you found your answer. This is a grade 8 item that is coded to depth of knowledge level three. This item requires students to perform calculations in order to respond to each part. In addition, students are required to provide an explanation for each answer.

Level 4 – Extended Thinking During the task, the student assumes the role of an architect who is responsible for designing the best plan for a park with area and financial restraints. The student completes tasks in which he/she compares the costs of different bids, determines what facilities should be given priority in the park, and then develops a scale drawing of the best design for the park and an explanation of the choices made. This investigation is done in class using a calculator, an applet to construct the scale drawing, and a spreadsheet. This is a description for a grade 8 performance task that is coded to depth of knowledge level four. This item requires students to use concepts of geometry, numbers and operations, and statistics to determine the best solution to a problem where all constraints cannot be satisfied at the same time. Additionally, the student must provide justifications to support reasoning. Students are expected to engage with the task for an extended period of time, up to 120 minutes.

Depth of Knowledge Look at the ‘Comparing the Claims’ and determine the Depth of Knowledge level of each. Be prepared to share.

How to read these documents Item Specifications How to read these documents

Item Specification Overview As each section of an item specification document is reviewed, find the same section in an item specification document at your grade-band level. Grade 3-5 – Grade 6-8 – Grade 11 –

Procedures to Support Validity Start with a clear explanation of what students are expected to know and be able to do (CCSS-M). Control irrelevant sources of variability by: Standardizing items using Task Models. Standardizing items using a Style Guide.

Description of the Target Targeted Standards This slide shows the typical manner in which the front matter for a target presents an overview of the focus for the target. The target is described in paragraph form and targeted standards are listed.

Relevant above and below grade standards Achievement Level Descriptors Depth of Knowledge Next you will find relevant above and below grade standards – again to orient the reader to the growth continuum that underlies the CCSS. The concept of Depth of Knowledge was introduced in the previous training module. DOK levels have been identified for each target and task models also indicate which DOK level is demonstrated in the item or task. Next you will find achievement level descriptors. These were used to inform the task models to ensure that the Smarter Balanced Assessments provide appropriate items for students at all proficiency levels. Achievement Level Descriptors

Statements of Required Evidence (Derived from Standards) Allowable Stimulus Materials and Mathematics Vocabulary Allowable Response Types Next you will see statements regarding the evidence required for the given target. These statements break the standards down into measurable units and become the focus of the task models. On this page you also see a list of appropriate stimulus materials and a list of vocabulary that is considered ‘fair game’ for test items. Specifications also address accessibility concerns that may arise for students with a variety of physical challenges.

Task Models Flow from Evidence Statements Required evidence is identified Details about stimuli for task model The image presents one task model for the evidence statement “The student computes unit rates.” The task model spells out the type of item in detail including a preamble and stem to show item writers how to phrase items so that the language is simple and straightforward and so that students know exactly what they are asked to do. Writing items requires closely following the model. The task model includes evidence statements and details about the stimuli that are allowed for the particular task model.

Task Models Flow from Evidence Statements Preamble orients student to problem The item sample preamble introduces the problem and the example stem asks for the required evidence. Stem asks for the required evidence

Item Specification - Claim 1Targets Grade Detailed Rapid Development 3 8 6 4 5 9 10 7 11 12

Rapid Development Specifications Detailed guidelines for stimuli for Task Model 1 This is an example of the first page from HS Claim 1 Target A which is focused on students’ abilities to extend properties of exponents to rational exponents and radicals. The stimulus specifications give specific details about the structure of the stimulus and the acceptable numbers for each part of (in this case) a radical.

Rapid Development Specifications TM Statement of required evidence Stimulus Format for Task Model Phrasing of Stem Rules for answer choices based on common errors related to standard The task model indicates the structure of the stimulus and the phrasing of the stem.

Claim 1 Activity Choose a task model from your item specification document and write an item to meet that task model. Share and review an item with a partner. How can you use this process to write items for your classroom or the district?

Claim 2-4 Item Writing

Targets for Claim 2 Target A: Apply mathematics to solve well-posed problems in pure mathematics and arising in everyday life, society, and the workplace. Target B: Select and use appropriate tools strategically. Target C: Interpret results in the context of a situation. Target D: Identify important quantities in a practical situation and map their relationships (e.g., using diagrams, two-way tables, graphs, flowcharts, or formulas).

Targets for Claim 3 Target A: Test propositions or conjectures with specific examples. Target B: Construct, autonomously, chains of reasoning that will justify or refute propositions or conjectures. Target C: State logical assumptions being used. Target D: Use the technique of breaking an argument into cases. Target E: Distinguish correct logic or reasoning from that which is flawed and – if there is a flaw in the argument – explain what it is. Target F: Base arguments on concrete referents such as objects, drawings, diagrams, and actions.

Targets for Claim 4 Target A: Apply mathematics to solve well-posed problems in pure mathematics and arising in everyday life, society, and the workplace. Target B: Construct, autonomously, chains of reasoning to justify mathematical models used, interpretations made, and solutions proposed for a complex problem. Target C: State logical assumptions being used. Target D: Interpret results in the context of a situation.

Targets for Claim 4 Target E: Analyze the adequacy of and make improvements to an existing model or develop a mathematical model of a real phenomenon. Target F: Identify important quantities in a practical situation and map their relationships. Target G: Identify, analyze, and synthesize relevant external resources to pose or solve problems.

Claim 2-4 Activity Take the item that you wrote for Claim 1. How might this item be modified to become Claim 2, 3, or 4? What changes would have to be made to move away from just Claim 1 items? What role does rigor play? How might these be included in a Performance Task? Share your thoughts with a partner. How can you use these thoughts to impact classroom instruction in your district?

Regional Assessment Workshops

Day 1 Wrap-Up What are the next steps you would do to use the information you learned today in your classroom, building and/or district? What role will you play in these next steps? Evaluations Clock hours Sign out

Thank you

Day 2 - Item Writing Regional Smarter Balanced Assessment Workshops

Schedule Start at 9 a.m. Preparation for Claim 1 item writing Adjourn by 4 p.m. With morning and afternoon breaks and lunch break as appropriate.

Day 2 Item Writing Agenda Welcome and Introductions Forms and Paperwork Group Norms Item-Writing for the Common Core Assessment Purposes and Goals Importance of Validity Documents Used to Support Validity Claim 1 Item Writing Considerations Procedure for Writing, Reviewing, Approving, and Entering Items Claim 1 Item Writing Claim 2, 3, 4 Item Writing

Forms and Paperwork Sign - in Non-Disclosure Agreement The items are secure, but the process is not. Substitute Reimbursement Evaluation Module Reviews Item Writing Clock Hours

Public Disclosure Please be aware that your participation on this committee is considered public record under Washington State Public Record Act RCW 42.56. Please be assured that your personal and residential information are not released.

Group Norms Be on time when returning from breaks. Use cell phones and personal electronic devices at breaks. Keep side conversations to a minimum. Respect each other’s work styles (need for quiet). Ask for and give feedback. Use the Parking Lot for non-item-writing questions.

Purposes and Goals of Workshop Write assessment items that are aligned to the Common Core State Standards for Mathematics Increase understanding of Smarter Balanced item specifications and other item-writing documents Involve Washington educators in the assessment process Provide participants with knowledge to share item-writing process with others and increase student achievement

Item Writing Considerations As reviewed yesterday: Alignment to CCSS-M Standards Item Specifications and Task Models Plain Language and Vocabulary Bias and Sensitivity Universal Design Cognitive Complexity And now the Style Guide Review 3rd – 5th bullet during training. Doesn’t seem to be a problem on example items. Have had experience with item specifications and task models.

Style Guide Highlights Use words for numbers that begin sentences. Consider rewording when the number is needed to solve mathematical problems. Use numerals for: numbers 10 and above numbers that precede abbreviated units of measure numbers used to solve mathematical problems Use a comma: in numbers with five or more digits numbers with four digits only if other numbers in the item have five or more digits. How many copies do we need? What is most important?

Style Guide Highlights Fractions: as words, fractions are hyphenated. as numerals, fractions are stacked vertically Time: use lower case abbreviations “a.m.” and “p.m.” use “noon” and “midnight” to avoid confusion Units: spell out units in item stems abbreviate units in options; do not include periods in abbreviated units page 73-74 give specific abbreviations

Style Guide Highlights Conditional clauses should be avoided if possible.

Claim 1 Item Writing Considerations What did we find out through item writing on moodle

Mathematical Practices Mathematical Practices that support Claim 1: MP 4: Model with mathematics MP 5: Use appropriate tools strategically. MP 6: Attend to precision. MP 7: Look for and make use of structure. MP 8: Look for and express regularity in repeated reasoning.

Metadata Table – Claim 1 Just given so can follow along….

Achievement Quintiles Five divisions are: 1 - Very easy 2 - Easy 3 - Medium 4 - Hard 5 - Very hard

Maximum, Minimum Grade Claim 1 items will often be written to a specific grade level so both maximum and minimum grade will be the same. Some items where there is a direct building of skills from grade to grade may make the maximum or minimum grade one above or below.

Score Points Maximum Points 0, 1 for a one-point item 0, 1, 2, for a two-point item. Maximum Points: 1 for a one-point item 2 for a two-point item.

Depth of Knowledge DOK 1—Recall; perform a rote procedure DOK 2—Application of Skill/Concept DOK 3—Strategic Thinking DOK 4—Synthesizing information and creating new example Most of Claim 1 items will be DOK 1 or 2 but watch what is written in the item specification documents. If multiple DOKs are listed, you must pick one.

Presentation Format T – Text TG – Text with Graphics

Allowable Response Types DAS Multiple Choice, single correct response (MC) Multiple Choice, multiple correct response (MS) Matching Tables T/F and Y/N variations (MA) ITS Equation/Numeric (EQ) Fill-in Table (TI) Drag and Drop, Hot Spot, and Graphing (GI) Used DAS for everything for the initial item writing. Now we will separate by authoring system.

Lessons Learned from Certification Set Successes Identifying the Task Model Following the Task Model rpdv, or rapid development, must be followed explicitly det, or detailed, has a bit more flexibility Alternatives for MC and MS items Rubric and Scoring information Foci of Attention Including only what you want the “student” to read in the item Rationales for MC and MS alternatives

Item Content - Student

Rationales MC and MS items need rationales for each alternative Use “Correct Answer” for correct answer(s) Item spec documents usually give examples of reasonable distractors (incorrect alternatives) You can include other types of distractors Describe, in positive terms, or show work as to why students would choose a distractor: The student added 3 + 4 before multiplying by 6 The student multiplied the exponents 4 and 7 instead of adding the exponents Student adds the leg lengths of the triangle and does not square the lengths first

Questions?

Steps to Write Item Work with a partner. Choose a standard from the folders and take an item-specific sheet. Check item specification for item considerations for the given target. Check task model rules for item type, cognitive complexity, required evidence, tools Write item. Follow Peer Review Process.

Peer Review Process Ask a pair of colleagues to work your item and make suggestions for edits. Submit for approval. Specialist will review item and provide feedback to writers and peer reviewers, if appropriate. Specialist may ask for revisions/edits. Writers make edits and resubmit item to same specialist.

Let the writing begin! Write 2 items in Claim 1

Claim 2 Item Writing

Claim 2 Students can solve a range of well-posed problems in pure and applied mathematics, making productive use of knowledge and problem-solving strategies.

Problem Solving Problem solving, which of course builds on a foundation of knowledge and procedural proficiency, sits at the core of doing mathematics. Proficiency at problem solving requires students to choose to use concepts and procedures from across the content domains and check their work using alternative methods. Smarter Balanced

Claim 2 Math Practices Mathematical Practices that support Claim 2: MP 1: Make sense of problems and persevere in solving them. MP 5: Use appropriate tools strategically. MP 7: Look for and make use of structure. MP 8: Look for and express regularity in repeated reasoning.

Secondary Claim Items written primarily to assess Claim 2 will necessarily involve some Claim 1 content targets. The content should draw upon the knowledge and skills articulated in the progression of standards leading up to and including the targeted grade Related Claim 1 targets should be listed below the Claim 2 targets in the metadata form.

Primary and Secondary Standard/Cluster While tasks developed to assess Claim 2 will have a primary content focus, components of these tasks will likely produce enough evidence for other content domains that a separate listing of these content domains needs to be included where appropriate.

Four Targets for Claim 2 Target A (TM1): Apply mathematics to solve well-posed problems in pure mathematics and arising in everyday life, society, and the workplace. Target B (TM2): Select and use appropriate tools strategically. Target C (TM3): Interpret results in the context of a situation. Target 4 (TM4): Identify important quantities in a practical situation and map their relationships (e.g., using diagrams, two-way tables, graphs, flowcharts, or formulas).

Assessment Targets Any given task should provide evidence for several of the preceding assessment targets; each of the targets should not lead to a separate task. Multiple targets should be listed in order of prominence as related to the task.

Activity Review the Claim 2 Document with your group. Distinguish between the various targets and related task models based on your grade band. Be prepared to speak to the differences between each target and task model.

Claim 2 Standards

Metadata One Claim 2 One Claim 1 Fill in rest of data

Metadata Two Claim 2 One Claim 1 Fill in rest of data

Metadata Two Claim 2 Two Claim 1 Fill in rest of data

A Review of Metadata Fields Mathematical Practices Achievement Quintiles Maximum, Minimum Grade Score Points Maximum Points Depth of Knowledge Presentation Formats Response Type

Scoring Scoring rules and answer choices will focus students’ ability to use the appropriate reasoning. Multiple correct responses are possible. MC and MS will be scored as correct/incorrect (1 point) If MA items require two skills, they will be scored as: All correct choices (2 points); at least ½ but less than all correct choices (1 point) Justification for more than 1 point must be clear in the scoring rules Where possible, include a “disqualifier” option that if selected would result in a score of 0 points, whether or not the student answered ½ correctly. EQ, GI, and TI items will be scored as: Single requirement items: will be scored as correct/incorrect (1 point) Multiple requirement items: All components correct (2 points); at least ½ but less than all correct choices (1 point) Justification for more than 1 point must be clear in the scoring rules. For a CAT item to score multiple points, either distinct skills must be demonstrated that earn separate points or distinct levels of understanding of a complex skill must be tied directly to earning one or more points.

Lessons Learned from Certification Set Successes Identifying the Task Model Following the Task Model rpdv, or rapid development, must be followed explicitly det, or detailed, has a bit more flexibility Alternatives for MC and MS items Rubric and Scoring information Foci of Attention Completing the metadata table Including only “student” information in item Rationales for MC and MS alternatives

Allowable Response Types DAS Multiple-Choice, single correct response (MC) Multiple Choice, multiple correct response (MS) Matching Tables T/F and Y/N variations (MA) ITS Equation/Numeric (EQ) Used DAS for everything for the initial item writing. Now we will separate by authoring system.

Problem Solving Contexts/Situations Brainstorm contexts/situations that are appropriate for your grade band Consider specific contexts from the designated standards for this claim Contexts/situations can include content from multiple domains.

Claim 2 Items Start with these item types: MC, MS, and EQ Do not copy examples or try to clone them. Items must be original and contain several standards that are in alignment with the chart for that grade level.

Item Writing Process Review Claim 2 document carefully. Work with a partner. Decide to which target you will write. Use the Claim 1 documents as a reference for how the problem is posed to assess Claim 1. Identify how the Claim 2 item you are writing differs from Claim 1 items.

Questions?

Debrief of Claim 2 How was this writing different from Claim 1: The process for Claims 3 and 4 is similar. How might you prepare to write these types of items? What are you next steps in your classroom, building, or district?

Day 2 Item Writing Wrap-up Parking Lot Questions/Comments Feedback Suggestions to improve process. Other questions? Evaluations Clock-hours Substitute reimbursement form

Thank you!

CLAIM 3 Item Writing

Claim 3 Students clearly and precisely construct viable arguments to support their own reasoning and to critique the reasoning of others.

Communicating Reasoning Communicating reasoning is the ability to construct and present a clear, logical, convincing argument. This is not just a requirement of the Standards for Mathematical Practice, it is also a recurrent them in the Standards for Mathematical Content. Many content standards call for students to explain, justify, or illustrate.

Claim 3 Mathematical Practices Mathematical Practices that support Claim 3: MP 3 – Construct viable arguments and critique the reasoning of others. MP 6 – Attend to precision.

Secondary Claim Items written primarily to assess Claim 3 will necessarily involve some Claim 1 content targets. The content should draw upon the knowledge and skills articulated in the progression of standards leading up to and including the targeted grade Related Claim 1 targets should be listed below the Claim targets in the metadata form.

Primary and Secondary Standard/Cluster While tasks developed to assess Claim 3 will have a primary content focus, components of these tasks will likely produce enough evidence for other content domains that a separate listing of these content domains needs to be included where appropriate.

Targets for Claim 3 Target A: Test propositions or conjectures with specific examples. Target B: Construct, autonomously, chains of reasoning that will justify or refute propositions or conjectures. Target C: State logical assumptions being used. Target D: Use the technique of breaking an argument into cases. Target E: Distinguish correct logic or reasoning from that which is flawed and – if there is a flaw in the argument – explain what it is. Target F: Base arguments on concrete referents such as objects, drawings, diagrams, and actions.

Assessment Targets Any given task should provide evidence for several of the preceding assessment targets; each of the targets should not lead to a separate task. Multiple targets should be listed in order of prominence as related to the task.

Task Models for Claim 3 TM1: Test propositions or conjectures with specific examples. TM2:Construct, autonomously, chains of reasoning that will justify or refute conjectures. TM3: Use the techniques of breaking an argument into cases. TM4: Determine conditions under which an argument does and does not apply. TM5: Distinguish correct reasoning from flawed reasoning.

Activity Review the Claim 3 document with your group. Distinguish between the various targets and related task models based on your grade band. Be prepared to speak to the differences between each target and task model.

Claim 3 CCSS Standards

Claim 3 CCSS Standards

Metadata One Claim 3 One Claim 1 Fill in rest of data

Metadata Two Claim 3 One Claim 1 Fill in rest of data

Metadata Two Claim 3 Two Claim 1 Fill in rest of data

A Review of Metadata Fields Mathematical Practices Achievement Quintiles Maximum, Minimum Grade Score Points Maximum Points Depth of Knowledge Presentation Formats Response Type

Scoring Scoring rules and answer choices will focus students’ ability to use the appropriate reasoning. Multiple correct responses are possible. MC and MS will be scored as correct/incorrect (1 point) If MA items require two skills, they will be scored as: All correct choices (2 points); at least ½ but less than all correct choices (1 point) Justification for more than 1 point must be clear in the scoring rules Where possible, include a “disqualifier” option that if selected would result in a score of 0 points, whether or not the student answered ½ correctly. EQ, GI, and TI items will be scored as: Single requirement items: will be scored as correct/incorrect (1 point) Multiple requirement items: All components correct (2 points); at least ½ but less than all correct choices (1 point) Justification for more than 1 point must be clear in the scoring rules. For a CAT item to score multiple points, either distinct skills must be demonstrated that earn separate points or distinct levels of understanding of a complex skill must be tied directly to earning one or more points.

Lessons Learned from Certification Set Successes Identifying the Task Model Following the Task Model Alternatives for MC and MS items Rubric and Scoring information Foci of Attention Completing the metadata table Including only “student” information in item Rationales for MC and MS alternatives

Communicating Reasoning in Contexts/Situations Brainstorm contexts/situations that are appropriate for your grade band Consider specific contexts from the designated standards for this claim Contexts/situations can include content from multiple domains.

Claim 3 Items Start with these item types: MC, MS, MA, and EQ Write items that are not copies or clones of the examples. Items must be original and contain several standards that are in alignment with the chart for that grade level. Check that the item doesn’t follow a Claim 1 task model.

Item Writing Process Review Claim 3 document carefully. Work with a partner. Decide to which target you will write. Use the Claim 1 documents as a reference for how the problem is posed to assess Claim 1. Claim 3 will NOT be assessed in that manner. Follow the peer review process.

Questions?

Debrief of Claim 3 How was this writing different from Claim 1? The process for Claims 2 and 4 is similar. How might you prepare to write these types of items? What are your next steps in your classroom, building, or district?

Day 2 Item Writing Wrap-up Parking Lot Questions/Comments Feedback Suggestions to improve process. Other questions? Evaluations Clock-hours Substitute reimbursement form

Thank you!

CLAIM 4 Item WRiting

Claim 4 Students can analyze complex, real-world scenarios and can construct and use mathematical models to interpret and solve problems.

Modeling and Data Analysis Modeling is the process of choosing and using appropriate mathematics and statistics to analyze empirical situations, to understand them better, and to improve decision-making. Smarter Balanced

Claim 4 Mathematical Practices Mathematical Practices that support Claim 4: MP 2 – Reason abstractly and quantitatively. MP 4 – Model with mathematics. MP 5 – Use appropriate tools strategically.

Secondary Claim Items written primarily to assess Claim 4 will necessarily involve some Claim 1 content targets. The content should draw upon the knowledge and skills articulated in the progression of standards leading up to and including the targeted grade Related Claim 1 targets should be listed below the Claim 4 targets in the metadata form.

Primary and Secondary Standard/Cluster While tasks developed to assess Claim 4 will have a primary content focus, components of these tasks will likely produce enough evidence for other content domains that a separate listing of these content domains needs to be included where appropriate.

Targets for Claim 4 Target A (TM1): Apply mathematics to solve well-posed problems in pure mathematics and arising in everyday life, society, and the workplace. Target B (TM 2): Construct, autonomously, chains of reasoning to justify mathematical models used, interpretations made, and solutions proposed for a complex problem. Target C (TM3): State logical assumptions being used. Target D (TM 4): Interpret results in the context of a situation.

Targets for Claim 4 Target E (TM 5): Analyze the adequacy of and make improvements to an existing model or develop a mathematical model of a real phenomenon. Target F (TM 6): Identify important quantities in a practical situation and map their relationships. Target G: Identify, analyze, and synthesize relevant external resources to pose or solve problems.

Assessment Targets Any given task should provide evidence for several of the preceding assessment targets; each of the targets should not lead to a separate task. Multiple targets should be listed in order of prominence as related to the task.

Distinguishing Claim 4 Distinguishing between Claim 4 and Claims 1 and 2: In early grades when equations are still new to students, an important distinction between Claim 2 and Claim 4 is requiring a model that would lead to a problem's solution. In Claim 2 problems are well posed, while in Claim 4 they may have extraneous or missing information. In Claims 1 and 2, measurements of objects or figures can be accurately determined. In Claim 4, modeling is used to make approximations. In Claim 1, data analysis is straightforward procedural. In Claim 4, the analysis should be tied to some useful purpose in the real world.

Activity Review the Claim 4 Document with your group. Distinguish between the various targets and related task models based on your grade band. Be prepared to speak to the differences between each target and task model.

Claim 4 CCSS Standards

Metadata One Claim 4 One Claim 1 Fill in rest of data

Metadata Two Claim 4 One Claim 1 Fill in rest of data

Metadata Two Claim 4 Two Claim 1 Fill in rest of data

A Review of Metadata Fields Mathematical Practices Achievement Quintiles Maximum, Minimum Grade Score Points Maximum Points Depth of Knowledge Presentation Formats Response Type

Scoring Scoring rules and answer choices will focus students’ ability to use the appropriate reasoning. Multiple correct responses are possible. MC and MS will be scored as correct/incorrect (1 point) If MA items require two skills, they will be scored as: All correct choices (2 points); at least ½ but less than all correct choices (1 point) Justification for more than 1 point must be clear in the scoring rules Where possible, include a “disqualifier” option that if selected would result in a score of 0 points, whether or not the student answered ½ correctly. EQ, GI, and TI items will be scored as: Single requirement items: will be scored as correct/incorrect (1 point) Multiple requirement items: All components correct (2 points); at least ½ but less than all correct choices (1 point) Justification for more than 1 point must be clear in the scoring rules. For a CAT item to score multiple points, either distinct skills must be demonstrated that earn separate points or distinct levels of understanding of a complex skill must be tied directly to earning one or more points.

Lessons Learned from Certification Set Successes Identifying the Task Model Following the Task Model rpdv, or rapid development, must be followed explicitly det, or detailed, has a bit more flexibility Alternatives for MC and MS items Rubric and Scoring information Foci of Attention Completing the metadata table Including only “student” information in item Rationales for MC and MS alternatives

Modeling Contexts/Situations Do we Brainstorm contexts/situations that are appropriate for your grade band Consider specific contexts from the designated standards for this claim Contexts/situations can include content from multiple domains.

Claim 4 Items Start with these item types: MC, MS, and EQ Do not copy examples or try to clone them. Items must be original and contain several standards that are in alignment with the chart for that grade level.

Item Writing Process Review Claim 4 document carefully. Work with a partner. Decide to which target you will write. Use the Claim 1 documents as a reference for how the problem is posed to assess Claim 1. Identify how the Claim 4 item you are writing differs from Claim 1 items.

Questions?

Debriefing Claim 4 How was this writing different from Claim 1? The process for Claims 2 and 3 is similar. How might you prepare to write these types of items? What are your next steps in your classroom, building, or district?

Day 2 Item Writing Wrap-up Parking Lot Questions/Comments Feedback Suggestions to improve process. Other questions? Evaluations Clock hours Substitute reimbursement form

Thank you