1. What Does Smarter Balanced Want Students to Know and Show? Heather Dallas, The Curtis Center at UCLA Shelbi Cole, Student Achievement Partners Judy Hickman, Smarter Balanced at UCLA California Math Council South Conference November 6, 2015

2. Why New Assessments?

3. Why are we changing what students need to know and be able to do? Me: My daughter’s fingernails appear to be detaching at the base. Could that be related to the illness she had four weeks ago? Doc: [Gets on his computer to examine current research.] Do you want a pediatrician who relies on the most recent information or what he/she learned 20 years ago?

4. What does his future look like?

5. What skill set do his future employers value?

6. A Balanced Assessment System Common Core State Standards specify K-12 expectations for college and career readiness Common Core State Standards specify K-12 expectations for college and career readiness All students leave high school college and career ready Teachers and schools have information and tools they need to improve teaching and learning Interim assessments Flexible, open, used for actionable feedback Summative assessments Benchmarked to college and career readiness Teacher resources for formative assessment practices to improve instruction

7. Building the Math Item Pool for an Adaptive Test 7

8. The CCSSM Requires Three Shifts in Mathematics Focus strongly where the standards focus Coherence: Think across grades and link to major topics within grades Rigor: In major topics, pursue conceptual understanding, procedural skill and fluency, and application with equal intensity 8

9. Mathematics topics intended at each grade by at least two- thirds of A+ countries Mathematics topics intended at each grade by at least two- thirds of 21 U.S. states Shift #1: Focus Strongly where the Standards Focus The shape of math in A+ countries 1 Schmidt, Houang, & Cogan, “A Coherent Curriculum: The Case of Mathematics.” (2002). 9

10. 10 Grade Focus Areas in Support of Rich Instruction and Expectations of Fluency and Conceptual Understanding K–2 Addition and subtraction - concepts, skills, and problem solving and place value 3–5 Multiplication and division of whole numbers and fractions – concepts, skills, and problem solving 6 Ratios and proportional reasoning; early expressions and equations 7 Ratios and proportional reasoning; arithmetic of rational numbers 8 Linear algebra and linear functions Shift #1: Focus Key Areas of Focus in Mathematics

11. 11 Shift #2: Coherence Think Across Grades 4.NF.4. Apply and extend previous understandings of multiplication to multiply a fraction by a whole number. 5.NF.4. Apply and extend previous understandings of multiplication to multiply a fraction or whole number by a fraction. 5.NF.7. Apply and extend previous understandings of division to divide unit fractions by whole numbers and whole numbers by unit fractions. 6.NS. Apply and extend previous understandings of multiplication and division to divide fractions by fractions. 6.NS.1. Interpret and compute quotients of fractions, and solve word problems involving division of fractions by fractions, e.g., by using visual fraction models and equations to represent the problem. Grade 4 Grade 5 Grade 6

12. Shift #3: Rigor Required Fluencies for Grades K-6 12 GradeStandardRequired Fluency KK.OA.5Add/subtract within 5 11.OA.6Add/subtract within 10 2 2.OA.2 2.NBT.5 Add/subtract within 20 (know single-digit sums from memory) Add/subtract within 100 3 3.OA.7 3.NBT.2 Multiply/divide within 100 (know single-digit products from memory) Add/subtract within 1000 44.NBT.4Add/subtract within 1,000,000 55.NBT.5Multi-digit multiplication 66.NS.2,3 Multi-digit division Multi-digit decimal operations

13. Using Computer Adaptive Technology for Summative and Interim Assessments Provides accurate measurements of student growth over time Increased precision Item difficulty based on student responses Tailored for Each Student Larger item banks mean that not all students receive the same questions Increased Security Fewer questions compared to fixed form tests Shorter Test Length Turnaround time is significantly reduced Faster Results GMAT, GRE, COMPASS (ACT), Measures of Academic Progress (MAP) Mature Technology

14. How CAT Works (Binet’s Test)

15. Rating Item Difficulty

16. Percentage of 4 th graders getting problems like these correct (Based on spring 2014 data) Enter a fraction equal to 1/3.

17. Implementing at Scale: Item Specifications 17

18. Difficulty vs. Complexity The adaptive test will select items of appropriate difficulty level, but all students will see items across the full range of available complexity. Difficulty is about how hard the item is (i.e., how many students got it right); while complexity is about the kind of thinking a student needs to do to solve the problem. 18

19. How do we ensure that “difficult” items are difficult for the right reasons? A set of item quality criteria is being applied to the evaluation of items in the Smarter Balanced bank, and one is about the time vs. information tradeoff from an item: 4a. Is the time spent on the item due to the required mathematics or due to the complexity of the item itself? 19

20. Will the CAT select items for a student aligned to standards in other grade levels? Once 2/3 of items required by the blueprint have been delivered in a student’s test, the pool may be expanded to include items with primary alignment to other grade levels for students performing at the highest and lowest performance levels –The goal is to increase precision of measurement for very high and very low achieving students (minimize error in student’s score) 20

21. Why Update Item Specifications? Increase transparency for stakeholders. Enhancements include: –More items to illustrate each assessment target –Enhanced descriptions for how to adjust the difficulty of items within a target –Clear labeling of which items allow calculators or other tools –Reveal new item types planned for future pilot testing –Remove any task models that were unsuccessful during pilot testing phase 21

22. Did you know? There are over 1,000 example items for mathematics across the Smarter Balanced grades 3-8 item specifications! What are we waiting for? Let’s take a look! 22

23. 23 Digging Into the Item Specifications

24. Did you know over 1000 math items can be found in the Mathematics Item Specifications? Mathematics Content Specifications CAT Item Specs, Version 3 (9/28/2015) Mathematics PT Item Specs All Grades (PDF) (5/27/14) Mathematics PT Item Specs All Grades Scoring Guide for Selected Short-Text Mathematics Items http://www.smarterbalanced.org/smarter-balanced-assessments

25. Eliciting Problem Solving: Claim 2 Problem solving sits at the core of doing mathematics. Proficiency at problem solving requires students to choose to use concepts and procedures from across the content domains and check their work using alternative methods. 25

26. Distinctives of Claim 2 Items: Multiple approaches are feasible or a range of responses is expected (e.g., if a student can solve a word problem by identifying a key word or words and selecting operations, then it is Claim 1). The use of tools in Claim 2 is intended to support the problem solving process. In some cases, students may be asked to display their answer on the tool (e.g., by clicking the appropriate point or interval on a number line or ruler). Assessing the reasonableness of answers to problems is a Claim 2 skill. 26

27. Claim 2 Exemplars 27

28. Claim 2 Exemplars 28

29. Claim 2 Exemplars 29

30. Claim 2 Exemplars 30

31. Claim 2 Exemplars 31

32. Claim 2 Exemplars 32

33. Claim 2 Exemplars 33

34. Eliciting Reasoning: Claim 3 This claim refers to a recurring theme in the CCSSM content and practice standards: the ability to construct and present a clear, logical, convincing argument. Assessment tasks that address this claim will typically present a claim or a proposed solution to a problem and will ask students to provide, for example, a justification, and explanation, or counter-example. 34

35. Distinctives of Claim 3 Items: Items and task assessing Claim 3 may involve application of more than one [content] standard. The focus is on communicating reasoning rather than demonstrating mathematical concepts or simple applications of mathematical procedures. Targeted content standards for Claim 3 should belong to the major work of the grade. 35

36. Claim 3 Exemplars 36

37. Claim 3 Exemplars 37

38. Claim 3 Exemplars 38

39. Claim 3 Exemplars 39

40. Eliciting Modeling: Claim 4 Modeling is the process of choosing and using appropriate mathematics and statistics to analyze empirical situations, to understand them better, and to improve decision-making 40

41. Distinctives of Claim 4 Items: Claim 4 differ from those in Claim 2, because while the goal is clear, the problems themselves are not yet fully formulated (well-posed) in mathematical terms. Claim 4 items and tasks should sample across the content domains, with many of these involving more than one domain. Because of the high strategic demand that substantial non-routine tasks present, the technical demand will be lower—normally met by content first taught in earlier grades. 41

42. Claim 4 Exemplars 42

43. Claim 4 Exemplars 43

44. Claim 4

45. Claim 4 Exemplars 45

46. Development Processes

47. Item Development Considerations Engaging for all students Range of items of high and low difficulty New item types for computer response capture and real-time scoring Robust tagging for interoperability

48. Operational Blueprint Requirements Field Test Item Development Plan Content and Item Specifications Tagging requirements Claim, Target, Task Model Item Type Scoring method Cognitive Complexity Difficulty Accessibility, Bias/Sensitivity, other Guidelines

49. Specifications/Archetypes ●Item Specifications ●Reflect lessons from pilot ●Content reviews ●Accessibility, bias, sensitivity reviews ●Archetypes ●Models ●Content reviews ●Accessibility, bias, sensitivity reviews ●Language Complexity Rubric Foundation of Development Approach

50. Item Types Multiple Choice Multiple Select Hot Spot Equation/Numeric Graphing Fill In Tables Matching Tables Drag and Drop Short text

51. Item Quality - Supporting Activities Item Reviews –Occur in advance of audits –Independent but managed by CTB –Items housed in DAS and ITS systems –Comments evaluated for item revision, approvals Item Audits –Final quality control activity before release –Managed by Smarter Balanced –Items housed in ITS, feedback collected on Google

52. Quality Criteria Purpose Define characteristics of Smarter Balanced items/tasks to ensure high quality items Provide item authors with quality guidelines to apply during development Provide item reviewers with quality guidelines for item/task review Role Expand traditional item development quality guidelines Ensure focus on important elements of evidence and alignment Provide gatekeeper criteria to focus authoring and review Guide overall quality training and processes

53. Language Complexity Cook, H. G. & MacDonald, R. (2013). Tool to Evaluate Language Complexity of Test Items (WCER Working Paper No. 2013-5). Retrieved from University of Wisconsin– Madison, Wisconsin Center for Education Research website: http://www.wcer.wisc.edu/publications/workingPapers/papers.phphttp://www.wcer.wisc.edu/publications/workingPapers/papers.php

54. Questions?

55. Heather Dallas: dallas@math.ucla.edu Shelbi Cole: scole@studentsachieve.net Judy Hickman: judy.hickman@smarterbalanced.org California Math Council South Conference November 6, 2015