1. Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 1 Building an Assessment System for Learning Paul Nichols, Ph.D. Vice President Assessment & Information Pearson Race To the Top Assessment Program Atlanta, GA November 17 & 18

2. Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 2 In my presentation, … Invitation to think differently about assessment –The distinction between formative and summative assessment –The attitude that educational measurement is quantification –The definition of an assessment as a set of items in a test form Vision of an assessment system as a system for learning Technical qualities of this vision

3. Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 3 Distinction between formative and summative assessment No such thing as a formative assessment Assessment by itself is like one hand clapping Assessment information must fit as a component of a system of coordinated assessment and instruction Information from the same assessment may be used for formative or summative purposes

4. Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 4 Attitude that educational measurement is quantification “It is only a slight exaggeration to describe the test theory that dominates educational measurement today as the application of 20th century statistics to 19th century psychology” (Mislevy, 1993) Modern theories portray learning as reorganizing, restructuring, evolving Assessment system requires fidelity between our understanding of learning & our method of assessing Consequences of lack of fidelity may be little, none, or even negative student learning

5. Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 5 Definition of an assessment Not a set of items in a test form Think about assessment as arranging opportunities for the student to show what they know and can do Encompasses a multiple-choice item, a teacher working one-on-one with a student and a computer-based, interactive simulation

6. Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 6 Vision of a system for learning

7. Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 7 Learning progression The glue for the system for learning Scale representing achievement and growth for both classroom and large-scale assessments Learning progressions can be broad and span years, exploded to more detail and span weeks Guides the types of assessment tasks to fit different points on the learning progression Supports pedagogical practices, e.g., selection of curricular tasks, differentiating instruction for small groups, and suggesting remedial interventions

8. Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 8 Large-scale assessment Computer-delivered and, wherever possible, objectively scored Includes variety of formats, e.g., simulation- and scenario-based tasks, multiple-choice questions, concepts mapping activities Goal is fidelity & efficiency in assessing targeted knowledge and skills Serve as external moderator for classroom assessment More narrow representation of the construct and a one-time sample of student performance Perceived as more reliable and independent

9. Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 9 Classroom assessment Classroom-situated, teacher-delivered but may not be teacher-developed Used for diagnostic & summative purposes Include variety of formats, e.g., performance-based tasks, multiple-choice questions and concepts mapping activities Teachers as primary assessors, must be given adequate support to make these judgments Potential to assess broader aspects and over time Not independent, susceptible to inconsistency & bias Challenge is the need to record sufficient information

10. Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 10 Online support bank Supports the everyday practices of teachers Teacher guidelines for instructional & assessment activities Learning examples such as annotated student performances in text and video formats Professional resources such as links to articles or presentations of professional interest An online forum for informal teacher collaboration and discussion about instruction and assessments

11. Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 11 Data capability Collect, aggregate and interpret data across observations, time, students and classrooms Common standard for data critical to aggregation Common understanding of learning critical to interpretation Data capture must not be intrusive for classroom assessment Summative information from classroom & large- scale assessment combined & weighted equally Information is ordinal in nature but more consumable by teachers, students, parents & policymakers

12. Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 12 Practicality System of coordinated assessment and instruction that I have described is more than a vision of a possible system for learning Existing systems demonstrate the possibilities of such a system The GISMO research laboratory at North Carolina State University working with North Carolina (http://gismo.fi.ncsu.edu/index.php)http://gismo.fi.ncsu.edu/index.php System from Queensland, Australia, as reviewed in Stanley, Gardner, Reynolds and Wild (2009)

13. Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 13 Technical qualities of this vision What is a reliable system for learning? –Reliability for system for learning focuses on outcome, i.e., student learning –How consistent is student learning across classrooms, ethnic groups, cohorts? How do you argue for validity? –Stress the consequences of test score use –Make clear the components that lead to learning gains How do you create vertical alignment of the entire system? –Learning progression aligns components with each other and across grades –Evidence-centered process linking components to the learning progression

14. Copyright © 2007 Pearson Education, inc. or its affiliates. All rights reserved. 14 Finally, technical qualities for developing this vision Investment Patience Perseverance