University of Colorado at Boulder Lorrie A. Shepard School of Education University of Colorado at Boulder Conceptual Coherence between Large-Scale and Classroom Assessments CRESST Conference UCLA September 9, 2004

University of Colorado at Boulder Contrasts between large-scale and classroom level assessments that imply that different assessments are needed standardized vs. dynamic uniform date vs. variable dates independent performance vs. assisted performance delayed vs. immediate feedback stringent requirements for technical accuracy vs. less stringent requirements The most important shared characteristics of the two levels of assessment should be alignment with the same curriculum standards and a shared model of learning.

University of Colorado at Boulder For an assessment system to support learning, it must have coherence. The conceptual base or models of student learning underlying both external and classroom assessments must be compatible. Knowing What Students Know Pellegrino, Chudowsky, & Glaser NRC, 2001

University of Colorado at Boulder Committee on Assessment in Support of Instruction and Learning A joint committee of Board on Testing and Assessment Committee on Science Education K-12 Mathematical Sciences Education Board Bridging the Gap criteria that characterize an ideal system, distilled from:  NRC Science Education Standards  NCTM Assessment Standards  Commission on Instructionally Supportive Assessment

University of Colorado at Boulder Shared Characteristics of Large-Scale and Classroom Assessments A shared model of student learning including developmental progressions over time A shared conception of disciplinary knowledge and competence  Focus on evaluating understanding and reasoning  Offer a clear vision of what constitutes mastery  Target both general and domain-specific forms of cognition  Select complex, context-based tasks that are open to multiple approaches

University of Colorado at Boulder This idea of alignment between large-scale and classroom assessments refers  Not to the limited alignment where test publishers show fit within test blueprints,  But to the more complete and substantive alignment that occurs when the tasks, problems, and projects in which students are engaged represent the range and depth of what we say we want students to understand and be able to do.  Perhaps a better word would be embodiment. Shepard, 2003, NSTA

University of Colorado at Boulder A Sampler of Science Assessment © California Department of Education Part B

University of Colorado at Boulder A Sampler of Science Assessment © California Department of Education

University of Colorado at Boulder © Wyoming Body Evidence Activities Consortium

University of Colorado at Boulder © Wyoming Body Evidence Activities Consortium

University of Colorado at Boulder Note that good assessment tasks are interchangeable with good instructional tasks. (The exact same task should not be used for both purposes, however.) We also have evidence that “teaching to” problem types like these improves learning.

University of Colorado at Boulder Shepard, Flexer, Hiebert, Marion, Mayfield, & Weston (1996).

University of Colorado at Boulder Effect Size: AP – PaceSetter Students =.64 on SAT I, 1.58 on AP Calculus, and.21 on Open-Ended Tasks

University of Colorado at Boulder Purposes Served by Large-Scale Assessment Data for monitoring and accountability Exemplification of learning goals Program “Diagnosis” Professional Development

University of Colorado at Boulder Program Diagnosis notes from Delaware’s Rachel Wood and Julie Schmidt …an 8 th -grade weather assessment revealed that students across the state had over-generalized their knowledge of the movement of all air masses as having to go from west to east. …lead teachers were stunned that their own students could not answer a question that involved drawing conclusions from a simple graph. A commentary around a released item on Punnett squares attempts to turn teachers’ attention to thinking about why students have acquired only a mechanical sense.

University of Colorado at Boulder Professional Development Assessment-focused professional development deepens understanding of standards and directs attention to what students are getting from new activities. Given time burdens, teachers can learn formative assessment strategies while working to raise achievement on accountability tests. Assessment efforts only make sense if they are intimately tied to content learning.

University of Colorado at Boulder  A developmental framework and progress variables provide a substantive basis for monitoring student progress over time.  “Large-scale assessment tools and supporting instructional materials should be developed so that clear learning goals and landmark performances along the way to competence are shared…”

University of Colorado at Boulder Towards Coherence Between Classroom Assessment and Accountability NSSE Yearbook 2004 M. Wilson, Editor Forster & Masters (Australia) Closely articulated, classroom-level developmental assessments National survey assessments Both tied to progress maps and accompanying curriculum materials Progress maps are broad, general descriptions of how proficiency typically develops in a curricular area. Accountability data collection embedded in classrooms with teachers as judges

University of Colorado at Boulder NSSE Yearbook 2004 Wilson & Draney (BEAR Assessment System) BEAR follows a developmental perspective, is built around a set of progress variables that reflect the way that increasing proficiency develops. (In contrast to cross-sectional and smorgasbord type assessments typical in the U.S.) Coherence between classroom and external tests requires a shared understanding of the construct at the level of the progress dimension and the level of specific assessment tasks and scoring guides.

University of Colorado at Boulder  Note that KWSK said that progress variables provide a substantive basis for monitoring student progress.  Developmental continua should not rely on either statistically derived trajectories or aspirational benchmarks. Instead progress maps should represent the pathways of real students’ learning gains under conditions of reasonably adequate instruction. Caution in Developing Growth Trajectories

University of Colorado at Boulder

Queensland Shavelson  Queensland replaced A levels with formative assessments to meet students’ needs and to serve as a basis for summative information beyond the classroom.  Queensland’s system is intended to audit local implementation of curriculum, assessment, and accountability.  A process for moderation of scores ensures consistency across classes and schools.  Australian examples are quite different from U.S. context because of the feasibility of a shared curriculum and trust in teachers’ judgments.

University of Colorado at Boulder Delaware: Comprehensive Science Assessment Rachel Wood  Curriculum materials developed to support state standards.  Prerequisites for meeting curricular objectives were identified grade-by-grade.  Then, end-of-unit assessments were developed.  Generic rubrics were not diagnostic, replaced with specific rubrics to identify misconceptions.  State, with some outside resources, provided intensive professional development for a cadre of teachers.

University of Colorado at Boulder Déjà Vu All Over Again  Assessment reformers used the terms “Performance,” (Resnick & Resnick, 1992), “Direct,” (Frederiksen & Collins, 1989), and “Authentic” Assessment (Wiggins, 1989) to convey the idea that assessments must capture real learning goals.  Embodying worthy goals is still a core issue.  KWSK’s ideal system will require much more than token constructed-response items on accountability tests.