2. School of Education, University of Colorado at Boulder Assessment in Support of Learning Lorrie A. Shepard University of Colorado at Boulder Lorrie.Shepard@Colorado.edu Math/Science Partnerships Workshop Assessment of Student Learning February 1, 2004 School of Education, University of Colorado at Boulder

3. Goals for my talk:  Provide a big-picture overview of changes need in classroom assessment to support learning  And correspondingly, the changes needed in large-scale assessment  Anticipate and set the stage for other presenters  Demonstrate the connection of these principles to research evidence

4. School of Education, University of Colorado at Boulder To support learning, classroom assessment must change in two fundamentally important ways. First, its form and content must be changed to better represent important thinking and problem solving skills in each of the disciplines. Second, the way that assessment is used in classrooms and how it is regarded by students must change. Shepard, 2000, “The Role of Assessment in a Learning Culture”

5. School of Education, University of Colorado at Boulder Correspondingly, the content of large-scale assessments must change to more fully capture rich learning goals and knowledge progressions. KWSK uses the term Coherence to talk about the need for conceptual alignment between the model of student learning underlying both classroom and large-scale assessments. Knowing What Students Know Pellegrino, Chudowsky, & Glaser NRC, 2001

6. School of Education, University of Colorado at Boulder In the U.S. assessment reform began in the late 1980s - early 1990s in response to evidence of the negative effects of high-stakes testing.  Teacher survey data showed declines in the teaching of science and social studies.  Teacher survey data and curriculum studies showed close parallels between test formats and instructional materials.  Test score gains were shown not to generalize to independent measures.  Research on teaching-the-test effects can be tied to cognitive studies on learning transfer (or lack of transfer).  Phil Sadler’s films, A Private Universe and Minds of Our Own are also examples.

7. School of Education, University of Colorado at Boulder According to research on basic-skills, standardized testing in the 1980s, test- driven reforms produce:  test score inflation and  curriculum distortion “Although embarrassing to some State policymakers, the Lake Wobegon report illustrated the potential mischief caused by high- stakes testing: higher test scores without more learning.” OTA, 1992

8. School of Education, University of Colorado at Boulder “The tests most commonly taken by students – both standardized tests and textbook tests – emphasize and mutually reinforce low level thinking and knowledge, and were found to have an extensive and pervasive influence on math and science instruction nationwide.” Madaus, West, Harmon, Lomax, & Viator, 1992.

13. Assessment reformers used the terms “Performance,” (Resnick & Resnick, 1992) “Direct,” (Frederiksen & Collins, 1989) and “Authentic” Assessment (Wiggins, 1989) to convey the idea that assessments must capture real learning if they are to avoid distorting instruction.

14. School of Education, University of Colorado at Boulder  The single most important shared characteristic of large-scale and classroom assessments should be their alignment with curriculum standards.  Not the limited alignment where test publishers show fit within test blueprints,  But the more complete and substantive alignment that occurs when the tasks, problems, and projects in which students are engaged represent the range and depth of what we say we want students to understand and be able to do.  Perhaps a better word would be embodiment. Shepard, 2003, NSTA

15. School of Education, University of Colorado at Boulder

16. A Sampler of Science Assessment © California Department of Education

17. School of Education, University of Colorado at Boulder  The single most important shared characteristic of large-scale and classroom assessments should be their alignment with curriculum standards.  Not the limited alignment where test publishers show fit within test blueprints,  But the more complete and substantive alignment that occurs when the tasks, problems, and projects in which students are engaged represent the range and depth of what we say we want students to understand and be able to do.  Perhaps a better word would be embodiment. Shepard, 2003, NSTA

18. School of Education, University of Colorado at Boulder

19. A Sampler of Science Assessment © California Department of Education

20. School of Education, University of Colorado at Boulder Note that good assessment tasks are interchangeable with good instructional tasks. (The exact same task should not be used for both purposes, however.) We also have evidence that “teaching to” problem types like these improves learning.

21. School of Education, University of Colorado at Boulder Shepard, Flexer, Hiebert, Marion, Mayfield, & Weston (1996).

22. School of Education, University of Colorado at Boulder Effect Size: AP – PaceSetter Students =.64 on SAT I, 1.58 on AP Calculus, and.21 on Open-Ended Tasks

23. School of Education, University of Colorado at Boulder

24. Formative Assessment in Other Countries Researchers in England, Australia, New Zealand, Scotland, Wales opposed standardized tests by focusing on formative assessment. (Crooks, 1988) motivational and cognitive effects of classroom assessment. (Sadler, 1989) a model of formative assessment that includes feedback and self-monitoring. (Assessment Reform Group, 1999; Black & Wiliam, 1998), “assessment for learning.”

25. School of Education, University of Colorado at Boulder Assessment of Learning vs. Assessment for Learning Assessment Reform Group England, Northern Ireland, Scotland, Wales

26. School of Education, University of Colorado at Boulder “the long-term exposure of students to defective patterns of formative assessment” has developed embedded coping responses that will take ingenuity, patience and time on the part of educators to reverse.” (Sadler, 1998) “…every teacher who wants to practice formative assessment must reconstruct the teaching contract…” (Perrenoud, 1991) “`Constructing the way forward’ was (a type of feedback) used by teachers to articulate future possibilities in learning in a way that looked like a partnership with the child.” (Tunstall & Gipps, 1996)

27. School of Education, University of Colorado at Boulder Paul Black & Dylan Wiliam (1998). Assessment and Classroom Learning  “Assessment becomes ‘formative assessment’ when evidence is actually used to adapt the teaching work to meet student needs.”  Formative assessment experiments produce typical effect sizes of between.40 and.70. Such effects are larger than most found for educational interventions.  Many of the studies reviewed show that improved formative assessment helps low achievers more than others, thereby reducing the spread of attainment while also raising achievement overall.

28. School of Education, University of Colorado at Boulder Atkin, Black, & Coffey, NRC, 2001 Illustrates use of basic formative assessment model:  Where are you trying to go?  Where are you now?  How can you get there?

29. School of Education, University of Colorado at Boulder Formative Assessment // Instructional Scaffolding  The formative assessment model is more than its data-gathering step.  FA model directly corresponds to the zone of proximal development (ZPD), the imaginary place on a learning continuum between what a child can do independently and what the same child can do with assistance.  Scaffolding refers to the support – guidance, coaching, hints, and encouragement – that adults provide in the ZPD to enable and challenge the learner to perform at a level that she would not otherwise be able to reach.

30. School of Education, University of Colorado at Boulder

32. Prior knowledge Prior knowledge techniques – Instructional conversations (Tharp & Gallimore, 1988) K-W-L (Ogle, 1986) are not seen as assessments but yield valuable data for revising instruction. Prior knowledge is more than facts amassed at home. It includes language patterns and ways of thinking developed through social roles and cultural experiences. Moll, Amanti, Neff, & Gonzalez (1992) use "funds of knowledge" to describe household knowledge of children from poor families based on farming,carpentry, childcare, cooking, medicine, religion, and budget management that can be used to support school knowledge.

33. School of Education, University of Colorado at Boulder Feedback Feedback is most effective when it focuses on correction of errors in relation to original learning goals. “A key role of the scaffolder is to summarize the progress that has been made and point out behaviors that led to the successes … One type of feedback is pointing out the distinction between performance and the ideal … another type of feedback is attributing success to effort … and explicitly restating the concept that has been learned” (Hogan & Pressley, 1997)

34. School of Education, University of Colorado at Boulder An Intervention Study Elawar & Corno (1985). A factorial experiment in teachers’ written feedback on student homework: changing teacher behavior a little rather than a lot, Journal of Educational Psychology. Study design: Teachers were trained to give written feedback focused on specific errors and poor strategy with suggestions about how to improve. The control group received grades on homework but no comments. Findings: The effect of focused feedback on final achievement was as great as the effect of prior achievement. There were also large positive effects on attitudes toward mathematics and the initial superiority of boys over girls was reduced.

35. School of Education, University of Colorado at Boulder Student Work as a Display of Thinking

36. School of Education, University of Colorado at Boulder

37. Teaching and Assessing for Transfer “Teaching for Robust Understandings” Multiple Ways to Ask about the Concept One-Half 1. Circle each shape that has one-half shaded Source: Assessing Mathematical Understanding, 1989

38. School of Education, University of Colorado at Boulder Measuring Up, Mathematical Sciences Education Board, National Research Council, 1993.

39. School of Education, University of Colorado at Boulder Measuring Up, Mathematical Sciences Education Board, National Research Council, 1993.

40. School of Education, University of Colorado at Boulder Explicit criteria The transition from feedback to self- monitoring can occur only when the student comes to know what constitutes quality. (Sadler, 1989) “The assessment system (should) provide a basis for developing a metacognitive awareness of what are important characteristics of good problem solving, good writing, good experimentation, good historical analysis, and so on.” (Frederiksen & Collins, 1989)

41. School of Education, University of Colorado at Boulder Student self assessment promises to increase students’ responsibility for their own learning. In case studies, students became more interested in the criteria and substantive feedback than grades… more honest about their own work, fair with other students, and able to defend their opinions in terms of the evidence. (Klenowski, 1995) Self assessment

42. School of Education, University of Colorado at Boulder An Intervention Study White & Frederiksen (1996). The Thinker Tools Inquiry Project: Making Scientific Inquiry Accessible to Students Center for Performance Assessment, Educational Testing Service. Assessment criteria were developed for attributes desired while conducting investigations in science. Students engaged in a set of activities to foster “reflective assessment.” è At several stages in the Inquiry Cycle curriculum, students evaluated their own work in terms of the criteria. è Each time they applied the criteria AND wrote a brief rationale pointing to the features of their work that supported their rating. è Students in the reflective assessment classrooms also used the criteria to give feedback to classmates after oral presentations. Compared to controls, students in reflective classrooms produced more highly rated projects (with the greatest gains for low-achieving students). Low-achieving students also showed dramatic gains on a measure of conceptual understanding.

43. School of Education, University of Colorado at Boulder Evaluation of teaching Assessment should be used to evaluate teaching as well as to improve student learning. If we want the cultural practices in the classroom to support development of students’ identities as learners – where students naturally seek feedback and critique their own work – then it is reasonable that teachers would model this same commitment to using data systematically as it applies to their own role in the teaching and learning process.

44. School of Education, University of Colorado at Boulder Classroom Assessment Practices Also Have a Profound Effect on Students’ Motivation to Learn.  Summative assessment and grading pose a serious threat to the learning purposes avowed for formative assessment.  If tests diverge from valued learning goals, students focus only on the graded portion of the curriculum.  The use of grades as rewards and punishments undermines intrinsic motivation to learn.

45. School of Education, University of Colorado at Boulder Students with a Performance Orientation (Extrinsically motivated)  Believe in fixed ability  Work toward “performance goals,” i.e., for grades, to please the teacher, and to appear competent  Focus on the “exchange value” of learning  Performance-oriented students pick easy task and are less likely to persist once they encounter difficulty.  Girls are over represented in this category. Stipek, 1996

46. School of Education, University of Colorado at Boulder Students with a Learning Orientation (Intrinsically motivated)  Attribute success to their own efforts  Work toward “learning goals,” i.e., to increase a sense of mastery and to become competent  Focus on the “use value” of learning  Learning-oriented students are more engaged in schoolwork, use more self-regulation, and develop deeper understanding of subject matter. Stipek, 1996

47. School of Education, University of Colorado at Boulder Mutually Supportive Formative and Summative Assessments To be mutually supportive, formative and summative assessments must be conceptually aligned. They should represent important learning goals using the same broad range of tasks and problem types to tap students’ understandings. Summative assessments should not be repeats of earlier formative tasks but should require students to use their knowledge in ways that generalize and extend what came before. Summative assessments are milestones on the same learning continua that under gird formative assessment.

48. School of Education, University of Colorado at Boulder To Foster a Learning Environment, Findings from Cognitive and Motivation Research Should Be Brought Together.  Focus efforts on developing competence.  Provide diverse opportunities to demonstrate mastery.  Adapt instruction to students’ knowledge and understanding.  Provide opportunities for students to take responsibility for learning.  Emphasize working hard and learning rather than right answers.  Make thinking visible in classroom discourse.  Treat errors and mistakes as a normal part of learning.  Evaluate progress as well as outcomes.  Help students learn the meaning of criteria for good work.  Provide opportunities for students to improve competence based on feedback.  Base grades on mastery standards rather than social norms.

49. School of Education, University of Colorado at Boulder