What kinds of assessments improve learning? Dylan Wiliam, ETS

Raising achievement matters For individuals –Increased lifetime salary –Improved health For society –Lower criminal justice costs –Lower health-care costs –Increased economic growth

Where’s the solution? Structure –Small high schools –K-8 schools Alignment –Curriculum reform –Textbook replacement Governance –Charter schools –Vouchers Technology

It’s the classroom Variability at the classroom level is up to 4 times greater than at school level It’s not class size It’s not the between-class grouping strategy It’s not the within-class grouping strategy It’s the teacher

Teacher quality: A labor force issue with 2 solutions Replace existing teachers with better ones? –No evidence that more pay brings in better teachers –No evidence that there are better teachers out there deterred by certification requirements Improve the effectiveness of existing teachers –The “love the one you’re with” strategy –It can be done –We know how to do it, but at scale? Quickly? Sustainably?

The design challenge Key metric: –Cost of buying one standard deviation of increased student achievement Constraints –Solution must be in principle scalable

Cost/effect comparisons InterventionEffect (sd) Cost/yr Class-size reduction (by 30%) 0.1$24k Increase teacher content knowledge by 1 sd 0.1? Formative assessment/ Assessment for learning 0.2<$1k

Effects of formative assessment Several major reviews of the research –Natriello (1987) –Crooks (1988) –Black & Wiliam (1998) –Nyquist (2003) All find consistent, substantial effects

Effects of feedback Kluger & DeNisi (1996) Review of 3000 research reports Excluding those: –without adequate controls –with poor design –with fewer than 10 participants –where performance was not measured –without details of effect sizes left 131 reports, 607 effect sizes, involving individuals Average effect size 0.4, but –Effect sizes very variable –40% of effect sizes were negative

Kinds of feedback (Nyquist, 2003) Weaker feedback only –Knowledge of results (KoR) Feedback only –KoR + clear goals/knowledge of correct results (KCR) Weak formative assessment –KCR+ explanation (KCR+e) Moderate formative assessment –(KCR+e) + specific actions for gap reduction Strong formative assessment –(KCR+e) + activity

Effect of formative assessment (HE) NEffect Weaker feedback only Feedback only Weaker formative assessment Moderate formative assessment Strong formative assessment160.51

Feedback and formative assessment “Feedback is information about the gap between the actual level and the reference level of a system parameter which is used to alter the gap in some way” (Ramaprasad, 1983 p. 4) Formative assessment requires –data on the actual level of some measurable attribute; –data on the reference level of that attribute; –a mechanism for comparing the two levels and generating information about the ‘gap’ between the two levels; –a mechanism by which the information can be used to alter the gap.

Formative assessment Frequent feedback is not necessarily formative Feedback that causes improvement is not necessarily formative Assessment is formative only if the information fed back to the learner is used by the learner in making improvements To be formative, assessment must include a recipe for future action

Assessment for learning & formative assessment (Black et al., 2002) Assessment for learning is any assessment for which the first priority in its design and practice is to serve the purpose of promoting pupils’ learning. It thus differs from assessment designed primarily to serve the purposes of accountability, or of ranking, or of certifying competence. An assessment activity can help learning if it provides information to be used as feedback, by teachers, and by their pupils, in assessing themselves and each other, to modify the teaching and learning activities in which they are engaged. Such assessment becomes ‘formative assessment’ when the evidence is actually used to adapt the teaching work to meet learning needs.

Feedback Feedback is therefore formative only if the information fed back is actually used in closing the gap. Three key instructional processes –Establishing where learners are in their learning –Establishing where they are going –Establishing how to get there

Aspects of formative assessment Where the learner is going Where the learner is How to get there Teacher Clarify learning intentions Engineering effective discussions Providing feedback that moves learners on Peer Understand/ clarify criteria for success Activating students as instructional resources for one another Learner Understand criteria for success Activating students as owners of their own learning

Five key strategies… Clarifying and understanding learning intentions and criteria for success Engineering effective classroom discussions that elicit evidence of learning Providing feedback that moves learners forward Activating students as instructional resources for each other Activating students as the owners of their own learning

…and one big idea Use evidence about learning to adapt instruction to meet student needs

Keeping Learning on Track (KLT) A pilot guides a plane or boat toward its destination by taking constant readings and making careful adjustments in response to wind, currents, weather, etc. Educational systems must do the same: –Plan a carefully chosen route ahead of time (in essence building the track) –Take readings along the way –Change course as conditions dictate

Regulation of learning Teaching as engineering learning environments Key features: –Create student engagement (pedagogies of engagement) –Well-regulated (pedagogies of contingency) Long feedback cycles vs. variable feedback cycles Quality control vs. quality assurance in learning Teaching vs. learning Regulation of activity vs. regulation of learning

Regulation of learning Proactive (upstream) regulation –Planning regulation into the learning environment –Planning for evoking information Interactive (downstream) regulation –‘Negotiating the swiftly-flowing river’ –‘Moments of contingency’ –Tightness of regulation (goals vs. horizons) Retrospective regulation –Structured reflection (e.g., lesson study)

Types of formative assessment Long-cycle –Focus: between units –Length: four weeks to one year Medium-cycle –Focus: within units, between lessons –Length: one day to two weeks Short-cycle –Focus: within lessons –Length: five seconds to one hour

Questioning in math: discussion Look at the following sequence: 3, 7, 11, 15, 19, …. Which is the best rule to describe the sequence? A.n + 4 B.3 + n C.4n - 1 D.4n + 3

Questioning in math: diagnosis In which of these triangles is a 2 + b 2 = c 2 ? A a c b C b c a E c b a B a b c D b a c F c a b

Questioning in science: discussion Ice-cubes are added to a glass of water. What happens to the level of the water as the ice-cubes melt? A.The level of the water drops B.The level of the water stays the same C.The level of the water increases D.You need more information to be sure

Wilson & Draney, 2004 Questioning in science: diagnosis The ball sitting on the table is not moving. It is not moving because: A. no forces are pushing or pulling on the ball. B. gravity is pulling down, but the table is in the way. C. the table pushes up with the same force that gravity pulls down D. gravity is holding it onto the table. E. there is a force inside the ball keeping it from rolling off the table

Questioning in English: discussion Macbeth: mad or bad?

Questioning in English: diagnosis Where is the verb in this sentence? The dog ran across the road ABCD

ABCD Questioning in English: diagnosis Where does the subject end and the predicate begin in this sentence? The dog ran across the road.

Questioning in English: diagnosis Which of these is a good thesis statement? A.The typical TV show has 9 violent incidents B.There is a lot of violence on TV C.The amount of violence on TV should be reduced D.Some programs are more violent than others E.Violence is included in programs to boost ratings F.Violence on TV is interesting G.I don’t like the violence on TV H.The essay I am going to write is about violence on TV

Questioning in history: discussion In which year did World War II begin? A.1919 B.1937 C.1938 D.1939 E.1941

Questioning in History Why are historians concerned with bias when analyzing sources? A.People can never be trusted to tell the truth B.People deliberately leave out important details C.People are only able to provide meaningful information if they experienced an event firsthand D.People interpret the same event in different ways, according to their experience E.People are unaware of the motivations for their actions F.People get confused about sequences of events

Why research hasn’t changed teaching Misunderstanding nature of teacher expertise Leaving teachers to “translate into practice” Failure to attend to both content and process

Klein & Klein (1981) Six video extracts of someone delivering cardio-pulmonary resuscitation (CPR) –5 of the videos are of CPR students –1 of the videos is of an expert paramedic Videos shown to three groups: –students, instructors and experts Success rate in identifying expert: –Experts: 90% –Students: 50% –Instructors: 30%

Teacher expertise (Berliner, 1994) –Experts excel mainly in their own domain –Experts often develop automaticity for the repetitive operations that are needed to accomplish their goals –Experts are more sensitive to the task demands and social situation when solving problems. –Experts are more opportunistic and flexible in their teaching than novices –Experts represent problems in qualitatively different ways than novices. –Experts have fast and accurate pattern recognition capabilities. Novices cannot always make sense of what they experience. –Experts perceive meaningful patterns in the domain in which they are experienced. –Experts begin to solve problems slower, but bring richer and more personal sources of information to bear on the problem that they are trying to solve.

Knowledge creation and transmission After Nonaka & Tageuchi, 1995

A model for teacher learning Content (what we want teachers to change) –Evidence –Ideas (strategies and techniques) Process (how to go about change) –Small steps –Flexibility –Choice –Accountability –Support

Discussion What are the implications of this research for publishers of formative assessment systems?

Questions? Don’t forget to fill out your evaluations…