ETS Confidential & Proprietary Using assessment to improve learning in science: why, what and how? Dylan Wiliam Director, Learning and Teaching Research Center ETS Sixth Annual Chemical Heritage Foundation Leadership in Science Education conference: Philadelphia, PA; April 2006

ETS Confidential & Proprietary 2 Overview of presentation Why raising achievement is important Why investing in teachers is the answer Why assessment for learning should be the focus How we can put this into practice

ETS Confidential & Proprietary 4...the model that says ‘learn while you are at school the skills that you will apply during your lifetime’ is no longer tenable. These skills will be obsolete by the time you get into the workplace and need them, except for one skill – the skill of being able to learn. It is the skill of being able, not to give the right answer to questions about what you were taught in school, but to make the right response to situations that are outside the scope of what you were taught in school. We need to produce people who know how to act when they are faced with situations for which they were not specifically prepared.(Papert, 1998) What do we need students to learn?

ETS Confidential & Proprietary 5 Preparation for future learning (PFL) Cannot be taught in isolation from other learning Students still need the basic skills of literacy, numeracy, concepts and facts Learning power is developed primarily through pedagogy, not curriculum We have to change the way teachers teach, not what they teach

ETS Confidential & Proprietary 6 Successful education The test of successful education is not the amount of knowledge that a pupil takes away from school, but his [sic] appetite to know and his capacity to learn. If the school sends out children with the desire for knowledge and some idea how to acquire it, it will have done its work. Too many leave school with the appetite killed and the mind loaded with undigested lumps of information. The good schoolmaster [sic] is known by the number of valuable subjects which he declines to teach. (Sir Richard Livingstone, President of Corpus Christi College, Oxford, 1941)

ETS Confidential & Proprietary 7 Different approaches to upper secondary schooling 20th century education –US: mass systems –Europe: elite systems 21st century education challenge –US: increase quality –Europe: scale up quality

ETS Confidential & Proprietary 8 Raising achievement matters For individuals –Increased lifetime salary –Improved health For society –Lower criminal justice costs –Lower health-care costs –Increased economic growth

ETS Confidential & Proprietary 9 Where’s the solution? Structure –Small high schools –K-8 schools Alignment –Curriculum reform –Textbook replacement Governance –Charter schools –Vouchers Technology

ETS Confidential & Proprietary 10 It’s the classroom Variability at the classroom level is up to 4 times greater than at school level It’s not class size It’s not the between-class grouping strategy It’s not the within-class grouping strategy It’s the teacher

ETS Confidential & Proprietary 11 Teacher quality: A labor force issue with 2 solutions Replace existing teachers with better ones? –No evidence that more pay brings in better teachers –No evidence that there are better teachers out there deterred by certification requirements Improve the effectiveness of existing teachers –The “love the one you’re with” strategy –It can be done –We know how to do it, but at scale? Quickly? Sustainably?

ETS Confidential & Proprietary 12 Learning power environments Key concept: –Teachers do not create learning –Learners create learning Teaching as engineering learning environments Key features: –Create student engagement (pedagogies of engagement) –Well-regulated (pedagogies of contingency)

ETS Confidential & Proprietary 13 Why pedagogies of engagement? Intelligence is partly inherited –So what? Intelligence is partly environmental –Environment creates intelligence –Intelligence creates environment Learning environments –High cognitive demand –Inclusive –Obligatory

ETS Confidential & Proprietary 14 Motivation: cause or effect? competence challenge Flow apathy boredom relaxation arousal anxiety worry control high low high (Csikszentmihalyi, 1990)

ETS Confidential & Proprietary 15 Why pedagogies of contingency? For evaluating institutions For describing individuals For supporting learning –Monitoring learning Whether learning is taking place –Diagnosing (informing) learning What is not being learnt –Forming learning What to do about it

ETS Confidential & Proprietary 16 Effects of formative assessment Several major reviews of the research –Natriello (1987) –Crooks (1988) –Kluger & DeNisi (1996) –Black & Wiliam (1998) –Nyquist (2003) All find consistent, substantial effects

ETS Confidential & Proprietary 17 Cost/effect comparisons InterventionEffect (sd) Cost/yr/ classroom Class-size reduction (by 30%)0.1$30k Increase teacher content knowledge by 1 sd 0.1? Formative assessment/ Assessment for learning 0.2~0.3$2k

ETS Confidential & Proprietary 18 Effects of feedback Kluger & DeNisi (1996) Review of 3000 research reports Excluding those: –without adequate controls –with poor design –with fewer than 10 participants –where performance was not measured –without details of effect sizes left 131 reports, 607 effect sizes, involving individuals Average effect size 0.4, but –Effect sizes very variable –40% of effect sizes were negative

ETS Confidential & Proprietary 19 Kinds of feedback (Nyquist, 2003) Weaker feedback only –Knowledge of results (KoR) Feedback only –KoR + clear goals or knowledge of correct results (KCR) Weak formative assessment –KCR+ explanation (KCR+e) Moderate formative assessment –(KCR+e) + specific actions for gap reduction Strong formative assessment –(KCR+e) + activity

ETS Confidential & Proprietary 20 Effect of formative assessment (HE) NEffect Weaker feedback only Feedback only Weaker formative assessment Moderate formative assessment Strong formative assessment160.51

ETS Confidential & Proprietary 21 Five key strategies… Clarifying and understanding learning intentions and criteria for success Engineering effective classroom discussions that elicit evidence of learning Providing feedback that moves learners forward Activating students as instructional resources for each other Activating students as the owners of their own learning

ETS Confidential & Proprietary 22 …and one big idea Use evidence about learning to adapt instruction to meet student needs

ETS Confidential & Proprietary 23 Keeping Learning on Track (KLT) A pilot guides a plane or boat toward its destination by planning a route, taking constant readings and making careful adjustments in response to wind, currents, weather, etc. A KLT teacher does the same: –Plans a carefully chosen (possibly differentiated) route ahead of time (in essence building the track) –Takes readings along the way –Changes course as conditions dictate

ETS Confidential & Proprietary 24 Types of formative assessment Long-cycle –Focus: between units –Length: four weeks to one year Medium-cycle –Focus: within units, between lessons –Length: one day to two weeks Short-cycle –Focus: within lessons –Length: five seconds to one hour

ETS Confidential & Proprietary Putting it into practice

ETS Confidential & Proprietary 26 A model for teacher learning Content (what we want teachers to change) –Evidence –Ideas (strategies and techniques) Process (how to go about change) –Choice –Flexibility –Small steps –Accountability –Support

ETS Confidential & Proprietary 27 Content: strategies and techniques Distinction between strategies and techniques –Strategies define the territory of AfL (no brainers) –Teachers are responsible for choice of techniques Allows for customization/ caters for local context Creates ownership Shares responsibility Key requirements of techniques –embodiment of deep cognitive/affective principles –relevance –feasibility –acceptability

ETS Confidential & Proprietary 28 Design and intervention Our design process Teachers’ implementation process cognitive/affective insights synergy/ comprehensiveness set of components set of components synergy/ comprehensiveness cognitive/affective insights

ETS Confidential & Proprietary 29 Practical techniques questioning Key idea: questioning should –cause thinking –provide data that informs teaching Improving teacher questioning –generating questions with colleagues –closed v open –low-order v high-order –appropriate wait-time Getting away from I-R-E –basketball rather than serial table-tennis –‘No hands up’ (except to ask a question) –class polls to review current attitudes towards an issue –‘Hot Seat’ questioning All-student response systems –ABCD cards, Mini white-boards, Exit passes

ETS Confidential & Proprietary 31 Questioning in science: discussion Ice-cubes are added to a glass of water. What happens to the level of the water as the ice-cubes melt? A.The level of the water drops B.The level of the water stays the same C.The level of the water increases D.You need more information to be sure

ETS Confidential & Proprietary 32 Wilson & Draney, 2004 Questioning in science: diagnosis The ball sitting on the table is not moving. It is not moving because: A. no forces are pushing or pulling on the ball. B. gravity is pulling down, but the table is in the way. C. the table pushes up with the same force that gravity pulls down D. gravity is holding it onto the table. E. there is a force inside the ball keeping it from rolling off the table

ETS Confidential & Proprietary 33 Questioning in science: diagnosis Which of these is living? A)A river B)An acorn C)Lichen D)A Xerox machine E)A broken-off branch of a tree

ETS Confidential & Proprietary 34 Questioning in science: diagnosis What proportion of the water taken in by a corn plant is lost through transpiration? A)10% B)30% C)50% D)70% E)90%

ETS Confidential & Proprietary 35 Questioning in science: diagnosis A.Joule B.Kilogram C.Newton D.Pascal E.Watt 1.Energy 2.Force 3.Mass 4.Pressure 5.Weight 6.Work

ETS Confidential & Proprietary 36 Questioning in science: diagnosis Take the nucleus of an atom of Sodium Add a proton Take away a neutron What do you get?

ETS Confidential & Proprietary 37 Dinosaur extinction Why did dinosaurs become extinct? –A) Humans destroyed their habitat –B) Humans killed them all for food –C) There was a major change in climate

ETS Confidential & Proprietary 38 Save the ozone layer What can we do to preserve the ozone layer? A.Reduce the amount of carbon dioxide produced by cars and factories B.Reduce the greenhouse effect C.Stop cutting down the rainforests D.Limit the numbers of cars that can be used when the level of ozone is high E.Properly dispose of air-conditioners and fridges

ETS Confidential & Proprietary 39 Hinge Questions A hinge question is based on the important concept in a lesson that is critical for students to understand before you move on in the lesson. The question should fall about midway during the lesson. Every student must respond to the question within two minutes. You must be able to collect and interpret the responses from all students in 30 seconds

ETS Confidential & Proprietary 40

ETS Confidential & Proprietary 41 Practical techniques: feedback Key idea: feedback should –cause thinking –provide guidance on how to improve Comment-only grading Focused grading Explicit reference to rubrics Suggestions on how to improve –‘Strategy cards’ ideas for improvement –Not giving complete solutions Re-timing assessment –(eg two-thirds-of-the-way-through-a-unit test)

ETS Confidential & Proprietary 42 Practical techniques: sharing learning intentions Explaining learning intentions at start of lesson/unit –Learning intentions –Success criteria Intentions/criteria in students’ language Posters of key words to talk about learning –eg describe, explain, evaluate Planning/writing frames Annotated examples of different standards to ‘flesh out’ assessment rubrics (e.g. lab reports) Opportunities for students to design their own tests

ETS Confidential & Proprietary 43 Practical techniques: peer and self-assessment Students assessing their own/peers’ work –with rubrics –with exemplars –“two stars and a wish” Training students to pose questions/identifying group weaknesses Self-assessment of understanding –Traffic lights –Red/green discs End-of-lesson students’ review

ETS Confidential & Proprietary 44 Process: Teacher Learning Communities Teacher as local expert Sustained over time Supportive forum for learning Embedded in day-to-day reality Domain-specific

ETS Confidential & Proprietary 45 A four-part model Initial workshops Monthly TLC meetings Peer observations Training for leaders

ETS Confidential & Proprietary 46 Summary Raising achievement is important Raising achievement requires improving teacher quality Improving teacher quality requires teacher professional development To be effective, teacher professional development must address –What teachers do in the classroom –How teachers change what they do in the classroom AfL + TLCs –A point of (uniquely?) high leverage –A “Trojan Horse” into wider issues of pedagogy, psychology, and curriculum

ETS Confidential & Proprietary 47 Questions? Comments?