1. ETS Confidential & Proprietary What kinds of assessment promote learning? Dylan Wiliam Director, Learning and Teaching Research Center ETS NARST Annual International Conference: San Francisco, CA; April 2006

2. ETS Confidential & Proprietary 2 Overview of presentation Why raising achievement is important Why investing in teachers is the answer Why assessment for learning should be the focus How we can put this into practice

3. ETS Confidential & Proprietary 3...the model that says ‘learn while you are at school the skills that you will apply during your lifetime’ is no longer tenable. These skills will be obsolete by the time you get into the workplace and need them, except for one skill – the skill of being able to learn. It is the skill of being able, not to give the right answer to questions about what you were taught in school, but to make the right response to situations that are outside the scope of what you were taught in school. We need to produce people who know how to act when they are faced with situations for which they were not specifically prepared.(Papert, 1998) What do we need students to learn?

4. ETS Confidential & Proprietary 4 The four Rs of “learning power” Resilience –Absorption, managing distractions, noticing, perseverance Resourcefulness –Questioning, making links, imagining, reasoning, capitalizing Reflectiveness –Planning, revising, distilling, meta-learning Reciprocity –Interdependence, collaboration, empathy and listening, imitation (Claxton 2002)

5. ETS Confidential & Proprietary 5 Preparation for future learning (PFL) Cannot be taught in isolation from other learning Students still need the basic skills of literacy, numeracy, concepts and facts Learning power is developed primarily through pedagogy, not curriculum We have to change the way teachers teach, not what they teach

6. ETS Confidential & Proprietary 6 Successful education The test of successful education is not the amount of knowledge that a pupil takes away from school, but his [sic] appetite to know and his capacity to learn. If the school sends out children with the desire for knowledge and some idea how to acquire it, it will have done its work. Too many leave school with the appetite killed and the mind loaded with undigested lumps of information. The good schoolmaster [sic] is known by the number of valuable subjects which he declines to teach. (Sir Richard Livingstone, President of Corpus Christi College, Oxford, 1941)

7. ETS Confidential & Proprietary 7 Different approaches to upper secondary schooling 20th century education –US: mass systems –Europe: elite systems 21st century education challenge –US: increase quality –Europe: scale up quality

8. ETS Confidential & Proprietary 8 Raising achievement matters For individuals –Increased lifetime salary –Improved health For society –Lower criminal justice costs –Lower health-care costs –Increased economic growth

9. ETS Confidential & Proprietary 9 Where’s the solution? Structure –Small high schools –K-8 schools Alignment –Curriculum reform –Textbook replacement Governance –Charter schools –Vouchers Technology

10. ETS Confidential & Proprietary 10 It’s the classroom Variability at the classroom level is up to 4 times greater than at school level It’s not class size It’s not the between-class grouping strategy It’s not the within-class grouping strategy It’s the teacher

11. ETS Confidential & Proprietary 11 Teacher quality: A labor force issue with 2 solutions Replace existing teachers with better ones? –No evidence that more pay brings in better teachers –No evidence that there are better teachers out there deterred by certification requirements Improve the effectiveness of existing teachers –The “love the one you’re with” strategy –It can be done –We know how to do it, but at scale? Quickly? Sustainably?

12. ETS Confidential & Proprietary 12 Learning power environments Key concept: –Teachers do not create learning –Learners create learning Teaching as engineering learning environments Key features: –Create student engagement (pedagogies of engagement) –Well-regulated (pedagogies of contingency)

13. ETS Confidential & Proprietary 13 Why pedagogies of engagement? Intelligence is partly inherited –So what? Intelligence is partly environmental –Environment creates intelligence –Intelligence creates environment Learning environments –High cognitive demand –Inclusive –Obligatory

14. ETS Confidential & Proprietary 14 Motivation: cause or effect? competence challenge Flow apathy boredom relaxation arousal anxiety worry control high low high (Csikszentmihalyi, 1990)

15. ETS Confidential & Proprietary 15 Why pedagogies of contingency? For evaluating institutions For describing individuals For supporting learning –Monitoring learning Whether learning is taking place –Diagnosing (informing) learning What is not being learnt –Forming learning What to do about it

16. ETS Confidential & Proprietary 16 Effects of formative assessment Several major reviews of the research –Natriello (1987) –Crooks (1988) –Kluger & DeNisi (1996) –Black & Wiliam (1998) –Nyquist (2003) All find consistent, substantial effects

17. ETS Confidential & Proprietary 17 Cost/effect comparisons InterventionEffect (sd) Cost/yr/ classroom Class-size reduction (by 30%)0.1$30k Increase teacher content knowledge by 1 sd 0.1? Formative assessment/ Assessment for learning 0.2~0.3$2k

18. ETS Confidential & Proprietary 18 Effects of feedback Kluger & DeNisi (1996) Review of 3000 research reports Excluding those: –without adequate controls –with poor design –with fewer than 10 participants –where performance was not measured –without details of effect sizes left 131 reports, 607 effect sizes, involving 12652 individuals Average effect size 0.4, but –Effect sizes very variable –40% of effect sizes were negative

19. ETS Confidential & Proprietary 19 Kinds of feedback (Nyquist, 2003) Weaker feedback only –Knowledge of results (KoR) Feedback only –KoR + clear goals or knowledge of correct results (KCR) Weak formative assessment –KCR+ explanation (KCR+e) Moderate formative assessment –(KCR+e) + specific actions for gap reduction Strong formative assessment –(KCR+e) + activity

20. ETS Confidential & Proprietary 20 Effect of formative assessment (HE) NEffect Weaker feedback only310.16 Feedback only480.23 Weaker formative assessment490.30 Moderate formative assessment410.33 Strong formative assessment160.51

21. ETS Confidential & Proprietary 21 Key processes Establishing where the learner is in her/his learning Establishing where she/he is going Establishing how to get there.

22. ETS Confidential & Proprietary 22 Aspects of formative assessment Where the learner is going Where the learner is How to get there Teacher Clarify learning intentions Engineering effective discussions Providing feedback that moves learners on Peer Understand/ clarify criteria for success Activating students as instructional resources for one another Learner Understand criteria for success Activating students as owners of their own learning

23. ETS Confidential & Proprietary 23 Five key strategies… Clarifying and understanding learning intentions and criteria for success Engineering effective classroom discussions that elicit evidence of learning Providing feedback that moves learners forward Activating students as instructional resources for each other Activating students as the owners of their own learning

24. ETS Confidential & Proprietary 24 …and one big idea Use evidence about learning to adapt instruction to meet student needs

25. ETS Confidential & Proprietary 25 Keeping Learning on Track (KLT) A pilot guides a plane or boat toward its destination by planning a route, taking constant readings and making careful adjustments in response to wind, currents, weather, etc. A KLT teacher does the same: –Plans a carefully chosen (possibly differentiated) route ahead of time (in essence building the track) –Takes readings along the way –Changes course as conditions dictate

26. ETS Confidential & Proprietary 26 Regulation of learning Proactive (upstream) regulation –Planning regulation into the learning environment –Planning for evoking information Interactive (downstream) regulation –‘Negotiating the swiftly-flowing river’ –‘Moments of contingency’ –Tightness of regulation (goals vs. horizons) Retrospective regulation –Structured reflection (e.g., lesson study)

27. ETS Confidential & Proprietary 27 Types of formative assessment Long-cycle –Focus: between units –Length: four weeks to one year Medium-cycle –Focus: within units, between lessons –Length: one day to two weeks Short-cycle –Focus: within lessons –Length: five seconds to one hour

28. ETS Confidential & Proprietary Putting it into practice

29. ETS Confidential & Proprietary 29 Why research hasn’t changed teaching Misunderstanding nature of teacher expertise Leaving teachers to “translate into practice” Failure to attend to both content and process

30. ETS Confidential & Proprietary 30 The nature of expertise in teaching Aristotle’s main intellectual virtues –Episteme: knowledge of universal truths –Techne: ability to make things –Phronesis: practical wisdom What works is not the right question –Everything works somewhere –Nothing works everywhere –What’s interesting is “under what conditions” does this work? Teaching is mainly a matter of phronesis, not episteme

31. ETS Confidential & Proprietary 31 Kinds of knowing Learning theories are not nested –Constructivist theories explain misconceptions better than associationist theories, but –Associationist theories explain acquisition of number facts better than constructivist theories –Each new theory provides better accounts of things the previous theory explained badly, but –Tends to explain poorly the things the previous theory explained well. The best descriptions of what happens when you come to know something depends on what it is you are learning For some kinds of learning, telling works Teaching is not one of them

32. ETS Confidential & Proprietary 32 Klein & Klein (1981) Six video extracts of a person delivering cardio-pulmonary resuscitation (CPR) 5 of the video extracts are students 1 of the video extracts is an expert Videos shown to three groups Students, experts, instructors Success rate in identifying expert: Experts:90% Students:50% Instructors:30%

33. ETS Confidential & Proprietary 33 25 3 9 4 Target number: 127 1 Countdown

34. ETS Confidential & Proprietary 34 Chess (Newell & Simon, 1973) 1684 33.54 Actual position Random positioning HighMiddle LowPositioning Expertise

35. ETS Confidential & Proprietary 35 Teacher expertise Resembles expertise in other domains (Berliner, 1994) –Experts excel mainly in their own domain –Experts often develop automaticity for the repetitive operations that are needed to accomplish their goals –Experts are more sensitive to the task demands and social situation when solving problems. –Experts are more opportunistic and flexible in their teaching than novices –Experts represent problems in qualitatively different ways than novices. –Experts have fast and accurate pattern recognition capabilities. Novices cannot always make sense of what they experience. –Experts perceive meaningful patterns in the domain in which they are experienced. –Experts begin to solve problems more slowly, but bring richer and more personal sources of information to bear on the problem that they are trying to solve.

36. ETS Confidential & Proprietary 36 Knowledge creation and transmission After Nonaka & Tageuchi, 1995

37. ETS Confidential & Proprietary 37 A model for teacher learning Content (what we want teachers to change) –Evidence –Ideas (strategies and techniques) Process (how to go about change) –Choice –Flexibility –Small steps –Accountability –Support

38. ETS Confidential & Proprietary 38 Content: strategies and techniques Distinction between strategies and techniques –Strategies define the territory of AfL (no brainers) –Teachers are responsible for choice of techniques Allows for customization/ caters for local context Creates ownership Shares responsibility Key requirements of techniques –embodiment of deep cognitive/affective principles –relevance –feasibility –acceptability

39. ETS Confidential & Proprietary 39 Design and intervention Our design process Teachers’ implementation process cognitive/affective insights synergy/ comprehensiveness set of components set of components synergy/ comprehensiveness cognitive/affective insights

40. ETS Confidential & Proprietary 40 Practical techniques questioning Key idea: questioning should –cause thinking –provide data that informs teaching Improving teacher questioning –generating questions with colleagues –closed v open –low-order v high-order –appropriate wait-time Getting away from I-R-E –basketball rather than serial table-tennis –‘No hands up’ (except to ask a question) –class polls to review current attitudes towards an issue –‘Hot Seat’ questioning All-student response systems –ABCD cards, Mini white-boards, Exit passes

42. ETS Confidential & Proprietary 42 Questioning in science: discussion Ice-cubes are added to a glass of water. What happens to the level of the water as the ice-cubes melt? A.The level of the water drops B.The level of the water stays the same C.The level of the water increases D.You need more information to be sure

43. ETS Confidential & Proprietary 43 Wilson & Draney, 2004 Questioning in science: diagnosis The ball sitting on the table is not moving. It is not moving because: A. no forces are pushing or pulling on the ball. B. gravity is pulling down, but the table is in the way. C. the table pushes up with the same force that gravity pulls down D. gravity is holding it onto the table. E. there is a force inside the ball keeping it from rolling off the table

44. ETS Confidential & Proprietary 44 Questioning in science: diagnosis Which of these is living? A)A river B)An acorn C)Lichen D)A Xerox machine E)A broken-off branch of a tree

45. ETS Confidential & Proprietary 45 Questioning in science: diagnosis A.Joule B.Kilogram C.Newton D.Pascal E.Watt 1.Energy 2.Force 3.Mass 4.Pressure 5.Weight 6.Work

46. ETS Confidential & Proprietary 46 Dinosaur extinction Why did dinosaurs become extinct? –A) Humans destroyed their habitat –B) Humans killed them all for food –C) There was a major change in climate

47. ETS Confidential & Proprietary 47 Save the ozone layer What can we do to preserve the ozone layer? A.Reduce the amount of carbon dioxide produced by cars and factories B.Reduce the greenhouse effect C.Stop cutting down the rainforests D.Limit the numbers of cars that can be used when the level of ozone is high E.Properly dispose of air-conditioners and fridges

48. ETS Confidential & Proprietary 48 Hinge Questions A hinge question is based on the important concept in a lesson that is critical for students to understand before you move on in the lesson. The question should fall about midway during the lesson. Every student must respond to the question within two minutes. You must be able to collect and interpret the responses from all students in 30 seconds

49. ETS Confidential & Proprietary 49 Practical techniques: feedback Key idea: feedback should –cause thinking –provide guidance on how to improve Comment-only grading Focused grading Explicit reference to rubrics Suggestions on how to improve –‘Strategy cards’ ideas for improvement –Not giving complete solutions Re-timing assessment –(eg two-thirds-of-the-way-through-a-unit test)

50. ETS Confidential & Proprietary 50 Practical techniques: sharing learning intentions Explaining learning intentions at start of lesson/unit –Learning intentions –Success criteria Intentions/criteria in students’ language Posters of key words to talk about learning –eg describe, explain, evaluate Planning/writing frames Annotated examples of different standards to ‘flesh out’ assessment rubrics (e.g. lab reports) Opportunities for students to design their own tests

51. ETS Confidential & Proprietary 51 Practical techniques: peer and self-assessment Students assessing their own/peers’ work –with rubrics –with exemplars –“two stars and a wish” Training students to pose questions/identifying group weaknesses Self-assessment of understanding –Traffic lights –Red/green discs End-of-lesson students’ review

52. ETS Confidential & Proprietary 52 Process: Teacher Learning Communities Teacher as local expert Sustained over time Supportive forum for learning Embedded in day-to-day reality Domain-specific

53. ETS Confidential & Proprietary 53 A four-part model Initial workshops Monthly TLC meetings Peer observations Training for leaders

54. ETS Confidential & Proprietary 54 Summary Raising achievement is important Raising achievement requires improving teacher quality Improving teacher quality requires teacher professional development To be effective, teacher professional development must address –What teachers do in the classroom –How teachers change what they do in the classroom AfL + TLCs –A point of (uniquely?) high leverage –A “Trojan Horse” into wider issues of pedagogy, psychology, and curriculum

55. ETS Confidential & Proprietary 55 Questions? Comments?