1. Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education, University of London www.dylanwiliam.net

2. Overview of presentation Some theoretical precepts About learning About teaching Pedagogies of engagement Pedagogies of contingency Pedagogies of formation The role of technology Supporting, rather than replacing, teachers Classroom aggregation technologies Some thoughts about supporting teachers in changing practice

3. How do we improve learning?

4. What gets learnt? (Denvir & Brown, 1986)

5. Key insights from C20th psychology What gets learned as a result of a particular sequence of instructional activities is impossible to predict, but Student errors are not random Conclusions Assessment is the bridge between teaching and learning Teaching is interesting because learners are so different, but only possible because they are so similar

6. Learning power environments Key concept: Teachers do not create learning Learners create learning Teaching as the engineering of learning environments Key features: Create student engagement (pedagogies of engagement) Well-regulated (pedagogies of contingency) Develop habits of mind (pedagogies of formation)

7. Why pedagogies of engagement? Intelligence is partly inherited So what? Intelligence is partly environmental Environment creates intelligence Intelligence creates environment Learning environments High cognitive demand Inclusive Obligatory

8. 264 low and high ability grade 6 students in 12 classes in 4 schools; analysis of 132 students at top and bottom of each class Same teaching, same aims, same teachers, same class work Three kinds of feedback: scores, comments, scores+comments [Butler(1988) Br. J. Educ. Psychol., 58 1-14] Engagement and feedback AchievementAttitude Scores no gainHigh scorers: positive Low scorers: negative Comments30% gainHigh scorers : positive Low scorers : positive

9. [Butler(1988) Br. J. Educ. Psychol., 58 1-14] Responses What do you think happened for the students given both scores and comments? A.Gain: 30%; Attitude: all positive B.Gain: 30%; Attitude: high scorers positive, low scorers negative C.Gain: 0%; Attitude: all positive D.Gain: 0%; Attitude: high scorers positive, low scorers negative E.Something else AchievementAttitude Scores no gainHigh scorers : positive Low scorers: negative Comments30% gainHigh scorers : positive Low scorers : positive

10. Kinds of feedback (Nyquist, 2003) Weaker feedback only Knowledge of results (KoR) Feedback only KoR + clear goals or knowledge of correct results (KCR) Weak formative assessment KCR+ explanation (KCR+e) Moderate formative assessment (KCR+e) + specific actions for gap reduction Strong formative assessment (KCR+e) + activity

11. Effect of formative assessment (HE) NEffect size Weaker feedback only310.14 Feedback only480.36 Weaker formative assessment490.26 Moderate formative assessment410.39 Strong formative assessment160.56 (Nyquist, 2003; revised values)

12. Effects of feedback Kluger & DeNisi (1996) review of 3000 research reports Excluding poorly designed studies left 131 reports, 607 effect sizes, involving 12652 individuals On average, feedback increases achievement, but Effect sizes highly variable 38% (50 out of 131) of effect sizes were negative

13. Engagement in learning Attribution (Dweck, 2000) Personalization (internal vs. external) Permanence (stable vs. unstable) Good learners attribute failure and success to internal, unstable causes. (Its down to you, and you can do something about it.) Views of ability Fixed (IQ) Incremental (untapped potential) Essential that teachers inculcate in students a view that ability is incremental rather than fixed (by working, youre getting smarter).

14. Dual-pathway theory (Boekaerts, 2006) Long-term learning goals are translated into short-term learning intentions Dynamic comparisons of task and situational demands with personal resources Resulting activation of energy along one of two pathways: Well-being Growth

15. Motivation: cause or effect? competence challenge Flow apathy boredom relaxation arousal anxiety worry control high low high (Csikszentmihalyi, 1990)

16. Why pedagogies of contingency? Fuchs & Fuchs (1986) Natriello (1987) Crooks (1988) Banger-Drowns, et al. (1991) Kluger & DeNisi (1996) Black & Wiliam (1998) Nyquist (2003) Dempster (1991, 1992) Elshout-Mohr (1994) Brookhart (2004) Allal & Lopez (2005) Köller (2005) Brookhart (2007) Wiliam (2007) Hattie & Timperley (2007) Shute (2008)

17. Hinge-point question An experimental study of problem-based learning in undergraduate medical education reports that a result was significant (p<0.05). This means that: A.The experimental group out-performed the control group by at least 5% B.There is only a 5% chance that the experimental group did not out- perform the control group C.There is a 5% chance that there is no difference between the experimental group and the control group D.There is only a 5% chance that the observed result would have happened if the experimental and control groups had the same achievement

18. Other supports for contingency All-student response systems ABCD cards Exit-pass questions

19. Exit-pass question Summarize the key principles of the following schools of psychology on the appropriate coloured card Associationism (blue) Information processing (orange) Constructivism (red) Situated approaches (green)

20. Pedagogies of formation Instilling disciplinary habits of mind History Philosophy Statistics Instilling critical perspectives Values

21. Three generations of pedagogy First generation Traditional pedagogy Negligible contingency Second generation All student response systems Contingency dependent entirely on teacher skill Third generation Automated aggregation technologies Contingency supported by technology

22. Evidence-centred design Quality in assessment is essentially a matter of validity Validity is a property of inferences, not of instruments Assessments should be designed backwards from the intended inferences Four-process architecture for assessment design Task selection Task presentation Evidence identification Evidence accumulation Almond, Steinberg and Mislevy (2002)

23. Task selection/ Task presentation

24. Hinge-point question Which of the following is the most important difference between the theories of Piaget and Vygotsky? A.Piaget places greater importance on the role of conservation in cognitive development B.Vygotsky places greater importance on the role of cultural artifacts in cognitive development. C.Vygotsky did not believe in distinct stages of cognitive development. D.Piaget was a social constructivist while Vygotsky placed greater emphasis on cultural-historical activity theory

25. Evidence identification

26. Single student response systems All-student response systems Flash-cards/dry erase boards Classroom clickers Traditional keyboards (wired/wireless) Anoto pens

27. Anoto pen Wireless pen Special coated paper Pen knows where it is

28. Palm with wireless keyboard Text-based input Limited task-presentation capability Portable

29. Classroom clickers (and their progeny)

30. Discourse ® www.ets.org/discourse

31. Evidence identification Automated essay scoring (e-rater) Paraphrase analysers (c-rater) Graphical and equation analysers (m-rater)

32. Evidence identification technologies unstructuredstructured evidence structure teacher- mediated automated clickers aggregationABCD cards dry-erase boards c-rater Discourse ® latent semantic analysis m-rater e-rater

33. Evidence accumulation

34. Unidimensional student models Useful for summative purposes Almost useless for formative purposes Multidimensional student models Evidence-centred design Bayesian inference networks Proficiency model Task model Evidence model Student model

35. Four-process architecture for ECD Mislevy, Almond and Lukas (2003) Task selection Task presentation Evidence accumulation Evidence identification

36. Evidence utilization Whole-class Sub-groups Homogenous Heterogenous Individualization

37. 37 Actually, the technology is the easy part Whats hard is changing practice Telling teachers what to do doesnt work Context always intrudes…

38. 38 The ancient yogis used logs of wood, stones, and ropes to help them practise asanas effectively. Extending this principle, Yogacharya Iyengar invented props which allow asanas to be held easily, and for a longer duration without strain. Yogacharya Iyengar in setubandha sarvangasana This version of the posture requires considerable strength in the neck, shoulders, and back requiring years of practice to achieve it. It should not be attempted without supervision.

39. 39

40. A model for teacher learning Content, then process Content (what we want teachers to change) Evidence Ideas (strategies and techniques) Process (how to go about change) Choice Flexibility Small steps Accountability Support

41. Summary Learning has to be done by the learner, not for the learner Teaching as engineering effective learning environments Features of effective learning environments Pedagogies of engagement Pedagogies of contingency All-student response systems Classroom aggregation technologies Pedagogies of formation Learning milieu focused on growth, rather than well-being Teachers supported to improve practice continually