Integrating assessment with instruction to keep learning on track Plenary address to NSTA Convention on Science assessment: research and practical approaches for classroom teachers, school administrators and school districts Chicago, IL; November 9, 2005 Dylan Wiliam, Educational Testing Service

Overview of presentation Why raising achievement is important Why investing in teachers is the answer Why assessment for learning should be the focus Why teacher learning communities should be the mechanism

Raising achievement matters For individuals Increased lifetime salary Improved health For society Lower criminal justice costs Lower health-care costs Increased economic growth

Where’s the solution? Structure Alignment Governance Technology Small high schools K-8 schools Alignment Curriculum reform Textbook replacement Governance Charter schools Vouchers Technology

It’s the classroom Variability at the classroom level is up to 4 times greater than at school level It’s not class size It’s not the between-class grouping strategy It’s not the within-class grouping strategy It’s the teacher

Teacher quality A labor force issue with 2 solutions Replace existing teachers with better ones? No evidence that more pay brings in better teachers No evidence that there are better teachers out there deterred by certification requirements Improve the effectiveness of existing teachers The “love the one you’re with” strategy It can be done We know how to do it, but at scale? Quickly? Sustainably?

Functions of assessment For evaluating institutions For describing individuals For supporting learning Monitoring learning Whether learning is taking place Diagnosing (informing) learning What is not being learnt Forming learning What to do about it

Effects of formative assessment Several major reviews of the research Natriello (1987) Crooks (1988) Black & Wiliam (1998) Nyquist (2003) All find consistent, substantial effects

Kinds of feedback (Nyquist, 2003) Weaker feedback only Knowledge of results (KoR) Feedback only KoR + clear goals or knowledge of correct results (KCR) Weak formative assessment KCR+ explanation (KCR+e) Moderate formative assessment (KCR+e) + specific actions for gap reduction Strong formative assessment (KCR+e) + activity

Effect of formative assessment (HE) Weaker feedback only 31 0.16 Feedback only 48 0.23 Weaker formative assessment 49 0.30 Moderate formative assessment 41 0.33 Strong formative assessment 16 0.51

Classroom assessment is not (necessarily) formative assessment Formative assessment is not (necessarily) classroom assessment

Formative assessment Assessment for learning is any assessment for which the first priority in its design and practice is to serve the purpose of promoting pupils’ learning. It thus differs from assessment designed primarily to serve the purposes of accountability, or of ranking, or of certifying competence. An assessment activity can help learning if it provides information to be used as feedback, by teachers, and by their pupils, in assessing themselves and each other, to modify the teaching and learning activities in which they are engaged. Such assessment becomes ‘formative assessment’ when the evidence is actually used to adapt the teaching work to meet learning needs. Black et al., 2002

Feedback and formative assessment “Feedback is information about the gap between the actual level and the reference level of a system parameter which is used to alter the gap in some way” (Ramaprasad, 1983 p. 4) Three key instructional processes Establishing where learners are in their learning Establishing where they are going Establishing how to get there

Aspects of formative assessment Where the learner is going Where the learner is How to get there Teacher Clarify learning intentions Engineering effective discussions Providing feedback that moves learners on Peer Understand/ clarify criteria for success Activating students as instructional resources for one another Learner Understand criteria for success Activating students as owners of their own learning

Five key strategies… Clarifying and understanding learning intentions and criteria for success Engineering effective classroom discussions that elicit evidence of learning Providing feedback that moves learners forward Activating students as instructional resources for each other Activating students as the owners of their own learning

Use evidence about learning to adapt instruction to meet student needs …and one big idea Use evidence about learning to adapt instruction to meet student needs

Keeping Learning on Track (KLT) A pilot guides a plane or boat toward its destination by taking constant readings and making careful adjustments in response to wind, currents, weather, etc. A KLT teacher does the same: Plans a carefully chosen route ahead of time (in essence building the track) Takes readings along the way Changes course as conditions dictate

Regulation of learning Teaching as engineering learning environments Key features: Create student engagement Well-regulated Long feedback cycles vs. variable feedback cycles Quality control vs. quality assurance in learning Teaching vs. learning Regulation of activity vs. regulation of learning

Regulation of learning Proactive (upstream) regulation Planning regulation into the learning environment Planning for evoking information Interactive (downstream) regulation ‘Negotiating the swiftly-flowing river’ ‘Moments of contingency’ Tightness of regulation (goals vs. horizons) Retrospective regulation Structured reflection (e.g., lesson study)

Types of formative assessment Long-cycle Focus: between units Length: four weeks to one year Medium-cycle Focus: within units Length: one day to two weeks Short-cycle Focus: within lessons Length: five seconds to one hour

Practical techniques: Questioning Improving teacher questioning Generating questions with colleagues Closed v open Low-order v high-order Appropriate wait-time Getting away from I-R-E (initiation-response-evaluation) Basketball rather than serial table-tennis ‘No hands up’ (except to ask a question) Class polls to review current attitudes towards an issue ‘Hot Seat’ questioning All-student response systems ABCD cards Mini white-boards Exit passes In many classrooms teachers use questions as a way of directing the attention of the class, and keeping students ‘on task’. This may keep students ‘on their toes’ but rarely helps learning. It may be better in a whole-class lesson, to have an extended exchange with a single student. This ‘hot seat’ questioning also has benefits for other students who appear to learn vicariously from the exchange. Other students may see extended exchanges between the teacher and another student as a chance to relax and go ‘off task’, but if theteacher asks them what they have learned from a particular exchange between another student and the teacher, their concentration is likely to be quite high! How much time a teacher allows a student to respond before evaluating the response is also important. It is well known that teachers do not allow students much time to answer questions, and, if they don’t receive a response quickly, they will ‘help’ the student by providing a clue or weakening the question in some way, or even moving on to another student. However, what is not widely appreciated is that the amount of time between the student providing an answer and the teacher’s evaluation of that answer is much more important. Of course, where the question is a simple matter of factual recall, then allowing a student time to reflect and expand upon the answer is unlikely to help much. But where the question requires thought, then increasing the time between the end of the student’s answer and the teacher’s evaluation from the average ‘wait-time’ of less than a second to three seconds, produces measurable increases in learning (although increases beyond three seconds have little effect, and may cause lessons to lose pace). In fact, questions need not always come from the teacher. There is substantial evidence that students’ learning is enhanced by getting them to generate their own questions. If instead of writing an end-of-topic test herself, the teacher asks the students to write a test that tests the work the class has been doing, the teacher can gather useful evidence about what the students think they have been learning, which is often very different from what the teacher thinks the class has been learning. This can be a particularly effective strategy with disaffected older students, who often feel threatened by tests. Asking them to write a test for the topic they have completed, and making clear that the teacher is going to mark the question rather than the answers, can be a hugely liberating experience for many students.

Practical techniques: feedback Comment-only grading Focused grading Explicit reference to rubrics Suggestions on how to improve ‘Strategy cards’ ideas for improvement Not giving complete solutions Re-timing assessment (eg two-thirds-of-the-way-through-a-unit test) The research evidence suggests that feedback in terms of scores, grades and levels is unlikely to improve achievement, but that feeding back in terms of comments (whether written or verbal) is. This immediately raises the question “what kind of comments”, and although there is no specific research evidence on this point, it seems that, to be useful, a comment should cause thinking to take place. This feature of ‘mindfulness’ is one of the crucial features of effective formative assessment—effective learning involves having most of the students thinking most of the time. This notion of ‘mindfulness’ also gives some clues about what sort of marking is most helpful. Many teachers say that formative feedback is less useful in mathematics, because an answer is either wrong or right. But even where answers are wrong or right, we can still encourage students to think. For example, rather than marking answers right and wrong and telling the students to do corrections, teachers could, instead, feed back saying simply “Three of these ten questions are wrong. Find out which ones and correct them”. After all, we are often telling our students to check their work, but rarely help them develop the skills to do so. Other strategies that are useful are focused gradin—ie grading a particular piece of work for one aspect (such as sentence structure, expression or spelling) rather than trying to correct everything. This is particularly useful if the comments can be related directly to the assessment criteria for the work. Of course, it is very difficult for feedback to function formatively at the end of a unit so rather than an ‘end-of-unit test’ it may be more useful to have a ‘two-thirds-of-the-way-through-a-unit test’. Those students who have understood something can then help those who haven’t. Many teachers sometimes worry that such strategies may hold back abler students, but the research evidence suggests that it is the students who give help who benefit most from such peer-tutoring. While this may not accelerate more able students through the curriculum, it does lead to better long-term retention.

Practical techniques: sharing learning expectations Explaining learning objectives at start of lesson/unit Criteria in students’ language Posters of key words to talk about learning eg describe, explain, evaluate Planning/writing frames Annotated examples of different standards to ‘flesh out’ assessment rubrics (e.g. lab reports) Opportunities for students to design their own tests Most of these strategies are self-explanatory. Planning and writing frames provide a structure to help students develop a response. While some teachers see such frames as constricting, for most students they provide valuable ‘scaffolding’ for their answers.

Practical techniques: peer and self-assessment Students assessing their own/peers’ work with scoring guides, rubrics or exemplars two stars and a wish Training students to pose questions Identifying group weaknesses Self-assessment of understanding Red/green discs Traffic lights Smiley faces Post-it notes End-of-lesson students’ review Again, most of these strategies are self-explanatory. With ‘traffic-lights’ the teacher identifies a small number of objectives for the lesson (perhaps only one), which are made as clear as possible to the students at the beginning of the lesson. At the end of the lesson, students are asked to indicate their understanding of each objective by a green, yellow or red circle, according to whether they feel they have achieved the objective fully, partially or not at all. This provides useful feedback to the teacher at two levels—to see if there are parts of the lesson that it would be worth re-doing with the whole class, but also to get feedback about which students would particularly benefit from individual support (one technique here has been to ask ambers and greens to work together, while the teacher works with the reds). However, the real benefit of such a system is that it forces the student to reflect on what she or he has been learning. Other teachers have used ‘smiley faces’ which have the advantage, if drawn in pencil, of being modifiable when the student is more confident about their understanding. The other strategy that may require explanation is that of ‘end-of-lesson’ reviews. The idea here is that at the beginning of the lesson, one student is appointed as a ‘rapporteur’ for the lesson. The teacher then teaches a whole-class lesson on some topic, and finishes the lesson ten or fifteen minutes before the end of the lesson. The student rapporteur then gives a summary of the main points of the lesson, and tries to answer any remaining questions that students in the class may have. If he or she can’t answer the questions, then the rapporteur asks members of the class to help out. What is surprising is that teachers who have tried this out have found that students are queuing up to play the role of rapporteur, provided this is started at the beginning of the school year, or even better, when students are new to the school.

Professional development must be Consistent with what we know about adult learning, incorporating choice respect for prior experience recognition of varied learning styles and motivation Sustained Contextualized Consistent with research on expertise

A model for teacher learning Ideas Evidence Small steps Flexibility Choice Accountability Support

Why Teacher Learning Communities? Teacher as local expert Sustained over time Supportive forum for learning Embedded in day-to-day reality Domain-specific

Initial workshops TLC meetings Peer observations Training for leaders A four-part model Initial workshops TLC meetings Peer observations Training for leaders

What formative assessment strategies do you use already? Three questions What formative assessment strategies do you use already? What new ideas do you want to add to your practice? What will you do less of to make time? During lessons Outside of lessons

Write a memo to yourself What can you do? Write a memo to yourself To be opened in the New Year Commit to making 2 or 3 changes Hold yourself accountable Set up a study group Set up peer observations