1. Integrating assessment with instruction to keep learning on track
Plenary address to NSTA Convention on
Science assessment: research and practical approaches for classroom teachers, school administrators and school districts
Chicago, IL; November 9, 2005
Dylan Wiliam, Educational Testing Service

2. Overview of presentation
Why raising achievement is important
Why investing in teachers is the answer
Why assessment for learning should be the focus
Why teacher learning communities should be the mechanism

3. Raising achievement matters
For individuals
Increased lifetime salary
Improved health
For society
Lower criminal justice costs
Lower health-care costs
Increased economic growth

4. Where’s the solution? Structure Alignment Governance Technology
Small high schools
K-8 schools
Alignment
Curriculum reform
Textbook replacement
Governance
Charter schools
Vouchers
Technology

5. It’s the classroom
Variability at the classroom level is up to 4 times greater than at school level
It’s not class size
It’s not the between-class grouping strategy
It’s not the within-class grouping strategy
It’s the teacher

6. Teacher quality A labor force issue with 2 solutions
Replace existing teachers with better ones?
No evidence that more pay brings in better teachers
No evidence that there are better teachers out there deterred by certification requirements
Improve the effectiveness of existing teachers
The “love the one you’re with” strategy
It can be done
We know how to do it, but at scale? Quickly? Sustainably?

7. Functions of assessment
For evaluating institutions
For describing individuals
For supporting learning
Monitoring learning
Whether learning is taking place
Diagnosing (informing) learning
What is not being learnt
Forming learning
What to do about it

8. Effects of formative assessment
Several major reviews of the research
Natriello (1987)
Crooks (1988)
Black & Wiliam (1998)
Nyquist (2003)
All find consistent, substantial effects

9. Kinds of feedback (Nyquist, 2003)
Weaker feedback only
Knowledge of results (KoR)
Feedback only
KoR + clear goals or knowledge of correct results (KCR)
Weak formative assessment
KCR+ explanation (KCR+e)
Moderate formative assessment
(KCR+e) + specific actions for gap reduction
Strong formative assessment
(KCR+e) + activity

10. Effect of formative assessment (HE)
Weaker feedback only 31
0.16
Feedback only 48
0.23
Weaker formative assessment 49
0.30
Moderate formative assessment 41
0.33
Strong formative assessment 16
0.51

11. Classroom assessment is not (necessarily) formative assessment
Formative assessment is not (necessarily) classroom assessment

12. Formative assessment
Assessment for learning is any assessment for which the first priority in its design and practice is to serve the purpose of promoting pupils’ learning. It thus differs from assessment designed primarily to serve the purposes of accountability, or of ranking, or of certifying competence. An assessment activity can help learning if it provides information to be used as feedback, by teachers, and by their pupils, in assessing themselves and each other, to modify the teaching and learning activities in which they are engaged.
Such assessment becomes ‘formative assessment’ when the evidence is actually used to adapt the teaching work to meet learning needs.
Black et al., 2002

13. Feedback and formative assessment
“Feedback is information about the gap between the actual level and the reference level of a system parameter which is used to alter the gap in some way” (Ramaprasad, 1983 p. 4)
Three key instructional processes
Establishing where learners are in their learning
Establishing where they are going
Establishing how to get there

14. Aspects of formative assessment
Where the learner is going
Where the learner is
How to get there
Teacher
Clarify learning intentions
Engineering effective discussions
Providing feedback that moves learners on
Peer
Understand/
clarify criteria for success
Activating students as instructional resources for one another
Learner
Understand criteria for success
Activating students as owners of their own learning

15. Five key strategies…
Clarifying and understanding learning intentions and criteria for success
Engineering effective classroom discussions that elicit evidence of learning
Providing feedback that moves learners forward
Activating students as instructional resources for each other
Activating students as the owners of their own learning

16. Use evidence about learning to adapt instruction to meet student needs
…and one big idea
Use evidence about learning to adapt instruction to meet student needs

17. Keeping Learning on Track (KLT)
A pilot guides a plane or boat toward its destination by taking constant readings and making careful adjustments in response to wind, currents, weather, etc.
A KLT teacher does the same:
Plans a carefully chosen route ahead of time (in essence building the track)
Takes readings along the way
Changes course as conditions dictate

18. Regulation of learning
Teaching as engineering learning environments
Key features:
Create student engagement
Well-regulated
Long feedback cycles vs. variable feedback cycles
Quality control vs. quality assurance in learning
Teaching vs. learning
Regulation of activity vs. regulation of learning

19. Regulation of learning
Proactive (upstream) regulation
Planning regulation into the learning environment
Planning for evoking information
Interactive (downstream) regulation
‘Negotiating the swiftly-flowing river’
‘Moments of contingency’
Tightness of regulation (goals vs. horizons)
Retrospective regulation
Structured reflection (e.g., lesson study)

20. Types of formative assessment
Long-cycle
Focus: between units
Length: four weeks to one year
Medium-cycle
Focus: within units
Length: one day to two weeks
Short-cycle
Focus: within lessons
Length: five seconds to one hour

21. Practical techniques: Questioning
Improving teacher questioning
Generating questions with colleagues
Closed v open
Low-order v high-order
Appropriate wait-time
Getting away from I-R-E (initiation-response-evaluation)
Basketball rather than serial table-tennis
‘No hands up’ (except to ask a question)
Class polls to review current attitudes towards an issue
‘Hot Seat’ questioning
All-student response systems
ABCD cards
Mini white-boards
Exit passes
In many classrooms teachers use questions as a way of directing the attention of the class, and keeping students ‘on task’. This may keep students ‘on their toes’ but rarely helps learning. It may be better in a whole-class lesson, to have an extended exchange with a single student. This ‘hot seat’ questioning also has benefits for other students who appear to learn vicariously from the exchange. Other students may see extended exchanges between the teacher and another student as a chance to relax and go ‘off task’, but if theteacher asks them what they have learned from a particular exchange between another student and the teacher, their concentration is likely to be quite high!
How much time a teacher allows a student to respond before evaluating the response is also important. It is well known that teachers do not allow students much time to answer questions, and, if they don’t receive a response quickly, they will ‘help’ the student by providing a clue or weakening the question in some way, or even moving on to another student. However, what is not widely appreciated is that the amount of time between the student providing an answer and the teacher’s evaluation of that answer is much more important. Of course, where the question is a simple matter of factual recall, then allowing a student time to reflect and expand upon the answer is unlikely to help much. But where the question requires thought, then increasing the time between the end of the student’s answer and the teacher’s evaluation from the average ‘wait-time’ of less than a second to three seconds, produces measurable increases in learning (although increases beyond three seconds have little effect, and may cause lessons to lose pace).
In fact, questions need not always come from the teacher. There is substantial evidence that students’ learning is enhanced by getting them to generate their own questions. If instead of writing an end-of-topic test herself, the teacher asks the students to write a test that tests the work the class has been doing, the teacher can gather useful evidence about what the students think they have been learning, which is often very different from what the teacher thinks the class has been learning. This can be a particularly effective strategy with disaffected older students, who often feel threatened by tests. Asking them to write a test for the topic they have completed, and making clear that the teacher is going to mark the question rather than the answers, can be a hugely liberating experience for many students.

22. Practical techniques: feedback
Comment-only grading
Focused grading
Explicit reference to rubrics
Suggestions on how to improve
‘Strategy cards’ ideas for improvement
Not giving complete solutions
Re-timing assessment
(eg two-thirds-of-the-way-through-a-unit test)
The research evidence suggests that feedback in terms of scores, grades and levels is unlikely to improve achievement, but that feeding back in terms of comments (whether written or verbal) is. This immediately raises the question “what kind of comments”, and although there is no specific research evidence on this point, it seems that, to be useful, a comment should cause thinking to take place.
This feature of ‘mindfulness’ is one of the crucial features of effective formative assessment—effective learning involves having most of the students thinking most of the time. This notion of ‘mindfulness’ also gives some clues about what sort of marking is most helpful. Many teachers say that formative feedback is less useful in mathematics, because an answer is either wrong or right. But even where answers are wrong or right, we can still encourage students to think. For example, rather than marking answers right and wrong and telling the students to do corrections, teachers could, instead, feed back saying simply “Three of these ten questions are wrong. Find out which ones and correct them”. After all, we are often telling our students to check their work, but rarely help them develop the skills to do so.
Other strategies that are useful are focused gradin—ie grading a particular piece of work for one aspect (such as sentence structure, expression or spelling) rather than trying to correct everything. This is particularly useful if the comments can be related directly to the assessment criteria for the work.
Of course, it is very difficult for feedback to function formatively at the end of a unit so rather than an ‘end-of-unit test’ it may be more useful to have a ‘two-thirds-of-the-way-through-a-unit test’. Those students who have understood something can then help those who haven’t. Many teachers sometimes worry that such strategies may hold back abler students, but the research evidence suggests that it is the students who give help who benefit most from such peer-tutoring. While this may not accelerate more able students through the curriculum, it does lead to better long-term retention.

23. Practical techniques: sharing learning expectations
Explaining learning objectives at start of lesson/unit
Criteria in students’ language
Posters of key words to talk about learning
eg describe, explain, evaluate
Planning/writing frames
Annotated examples of different standards to ‘flesh out’ assessment rubrics (e.g. lab reports)
Opportunities for students to design their own tests
Most of these strategies are self-explanatory. Planning and writing frames provide a structure to help students develop a response. While some teachers see such frames as constricting, for most students they provide valuable ‘scaffolding’ for their answers.

24. Practical techniques: peer and self-assessment
Students assessing their own/peers’ work
with scoring guides, rubrics or exemplars
two stars and a wish
Training students to pose questions
Identifying group weaknesses
Self-assessment of understanding
Red/green discs
Traffic lights
Smiley faces
Post-it notes
End-of-lesson students’ review
Again, most of these strategies are self-explanatory. With ‘traffic-lights’ the teacher identifies a small number of objectives for the lesson (perhaps only one), which are made as clear as possible to the students at the beginning of the lesson. At the end of the lesson, students are asked to indicate their understanding of each objective by a green, yellow or red circle, according to whether they feel they have achieved the objective fully, partially or not at all. This provides useful feedback to the teacher at two levels—to see if there are parts of the lesson that it would be worth re-doing with the whole class, but also to get feedback about which students would particularly benefit from individual support (one technique here has been to ask ambers and greens to work together, while the teacher works with the reds). However, the real benefit of such a system is that it forces the student to reflect on what she or he has been learning. Other teachers have used ‘smiley faces’ which have the advantage, if drawn in pencil, of being modifiable when the student is more confident about their understanding.
The other strategy that may require explanation is that of ‘end-of-lesson’ reviews. The idea here is that at the beginning of the lesson, one student is appointed as a ‘rapporteur’ for the lesson. The teacher then teaches a whole-class lesson on some topic, and finishes the lesson ten or fifteen minutes before the end of the lesson. The student rapporteur then gives a summary of the main points of the lesson, and tries to answer any remaining questions that students in the class may have. If he or she can’t answer the questions, then the rapporteur asks members of the class to help out. What is surprising is that teachers who have tried this out have found that students are queuing up to play the role of rapporteur, provided this is started at the beginning of the school year, or even better, when students are new to the school.

25. Professional development must be
Consistent with what we know about adult learning, incorporating
choice
respect for prior experience
recognition of varied learning styles and motivation
Sustained
Contextualized
Consistent with research on expertise

26. A model for teacher learning
Ideas
Evidence
Small steps
Flexibility
Choice
Accountability
Support

27. Why Teacher Learning Communities?
Teacher as local expert
Sustained over time
Supportive forum for learning
Embedded in day-to-day reality
Domain-specific

28. Initial workshops TLC meetings Peer observations Training for leaders
A four-part model
Initial workshops
TLC meetings
Peer observations
Training for leaders

29. What formative assessment strategies do you use already?
Three questions
What formative assessment strategies do you use already?
What new ideas do you want to add to your practice?
What will you do less of to make time?
During lessons
Outside of lessons

30. Write a memo to yourself
What can you do?
Write a memo to yourself
To be opened in the New Year
Commit to making 2 or 3 changes
Hold yourself accountable
Set up a study group
Set up peer observations