Formative Assessment: The most fruitful ground for improving student achievement Dylan Wiliam Renaissance Learning Leadership Conference, Oklahoma City, OK, December 10,

Outline: three issues  Teacher quality  what it is  why it matters  how to get more of it  Assessment  The bridge between teaching and learning  Different approaches to formative assessment  Leadership for teacher learning  Content, then process  Making teacher learning the priority 2

Teacher quality 3

Teaching quality vs. teacher quality 4  Teaching quality depends on a number of factors  The time teachers have to plan teaching  The size of classes  The resources available Material resources Collegial support  The skills of the teacher  All of these are important, but the quality of the teacher seems to be especially important

Teacher quality and student learning SubjectCorrelation WoodheadAll0* Rivkin, Hanushek and Kain (2005)Reading>0.10 Rivkin, Hanushek and Kain (2005)Mathematics>0.11 Rockoff (2004)Reading0.20 Rockoff (2004)Mathematics0.25 Aaronson, Barrow and Sander (2007)Mathematics0.13

Annual growth in achievement, by age 6 Bloom, Hill, Black, and Lipsey (2008)

7  Assuming that  one year’s student growth is 0.3 standard deviations  the correlation between teacher quality and student achievement is 0.15  Then with a good teacher (1sd above the mean) students learn 50% more  And with an outstanding teacher (2sd above the mean) they learn 100% more  Note also that students make some progress through maturation so these are probably underestimates of the true effect (Fitzpatrick, Grissmer, & Hastedt, 2011)

Teacher quality 8  The impact of teacher quality (Hanushek & Rivkin, 2006)  In the classroom of the best teacher in a group of 50 teachers, students learn twice as fast as average.  In the classroom of the least effective teacher in a group of 50, students learn half as fast as average  And in the classrooms of the best teachers, students from disadvantaged backgrounds learn as much as others (Hamre & Pianta, 2005)  Teachers make a difference  But what makes the difference in teachers?  In particular, can we predict student progress from: Teacher qualifications? Value-added? Teacher observation?

Teacher qualifications 9

Teacher qualifications and student progress 10 MathematicsReading PrimaryMiddleHighPrimaryMiddleHigh General theory of education courses Teaching practice courses Pedagogical content courses Advanced university courses Aptitude test scores Harris and Sass (2007) MathematicsReading PrimaryMiddleHighPrimaryMiddleHigh General theory of education courses — Teaching practice courses —+ Pedagogical content courses ++ Advanced university courses —+ Aptitude test scores —

Teacher observations 11

Framework for teaching (Danielson 1996)  Four domains of professional practice 1.Planning and preparation 2.Classroom environment 3.Instruction 4.Professional responsibilities  Links with student achievement (Sartain, et al. 2011)  Domains 1 and 4: no impact on student achievement  Domains 2 and 3: some impact on student achievement 12

A framework for teaching (Danielson, 1996)  Domain 2: The classroom environment  2a: Creating an environment of respect and rapport  2b: Establishing a culture for learning  2c: Managing classroom procedures  2d: Managing student behavior  2e: Organizing physical space  Domain 3: Instruction  3a: Communicating with students  3b: Using questioning and discussion techniques  3c: Engaging students in learning  3d: Using assessment in instruction  3e: Demonstrating flexibility and responsiveness

Observations and teacher quality 14 Sartain, Stoelinga, Brown, Luppescu, Matsko, Miller, Durwood, Jiang, and Glazer (2011) So, the highest rated teachers are 30% more productive than the lowest rated But the best teachers are 400% more productive than the least effective

Imprecision of lesson observations 15 Achieving a reliability of 0.9 in judging teacher quality through lesson observation is likely to require observing a teacher teaching 6 different classes, and for each lesson to be judged by 5 independent observers. Hill, Charalambous and Kraft (2012)

16  To recap  The highest rated teachers generate learning 30% faster than the lowest rated teachers  But the most effective teachers generate learning 400% faster than the least effective teachers  So the best observation systems we have capture less than 10% of teacher quality (in reality, less than this because most teachers are in the middle two categories)  There is therefore a real danger that teachers will “game the system” by aping features of observation protocols while becoming less effective.

Teacher value-added 17

Teacher value-added Value-added quintile: Student fixed-effects model Value-added quintile: Traditional model Goldhaber, Goldschmidt, and Tseng (2013)

Mathematics Atteberry, Loeb and Wyckoff (2013)

Correlation of initial and later performance 20 Mathematics

Correlation of initial and later performance 21 Reading

Issues with value-added models for teachers 22  Different (reasonable) models of value-added give very different estimates of teacher quality (Goldhaber, Goldschmidt and Tseng, 2013)  Teacher value-added in their first year accounts for less than 5% of the variation in teacher quality in their fifth year of teaching (Atteberry, Loeb and Wyckoff, 2013)  Teachers benefit students for at least three years after they stop teaching them (Rothstein, 2010)

Control and impact Control InsideOutside Impact Low Ability groupingSchool buildings High Classroom practicePoverty 23

The relationship of formative assessment to other policy priorities 24

The evidence base for formative assessment  Fuchs & Fuchs (1986)  Natriello (1987)  Crooks (1988)  Bangert-Drowns, et al. (1991)  Dempster (1991, 1992)  Elshout-Mohr (1994)  Kluger & DeNisi (1996)  Black & Wiliam (1998)  Nyquist (2003)  Brookhart (2004)  Allal & Lopez (2005)  Köller (2005)  Brookhart (2007)  Wiliam (2007)  Hattie & Timperley (2007)  Shute (2008) 25

The Formative Assessment Hijack Long-Cycle Across teaching units, terms Four weeks to one year Student monitoring; curriculum alignment Medium-Cycle Within and between teaching units One to four weeks Improved, student-involved assessment; teacher cognition Short-Cycle Within and between lessons Day-by-day or Minute-by- minute Classroom practice; student engagement Span Length Impact

Main Approaches to Formative Assessment  Professional Learning Communities  “…an inclusive group of people, motivated by a shared learning vision, who support and work with each other, finding ways, inside and outside their immediate community, to enquire on their practice and together learn new and better approaches that will enhance all pupils’ learning.” (Stoll et al., 2006)  Two main approaches  Focus on outcomes for students  Focus on increased teacher capacity

Complementary Processes Instructional Data Teams  Quality control  Common assessments  Improvement through better team work and systems  Focus on individual outcomes for students  Regular meetings focused on data  16 points on PISA (in two to three years) Teacher Learning Communities  Quality assurance  Highly structured meetings  Improvement through increased teacher capacity  Focus on teachers’ individual accountability for change  Regular meetings focused on teacher change  30 points on PISA (in two to three years)

Where the learner is going Where the learner is now How to get the learner there Teacher Peer Student Unpacking Formative Assessment Clarifying, sharing, and understanding learning intentions Engineering effective discussions, tasks, and activities that elicit evidence of learning Providing feedback that moves learners forward Activating students as resources for one another Activating students as resources for one another Activating students as owners of their own learning Activating students as owners of their own learning

Where the learner is going Where the learner is now How to get the learner there Teacher Peer Student Unpacking Formative Assessment Using evidence of achievement to adapt what happens in classrooms to meet learner needs

An educational positioning system  A good teacher:  Establishes where the students are in their learning  Identifies the learning destination  Carefully plans a route  Begins the learning journey  Makes regular checks on progress on the way  Makes adjustments to the course as conditions dictate 31

Formative assessment and other priorities  Formative assessment is an integral part of many current policy priorities:  Framework for teaching (Danielson)  Differentiated instruction (Tomlinson)  Response to (instruction and) intervention  Common formative assessments (DuFour) 32

The Danielson framework in detail  Domain 2: The classroom environment  2a: Creating an environment of respect and rapport  2b: Establishing a culture for learning  2c: Managing classroom procedures  2d: Managing student behavior  2e: Organizing physical space  Domain 3: Instruction  3a: Communicating with students  3b: Using questioning and discussion techniques  3c: Engaging students in learning  3d: Using assessment in instruction  3e: Demonstrating flexibility and responsiveness 33

Formative assessment and domain 3  Communicating with students  Using questioning and discussion techniques  Engaging students in learning  Using assessment in instruction  Demonstrating flexibil- ity and responsiveness  Sharing learning intentions with students  Eliciting evidence  Feedback  Students as learning resources  Students as owners of their learning Framework for teachingClassroom formative assessment 34

Differentiated instruction: not a new idea  Differentiation in action (Stradling & Saunders, 1993)  Differences in  educational goals  curriculum structure  course content  learning tasks  teaching approach  pace of learning  assessment  review 35

Most definitions of DI are vague “While the concept of ‘differentiated instruction’ can be defined in many ways, as good a definition as any is ensuring that what a student learns, how he/she learns it, and how the student demonstrates what he/she has learned is a match for that student's readiness level, interests, and preferred mode of learning.” (Tomlinson, 2004 p. 188) “To differentiate instruction is to recognize students' varying background knowledge, readiness, language, preferences in learning and interests; and to react responsively. Differentiated instruction is a process to teaching and learning for students of differing abilities in the same class.” (Hall, Strangman, & Meyer, 2011) 36

Differentiated instruction and formative assessment 37 Aspects of differentiated instruction (Hall, Strangman & Meyer, 2008)FA? Content Several elements and materials are used Align tasks and objectives to learning goals 3 Instruction is concept-focused and principle-driven Process Flexible grouping is consistently used Classroom management benefits students and teachers Products Initial and on-going assessment of student readiness and growth 3 Students are active and responsible explorers 3 Vary expectations and requirements for student responses 3 Misccelaneous Clarify key concepts and generalizations Use assessment as a teaching tool 3 Emphasize critical and creative thinking as a goal in lesson design Engaging all learners is essential 3 Balance between teacher-assigned and student-selected tasks

Response to (instruction and) intervention “Response to intervention integrates assessment and intervention within a multi-level prevention system to maximize student achievement and reduce behavior problems. With RTI, schools identify students at risk for poor learning outcomes, monitor student progress, provide evidence-based interventions and adjust the intensity and nature of those interventions depending on a student’s responsiveness, and identify students with learning disabilities.” (National Center on Response to Intervention, 2010)  Two “creation myths” for RT(I)I  A protocol for preventing academic failure (progress monitoring, early—research-based—intervention)  An alternative to IQ testing in the identification of learning disabilities 38

Response to (instruction and) intervention  Key points  Tier 1 must be high-quality, evidence-based instruction  Student progress must be monitored  Failure to progress triggers additional support  Formative assessment  Makes tier 1 instruction as effective as it can be  Allows assessment of progress (for tier 2 assessment) 39

So much for the easy bit

A model for teacher learning  Content, then process  Content (what we want teachers to change):  Evidence  Ideas (strategies and techniques)  Process (how to go about change):  Choice  Flexibility  Small steps  Accountability  Support 41

Choice

A strengths-based approach to change 43  Belbin inventory (Management teams: Why they succeed or fail):  Eight team roles (defined as “a tendency to behave, contribute and interrelate with others in a particular way”): Company worker; innovator; shaper; chairperson; resource investigator; monitor/evaluator; completer/finisher; team worker  Key ideas: Each role has strengths and allowable weaknesses. People rarely sustain “out-of-role” behavior, especially under stress.  Each teacher’s personal approach to teaching is similar:  Some teachers’ weaknesses require immediate attention.  For most, however, students benefit more from the development of teachers’ strengths.

Flexibility

Examples of techniques 45  Clarifying, sharing and understanding learning intentions and success criteria  Comparing work samples  Engineering effective tasks, discussions and activities that elicit evidence of achievement  Mini white-boards  Feedback  Match the comments to the essays  Activating students as learning resources for one another  Two stars and a wish  Activating students as owners of their own learning  Colored cups

Strategies and techniques  Distinguish between strategies and techniques:  Strategies define the territory of formative assessment (no-brainers).  Teachers are responsible for choice of techniques: Allows for customization; caters for local context Creates ownership; shares responsibility  Key requirements of techniques:  They embody the deep cognitive and affective principles that research shows are important.  They are seen as relevant, feasible, and acceptable. 46

Small steps

Expertise  According to Berliner (1994), experts:  Excel mainly in their own domain  Often develop automaticity for the repetitive operations that are needed to accomplish their goals  Are more sensitive to the task demands and social situation when solving problems  Are more opportunistic and flexible in their teaching than novices  Represent problems in qualitatively different ways than novices  Have faster and more accurate pattern recognition capabilities  Perceive meaningful patterns in the domain in which they are experienced  Begin to solve problems slower but bring richer and more personal sources of information to bear 48

Knowing more than we can say 49  Six video extracts of a person delivering cardiopulmonary resuscitation (CPR):  Five of the video extracts feature students.  One of the video extracts feature an expert.  Videos shown to three groups  students, experts, instructors  Success rate in identifying the expert:  Experts:90%  Students:50%  Instructors:30% Klein and Klein (1981)

Looking at the wrong knowledge 50  The most powerful teacher knowledge is not explicit:  That’s why telling teachers what to do doesn’t work.  What we know is more than we can say.  And that is why most professional development has been relatively ineffective.  Improving practice involves changing habits, not adding knowledge:  That’s why it’s hard: And the hardest bit is not getting new ideas into people’s heads. It’s getting the old ones out.  That’s why it takes time.  But it doesn’t happen naturally:  If it did, the most experienced teachers would be the most productive, and that’s not true (Hanushek & Rivkin, 2006).

Sensory capacity Nørretranders (1998) Sensory system Total bandwidth (in bits/second) Conscious bandwidth (in bits/second) Eyes10,000,00040 Ears100,00030 Skin1,000,0005 Taste1,0001 Smell100,

Hand hygiene in hospitalsStudyFocus Compliance rate Preston, Larson, & Stamm (1981)Open ward16% ICU30% Albert & Condie (1981)ICU28% to 41% Larson (1983)All wards45% Donowitz (1987)Pediatric ICU30% Graham (1990)ICU32% Dubbert (1990)ICU81% Pettinger & Nettleman (1991)Surgical ICU51% Larson, et al. (1992)Neonatal ICU29% Doebbeling, et al. (1992)ICU40% Zimakoff, et al. (1992)ICU40% Meengs, et al. (1994)ER (Casualty)32% Pittet, Mourouga, & Perneger (1999)All wards48% ICU36% Pittet (2001)

Accountability

Making a commitment 54  Action planning:  Forces teachers to make their ideas concrete and creates a record  Makes the teachers accountable for doing what they promised  Requires each teacher to focus on a small number of changes  Requires the teachers to identify what they will give up or reduce  A good action plan:  Does not try to change everything at once  Spells out specific changes in teaching practice  Relates to the five “key strategies” of AFL  Is achievable within a reasonable period of time  Identifies something that the teacher will no longer do or will do less of

And being held to it “I think specifically what was helpful was the ridiculous NCR [No Carbon Required] forms. I thought that was the dumbest thing, but I’m sitting with my friends and on the NCR form I write down what I am going to do next month. “Well, it turns out to be a sort of ‘I’m telling my friends I’m going to do this’ and I really actually did it and it was because of that. It was because I wrote it down. “I was surprised at how strong an incentive that was to do actually do something different…that idea of writing down what you are going to do and then because when they come by the next month you better take out that piece of paper and say ‘Did I do that?’…just the idea of sitting in a group, working out something, and making a commitment…I was impressed about how that actually made me do stuff.” —Tim, Spruce Central High School 55

Support

Supportive accountability  What is needed from teachers:  A commitment to: The continual improvement of practice Focus on those things that make a difference to students  What is needed from leaders:  A commitment to engineer effective learning environments for teachers by: Creating expectations for continually improving practice Keeping the focus on the things that make a difference to students Providing the time, space, dispensation, and support for innovation Supporting risk-taking 57

Teacher learning communities 58

 We need to create time and space for teachers to reflect on their practice in a structured way, and to learn from mistakes. (Bransford, Brown & Cocking, 1999)  “Always make new mistakes.” —Esther Dyson  “Ever tried. Ever failed. No matter. Try again. Fail again. Fail better.” (Beckett, 1984) 59

Teacher learning communities  Plan that the TLC will run for two years.  Identify 10 to 12 interested colleagues:  Composition: Similar assignments (e.g., early years, math/science) Mixed subject/mixed phase Hybrid  Secure institutional support for:  Monthly meetings (75–120 minutes each, inside or outside school time)  Time between meetings (two hours per month in school time): Collaborative planning Peer observation  Any necessary waivers from school policies 60

A “signature pedagogy” for teacher learning  Every monthly TLC meeting should follow the same structure and sequence of activities:  Activity 1: Introduction (5 minutes)  Activity 2: Starter (5 minutes)  Activity 3: How’s it going? (25–50 minutes)  Activity 4: New learning about formative assessment (20–40 minutes)  Activity 5: Personal action planning (15 minutes)  Activity 6: Review of learning (5 minutes) 61

Every TLC needs a leader  The job of the TLC leader(s):  To ensure that all necessary resources (including refreshments!) are available at meetings  To ensure that the agenda is followed  To maintain a collegial and supportive environment  But most important of all:  It is not to be the formative assessment “expert.” 62

Peer observation  Run to the agenda of the observed, not the observer:  Observed teacher specifies focus of observation: E.g., teacher wants to increase wait time.  Observed teacher specifies what counts as evidence: Provides observer with a stopwatch to log wait times.  Observed teacher owns any notes made during the observation. 63

Summary  Raising achievement is important.  Raising achievement requires improving teacher quality.  Improving teacher quality requires teacher professional development.  To be effective, teacher professional development must address:  What teachers do in the classroom  How teachers change what they do in the classroom  Classroom formative assessment + teacher learning communities:  A point of (uniquely?) high leverage  A “Trojan horse” into wider issues of pedagogy, psychology, and curriculum 64

Thank you

For more information, please visit DylanWiliamCenter.com