1. Assessment as a washback tool: is it beneficial or harmful? Nick Saville Director, Research and Validation University of Cambridge ESOL Examinations October 2008

2. 2 Nick Saville, Cambridge ESOL Outline 1.Introduction 2.Washback and impact: some definitions 3.Impact and the law of unintended consequences 4.Researching impact: the Cambridge ESOL approach 5.Conclusion

3. 3 Nick Saville, Cambridge ESOL 1. Introduction Examinations access to opportunity fairness Exert a powerful influence on educational processes possibly negative? concern for social consequences, ethicality, accountability

4. 4 Nick Saville, Cambridge ESOL Questions to be addressed: How can educators and assessment providers foster beneficial effects from their tests and examinations? How can harmful consequences be avoided or their impact mitigated?

5. 5 Nick Saville, Cambridge ESOL “Impact by design” Integral part of a framework for developing and validating examination systems A concept akin to social impact assessment

6. 6 Nick Saville, Cambridge ESOL 2. Washback and impact: definitions Washback (or backwash) has been broadly defined in the assessment literature as the effect of testing on teaching and learning One aspect of the broader phenomenon known as impact – as we shall see later

7. 7 Nick Saville, Cambridge ESOL Washback Alderson and Wall, 1993: 15 washback hypotheses Based on who or what might be affected: Teaching Learning Content Rate of learning Sequence of teaching/learning Degree/depth of curriculum coverage Attitudes of teachers/learners Etc.

8. 8 Nick Saville, Cambridge ESOL Washback A continuum - stretching from harmful at one end, through neutral to beneficial at the other end NegativeNeutralPositive - +

9. 9 Nick Saville, Cambridge ESOL Washback Negative? Restriction of content – narrowing of curriculum Too much time practising for the test Positive? Transparent objectives and outcomes Increased motivation of learners Increased accountability of teachers (?) BUT – cause and effect explanations are rarely adequate …..

10. 10 Nick Saville, Cambridge ESOL Washback Models In the language testing literature: Bailey (1996) Watanabe (2004) Cheng (2004, 2005) Green (2007)

11. 11 Nick Saville, Cambridge ESOL Bailey’s Model (1996) - based on Hughes,1993 3 Ps: Participants students teachers Processes Products learning teaching materials curricula

12. 12 Nick Saville, Cambridge ESOL Watanabe’s Model (2004) Five dimensions : Specificity Intensity Length Intentionality Value Factors influencing the process of washback are related to: the test itself status stakeholders

13. 13 Nick Saville, Cambridge ESOL Green’s Model See: Studies in Language Testing, 25, 2007: IELTS Washback in Context Washback will be most intense – have the most powerful effects on teaching and learning behaviours – where participants see the test as challenging and the results as important (high stakes) SEE BLUE ARROW

14. 14 Nick Saville, Cambridge ESOL Impact Impact concerns the effects and consequences a test can have beyond the classroom and immediate learning context: On individual career or life chances In educational systems and in society

15. 15 Nick Saville, Cambridge ESOL Impact Impact deals with wider influences and includes the “macro contexts” - tests and examinations in society Washback is an aspect of impact related to the “micro contexts” of the classroom and the school

16. 16 Nick Saville, Cambridge ESOL Investigating impact is integral to validation an essential component in establishing the usefulness of an assessment system - fitness for specific purposes and contexts of use Consistent with Messick views of validity (1989, 1996) Consequential aspects of validity fairness and ethics Impact

17. 17 Nick Saville, Cambridge ESOL Educational systems – complex phenomena Stakeholder constituencies many different stakeholders in educational processes complex network of relationships Impact

18. 18 Nick Saville, Cambridge ESOL Learners Teachers Test writers/examiners Receiving institutions School owners Future employers Government agencies Professional bodies Test centre administrators Materials writers Publishers etc Learners Parents/carers Teachers Receiving institutions Employers School owners Examiners Government agencies Professional bodies Academic researchers Test writers/ Examiners etc Test constructs Test format Test conditions Test assessment criteria Test scores Stakeholders in the testing constituency Testing System Contexts of test use - consequences Inputs to test design

19. 19 Nick Saville, Cambridge ESOL Learners Teachers Test writers/examiners Receiving institutions School owners Future employers Government agencies Professional bodies Test centre administrators Materials writers Publishers etc Learners Parents/carers Teachers Receiving institutions Employers School owners Examiners Government agencies Professional bodies Academic researchers Test writers/ Examiners etc Test constructs Test format Test conditions Test assessment criteria Test scores Testing System Contexts of test use - consequences Inputs to test design Stakeholders in the testing constituency

20. 20 Nick Saville, Cambridge ESOL Learners Teachers Test writers/examiners Receiving institutions School owners Future employers Government agencies Professional bodies Test centre administrators Materials writers Publishers etc Learners Parents/carers Teachers Receiving institutions Employers School owners Examiners Government agencies Professional bodies Academic researchers Test writers/ Examiners etc Test constructs Test format Test conditions Test assessment criteria Test scores Testing System Contexts of test use - consequences Inputs to test design Stakeholders in the testing constituency

21. 21 Nick Saville, Cambridge ESOL Impact Multiple voices views, beliefs and attitudes Important for examination providers to collect feedback from the stakeholder to take stakeholder perspectives into account Dynamic relationship between micro and macro contexts

22. 22 Nick Saville, Cambridge ESOL 3. The “law” of unintended consequences “Any purposeful action will produce some unintended consequences” or side-effects “Goodhart’s Law” (or “Campbell’s Law” in the USA) a variant of the “law” of unintended consequences

23. 23 Nick Saville, Cambridge ESOL “Goodhart’s Law” “All performance indicators lose their meaning when adopted as policy targets” Examples: England - school achievement targets - school league tables USA – No Child Left Behind (NCLB) The clearer you are about what you want, the more likely you are to get it – but the less likely it is to mean what you wanted it to! (Dylan Wiliam, 2008)

24. 24 Nick Saville, Cambridge ESOL Perverse incentives? Assessment policy can create a tension between educational objectives at the micro level (teaching and learning in schools) and a requirement for accountability at the macro level

25. 25 Nick Saville, Cambridge ESOL What role can examination providers play? How can examination providers ensure that : examination systems are “fit for purpose”? research is carried out to “find out what is going on” in contexts of use? mitigating action is are carried out if/when negative effects and consequences occur?

26. 26 Nick Saville, Cambridge ESOL 4. Researching impact Cambridge ESOL an international examinations board Maxims of Test Impact An attempt to integrate an action-oriented approach to investigating impact into working practices Milanovic and Saville, 1996

27. 27 Nick Saville, Cambridge ESOL Maxims for achieving/monitoring impact Maxim 1PLAN Use a rational and explicit approach to test development Maxim 2SUPPORT Support stakeholders in the testing process Maxim 3COMMUNICATE Provide comprehensive, useful and transparent information Maxim 4MONITOR and EVALUATE Collect all relevant data and analyse as required. Milanovic and Saville, 1996 - Considering the impact of the Cambridge examinations

28. 28 Nick Saville, Cambridge ESOL Maxim 1Plan Use a rational and explicit approach to test development a process model – cyclical and iterative creates the necessary conditions for appropriate tests to be developed and for on-going validation to take place begins with the purpose - including anticipating how the test should (or might) be used how relevant and useful it is likely to be - social consequences and value implications potential (unplanned) side-effects

29. 29 Nick Saville, Cambridge ESOL Maxim 2Support Support stakeholders in the testing process Involvement of stakeholders - during test design and development consultation on specifications/syllabus design participation in pilot tests Professional support programmes for institutions and individual teachers/students etc. Training of suitable personnel to work on all aspects of the examination cycle item writers, examiners, etc.

30. 30 Nick Saville, Cambridge ESOL Learners Teachers Test writers/examiners Receiving institutions School owners Future employers Government agencies Professional bodies Test centre administrators Materials writers Publishers etc Learners Parents/carers Teachers Receiving institutions Employers School owners Examiners Government agencies Professional bodies Academic researchers Test writers/ Examiners etc Test constructs Test format Test conditions Test assessment criteria Test scores Stakeholders in the testing constituency Testing System Contexts of test use - consequences Inputs to test design

31. 31 Nick Saville, Cambridge ESOL Maxim 3Communicate Provide comprehensive, useful and transparent information Explain issues related to assessment adapted for the different stakeholders the nature of the language constructs being assessed the meaning of language test results etc. A major challenge for all test providers!

32. 32 Nick Saville, Cambridge ESOL Maxim 4 Monitor and Evaluate Collect all relevant data and analyse as required For example, feedback from the candidates and their teachers from local contexts where the test is used (i.e. at the micro level) A long-term endeavour as it involves: the development of suitable instruments for the collection of adequate data appropriate research methodologies e.g. mixed method designs, case studies etc. Evaluate the test’s usefulness routinely determine the need for changes and periodic revisions

33. 33 Nick Saville, Cambridge ESOL Outcomes of impact studies The World-wide survey of the impact of IELTS The Italian Progetto Lingue 2000 (PL2000 project) See Hawkey, (2006) Studies in Language Testing, 24 The theory and practice of impact studies: Messages from studies of the IELTS test and Progetto Lingue 2000

34. 34 Nick Saville, Cambridge ESOL Progetto Lingue 2000 The policy was “.... to introduce innovation into the teaching and learning of other languages by putting greater emphasis on the development of communicative competence in all grades of the school system” It included: the adoption of the Council of Europe’s Common European Framework of Reference (CEFR) as the basis for learning objectives and standards certification of proficiency – the testing (by a certificating body recognised internationally)

35. 35 Nick Saville, Cambridge ESOL Italy - PL2000 Impact Project Learning goals, curriculum, syllabus Students Parents Teachers Teacher- trainers Curriculum developers Testers Publishers Receiving institutions Employers Students Parents Teachers Teacher- trainers Curriculum developers Testers Publishers Receiving institutions Employers Materials Methodology Teacher Support Testing

36. 36 Nick Saville, Cambridge ESOL 5. Conclusion Impact research now reflects the growing importance of evidence-based approaches to educational planning and evaluation By adopting an “impact by design” approach - through careful test development and validation strategies, and by using the findings from impact research to guide future actions - more effective assessment policies and practices can be developed to meet the needs of contemporary education

37. 37 Nick Saville, Cambridge ESOL Thank you

38. 38 Nick Saville, Cambridge ESOL