1. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 1 The Place of Intended Impact in Assessment Use Arguments * Lyle F. Bachman Department of Applied Linguistics U.C.L.A. Los Angeles, California Adrian Palmer Department of Linguistics University of Utah Salt Lake City, Utah *The material in this presentation and handout is based upon the books Language Testing in Practice, Lyle F. Bachman & Adrian Palmer. © Oxford University Press (1996) and Language Assessment in Action, Oxford University Press (forthcoming) as well as on various other articles by Lyle F. Bachman.

2. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 2 References Bachman, L. F. "Building and supporting a case for assessment use." Language Assessment Quarterly, 2(1). 2005. Bachman, Lyle F and Adrian Palmer. Language Testing In Practice. Oxford University Press. 1996. http://www.oup.co.uk/http://www.oup.co.uk/ Bachman, Lyle F and Adrian Palmer. Language Assessment In Action. Oxford University Press. Forthcoming. Toulmin, S. E. The Uses of Argument. Cambridge: Cambridge University Press. 2003. Watson, Jenny Peterson & Sindhvananda, Kanchana. "Notes on the Thammasat University English Program". Bangkok: Thammasat University Faculty of Liberal Arts. 1972. Palmer, Adrian. "Procedures for student classification and grading in courses I-IV". Bangkok: Thammasat University Faculty of Liberal Arts. 1972.

3. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 3 Outline of Presentation How to make an Assessment Use Argument to justify using a test to have specific types of intended impact in a specific situation. How to use this argument to argue for two different testing options (different methods of testing). How to go about making a decision to use one option or the other.

4. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 4 Four Qualities of Useful Language Assessments 1.Reliability: consistency of measurement 2.Construct validity: the meaningfulness of the interpretations that we make on the basis of assessment scores 3.Authenticity: the degree of correspondence between the characteristics of a given assessment task and the characteristics of a relevant non-assessment language use task 4.Intended Impact: the intended effects that taking a assessment, administering and taking a assessment, and using assessment results have on students, teachers, educational systems, and society

5. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 5 Qualities of Usefulness Associated With Links in Assessment Use Argument Bachman & Palmer (Forthcoming)

6. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 6 Summary of Reasoning in Example Assessment Use Argument 4. USE/DECISIONS Assign grades at end of grammar unit. 3. INTERPRETATION Numbers are interpreted as students' knowledge of grammar 2. RESULTS/SCORES Numbers are assigned to performance 1. PERFORMANCE ON ASSESSMENT TASK Students select answers on M-C Grammar Test Tasks Reliability For the following reasons … we can consistently associate grammar scores with students' performance on M-C tasks Construct Validity For the following reasons … scores can be interpreted in terms of "knowledge of grammar Authenticity For the following reasons … the M-C task is appropriate for measuring the students' knowledge of grammar in this situation. Intended Impact For the following reasons … using the interpretations of the students' knowledge of grammar to assign grades will have the intended impact on test takers and test users.

7. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 7 Backing (Supporting Evidence) for Warrants (Reasoning) 2. RESULTS/SCORES Scores (numbers) are assigned to performance. 1. PERFORMANCE ON ASSESSMENT TASKS Students check answers on M-C answer sheet. Reliability Warrants (reasons) Backing (supporting evidence)

8. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 8 Kinds of Backing Prior research Evidence specifically collected for this purpose Accepted community social practice and values Government regulations Laws Legal precedents

9. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 9 Example of Backing (Evidence) for Specific Reliability Warrant (Reasoning) 2. RESULTS/SCORES Scores (numbers) are assigned to performance. 1. PERFORMANCE ON ASSESSMENT TASKS Students mark answers on M-C grammar test. Backing On 2/34/06, measured test/retest reliability =.91 Reliability Warrant Scores are consistent from one administration to another.

10. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 10 Complete Assessment Use Argument Bachman & Palmer (Forthcoming)

11. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 11 Thammasat University Proficiency Test (TUPT) Kanchana Sindhvananda, J. Peterson, A. Palmer, and Thammasat Faculty of Liberal Arts Ajarns. (1971) High-stakes test used to make decisions affecting all students in Thammasat University Purpose –Measure knowledge of grammar, vocabulary reading comprehension –To make decisions about exemption from university ESL courses primarily involving reading placement in required ESL courses primarily involving reading grading in required ESL courses primarily involving reading

12. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 12 Criteria for Student Classification and Grading in Courses I-IV

13. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 13 Intended Impact & Options SituationsTest Method Intended Impact Situation 1 Thammasat 1971 Multiple- choice Efficient and hassle- free placement and grading in reading- based ESL program Situation 2 Thammasat 1973 (hypothet.) Option 1 Multiple- choice 1. Efficient and hassle- free placement and grading in reading and writing-based ESL program 2. Washback: teachers and students Situation 2 Thammasat 1973 (hypothet.) Option 2 Multiple- choice and essay 1. Efficient and hassle- free placement and grading in reading and writing-based ESL program 2. Washback: teachers and students

14. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 14 Intended Impact Argument Warrants

15. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 15 Intended Impact Argument Backing

16. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 16 Authenticity Argument Warrants

17. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 17 Authenticity Argument Backing

18. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 18 Construct Validity Warrants

19. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 19 Construct Validity Backing

20. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 20 Reliability Warrants

21. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 21 Reliability Backing

22. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 22 Situation 2: Same as for Situation 1 With The Following Additions Purpose –Also to measure knowledge of the following constructs in task involving essay writing: grammar vocabulary rhetorical organization –To make decisions about… exemption from new university ESL writing courses placement in new required ESL writing courses grading in new required ESL writing courses Additional intended impact: promote positive washback on writing teachers and students in writing courses

23. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 23 Additional Intended Impact Argument Warrants

24. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 24 Additional Intended Impact Argument Backing

25. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 25 Additional Authenticity Argument Warrants

26. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 26 Additional Authenticity Argument Backing

27. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 27 Additional Construct Validity Warrants

28. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 28 Additional Construct Validity Backing

29. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 29 Comparative Assessment Use Arguments Option #1Option #2

30. Updated 11/16/06©1996 & forthcoming, Bachman & Palmer & OUPPage 30 How to Decide Between Alternatives Describe additional decisions and intended impact –Program directors need to make the following decision: Should they add an essay writing task to the English test given to all students entering Thammasat University? –Program directors want to increase students' ability to write essays because essay writing is an ability that students currently lack. This ability is needed both in instructional and real-life language use tasks that the students need to perform. To make this decision, they need to develop Assessment Use Arguments for two alternatives: 1.Do not add an essay writing task. Continue to use only the M-C tasks to place and grade students in essay writing classes. 2.Add an additional essay writing task and use this to place and grade students in essay writing classes. Then decide 1.which argument they prefer and can live with… 2.on the basis of whether developing the test according to the preferred argument is worth the cost.