Developing a task-based assessment of EAP pragmatics Soo Jung Youn (soojung@hawaii.edu) University of Hawai‘i at Mānoa 3rd International Conference on Task-Based Language Teaching Lancaster University, UK

Investigating EAP Pragmatics Even with high L2 proficiency, L2 are in great need of L2 pragmatics instruction Advising sessions, e-mails to professor, writing tutorials (Bardovi-Harlig & Hartford, 1990, 1996, 2005) A longitudinal case study of e-mail literacy dvlpmt (Chen, 2006) “the study of language from the point of view of the users, especially of the choices they make, the constraints they encounter in using language in social interaction, and the effects their use of language has on the other participants in the act of communication” (Crystal, 1997, p. 301, italics added) little attention on systematic investigation of learning needs + pedagogic & assessment tasks “referring to nonnative speakers’ comprehension and production of speech acts, and how that L2-related speech act knowledge is acquired” (Kasper & Dahl, 1991, p. 216)

Assessment of Pragmatics Development of six prototype pragmatic measures and rating scales (Hudson, Detmer, & Brown, 1992, 1995) Development of general pragmatic tests (Roever, 2005, 2006; Liu, 2007; Tada, 2005) Remaining Concerns: how the test batteries are being used and their pedagogical value not fully aligned with pragmatic needs of a specific context How can these concerns be addressed?

Task-based Language Assessment Fundamental Question: Why and how are task-based assessment being used in particular L2 educational contexts? (Norris, 2002, 2008) Deal with complex, integrative, task-specific needs Examples of empirical TBLA studies University-level second & foreign TBLA (Norris et al., 1998; Brown et al., 2002) DSL in Belgium for high-stakes decisions with summative uses of TBLA (Gysen & Van Avermaet, 2005) Reform an entire foreign language curriculum - both pedagogical & assessment (Byrnes, 2002) TBLA Provide learning opportunities Evaluate effectiveness Rethink the program …

Study Purpose & Context To engage in the process of identifying EAP pragmatic needs, specifying intended use of assessment tasks, developing assessment tasks & rating scales EAP program context: - English Language Institute (ELI) at the University of Hawai‘i at Mānoa - provides academic English instruction for international & immigrant & (U.S. citizens) – in need of assistance with EAP - academic skill-based curriculum

Research Questions What are major EAP pragmatic needs as perceived by students, instructors, and the program? What are the intended uses of assessments of EAP pragmatics within a university EAP program? What kinds of assessment tasks and rating criteria can be developed to meet both the EAP pragmatic learning needs and intended assessment uses in the target program?

Study Procedure 1) Needs analysis 2) A specification of intended uses of EAP pragmatic assessment 3) Development of pragmatic learning outcomes proposal 4) Development of pragmatic target tasks 5) Development and validation of EAP pragmatic assessment tasks and task-dependent rating scales

Needs Analysis - Procedure Semi-structured interviews Questionnaire 12 ELI Students a) Focus group b) Interviews 20 EAP pragmatics-related situations 102 ELI Students (Grad/UG: 43.1%/ 56.9%) synthesis 7 ELI Instructors 3 ELI Administrators

Questionnaire Main Findings 1(not at all necessary)-2-3-4 (very necessary) scale: “learning need” Cronbach alpha = 0.9 Lowest mean = 2.9 (out of 4) Five most needed tasks (by students) 1. Refuse politely to professor’s request 2. Request a recommendation letter 3. Write a cover letter to apply for a job 4. Write an appropriate e-mail to professor 5. Ask clarifying questions to professors appropriately

Any gaps among informants? Do all informants mention same needs? Among the least needed tasks by students.. But, mentioned and emphasized by ELI teachers!! Know how to appropriately comment, criticize, and compliment on classmates’ in class work

Graduate vs. Undergraduate Students Recommendation letter request Write an e-mail Write a cover letter

Across Different Levels Recommendation letter request Write a cover letter

Intended Uses of EAP Pragmatic Assessment Who use? ELI students ELI instructors Prospective instructors & students What is being assessed? Overall pragmatic ability with a range of EAP tasks that place common pragmatic demands on students Why assessment? Diagnose Self-assessment Pedagogical tasks Measure students’ learning progress and achievement Teaching materials Who/What is being impacted? Not intended to reform, but strengthen the program Satisfaction of needs Evaluate effectiveness Professional development Raising awareness

Assessment Tasks based on identified target tasks: 2. Write an e-mail to a potential employer to send application packet 3. Write an e-mail to refuse professor’s request Write a recommendation request e-mail to professor Task 1 4. Write constructive comments on cover letter written by classmate based on identified target tasks: 7. Role-play with a professor in situations of making requests and refusal 5. Give oral peer-feedback on classmate’s request e-mail to professor 6. Role-play with a classmate in situations of making suggestions and disagreement

Task-Dependent Rating Criteria Analytical rating criteria with detailed descriptions Each task has different rating criteria Feedback from domain experts 3 (good) – 2 (able) – 1 (inadequate) e.g.) 1. Write a recommendation letter request e-mail to professor Task rating criteria

Examinee Participation Procedure Examinee Participation Rating 3 raters (experienced ESL/EFL teachers with MA in ESL degrees) Had three consecutive separate training sessions Asked to keep monitoring, to document any difficulties, reasoning of rating 40 students participated (low – high proficiency) Individually completed all 7 tasks (about 1 hour to 1 hour 15 min) Either use a computer or hand-write Role-play audio-recorded

FACETS summary Write a constructive comment on cover ltr written by a classmate most difficult 0.90 easiest Role-play with a professor (request, refusal)

Concluding Marks - so what? Ongoing process Future investigations (e.g., how these assessment are actually put to use, helpful to teachers & learners?, impact on the program) Need more empirical studies (e.g., do actually assessment tasks promote L2 pragmatic learning? If so, how?) Rethink roles of assessment

Mahalo. (soojung@hawaii. edu) Sincere appreciation to: Dr. John M Mahalo! (soojung@hawaii.edu) Sincere appreciation to: Dr. John M. Norris & ELI, HELP, Research Participants Funded by University of Hawai‘i at Mānoa Art & Sciences Student Research Award Graduate Student Organization Grant

Reference Bardovi-Harlig, K., & Hartford, B. S. (1990). Congruence in native and nonnative conversations: Status balance in the academic advising session, Language Learning, 40, 467-501. Bardovi-Harlig, K., & Hartford, B. S. (1996). Input in an institutional setting, Studies in Second Language Acquisition, 18, 171-188. Bardovi-Harlig, K., & Hartford, B. S. (Eds.) (2005). Interlanguage pragmatics: Exploring institutional talk. Mahwah, New Jersey: Lawrence Erlbaum Associates. Brown, J. D., Hudson, T., Norris, J., & Bonk, W. J. (2002). An investigation of second language task-based performance assessments. Honolulu, HI: Second Language Teaching and Curriculum Center. Byrnes, H. (2002). The role of task and task-based assessment in a content-oriented collegiate foreign language curriculum. Language Testing, 19, 419-437. Chen, C.-F. E. (2006) The development of e-mail literacy: From writing to peers to writing to authority figures. Language Learning & Technology, 10(2), 35-55. Gysen, S., & Van Avermaet, P. (2005). Issues in functional language performance assessment: The case of the certificate Dutch as a foreign language, Language Assessment Quarterly, 2, 51-68. Hudson, T., Detmer, E., & Brown, J. D. (1992). A framework for testing cross-cultural pragmatics (Technical Report #2). Honolulu: University of Hawai‘i, Second Language Teaching and Curriculum Center. Hudson, T., Detmer, E., & Brown, J. D. (1995). Developing prototypic measures of cross-cultural pragmatics (Technical Report #7). Honolulu: University of Hawai‘i, Second Language Teaching and Curriculum Center. Liu, J. (2007). Developing a pragmatic test for Chinese EFL learners. Language Testing, 24, 391-415. Norris, J. M. (2000). Purposeful language assessment. English Teaching Forum, 38(1), 18-23. Norris, J. M. (2002). Interpretations, intended uses and designs in task-based language assessment, Language Testing, 19, 337-346. Norris, J. M., Brown, J. D., Hudson, T., & Yoshioka, J. (1998). Designing second language performance assessments. Technical Report #18. Honolulu, HI: Second Language Teaching and Curriculum Center. Roever, C. (2006). Validation of a web-based test of ESL pragmalinguistics. Language Testing, 23, 229-256.