Presentation is loading. Please wait.

Presentation is loading. Please wait.

Developing a task-based assessment of EAP pragmatics Soo Jung Youn University of Hawaii at M ā noa 3 rd International Conference on.

Similar presentations

Presentation on theme: "Developing a task-based assessment of EAP pragmatics Soo Jung Youn University of Hawaii at M ā noa 3 rd International Conference on."— Presentation transcript:

1 Developing a task-based assessment of EAP pragmatics Soo Jung Youn University of Hawaii at M ā noa 3 rd International Conference on Task-Based Language Teaching Lancaster University, UK

2 2 Investigating EAP Pragmatics Pragmatics the study of language from the point of view of the users, especially of the choices they make, the constraints they encounter in using language in social interaction, and the effects their use of language has on the other participants in the act of communication (Crystal, 1997, p. 301, italics added) referring to nonnative speakers comprehension and production of speech acts, and how that L2-related speech act knowledge is acquired (Kasper & Dahl, 1991, p. 216) Advising sessions, s to professor, writing tutorials (Bardovi-Harlig & Hartford, 1990, 1996, 2005) A longitudinal case study of literacy dvlpmt (Chen, 2006) Even with high L2 proficiency, L2 are in great need of L2 pragmatics instruction little attention on systematic investigation of learning needs + pedagogic & assessment tasks

3 3 Assessment of Pragmatics Development of six prototype pragmatic measures and rating scales (Hudson, Detmer, & Brown, 1992, 1995) Development of general pragmatic tests (Roever, 2005, 2006; Liu, 2007; Tada, 2005) Remaining Concerns: 1)how the test batteries are being used and their pedagogical value 2)not fully aligned with pragmatic needs of a specific context How can these concerns be addressed?

4 4 Task-based Language Assessment Fundamental Question: Why and how are task-based assessment being used in particular L2 educational contexts? (Norris, 2002, 2008) Deal with complex, integrative, task-specific needs Examples of empirical TBLA studies University-level second & foreign TBLA (Norris et al., 1998; Brown et al., 2002) DSL in Belgium for high-stakes decisions with summative uses of TBLA (Gysen & Van Avermaet, 2005) Reform an entire foreign language curriculum - both pedagogical & assessment (Byrnes, 2002) TBLA Provide learning opportunities Evaluate effectiveness Rethink the program …

5 5 Study Purpose & Context To engage in the process of identifying EAP pragmatic needs, specifying intended use of assessment tasks, developing assessment tasks & rating scales EAP program context: - English Language Institute (ELI) at the University of Hawaii at M ā noa - provides academic English instruction for international & immigrant & (U.S. citizens) – in need of assistance with EAP - academic skill-based curriculum

6 6 Research Questions (a)What are major EAP pragmatic needs as perceived by students, instructors, and the program? (b)What are the intended uses of assessments of EAP pragmatics within a university EAP program? (c)What kinds of assessment tasks and rating criteria can be developed to meet both the EAP pragmatic learning needs and intended assessment uses in the target program?

7 7 Study Procedure 1) Needs analysis 2) A specification of intended uses of EAP pragmatic assessment 3) Development of pragmatic learning outcomes proposal 4) Development of pragmatic target tasks 5) Development and validation of EAP pragmatic assessment tasks and task-dependent rating scales

8 8 Needs Analysis - Procedure 12 ELI Students a) Focus group b) Interviews 7 ELI Instructors 3 ELI Administrators Questionnaire Semi-structured interviews 20 EAP pragmatics-related situations 102 ELI Students (Grad/UG: 43.1%/ 56.9%) synthesis

9 9 Questionnaire Main Findings 1 (not at all necessary) (very necessary) scale: learning need Cronbach alpha = 0.9 Lowest mean = 2.9 (out of 4) Five most needed tasks (by students) 1. Refuse politely to professors request 2. Request a recommendation letter 3. Write a cover letter to apply for a job 4. Write an appropriate to professor 5. Ask clarifying questions to professors appropriately

10 10 Any gaps among informants? Do all informants mention same needs? Among the least needed tasks by students.. But, mentioned and emphasized by ELI teachers!! Know how to appropriately comment, criticize, and compliment on classmates in class work

11 11 Graduate vs. Undergraduate Students Write an Write a cover letter Recommendation letter request

12 12 Across Different Levels Write a cover letter Recommendation letter request

13 13 Intended Uses of EAP Pragmatic Assessment Who use? ELI students ELI instructors Prospective instructors & students What is being assessed? Overall pragmatic ability with a range of EAP tasks that place common pragmatic demands on students Who/What is being impacted? Not intended to reform, but strengthen the program Satisfaction of needs Evaluate effectiveness Professional development Raising awareness Why assessment? Diagnose Self-assessment Pedagogical tasks Measure students learning progress and achievement Teaching materials

14 14 Assessment Tasks based on identified target tasks: 1.Write a recommendation request to professor Task 1 2. Write an to a potential employer to send application packet 3. Write an to refuse professors request 4. Write constructive comments on cover letter written by classmate 5. Give oral peer-feedback on classmates request to professor 6. Role-play with a classmate in situations of making suggestions and disagreement 7. Role-play with a professor in situations of making requests and refusal

15 15 Task-Dependent Rating Criteria Analytical rating criteria with detailed descriptions Each task has different rating criteria Feedback from domain experts 3 (good) – 2 (able) – 1 (inadequate) e.g.) 1. Write a recommendation letter request to professor Task rating criteria

16 16 Procedure Examinee Participation Rating 40 students participated (low – high proficiency) Individually completed all 7 tasks (about 1 hour to 1 hour 15 min) Either use a computer or hand-write Role-play audio-recorded 3 raters (experienced ESL/EFL teachers with MA in ESL degrees) Had three consecutive separate training sessions Asked to keep monitoring, to document any difficulties, reasoning of rating

17 most difficult easiest FACETS summary Write a constructive comment on cover ltr written by a classmate Role-play with a professor (request, refusal)

18 18 Concluding Marks - so what? Ongoing process Future investigations ( e.g., how these assessment are actually put to use, helpful to teachers & learners?, impact on the program) Need more empirical studies (e.g., do actually assessment tasks promote L2 pragmatic learning? If so, how?) Rethink roles of assessment

19 19 Mahalo! Sincere appreciation to: Dr. John M. Norris & ELI, HELP, Research Participants Funded by University of Hawaii at Mānoa Art & Sciences Student Research Award Graduate Student Organization Grant

20 20 Reference Bardovi-Harlig, K., & Hartford, B. S. (1990). Congruence in native and nonnative conversations: Status balance in the academic advising session, Language Learning, 40, Bardovi-Harlig, K., & Hartford, B. S. (1996). Input in an institutional setting, Studies in Second Language Acquisition, 18, Bardovi-Harlig, K., & Hartford, B. S. (Eds.) (2005). Interlanguage pragmatics: Exploring institutional talk. Mahwah, New Jersey: Lawrence Erlbaum Associates. Brown, J. D., Hudson, T., Norris, J., & Bonk, W. J. (2002). An investigation of second language task- based performance assessments. Honolulu, HI: Second Language Teaching and Curriculum Center. Byrnes, H. (2002). The role of task and task-based assessment in a content-oriented collegiate foreign language curriculum. Language Testing, 19, Chen, C.-F. E. (2006) The development of literacy: From writing to peers to writing to authority figures. Language Learning & Technology, 10(2), Gysen, S., & Van Avermaet, P. (2005). Issues in functional language performance assessment: The case of the certificate Dutch as a foreign language, Language Assessment Quarterly, 2, Hudson, T., Detmer, E., & Brown, J. D. (1992). A framework for testing cross-cultural pragmatics (Technical Report #2). Honolulu: University of Hawaii, Second Language Teaching and Curriculum Center. Hudson, T., Detmer, E., & Brown, J. D. (1995). Developing prototypic measures of cross-cultural pragmatics (Technical Report #7). Honolulu: University of Hawaii, Second Language Teaching and Curriculum Center. Liu, J. (2007). Developing a pragmatic test for Chinese EFL learners. Language Testing, 24, Norris, J. M. (2000). Purposeful language assessment. English Teaching Forum, 38(1), Norris, J. M. (2002). Interpretations, intended uses and designs in task-based language assessment, Language Testing, 19, Norris, J. M., Brown, J. D., Hudson, T., & Yoshioka, J. (1998). Designing second language performance assessments. Technical Report #18. Honolulu, HI: Second Language Teaching and Curriculum Center. Roever, C. (2006). Validation of a web-based test of ESL pragmalinguistics. Language Testing, 23,

Download ppt "Developing a task-based assessment of EAP pragmatics Soo Jung Youn University of Hawaii at M ā noa 3 rd International Conference on."

Similar presentations

Ads by Google