1. Developing a task-based assessment of EAP pragmatics
Soo Jung Youn University of Hawai‘i at Mānoa
3rd International Conference on Task-Based Language Teaching
Lancaster University, UK

2. Investigating EAP Pragmatics
Even with high
L2 proficiency,
L2 are in great
need of
L2 pragmatics
instruction
Advising sessions,
s to professor,
writing tutorials
(Bardovi-Harlig
& Hartford,
1990, 1996, 2005)
A longitudinal
case study of
literacy dvlpmt
(Chen, 2006)
“the study of language from the point of view of the users, especially of the choices they make, the constraints they encounter in using language in social interaction, and the effects their use of language has on the other participants in the act of communication” (Crystal, 1997, p. 301, italics added)
little attention
on systematic
investigation of
learning needs +
pedagogic &
assessment
tasks
“referring to nonnative speakers’ comprehension and production of speech acts, and how that L2-related speech act knowledge is acquired” (Kasper & Dahl, 1991, p. 216)

3. Assessment of Pragmatics
Development of six prototype pragmatic measures and rating scales (Hudson, Detmer, & Brown, 1992, 1995)
Development of general pragmatic tests (Roever, 2005, 2006; Liu, 2007; Tada, 2005)
Remaining Concerns:
how the test batteries are being used and their pedagogical value
not fully aligned with pragmatic needs of a specific context
How can these concerns be addressed?

4. Task-based Language Assessment
Fundamental Question:
Why and how are task-based assessment being used in particular L2 educational contexts? (Norris, 2002, 2008)
Deal with complex, integrative, task-specific needs
Examples of empirical TBLA studies
University-level
second & foreign
TBLA
(Norris et al., 1998;
Brown et al., 2002)
DSL in Belgium
for high-stakes
decisions with
summative uses
of TBLA
(Gysen &
Van Avermaet,
2005)
Reform an entire
foreign language
curriculum
- both pedagogical
& assessment
(Byrnes,
2002)
TBLA
Provide learning
opportunities
Evaluate
effectiveness
Rethink
the program …

5. Study Purpose & Context
To engage in the process of
identifying EAP pragmatic needs,
specifying intended use of assessment tasks,
developing assessment tasks & rating scales
EAP program context:
- English Language Institute (ELI)
at the University of Hawai‘i at Mānoa
- provides academic English instruction for international & immigrant & (U.S. citizens) – in need of assistance with EAP
- academic skill-based curriculum

6. Research Questions
What are major EAP pragmatic needs as perceived by students, instructors, and the program?
What are the intended uses of assessments of EAP pragmatics within a university EAP program?
What kinds of assessment tasks and rating criteria can be developed to meet both the EAP pragmatic learning needs and intended assessment uses in the target program?

7. Study Procedure 1) Needs analysis
2) A specification of intended uses of EAP pragmatic assessment
3) Development of pragmatic learning outcomes proposal
4) Development of pragmatic target tasks
5) Development and validation of EAP pragmatic assessment tasks and task-dependent rating scales

8. Needs Analysis - Procedure
Semi-structured interviews
Questionnaire
12 ELI
Students
a) Focus group
b) Interviews
20 EAP
pragmatics-related
situations
102 ELI
Students
(Grad/UG:
43.1%/ 56.9%)
synthesis
7 ELI
Instructors
3 ELI
Administrators

9. Questionnaire Main Findings
1(not at all necessary) (very necessary) scale: “learning need”
Cronbach alpha = 0.9
Lowest mean = 2.9 (out of 4)
Five most needed tasks (by students)
1. Refuse politely to professor’s request
2. Request a recommendation letter
3. Write a cover letter to apply for a job
4. Write an appropriate to professor
5. Ask clarifying questions to professors appropriately

10. Any gaps among informants?
Do all informants mention same needs?
Among the least needed tasks by students..
But, mentioned and emphasized by
ELI teachers!!
Know how to appropriately comment, criticize, and compliment on classmates’ in class work

11. Graduate vs. Undergraduate Students
Recommendation letter request
Write an
Write a cover letter

12. Across Different Levels
Recommendation letter request
Write a cover letter

13. Intended Uses of EAP Pragmatic Assessment
Who use?
ELI students
ELI instructors
Prospective instructors
& students
What is being
assessed?
Overall pragmatic ability with
a range of EAP tasks that
place common pragmatic
demands on students
Why
assessment?
Diagnose
Self-assessment
Pedagogical tasks
Measure students’ learning
progress and achievement
Teaching materials
Who/What is
being impacted?
Not intended to reform, but
strengthen the program
Satisfaction of needs
Evaluate effectiveness
Professional development
Raising awareness

14. Assessment Tasks based on identified target tasks: 2. Write an e-mail
to a potential
employer
to send application
packet
3. Write an
to refuse professor’s
request
Write a
recommendation
request
to professor
Task 1
4. Write constructive
comments on cover letter
written by classmate
based on identified target tasks:
7. Role-play with a professor
in situations of
making requests and refusal
5. Give oral peer-feedback
on classmate’s request
to professor
6. Role-play with a
classmate
in situations of making
suggestions and
disagreement

15. Task-Dependent Rating Criteria
Analytical rating criteria with detailed descriptions
Each task has different rating criteria
Feedback from domain experts
3 (good) – 2 (able) – 1 (inadequate)
e.g.) 1. Write a recommendation letter request
to professor
Task rating criteria

16. Examinee Participation
Procedure
Examinee Participation
Rating
3 raters
(experienced ESL/EFL teachers with MA in ESL degrees)
Had three consecutive separate training sessions
Asked to keep monitoring, to document any difficulties, reasoning of rating
40 students participated
(low – high proficiency)
Individually completed all 7 tasks
(about 1 hour to 1 hour 15 min)
Either use a computer or hand-write
Role-play audio-recorded

17. FACETS summary
Write a constructive comment on cover ltr written by a classmate
most difficult
0.90
easiest
Role-play with a professor
(request, refusal)

18. Concluding Marks - so what?
Ongoing process
Future investigations (e.g., how these assessment are actually put to use, helpful to teachers & learners?, impact on the program)
Need more empirical studies (e.g., do actually assessment tasks promote L2 pragmatic learning? If so, how?)
Rethink roles of assessment

19. Mahalo. (soojung@hawaii. edu) Sincere appreciation to: Dr. John M
Mahalo! Sincere appreciation to: Dr. John M. Norris & ELI, HELP, Research Participants Funded by University of Hawai‘i at Mānoa Art & Sciences Student Research Award Graduate Student Organization Grant

20. Reference
Bardovi-Harlig, K., & Hartford, B. S. (1990). Congruence in native and nonnative conversations: Status balance in the academic advising session, Language Learning, 40,
Bardovi-Harlig, K., & Hartford, B. S. (1996). Input in an institutional setting, Studies in Second Language Acquisition, 18,
Bardovi-Harlig, K., & Hartford, B. S. (Eds.) (2005). Interlanguage pragmatics: Exploring institutional talk. Mahwah, New Jersey: Lawrence Erlbaum Associates.
Brown, J. D., Hudson, T., Norris, J., & Bonk, W. J. (2002). An investigation of second language task-based performance assessments. Honolulu, HI: Second Language Teaching and Curriculum Center.
Byrnes, H. (2002). The role of task and task-based assessment in a content-oriented collegiate foreign language curriculum. Language Testing, 19,
Chen, C.-F. E. (2006) The development of literacy: From writing to peers to writing to authority figures. Language Learning & Technology, 10(2),
Gysen, S., & Van Avermaet, P. (2005). Issues in functional language performance assessment: The case of the certificate Dutch as a foreign language, Language Assessment Quarterly, 2,
Hudson, T., Detmer, E., & Brown, J. D. (1992). A framework for testing cross-cultural pragmatics (Technical Report #2). Honolulu: University of Hawai‘i, Second Language Teaching and Curriculum Center.
Hudson, T., Detmer, E., & Brown, J. D. (1995). Developing prototypic measures of cross-cultural pragmatics (Technical Report #7). Honolulu: University of Hawai‘i, Second Language Teaching and Curriculum Center.
Liu, J. (2007). Developing a pragmatic test for Chinese EFL learners. Language Testing, 24,
Norris, J. M. (2000). Purposeful language assessment. English Teaching Forum, 38(1),
Norris, J. M. (2002). Interpretations, intended uses and designs in task-based language assessment, Language Testing, 19,
Norris, J. M., Brown, J. D., Hudson, T., & Yoshioka, J. (1998). Designing second language performance assessments. Technical Report #18. Honolulu, HI: Second Language Teaching and Curriculum Center.
Roever, C. (2006). Validation of a web-based test of ESL pragmalinguistics. Language Testing, 23,