Presentation on theme: "UNDERSTANDING DIFFICULTY & COMPLEXITY IN WRITING TASKS: TOWARD A USEFUL ANALYTIC FRAMEWORK Anne Nebel Lancaster University"— Presentation transcript:
UNDERSTANDING DIFFICULTY & COMPLEXITY IN WRITING TASKS: TOWARD A USEFUL ANALYTIC FRAMEWORK Anne Nebel Lancaster University email@example.com@lancaster.ac.uk Georgetown University firstname.lastname@example.org@georgetown.edu School of Foreign Service in Qatar TBLT 2009 Lancaster
Writing assessment tasks My research investigates the difficulty and complexity of a set of 8 writing tasks Not experimentally designed tasks Fixed set of retired test tasks (ECCE) Writing task option pair: letter or essay
Task choice? Challenge of task equivalence Within each test and across versions Potential for eliciting sufficient output equally rich in features under assessment Raises validity & reliability issues Fairness is a primary concern What decisions and conclusions are reached from the test results? (Bachman & Palmer, forthcoming)
Research context (1) High-stakes nature of language proficiency assessment (e.g. Greece) Employment opportunities Career advancement Pay scale May impact performance (affective variables) Over-trained and test-savvy Test context as a facet in the complexity equation?
Research context (2) Low-stakes environments (e.g. Qatar) Opportunity to look at test tasks outside the test context Test-like conditions (timed, no resources, etc.) None of the affective and pragmatic interference of a real test situation Motivation?
My research: writing assessment tasks Perceptions of task difficulty Student (N=20 pilot questionnaire and raking activity) Teacher (N=5, 30-minute interviews & ranking activity) Student (N=? interviews and raking activity) Analysis of task complexity 8 task set of 4 pairs (letter & essay) Various model adaptations – toward the development of a framework Analysis of performances on tasks Measures of CAF (which, why) Interactions between perceptions, task complexity and performance (what, how)
How to analyze difficulty & complexity? Expert judgments, VPAs, interviews, surveys and questionnaires. (e.g. Elder, Iwashita and McNamara, 2002; Hamp-Lyons, 1994) Existing models (Robinson, 2001, 2005; Skehan & Foster, 1999, 2001; and Ishikawa, 2008; Kuiken & Vedder, 2007) Taxonomies of characteristics (Bachman & Palmer, 1990; Carr, 2006) Coding & classification systems (Silver et al., 2009) DIFFICULTY learner factors / perceptions COMPLEXITY task structure / analysis
Student perceptions of difficulty (phase I) Task choice? letter (18), essay (2) S1: “I chose the letter because it gives me the way how to write and explain my opinion.” S4: “Letter, because it’s easiest.” S14: “Because I can explain a situation better in a letter than in the essay.” S17: “It’s easier to give an opinion on a letter (less formal) than writing an essay.” S18: “In my view a letter is more personal and it is easy to write their own opinion.” S6*: “Because I don’t know how to write an essay.”
“View from the learner” (Ortega, 2005) “Whether through discourse analysis, psychological questionnaires, or post-task interviews, a process- product approach that considers the full landscape of variables contributed by task, leaner and linguistics outcomes, has the potential to illuminate our theories of the role of planning in task-based language learning” (Ortega, 2005: 116). …of task complexity?
Approach to perceptions data (phase II) Phenomenography Qualitative approach that makes use of in-depth interview data to explore how participants conceive of a phenomenon (Marton, 1981); Reveals “categories of understanding” that can be ordered in an outcome space (Micari et al., 2007); Characterizes “ways of seeing something in terms of the critical aspects of the phenomenon” (Pang, 2003: 145)
Theory and method PCP (Kelly, 1955) / Repertory Grid Technique Personal construct theory is a theoretical framework that “provides a vocabulary with which to interpret the constituents and processes by which a person construes his or her experience” (Roberts, 1999:119). Dyadic difference elicitation technique Elicit constructs based on a comparison of elements. “There is nothing sacrosanct about the triad” (Fransella, Bell and Bannister, 2004: 28).
Elements: 8 tasks Construct elicited (sample) : Specific / non-specific Personal / general (“private/public orientation” Hamp- Lyons & Mathias,1994: 54-55) Real life / artificial or hypothetical Meaningful / meaningless Concrete / general or vague Specified content / need to invent content Voc-ed domain / academic domain 1 st person / 3 rd person Suggested structure / no suggestion of structure
Task ranking Teachers (N=5) ranked 8 tasks in order of difficulty. In every case, task 6 was ranked as the most difficult or the second most difficult. In every case (but one), task 1 was ranked as the easiest or the second easiest task of the set.
Task analysis (essay option, task pair 3) Should public and private resources be used to support development in locations where natural disasters, such as floods, forest fires, and volcano eruptions, could occur? Explain your views. Give specific reasons and examples. (http://www.lsa.umich.edu/UMICH/eli/Home/Test%20P rograms/ECCE/General%20Information/2000ecc eformb.pdf) (letter option, task pair 1) Write a letter to the Editor of the City Times. Name one person you think should go on the Millennium Exchange Program. Describe the person, and tell why you think this person should go. You may nominate yourself or someone else you know. (http://www.lsa.umich.edu/UMICH/eli/Home/Test%20Program s/ECCE/General%20Information/ECCE%20Practice%20T ests/ECCE_06NDWriting3.pdf) Task 6 (most difficult)Task 1 (easiest)
Task complexity analysis: constructs -specific -personal -concrete (abstract) -meaningful 6 = more complex task + specific + personal + concrete + meaningful Task 6 (rebuild city with funds essay) Task 1 (Millennium nominee letter)
Task complexity analysis: LACM and TCF (compared in Kuiken & Vedder, 2007) Code complexity Vocab Cognitive complexity Familiarity of topic Predictability Information organization 6=more complex task Resource directing Few elements Here and now No reasoning demands Resource dispersing Single task Prior knowledge Task difficulty LACM (Skehan)TCF (Robinson)
Task complexity analysis: 1 adaptation for speaking (Elder, Iwashita and McNamara, 2002) 2 performance conditions (+/-) 4 task dimensions: Perspective: your own/another’s Immediacy: there-then/here-now Adequacy: complete/incomplete input Planning time: [constant] Which is more complex: 6? Equal? Inconclusive
Complex tasks, complex writing? Although they don’t always take into account the same task dimensions, the models are capable of accounting for complexity in writing tasks; which dimensions are essential and how do we operationalize them? Can the models predict the corresponding features of writing that will demonstrate complexity of thinking in writing? Which features and why? In language testing, an equally important consideration is how well we can measure these features with a scoring rubric or scale.
References Bachman, L. and Palmer, A. (1990) Language Testing in Practice. Oxford: Oxford University Press. Carr, N. (2006) The factor structure of test task characteristics and examinee performance. Language Testing 23, 269-289. Chalhoub-Deville, M. (2001) Task –based assessments: characteristics and validity evidence. In Bygate, M. Skehan, P. and Swain, M. (2001) Researching pedagogic tasks: second language learning, teaching and testing. Essex: Pearson Education Limited. Ch 10, pp. 210-228. Elder, K, Iwashita, N. and McNamara, T. (2002) Estimating difficulty or oral proficiency tasks: what does the test-taker have to offer? Language Testing 19 (4); 347-368. Fransella, F., Bell, R. and Bannister, D. (2004) A Manual for Repertory Grid Technique. West Sussex: John Wiley and Sons. Hamp-Lyons, L. and Mathias, S. (1994) Examining expert judgments of task difficulty on essay tests. Journal of Second Language Writing 3 (1); 49-68. Ishikawa, T. (2008) The effect of manipulating task complexity along the [+/- here and now] dimensions on L2 written narrative discourse. In M. Garcia May (ed) Investigating tasks in formal language learning. Clevedon, UK: Multilingual Matters. Kelly, G. (1955) The psychology of personal constructs. New York: Norton. Kuiken, F. and Vedder, I. (2007) Task complexity and measures of linguistics performance in L2 writing. IRAL 45, 261-284. Micari, M., Light, G., Calkins, S. and Streetweiser (2007) Assessment beyond performance: phenomenography in educational evaluation. American Journal of Evaluation 28 (40); 458-476. Marton, F. (1981) Phenomenography: describing conceptions of the world around us [online version]. Instructional Science 10, 177-200. Retrieved 2/12/08 from http://www.ped.gu.se/bjorn/phgraph/misc/constr/html) http://www.ped.gu.se/bjorn/phgraph/misc/constr/html
References Ortega, L. (2005) What do learners plan? Leaner-driven attention to form during pre-task planning. In Ellis, R. (ed) Planning and task performance in a second language. Amsterdam/Philadelphia: John Benjamins. Pp. 77-110. Pang, M. F. (2003) Two faces of variation—on continuity in the phenomenographic movement. Scandinavian Journal of Educational Research 47 (2); 14-156. Roberts, J. (1999) PCP as a framework for research into teacher learning and thinking. Language Teaching Research 3 (2); 117-144. Robinson, P. (2001) Task complexity, task difficulty and task production: exploring interactions in a componential framework. Applied Linguistics 22, 27-57. Robinson, P. (2005) Cognitive complexity and task sequencing: studies in a componential framework for second language task design. IRAL 43, 1-32. Silver, E., Mesa, W., Morris, K., Star, J. and Benken, B. (2009) Teaching mathematics for understanding: an analysis of lessons submitted by teachers seeking NBPTS certification. American Educational Research Journal, 46 (2); 501-535. Skehan and Foster (1999) The influence of task structure and processing conditions on narrative retellings. Language Learning 49, 93-100. Skehan and Foster (2001) Cognition and tasks. In P. Robinson (ed) Cognition in second language instruction (pp. 185-205). Cambridge: Cambridge University Press. Wigglesworth, g. (2001) Influences on performance in task-based oral assessments. In Bygate, M. Skehan, P. and Swain, M. (2001) Researching pedagogic tasks: second language learning, teaching and testing. Essex: Pearson Education Limited. Ch 9, pp. 186-209.
Anne Nebel email@example.com firstname.lastname@example.org Thank you.