1. SLRF 2010 Slide 1 Oct 16, 2010 What is the construct in task-based language assessment? Robert J. Mislevy Professor, Measurement, Statistics and Evaluation Affiliated Professor, Second Language Acquisition University of Maryland, College Park Presented in the invited colloquium “Reprising the role of tasks in language assessment” organized by John Norris and Steven Ross at the Second Language Research Forum 2010, October 14-17, 2010, University of Maryland, College Park, MD. Supported by a grant from the Spencer Foundation.

2. SLRF 2010 Slide 2 Oct 16, 2010 What is the construct? Bachman 2005 LTRC plenary address: »What is the construct? The dialectic of abilities and contexts in defining constructs in language assessment. Challenges from a sociocognitive perspective (Atkinson, 2002; Chalhoub-Deville, 2003) »Interplay of extrapersonal and intrapersonal patterns. »Capabilities as resources to construct and act through relevant patterns in meaningful situations. Challenges in task-based language testing »Many patterns at many levels; interaction; evolving

3. SLRF 2010 Slide 3 Oct 16, 2010 [I]t seems to me that the critical issue is how we define the construct to be assessed — as ability or as task. Bachman, 2007, p. 71.

4. SLRF 2010 Slide 4 Oct 16, 2010 [T]he construct of interest in task-based assessment is performance of the task itself. Long & Norris, 2000, p. 600.

5. SLRF 2010 Slide 5 Oct 16, 2010 The final form of a sentence in ordinary conversation [has] to be understood as an interactional product. Schegloff, 1995, p.192. To adapt a social view of performance… is at some level incompatible with taking the traditional view of performance as a simple projection or display of individual competence. Macnamara & Roever, 2006, p. 46. Is the construct co-constructed by all of the participants in the discursive practice ? Bachman, 2007 Is the construct co-constructed by all of the participants in the discursive practice ? Bachman, 2007

6. SLRF 2010 Slide 6 Oct 16, 2010 The ability components the language user brings to the situation … interact with situational facets to change those facets as well as to be changed by them. [The construct is] “ability – in language user – in context.” Chalhoub-Deville, 2003, p. 372. Is the construct is strictly local? Bachman, 2007 Is the construct is strictly local? Bachman, 2007

7. SLRF 2010 Slide 7 Oct 16, 2010 My Objective Propose an consistent sense of “construct” for assessment, including language testing. Ground it in … »a sociocognitive perspective and »the structure of assessment design and use arguments. Show how it … »encompasses most of the senses of construct in Bachman’s analysis, »helps answer the problematic questions about constructs.

8. SLRF 2010 Slide 8 Oct 16, 2010 The Situative Stance Affordances and abilities … are … inherently relational. An affordance relates attributes of something in the environment to an interactive activity by an agent who has some ability, and an ability relates attributes of an agent to an interactive activity with something in the environment that has some affordance. … It does not go far enough to say that an ability depends on the context of environmental characteristics, or that an affordance depends on the context of an agent's characteristics. The concepts are codefining... Greeno, 1994, p. 338.

9. SLRF 2010 Slide 9 Oct 16, 2010 The assessment [design] argument (Messick, 1994) What complex of knowledge, skills, or other attributes should be assessed? What behaviors or performances should reveal those constructs? What tasks or situations should elicit those behaviors?

10. SLRF 2010 Slide 10 Oct 16, 2010 Toulmin’s Argument Claim Backing unless since Warrant Alternative explanation so Data Structure

11. Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation

12. Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Macro features of performance Micro features of performance Unfolding situated performance Micro features of situation as it evolves Macro features of situation Time Features of context arise over time as student acts / interacts. Features of performance evaluated in light of emerging context. Especially important in interactive and extended performance contexts Bachman / Macnamara / Chalhoub-Deville consider: Is the construct … co-constructed by all of the participants? Me: The activity and its meaning are co- constructed, but the assessment construct is the examinee’s capability to act in ways that productively contribute the construction. Bachman / Macnamara / Chalhoub-Deville consider: Is the construct … co-constructed by all of the participants? Me: The activity and its meaning are co- constructed, but the assessment construct is the examinee’s capability to act in ways that productively contribute the construction.

13. Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Concerns features of (possibly evolving) context as seen from the view of the assessor – in particular, those seen as relevant to targets of inference: Important because of task qua task? Import as opportunities to exhibit pattern attunement in context? Where do they match / mismatch features of use situations? Concerns features of (possibly evolving) context as seen from the view of the assessor – in particular, those seen as relevant to targets of inference: Important because of task qua task? Import as opportunities to exhibit pattern attunement in context? Where do they match / mismatch features of use situations? Evaluation of performance concerns clues that suggest attunement to features of cultural / linguistic models of interest: Aspects of success in task? Aspects of broader L/C/S models? Evaluation of performance concerns clues that suggest attunement to features of cultural / linguistic models of interest: Aspects of success in task? Aspects of broader L/C/S models?

14. Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Design Argument … D p1 OI 1 A1A1 D s1 D p1 OI 2 A2A2 D s2 D p2 D p1 OI n AnAn D sn D pn Claim about student Multiple tasks: What do they have in common / i.e., sampling from what domain?

15. Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation (Bachman)

16. Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation Actions in assessment situations and use situations are always understood through an interactionalist / sociocognitive lens.

17. Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation Claim about student phrased in terms of score(s) on some variable(s) – conduit from assessment observations to use situations The values that the variable(s) can take induce a simplified view of some aspects of peoples’ capabilities from some perspective. The analyst’s interpretation, backed up by a compatible operationalization, is the construct the assessment seeks to measure. The values that the variable(s) can take induce a simplified view of some aspects of peoples’ capabilities from some perspective. The analyst’s interpretation, backed up by a compatible operationalization, is the construct the assessment seeks to measure.

18. Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation Student acting in assessment situation Claim about student Claim about student in use situation This is inherently a statement about the capabilities or propensities of the examinee. Its nature and situated meaning depend on … Design choices about the features of the situation and Features of performance to evaluate; and Choice about the set of task situations; and The relationship of task situations to use situations. This is inherently a statement about the capabilities or propensities of the examinee. Its nature and situated meaning depend on … Design choices about the features of the situation and Features of performance to evaluate; and Choice about the set of task situations; and The relationship of task situations to use situations. Bachman asks: Construct defined in terms of abilities or tasks? Me: The assessment construct is always about examinees’ capabilities, but can be organized around traits or capabilities to perform in various senses in task situations. Bachman asks: Construct defined in terms of abilities or tasks? Me: The assessment construct is always about examinees’ capabilities, but can be organized around traits or capabilities to perform in various senses in task situations.

19. Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation Student acting in assessment situation Claim about student Claim about student in use situation Trait-based testing From situative p.o.v., there many situations with similar affordances, amenable to similar capabilities [“invariant”? Me: too strong] Trait-based construct presumes stability of certain level/kind of pattern use & capabilities across such situations. Situation features designed to evoke evidence of traits as conceived. Correspondence to features of use situation not critical. Performance features identified as evidence of traits. Can be wide variety of use situations, meant to require traits. Trait-based testing From situative p.o.v., there many situations with similar affordances, amenable to similar capabilities [“invariant”? Me: too strong] Trait-based construct presumes stability of certain level/kind of pattern use & capabilities across such situations. Situation features designed to evoke evidence of traits as conceived. Correspondence to features of use situation not critical. Performance features identified as evidence of traits. Can be wide variety of use situations, meant to require traits.

20. Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation Student acting in assessment situation Claim about student Claim about student in use situation Task-based testing I: Focus on competences in performance Task features designed to evoke evidence of traits as conceived, in the context of valued real-world use tasks. Can be language centric, but also pragmatic, sociolinguistic. Correspondence to selected features of use situation important. Performance features identified as evidence of traits. Can be wide variety of use situations, meant to require traits. Construct allows more context dependence of pattern use & capabilities. Task-based testing I: Focus on competences in performance Task features designed to evoke evidence of traits as conceived, in the context of valued real-world use tasks. Can be language centric, but also pragmatic, sociolinguistic. Correspondence to selected features of use situation important. Performance features identified as evidence of traits. Can be wide variety of use situations, meant to require traits. Construct allows more context dependence of pattern use & capabilities.

21. Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation Student acting in assessment situation Claim about student Claim about student in use situation Task-based testing II: Focus on aspects of performance Task features designed to reflect features of important real-world tasks. Correspondence to selected features of use situation important. Performance features identified as evidence of capabilities to act effectively in use situations. Assessment construct: capability to perform in corresponding ways in corresponding real-world situations. Task-based testing II: Focus on aspects of performance Task features designed to reflect features of important real-world tasks. Correspondence to selected features of use situation important. Performance features identified as evidence of capabilities to act effectively in use situations. Assessment construct: capability to perform in corresponding ways in corresponding real-world situations. Long & Norris propose: Construct is task performance? Me: Task performance is of central interest, but assessment argument construct is capability for targeted aspects of task performance, as observed in task and inferred to use situations. Long & Norris propose: Construct is task performance? Me: Task performance is of central interest, but assessment argument construct is capability for targeted aspects of task performance, as observed in task and inferred to use situations. Bachman, Chalhoub-Deville consider: Is the construct … strictly local? Me: The creation of performance is strictly local in every instance, but the construct is capability of doing such -- stability / variability across situations is an empirical question. Bachman, Chalhoub-Deville consider: Is the construct … strictly local? Me: The creation of performance is strictly local in every instance, but the construct is capability of doing such -- stability / variability across situations is an empirical question.

22. SLRF 2010 Slide 22 Oct 16, 2010 Conclusion I Q: What is the construct? A: What do you want it to be? In any given application, the assessment-argument construct concerns capabilities of the individual (as opposed to abilities or traits). It is operationalized by choices wrt assessment design and intended inferences, which can be grounded in a SC perspective of capabilities, made explicit in argument framework, and embodied in the elements and processes of the assessment machinery.

23. SLRF 2010 Slide 23 Oct 16, 2010 Conclusion II Q: Where is the construct? A: The construct is a frame in the analyst’s cognition: to recognize, make sense, and reason from patterns and regularities in peoples’ behaviors in unique situations. The regularities arise from the way people and situations work in the real world, through the interplay of extrapersonal and intrapersonal patterns. Within this frame, we summarize the ways or extents examinees act in assessment situations that we (in part) shape and decide how to characterize in accordance with the frame. We use this synthesis to reason about use situations.