1. (1) If Language is a Complex Adaptive System, What is Language Assessment? Presented at “Language as a Complex Adaptive System”, an invited conference celebrating the 60 th Anniversary of Language Learning, at the University of Michigan, Ann Arbor, MI, November 7-9, 2008. The first author was supported by a grant from the Spencer Foundation. Robert J. Mislevy University of Maryland Chengbin Yin Center for Applied Linguistics

2. (2) Key Ideas Assessment as evidentiary argument, not simply as measurement. Arguments constructed around … »View of the nature of proficiency. »Situations and ways people acquire it and use it. Relevant work taking place in language testing from an interactionist perspective. Reconceiving measurement models

3. (3) The Assessment Argument (Messick, 1994) What complex of knowledge, skills, or other attributes should be assessed? What behaviors or performances should reveal those constructs? What tasks or situations should elicit those behaviors? We’ll look at a more technical representation in a little while.

4. (4) Snow & Lohman, 1989 Summary test scores, and factors based on them, have often been though of as “signs” indicating the presence of underlying, latent traits. [  ] … An alternative interpretation of test scores as samples of cognitive processes and contents … is equally justifiable and could be theoretically more useful.

5. (5) LaCAS and Assessment Arguments Interactionalist perspective in language testing: »Communicative competence »Contextual features of tasks »Language tests for specific purposes

6. (6) An Interactionalist Perspective (Young, 2000, 2008) … language used in specific discursive practices rather than … language ability independent of context. Focus on the co-construction of discursive practices by all participants... A set of general interactional resources that participants draw upon in specific ways in order to co-construct a discursive practice.

7. (7) An Interactionalist Perspective (Young, 2000, 2008) Relationship between participants’ employment of interactional resources and the context in which they are employed. Varying with the practice and the participants…

8. (8) Challenges for Assessment (Chalhoub-Deville, 2003) Amending the construct of individual ability to accommodate [how] language use in a communicative event reflects dynamic discourse, which is co-constructed among participants; and … reconciling [the notion that language ability is local] with the need for assessments to yield scores to generalize across contextual boundaries.

9. (9) Sociocognitive Foundations Themes from, e.g., cognitive psychology, literacy, neuroscience, anthropology: »Connectionist metaphor, associative memory Situated cognition & information processing »Construction-Integration (CI) theory of comprehension (Kintsch and others) Individual  Sociocultural perspectives »A cognitive theory of cultural meaning (Strauss & Quinn, 1997)

10. (10) A Cognitive Theory of Cultural Meaning  “Interactional Resources” Extrapersonal: »Cultural models: What ‘being sick’ means, restaurant script, Newton’s laws, complaints »Linguistic: Grammar, conventions, constructions Intrapersonal: »Patterns from experience at many levels »Schemas / frames / understandings / assumptions Interplay  Situated understandings »Access to, and ways of interacting with, shared structures in order to accomplish goals The user’s knowledge of the language rules is interlocked with his knowledge of when, where, and with whom to use them. (R. Ellis, 1985)

11. (11) B Inside B Inside AA observablenot observable

12. (12) B Inside B Inside A Context A A la Kintsch: Propositional content of text / speech… and internal and external aspects of context …

13. (13) B Inside B Inside A Context A The C in CI theory, Construction: Activation of both relevant and irrelevant bits of cultural models, experiences, e.g., Restaurant script, Human motivation Guided in part by linguistic models, e.g. Conventions, constructions, rhetorical frames Content of utterance History with interlocutor Conversation thus far The C in CI theory, Construction: Activation of both relevant and irrelevant bits of cultural models, experiences, e.g., Restaurant script, Human motivation Guided in part by linguistic models, e.g. Conventions, constructions, rhetorical frames Content of utterance History with interlocutor Conversation thus far E.g., tasks in Occupational English Test (OET; McNamara, 1996) call upon patterns re language, but also genre, medical knowledge, use of information in clinical settings. If a pattern hasn’t been developed in past experience, it can’t be activated (although it may get constructed in the interaction). A relevant pattern from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)). If a pattern hasn’t been developed in past experience, it can’t be activated (although it may get constructed in the interaction). A relevant pattern from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)).

14. (14) B Inside B Inside A Context A The I in CI theory, Integration: Situation model: synthesis of coherent / reinforced activated cultural / linguistic / situational patterns Situation model is basis of understanding The I in CI theory, Integration: Situation model: synthesis of coherent / reinforced activated cultural / linguistic / situational patterns Situation model is basis of understanding

15. (15) B Inside B Inside A Context A Situation model is also the basis of planning and action.

16. (16) B Inside B Inside A Context A Previous situation models are input to subsequent situation models.

17. (17) B Inside B Inside A Context A Ideally, activation of relevant and compatible cultural & linguistic models…

18. (18) B Inside B Inside A Context A to lead to (sufficiently) shared understanding; i.e., co-constructed meaning. Kramsch’s "shared internal context "

19. (19) Comments about context… A B Inside B Inside A Context Can distinguish external and internal aspects of context (e.g., Douglas, 2000) External / pubic aspects of context, e.g., Setting Physical attributes External / pubic aspects of context, e.g., Setting Physical attributes Re assessment, Target language use (TLU) features Task features Re assessment, Target language use (TLU) features Task features

20. (20) A B Inside B Inside A Context Aspects of cultural/linguistic/interaction context as interpreted by an external observer. Used to determine what actions signal recognition, comprehension, action through targeted cultural /linguistic models. As such, in assessment, plays role in Evaluation, hence Observable variables As such, in assessment, plays role in Evaluation, hence Observable variables

21. (21) What can we say about individuals? Use of resources in appropriate contexts in appropriate ways; i.e., Attunement to cultural/linguistic patterns: Recognize markers of externally-viewed patterns? Construct internal meanings in their light? Act in ways appropriate to targeted cultural/linguistic models? What is the range and circumstances of activation? (variation of performance across contexts)

22. (22) The Assessment Argument (Messick, 1994) What complex of knowledge, skills, or other attributes should be assessed? What behaviors or performances should reveal those constructs? What tasks or situations should elicit those behaviors?

23. (23) Toulmin’s Argument Claim Backing unless since Warrant Alternative explanation so Data Structure

24. (24) Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation

25. (25) Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation In interactive task, performance flows in time, performance changes situation, may or may not be series of task and observable variables In interactive task, performance flows in time, performance changes situation, may or may not be series of task and observable variables

26. (26) Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Concerns features of (possibly evolving) context as seen from the view of the assessor – in particular, those seen as relevant to targets of inference.

27. (27) Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Evaluation of performance concerns context features indirectly: clues that suggest attunement to features of cultural / linguistic models of interest. (did examinee recognize, comprehend, act accordingly?)

28. (28) Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation “Hidden” aspects of context—not in test theory model but essential to argument: What attunements to cultural / linguistic models can be presumed among examinees, to condition inference re targeted l/c models? “Hidden” aspects of context—not in test theory model but essential to argument: What attunements to cultural / linguistic models can be presumed among examinees, to condition inference re targeted l/c models? Fundamental to situated meaning of student variables in measurement models; Both critical and implicit. Fundamental to situated meaning of student variables in measurement models; Both critical and implicit.

29. (29) Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Assessment context always has its own features that activate some cultural / linguistic models and suppress others in different ways for different examinees. (i.e., “method effects”) Assessment context always has its own features that activate some cultural / linguistic models and suppress others in different ways for different examinees. (i.e., “method effects”) Important for … Alternative explanations Variable performance Important for … Alternative explanations Variable performance

30. (30) Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Design Argument

31. (31) Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation (Bachman)

32. (32) Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation

33. (33) Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation

34. (34) Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation

35. (35) Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation Shared backing for test and use arguments grounds warrant for presumed appropriate activation in TLU context. What features do tasks and TLUs share? Implicit in trait arguments Explicit in interactionalist arguments What features do tasks and TLUs share? Implicit in trait arguments Explicit in interactionalist arguments This is the essence of warrant for claim in use argument.

36. (36) Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation What features do tasks and TLUs not have in common? Questions of validity / generalizability: TLU features that call for other cultural / linguistic models that weren’t in task and may or may not be in examinee’s resources. Target models not activated in LTM in TLU context. Questions of validity / generalizability: TLU features that call for other cultural / linguistic models that weren’t in task and may or may not be in examinee’s resources. Target models not activated in LTM in TLU context. Knowing about target examinees and TLUs is key to strong inferences (Douglas, 1998)

37. (37) Implications for measurement models Basic form: Probability of aspects of performance X ij given parameters for person i and situation j (all could be vector-valued) Way too simple No explicit connection with CI comprehension model, interaction processes, etc. Apparent separation of person and situation characteristics These are indeed properties of the conventional meaning of the measurement model and parameters.

38. (38) An Interactionalist Perspective: Instantiation in a Context Xs result from particular persons calling upon resources in particular contexts (or not, or how) Mechanically  s simply accumulate info across situations Our choosing situations and what to observe drives the situated meaning of  s. Situated meaning of  s are tendencies toward these actions in these situations that call for certain interactional resources, via l/c models.

39. (39) How to model inconsistent performance? Traditional: Model as “noise” / unreliability Promising direction: Model individual’s degree or pattern in variation in terms of context features If “motivated”: Model in terms of  s »Divide & Conquer: Multiple unidimensional tests (OET) »Exploratory multidimensional models »Controlled : Structured multidimensional models »Critical importance of what else you know Structured multidimensional

40. (40) How much can testing gain from modern cognitive psychology? So long as testing is viewed as something that takes place in a few hours, out of the context of instruction, and for the purpose of predicting a vaguely stated criterion, then the gains to be made are minimal. Buzz Hunt (1986) Conclusion

41. (41) I have argued that we need to capitalize on [method effects] by designing tests for specific populations -- tests that contain instructions, content, genre, and language directed toward that population. The goal is to produce tests … that would provide information interpretable as evidence of communicative competence in context. Douglas (1998) Conclusion

42. (42) Conclusion Interactionalist view of test theory… for arguments in interactionalist view of language to assemble, analyze, & interpret assessments in light of context and purpose. Methods and exemplars needed, but pressing need is interpretive frame … »To connect view of language proficiency with the machinery of test theory, »Toward modeling students’ (inter)actions in purposeful variations in task contexts.

43. (43) Thank you!