Language Issues Constructs, Theories, and Scales

Construct = The trait or traits that a test intends to measure. A construct can be defined as an ability or a set of abilities that will be reflected in test performance. A construct is generally defined in terms of a theory; in the case of language, a theory of language. A test, then, represents an operationalization of the theory. Davies, A. et. al. (1999). Dictionary of language testing. Cambridge University Presss.

Bachman & Palmer (p. 61) “[O]ur approach to the design and development of language tests is based on the premise that if we want to use the scores from a language test to make decisions about individuals or inferences about their language ability, we must be able to demonstrate how performance on that language test is related to language use in specific settings other than the language test itself.”

Organizational knowledge – how utterances or sentences and texts are organized Grammatical knowledge – how individual utterances or sentences are organized Textual knowledge – how utterances are organized to form texts Pragmatic knowledge – how utterances or sentences and texts are related to the communicative goals of the language user and to the features of the language use setting Functional knowledge – how utterances or sentences and texts are related to the communicative goals of language users Sociolinguistic knowledge – how utterances or sentences and texts are related to features of the language use setting Areas of language knowledge

Goal setting – deciding what one is going to do Assessment – taking stock of what is needed, what one has to work with, and how well one has done Planning – deciding how to use what one has Strategic Competence

Language “Skills” “It is this conception of language use as the performance of specific situated language use tasks that provides, we believe, a much more useful means for characterizing what have traditionally been called language skills. We would thus not consider language skills to be part of language ability at all, but to be the contextualized realization of the ability to use language in the performance of specific language use tasks…Thus, rather than attempting to define ‘speaking’ as an abstract skill, we believe that it is more useful to identify a specific language use task that involves the activity of speaking, and describe it in terms of its task characteristics and the areas of language ability it engages.” (Bachman & Palmer, pp. 75—76)

Language Scales

Table 1: Sample of Bachman and Palmer decontextualized scale Pragmatic competence RatingVocabularyCohesion 0 Extremely limited vocabulary (A few words and formulaic phrases. Not possible to discuss any topic, due to limited vocabulary) No cohesion (Utterances completely disjointed, or discourse too short to judge.) 1 Small vocabulary (Difficulty in talking with examinee because of vocabulary limitations.) Very little cohesion (Relationships between utterances not adequately marked; frequent confusing relationships among ideas.) 2 Vocabulary of moderate size (Frequently misses or searches for words.) Moderate cohesion (Relationships between utterances generally marked; sometimes confusing relationships among ideas.) 3 Large vocabulary (Seldom misses or searches for words.) Good cohesion (Relationships between utterances well-marked.) 4 Extensive vocabulary (Rarely, if ever, misses or searchers for words. Almost always uses appropriate word.) Excellent cohesion (Uses a variety of appropriate devices; hardly ever confusing relationships among ideas.) Adapted from Bachman (1990).

Table 2: Interagency Language Roundtable Levels and selected contexts - SPEAKING ___________________________________________________________________________________ Speaking 0 No Proficiency Speaking 0+ Memorized ProficiencyDealing with requests Speaking 1Elementary ProficiencyGetting facts over telephone Speaking 1+Elementary Proficiency, PlusMaking arrangements Speaking 2Limited Working ProficiencyEliciting and informed opinion Speaking 2+Limited Working Proficiency, PlusDefending a point of view Speaking 3 General Professional ProficiencyClarifying, Answering objections Speaking 3+ General Professional Proficiency, Plus Speaking 4Advanced Professional Proficiency Speaking 4+Advanced Professional Proficiency, Plus Speaking 5Functionally Native Proficiency

Table 3: The ILR to ACTFL concordance ILRACTFL Levels 0 –0+Novice-Low Novice-Mid Novice-High Level 1Intermediate-low Intermediate-mid Levels 1+Intermediate-high Level 2Advanced Level 2+Advanced Plus Levels 3-5Superior

Table 6: Organization of Canadian Language Benchmark components AN OVERVIEW BenchmarkProficiency LevelSpeaking and Listening Competencies Reading Competencies Writing Competencies STAGE I: BASIC PROFICIENCY 1Initial Creating/interpreting oral discourse in routine non- demanding contexts of language use in:  Social interaction  Instructions  Suasion (getting things done)  Information Interpreting simple texts:  Social interaction texts  Instructions  Business/service texts  Information texts Creating simple texts: Social interaction Recording information Business/service messages Presenting information 2Developing 3Adequate 4Fluent STAGE II: INTERMEDIATE PROFICIENCY 5Initial Creating/interpreting oral discourse in moderately demanding contexts of language use in:  Social interaction  Instructions  Suasion (getting things done)  Information Interpreting moderately complex texts  Social interaction texts  Instructions  Business/service texts  Information texts Creating moderately complex texts: Social interaction Recording information Business/service messages Presenting information/ ideas 6Developing 7Adequate 8Fluent STAGE III: ADVANCED PROFICIENCY 9Initial Creating/interpreting oral discourse in very demanding contexts of language use in:  Social interaction  Instructions  Suasion (getting things done) Information Interpreting complex and very complex texts  Social interaction texts  Instructions  Business/service texts  Information texts Creating complex and very complex texts: Social interaction Recording information Business/service messages Presenting information/ ideas 10Developing 11Adequate 12Fluent (adapted from Pawlikowska-Smith, 2002)

Figure 2: Sample tasks (Norris, et al., 1998) ITEM 1 Situation: Your friend John has broken a bone in his hand. He cannot write (see photo of John). You told him that you would help him with writing. Now, he wants you to fill out a change of address form for him. Study the form provided. Be prepared to listen for the information requested on the form. John said he would leave the information on your answering machine. Task: Play the message from John. Listen for the information from the change of address form. Fill in the form for John. You may listen to the message as many times as you need to get the correct information. Time: You have 10 minutes to complete this task. Product: Completed change of address form.

Figure 4: Example Task-Dependent Rating Scale for Task F05 Item 1inadequate able adept descrIptors descrIptors Examinee incorrectly fills out change of address form such that any essential elements (listed in the able descriptor) are not processable by the post office (this might include illegibility, incorrect placement of information, absence of information, etc. Examinee performance contains some elements from the inadequate descriptor and some elements from the able descriptor. Examinee fills out change of address form according to information given by John, minimally including with correct spelling and correct locations (see form for details) --name --new address --old address --starting date --signature and printed name (either John Harris or examinee’s own name). Examinee performance contains some elements from the able descriptor and some elements from the adept descriptor. Examinee correctly fills out change of address form with ALL applicable information given by John on the answering machine message (see form for details). Rating12345 Figure 5: Example Task-Independent Rating Scale across all task performances Inadequate: A rating of insufficient indicates that the student seems generally incapable of coming to terms with the particular processing component (code, cognitive, communicative) on task like those found on the (test). Student performance contains some elements from the inadequate descriptor and some elements from the able descriptor. Able: A rating of able indicates that the student seems generally capable of coming to terms with the particular processing component on tasks like those found on the (test). Student performance contains some elements from the able description and some elements from the adept descriptor. Adept: A rating of adept indicates that the student seems quite capable of coming to terms with the particular processing component on tasks like those found on the (test); additionally, the student seems to have little or no difficulty in accomplishing such tasks in terms of the processing component.