Presentation on theme: "1 Production of comparative language tests The European Survey on Language Competences Neil Jones Cambridge ESOL SQA 28 february 2013."— Presentation transcript:
1 Production of comparative language tests The European Survey on Language Competences Neil Jones Cambridge ESOL SQA 28 february 2013
Cambridge ESOLProject Management, English Language tests, Language Test Coordination Centre international détudes pédagogiques (CIEP) Centre international détudes pédagogiques French Language tests Goethe InstitutGerman Language tests Università per Stranieri di PerugiaItalian Language tests Universidad de Salamanca/ Instituto Cervantes Spanish Language tests Gallup EuropeSampling + Testing tool + Translation National Institute for Educational Measurement (Cito) Questionnaire design, Analysis
Key aims: The ESLC set out to: provide information on the general level of foreign language knowledge of pupils provide strategic information to policy makers, teachers and learners Using the following instruments: Language Tests: English, French, German, Italian, Spanish 3 skills (Reading, Listening, Writing) A1 to B2 levels of CEFR Contextual questionnaires: addressing 13 language policy issues for students, teachers, principals and countries
Validity as inference to some real world.. World of the test.. Real World of language use Inference
Test performance Test/ task features Processes, knowledge Learner features Real world (target situation of use) MeasureTest score Test construction 1 23 4 Inference to some real world: a sequence of steps
Test/ task features Processes, knowledge Learner features Context validityTheory-based validity What to observe? How? Test construction Scale construction, measurementStandard setting, interpretation Context-specificContext-neutral Test performance 1 Test score How can we score what we observe? Evaluation Scoring validity 2 Generalization Measure Are scores consistent and interpretable? Measurement validity 3 Real world (target situation of use) Extrapolation Does the test score reflect the candidates actual ability? 4 Frame- work levels Alignment How does the specific learning/testing context relate to a more general proficiency framework? 5
Approach to developing the language testing framework Identify the language testing objectives of the ESLC. For each skill, identify test content and testable subskills derived from: a socio-cognitive model of language proficiency language functions or competences salient at levels A1 to B2 in CEFR identify appropriate task types to test these subskills develop specifications, item writer guidelines and a collaborative test development process that are shared across languages in order to produce comparable language tests.
Common European Framework model of language use/learning …the actions performed by persons who as individuals and as social agents develop a range of competences, both general and in particular communicative language competences. They draw on the competences at their disposal in various contexts under various conditions and constraints to engage in language activities involving language processes to produce and/or receive texts in relation to themes in specific domains, activating those strategies which seem most appropriate for carrying out the tasks to be accomplished. The monitoring of these actions by the participants leads to the reinforcement or modification of their competences. (Council of Europe 2001:9, emphasis in original).
CEFRs model of language use and learning Domain of use The language learner/ user Knowledge Processes Strategies Monitoring, assessment Language activity Topic (situation, theme…) Task
Test Task An interactional view Domain of use (TLU) The language learner/ user Knowledge Processes Strategies Language activity Topic (situation, theme…) Task Learners engagement with tasks has interactional authenticity. Test tasks reflect TLU tasks. Test performance enables inference to performance in TLU.
The language learner/ user Knowledge Processes Strategies Building a mental model Integrating new information Enriching the proposition Establishing propositional meaning at clause and sentence levels Creating a text level structure: Construct an organised representation of the text [or texts] Inferencing Parsing Lexical access Word recognition Visual input Central processing core Goal setter Selecting appropriate type of reading: Careful reading Local: Understand sentence GlobaI Comprehend main idea(s) Comprehend overall text Comprehend overall texts Expeditious reading Local: Scan for specifics Global: Skim for gist Search for main ideas and important detail Monitor: goal checking Remediation where necessary Metacognitive mechanisms/ Strategies General knowledge of the world Topic knowledge Meaning representation of text(s) so far Syntactic knowledge Lexicon Lemma: Meaning Word class Lexicon Form: Orthography Phonology Morphology Text structure knowledge: Genre Rhetorical tasks Knowledge A model for reading (after Weir 2005)
Domains of language use A1A2B1B2 personal60%50%40%25% public30%40% 50% educational10% 20% professional0% 5%
Features of approach Implementation of construct: subskills mapped to specific task types Reading and Listening: objectively marked; Writing: subjectively marked Four task development stages: Pilot (2008), Pretesting (2009) Field Trial (2010), Main study (2011) Task adaptation across languages Cross-language vetting
Reading – an A1 task You will read a notice about a cat. For the next 4 questions, answer A, B or C. Leo is lost. Hes my little cat. Hes white with black paws. Hes small and very sweet. He has brown eyes. He wears a grey collar. He didnt come home on Monday and its Thursday today. Thats a long time for a little cat! Leo often sits on top of the houses near here between Smiths bakers shop and King Street. If you find him in your garden or under your car, please telephone me immediately. Please note – Leo doesnt like it when people pick him up, and he doesnt like milk. Thank you for your help! Sophie Martin tel: 798286 Busco a mi gato Leo. Ha desaparecido. Es blanco con las patas negras. Es pequeño, tiene 7 meses y es muy bonito. Tiene los ojos marrones. Lleva un collar gris. Le gusta sentarse en los tejados de las casas que están entre la panadería García y la calle de la Victoria. No veo a Leo desde el lunes y hoy es jueves. Es mucho tiempo para un gato tan pequeño. Leo no bebe leche y no come pan. Si lo ves cerca de tu casa o debajo de un coche, llámame. Gracias por tu ayuda. Sofía Alonso 626 537 548
ES - Foto de vacaciones Estás de vacaciones. Envía un e-mail a un amigo español con esta foto de tus vacaciones. Escribe sobre: el hotel el tiempo qué hace la gente Escribe 20–30 palabras.
Marking of Writing Responsibility of countries Central trickle-down training sessions held for national coordinators A proportion of multiple marking in each country: check on in- country rater agreement But (all) multiple-marked scripts also centrally marked: additional check on leniency/severity
Central marking Country A Country C Country B Multiple marking Single marking Central markers
A. Communication how many of the content points are dealt with (clearly) how well the points are expanded style – register B. Language coherence vocabulary cohesion accuracy
Measurement scale B1 A2 A1 Standards consistently applied Test 3 Test 1 Test 2 Tests at appropriate level 90 80 70 60 50 40 30 Item bank links all levels. Learners located on scale Item response theory and item-banking
Targeted language testing A1A2 B1 B2 Routing test
Standard setting to the CEFR Standard reference: the CoE Manual for relating language exams to the CEFR; http://www.coe.int/t/dg4/linguistic/manuel1_en.asp Jones, N (2009) A comparative approach to constructing a multilingual proficiency framework: constraining the role of standard setting http://www.coe.int/t/dg4/linguistic/Proceedings_CITO_EN.pdf See too the CoE Manual for language test development and examining (ALTE) http://www.coe.int/t/dg4/linguistic/ManualtLangageTest- Alte2011_EN.pdf
Standard setting to the CEFR My conclusions: Build on what you already know; Performance skills are a more practical target for standard setting judgment than indirectly observable, objectively marked skills; Comparative judgments are easier than absolute judgments, and therefore ranking may offer more than rating; In a multilingual framework it is essential to minimize the role of subjective judgment.
Cross-language alignment In ESLC a study was possible for Writing. A ranking study, cf Sevres (2008) for Speaking
Ranking approach to cross-language comparison (Speaking, CIEP 2008) A1A2B1B2C1Levels from rating C1 B2 B1 A2 A1 Standard Set for Rankings
ESLC Writing alignment: five languages on a single scale Level