1. 1 Production of comparative language tests The European Survey on Language Competences Neil Jones Cambridge ESOL SQA 28 february 2013

2. Cambridge ESOLProject Management, English Language tests, Language Test Coordination Centre international détudes pédagogiques (CIEP) Centre international détudes pédagogiques French Language tests Goethe InstitutGerman Language tests Università per Stranieri di PerugiaItalian Language tests Universidad de Salamanca/ Instituto Cervantes Spanish Language tests Gallup EuropeSampling + Testing tool + Translation National Institute for Educational Measurement (Cito) Questionnaire design, Analysis

3. Key aims: The ESLC set out to: provide information on the general level of foreign language knowledge of pupils provide strategic information to policy makers, teachers and learners Using the following instruments: Language Tests: English, French, German, Italian, Spanish 3 skills (Reading, Listening, Writing) A1 to B2 levels of CEFR Contextual questionnaires: addressing 13 language policy issues for students, teachers, principals and countries

4. Validity as inference to some real world.. World of the test.. Real World of language use Inference

5. Test performance Test/ task features Processes, knowledge Learner features Real world (target situation of use) MeasureTest score Test construction 1 23 4 Inference to some real world: a sequence of steps

6. Test/ task features Processes, knowledge Learner features Context validityTheory-based validity What to observe? How? Test construction Scale construction, measurementStandard setting, interpretation Context-specificContext-neutral Test performance 1 Test score How can we score what we observe? Evaluation Scoring validity 2 Generalization Measure Are scores consistent and interpretable? Measurement validity 3 Real world (target situation of use) Extrapolation Does the test score reflect the candidates actual ability? 4 Frame- work levels Alignment How does the specific learning/testing context relate to a more general proficiency framework? 5

7. Approach to developing the language testing framework Identify the language testing objectives of the ESLC. For each skill, identify test content and testable subskills derived from: a socio-cognitive model of language proficiency language functions or competences salient at levels A1 to B2 in CEFR identify appropriate task types to test these subskills develop specifications, item writer guidelines and a collaborative test development process that are shared across languages in order to produce comparable language tests.

8. Common European Framework model of language use/learning …the actions performed by persons who as individuals and as social agents develop a range of competences, both general and in particular communicative language competences. They draw on the competences at their disposal in various contexts under various conditions and constraints to engage in language activities involving language processes to produce and/or receive texts in relation to themes in specific domains, activating those strategies which seem most appropriate for carrying out the tasks to be accomplished. The monitoring of these actions by the participants leads to the reinforcement or modification of their competences. (Council of Europe 2001:9, emphasis in original).

9. CEFRs model of language use and learning Domain of use The language learner/ user Knowledge Processes Strategies Monitoring, assessment Language activity Topic (situation, theme…) Task

10. Test Task An interactional view Domain of use (TLU) The language learner/ user Knowledge Processes Strategies Language activity Topic (situation, theme…) Task Learners engagement with tasks has interactional authenticity. Test tasks reflect TLU tasks. Test performance enables inference to performance in TLU.

11. The language learner/ user Knowledge Processes Strategies Building a mental model Integrating new information Enriching the proposition Establishing propositional meaning at clause and sentence levels Creating a text level structure: Construct an organised representation of the text [or texts] Inferencing Parsing Lexical access Word recognition Visual input Central processing core Goal setter Selecting appropriate type of reading: Careful reading Local: Understand sentence GlobaI Comprehend main idea(s) Comprehend overall text Comprehend overall texts Expeditious reading Local: Scan for specifics Global: Skim for gist Search for main ideas and important detail Monitor: goal checking Remediation where necessary Metacognitive mechanisms/ Strategies General knowledge of the world Topic knowledge Meaning representation of text(s) so far Syntactic knowledge Lexicon Lemma: Meaning Word class Lexicon Form: Orthography Phonology Morphology Text structure knowledge: Genre Rhetorical tasks Knowledge A model for reading (after Weir 2005)

12. Domains of language use A1A2B1B2 personal60%50%40%25% public30%40% 50% educational10% 20% professional0% 5%

13. Features of approach Implementation of construct: subskills mapped to specific task types Reading and Listening: objectively marked; Writing: subjectively marked Four task development stages: Pilot (2008), Pretesting (2009) Field Trial (2010), Main study (2011) Task adaptation across languages Cross-language vetting

14. Reading – an A1 task You will read a notice about a cat. For the next 4 questions, answer A, B or C. Leo is lost. Hes my little cat. Hes white with black paws. Hes small and very sweet. He has brown eyes. He wears a grey collar. He didnt come home on Monday and its Thursday today. Thats a long time for a little cat! Leo often sits on top of the houses near here between Smiths bakers shop and King Street. If you find him in your garden or under your car, please telephone me immediately. Please note – Leo doesnt like it when people pick him up, and he doesnt like milk. Thank you for your help! Sophie Martin tel: 798286 Busco a mi gato Leo. Ha desaparecido. Es blanco con las patas negras. Es pequeño, tiene 7 meses y es muy bonito. Tiene los ojos marrones. Lleva un collar gris. Le gusta sentarse en los tejados de las casas que están entre la panadería García y la calle de la Victoria. No veo a Leo desde el lunes y hoy es jueves. Es mucho tiempo para un gato tan pequeño. Leo no bebe leche y no come pan. Si lo ves cerca de tu casa o debajo de un coche, llámame. Gracias por tu ayuda. Sofía Alonso 626 537 548

15. Reading – an A1 task

16. ES - Foto de vacaciones Estás de vacaciones. Envía un e-mail a un amigo español con esta foto de tus vacaciones. Escribe sobre: el hotel el tiempo qué hace la gente Escribe 20–30 palabras.

17. Marking of Writing Responsibility of countries Central trickle-down training sessions held for national coordinators A proportion of multiple marking in each country: check on in- country rater agreement But (all) multiple-marked scripts also centrally marked: additional check on leniency/severity

18. Central marking Country A Country C Country B Multiple marking Single marking Central markers

19. A. Communication how many of the content points are dealt with (clearly) how well the points are expanded style – register B. Language coherence vocabulary cohesion accuracy

20. 1 3 ~~~~~~~~~ ~~ ~~~~~~~~~ ~~ 2 lowerhigher

21. 1 3 ~~~~~~~~~ ~~ ~~~~~~~~~ ~~ 2 Lower exemplar lowerhigher 5 ~~~~~~~~~ ~~ ~~~~~~~~~ ~~ 4 Higher exemplar

22. Measurement scale B1 A2 A1 Standards consistently applied Test 3 Test 1 Test 2 Tests at appropriate level 90 80 70 60 50 40 30 Item bank links all levels. Learners located on scale Item response theory and item-banking

23. Targeted language testing A1A2 B1 B2 Routing test

24. Test design

25. Standard setting to the CEFR Standard reference: the CoE Manual for relating language exams to the CEFR; http://www.coe.int/t/dg4/linguistic/manuel1_en.asp Jones, N (2009) A comparative approach to constructing a multilingual proficiency framework: constraining the role of standard setting http://www.coe.int/t/dg4/linguistic/Proceedings_CITO_EN.pdf See too the CoE Manual for language test development and examining (ALTE) http://www.coe.int/t/dg4/linguistic/ManualtLangageTest- Alte2011_EN.pdf

26. Standard setting to the CEFR My conclusions: Build on what you already know; Performance skills are a more practical target for standard setting judgment than indirectly observable, objectively marked skills; Comparative judgments are easier than absolute judgments, and therefore ranking may offer more than rating; In a multilingual framework it is essential to minimize the role of subjective judgment.

27. Cross-language alignment In ESLC a study was possible for Writing. A ranking study, cf Sevres (2008) for Speaking

28. Ranking approach to cross-language comparison (Speaking, CIEP 2008) A1A2B1B2C1Levels from rating C1 B2 B1 A2 A1 Standard Set for Rankings

29. ESLC Writing alignment: five languages on a single scale Level

31. First target language (Skills averaged)

32. Second target language (Skills averaged)

33. Asset Languages link between GCSE and CEFR NQF level General qualifications Asset LanguagesAsset CEFR levels Cambridge CEFR levels Cambridge ESOL exams Level 7-8 MasteryC2 Levels 4-6 ProficiencyC1 Level 3 AS/A/AEAAdvancedB2C2CPE Level 2 Higher GCSEIntermediateB1C1CAE Level 1 Foundation GCSEPreliminaryA2B2FCE Entry 3 Level Entry 1-3BreakthroughA1B1PET Entry 2 Level A2KET Entry 1 Level A1

34. GCSE grades and CEFR levels

35. http://www.surveylang.org