Ensuring quality in language testing and assessment: the contribution of the CEFR ECML Colloquium2016 The experience of the ECML RELANG team over the three years of delivering RELANG Training and Consultancy: Aligning language curricula, tests and examinations to the CEFR in a total of 20 European countries  

on Innovative Methodologies and Assessment in Language Learning RELANG Relating language curricula and examinations to the common European reference levels of language proficiency Promoting quality assurance in education and facilitating mobility European Centre for Modern Languages and European Commission Cooperation on Innovative Methodologies and Assessment in Language Learning  

RELANG Team Members   Évelyne Bérard: (formerly) Université de Franche-Comté, Centre de linguistique appliquée, France Jana Bérešová: Trnava University, Slovak Republic José Noijons: (formerly) Cito, Institute for Educational Measurement, The Netherlands (Coordinator) Gábor Szabó: University of Pécs, Hungary

Modules on Offer (1)   Developing valid language tests for the use in the classroom Target group: teacher trainers (both pre-serviced and in-service) Constructing valid language tests and examinations based on the CEFR Target group: language testers and examiners, item writers and curriculum developers Relating existing tests and examinations to the CEFR Target group: language testers and examiners, item writers and curriculum developers, policy makers

Modules on Offer (2)   Screening existing tests and examinations for their relations to the CEFR Target group: examination centres and policy makers Relating foreign language curricula to the CEFR Target group: curriculum developers, teacher trainers and policy makers Applying the CEFR to the development of tests of second language competence Target group: policy makers, language test developers, teacher trainers

Basic Documentation The Training and Consultancy is based on two Council of Europe / ECML publications dealing with the construction and validation of language tests linked to the CEFR: The ALTE Manual for Language Test Development and Examining Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEFR): Highlights  

Linking Procedures in the Manual Familiarisation with the CEFR Linking on the basis of specification of curriculum/examination content Standardization and Benchmarking Standard setting Validation: checking that curricula and exam results relate to CEFR levels as intended

The CEFR Model of Language Use Elements in action-oriented approach: Actions performed by persons - individuals and social agents A range of competences, both general and in particular communicative language competences Various contexts under various conditions and constraints to engage in language activities Language processes to produce and/or receive texts in relation to themes in specific domains

Frequently Asked Questions How do I know whether a test is at the CEFR level it claims to be at? Can an exam that tests linguistic competence (language use) be linked to the CEFR? How do we make sure that our pass/fail decisions are related to the CEFR?  Can we test at more than one CEFR level in one test? Can we link our tests to the CEFR without statistics on student performance? Can tests of different skills be summarized at one CEFR level?  

How do I know whether a test is at the CEFR level it claims to be at? Examination centres, testing organisations and publishing companies claim that the tests they administer or publish are at a given CEFR level The validity of such claims may be very important for stakeholders On the basis of their results, students may be admitted to further education or hired for a job. There is also a need for institutes and employers to be able to depend on the validity of claims Yet some published information on links should be available: Field testing has shown the test is valid and reliable The test has been specified in terms of the CEFR The test has been standardized There is empirical validity (from outside)  

Can an exam that tests linguistic competence (language use) be linked to the CEFR?   The CEFR acknowledges that linguistic competence is an important aspect of language competence It may be difficult to link sections in an exam that test subskills such as grammar or vocabulary to the CEFR Such sections may focus on the structure of a language rather than on the communicative aspects of it. Such sections do not necessarily focus on grammatical constructions that are typical of texts produced in various contexts at various levels, and the vocabulary going with those. Such sections are not usually linked to specific linguistic CEFR descriptors at various levels.

How do we make sure that our pass/fail decisions are related to the CEFR?   Pass/fail scores in exams are often determined without reference to the CEFR Students may pass an exam at a given CEFR level without attaining a score that indicates a proficiency at the desired CEFR level Pass/fail scores related to the CEFR need Standard-setting procedure (for receptive skills) Benchmarking procedure (for productive skills) A score on the exam may indicate two things: the student has or has not passed the exam from a legal perspective the student has or has not reached the desired CEFR level

Can we test at more than one CEFR level in one test?   In theory this is possible, in practice this may be challenging A test that is linked to the CEFR is supposed to tap a representative number of descriptors at each desired CEFR level Tests of Writing, Reading and Listening may become too long Certain test formats for Speaking are suitable A low score at a higher level tests does not mean a pass at a lower level Insufficient score on a B1 test does NOT imply a pass at A2 level

Can we link our tests to the CEFR without statistics on student performance? Test validity will be enhanced when on the basis of data on student performance (pre-testing) the test is adapted: quality of items, difficulty of tasks, type of tasks, length of the test etc. However, there are situations where such a procedure is not possible or too costly, such as in classroom-based testing If it is not possible to collect evidence that a test is valid and reliable through statistics, we can try and link the test to the CEFR through specification. In fact specification is a phase in the linking process that always needs to be carried out.  

Can tests of different skills be summarized at one CEFR level? We cannot simply “average out” performances in different skills In real life most learners are better at one skill than another, certainly at the lower CEFR levels We may be able to average score points, but we cannot average CEFR levels We may be able to say that the student is at B2 for reading and at A2 for writing. We cannot then say that the student is at B1 for reading and writing combined The Council of Europe has propagated profiles, in which the student’s proficiencies in the various language skills are described. The European Language Portfolio has also adopted this approach. Suggestions for action: see http://relang.ecml.at/