Key findings on comparability of language testing in Europe ECML Colloquium 7th December 2016 Dr Nick Saville

Policy context EU - “United in diversity” Multilingualism policy at EU level… http://www.europarl.europa.eu/RegData/etudes/STUD/2016/573456/IPOL_STU(2016)573456_EN.pdf  … and implementation in the Member States http://www.europarl.europa.eu/RegData/etudes/IDAN/2016/573458/IPOL_IDA(2016)573458_EN.pdf 1995 - Mother-tongue plus 2 (L1+2) 2002 - Barcelona Conclusions 2005 - European Indicator on Language Competences (EILC) 2008 - 2012 - European Survey on Language Competences (ESLC) 2015 - Study on the Comparability of Language Testing in Europe

% achieving each CEFR Level European Survey on Language Competences 2012 % achieving each CEFR Level

ESLC - main achievements & limitations Hard evidence on outcomes of language education in a range of European countries Reliable data collected through standardised exams Questionnaire data to support and expand findings Limitations Only 14 countries Only two languages per country Only the foreign languages taught at school No test of speaking Ad hoc choice of testing mode: computer- or paper-based Standard setting to the CEFR done language by language “One fits all” model (vs European diversity)

ESLC - Conclusions A simple and intuitive recipe for success: language is learned better where motivation is high, where learners perceive it to be useful, and where it is indeed used outside school e.g. in communicating over the internet, for watching TV, or travelling on holiday The more teachers and students use the language in class, the better it is learned Language competences still need to be significantly improved Rich potential for peer learning in language policy and learning Complex challenges, require by-country qualitative analysis, mixed methods approaches

Policy context – 2012 to 2015 2012 2014 Invited the Commission European benchmark on languages The results of the ESLC followed up by ‘Rethinking Education’ 2014 EU Council conclusion on Multilingualism and the Development of Language Competences Invited the Commission to assess the feasibility of using national data to monitor progress towards the L1+2 goal to strengthen cooperation with the Council of Europe - ECML to continue the Open Method of Coordination for exchange of best practices and to enhance the effectiveness of language education

Policy context – 2012 to 2015 2015 - Study on the Comparability of Language Testing in Europe

“Study on comparability of language testing in Europe” Assessment of comparability of existing national language tests. Proposals for: procedures to increase comparability of existing results. developing existing tests to increase comparability. EU MSs who might wish to develop language exams. Summary table of existing results for the first foreign language in each country, adjusted on the basis of the comparability study.

Goals of the study A new approach for measuring language competences at the European level To develop an evidence base and understanding of language competences making use of existing national language tests

Main research questions How comparable are existing language tests across EU Member States? How could comparability be improved? Focusing on: Existing national language tests In 28 EU Member States At ISCED 2 and ISCED 3 levels

Scope of the study To complete five main tasks: Assessment of comparability of the existing national language tests Proposals for ex-post adjustment of existing results Proposals for development work in existing exams Proposals for the development of new national language exams Comparative overview of existing national results Based on the goals of the study, we were asked to complete 5 tasks

Comparative judgement judges Organisation Project Board Project Team Project Director Project Coordinator Data Management Officer Project Assistant Analysts Content analysts Comparative judgement judges + country representatives + SurveyLang language partners

Existing national tests 133 exams included in total 77 - ISCED 2 56 - ISCED 3 English French German Spanish Italian Swedish Dutch ISCED 2 29 15 24 6 2 1 ISCED 3 23 13 3

Task 1 - Comparability of existing national tests QUALITATIVE DATA Data collected using an online content analysis tool Focused on comparability of Constructs Interpretations Populations Measurement characteristics Used CEFR as framework of reference Employed expert content analysts

Task 1 - Comparability of existing national tests QUANTITATIVE DATA Data collected using Comparative Judgement Innovative online approach using No More Marking website Focused on first foreign language in each jurisdiction i.e. English for most, French for some Only reading + writing tasks 2 tasks per skill per exam Involved 49 raters from across Europe

Task 2 - Proposals for ex-post adjustment Comparative judgement is a suitable methodology for ex-post adjustment of existing results This builds a common scale of language proficiency to which national language exams could be mapped But a number of conditions would need to be met: a common approach to reporting national results a commitment to provide relevant evidence (more data) an agreed schedule to carry it out and a responsible body to monitor the work

Task 3 – Proposals to increase comparability Main recommendations: Construct Interpretations Population Measurement Characteristics

Task 4 - Proposals for new developments Design the CEFR into the test Develop procedures to continually improve the test Develop a process to maintain standards E.g. through item banking

Task 5 - Comparative overview of existing national results Out of 62 exams (33 jurisdictions)… 4 jurisdictions mentioned CEFR levels in their reports. National results for 16 tests were reported by what percentage of students passed the test (10 jurisdictions). National results for 28 tests were reported by what percentage of students achieved each grade (20 jurisdictions). National results for 20 tests reported results for each skill (12 jurisdictions) National results for 19 tests reported results by different levels or candidate groups (12 jurisdictions). The data for 20 jurisdictions (34 tests) contained information including average, maximum, minimum and standard deviation of test scores.

Conclusions However, such comparability will only become feasible if: The study showed how an approach to comparability using national results of language examinations across EU Member States could be successfully implemented in future. However, such comparability will only become feasible if: The data collected have sufficient elements in common The quality of the language examinations is addressed

European Commission’s follow-up Better understanding and use of the CEFR within national education systems Effective methods of formative assessment in national education systems Focus on developing students’ language learning skills Online Linguistic Support tool

Implementation in the Member States: Thank you! Saville.n@cambridgeenglish.org Full report: http://ec.europa.eu/languages/library/documents/edl-report_en.pdf Multilingualism policy at EU level: http://www.europarl.europa.eu/RegData/etudes/STUD/2016/573456/IPOL_STU(2016)573456_EN.pdf  Implementation in the Member States: http://www.europarl.europa.eu/RegData/etudes/IDAN/2016/573458/IPOL_IDA(2016)573458_EN.pdf

Flourishing diversity