1. Key findings on comparability of language testing in Europe ECML Colloquium 7th December Dr Nick Saville

2. Policy context EU - “United in diversity”
Multilingualism policy at EU level…
… and implementation in the Member States
Mother-tongue plus 2 (L1+2)
Barcelona Conclusions
European Indicator on Language Competences (EILC)
European Survey on Language Competences (ESLC)
Study on the Comparability of Language Testing in Europe

3. % achieving each CEFR Level
European Survey on Language Competences 2012
% achieving each CEFR Level

5. ESLC - main achievements & limitations
Hard evidence on outcomes of language education in a range of European countries
Reliable data collected through standardised exams
Questionnaire data to support and expand findings
Limitations
Only 14 countries
Only two languages per country
Only the foreign languages taught at school
No test of speaking
Ad hoc choice of testing mode: computer- or paper-based
Standard setting to the CEFR done language by language
“One fits all” model (vs European diversity)

6. ESLC - Conclusions A simple and intuitive recipe for success:
language is learned better where motivation is high, where learners perceive it to be useful, and where it is indeed used outside school
e.g. in communicating over the internet, for watching TV, or travelling on holiday
The more teachers and students use the language in class, the better it is learned
Language competences still need to be significantly improved
Rich potential for peer learning in language policy and learning
Complex challenges, require by-country qualitative analysis, mixed methods approaches

7. Policy context – 2012 to 2015 2012 2014 Invited the Commission
European benchmark on languages
The results of the ESLC followed up by ‘Rethinking Education’
2014
EU Council conclusion on Multilingualism and the Development of Language Competences
Invited the Commission
to assess the feasibility of using national data to monitor progress towards the L1+2 goal
to strengthen cooperation with the Council of Europe - ECML
to continue the Open Method of Coordination for exchange of best practices and to enhance the effectiveness of language education

8. Policy context – 2012 to 2015
Study on the Comparability of Language Testing in Europe

10. “Study on comparability of language testing in Europe”
Assessment of comparability of existing national language tests.
Proposals for:
procedures to increase comparability of existing results.
developing existing tests to increase comparability.
EU MSs who might wish to develop language exams.
Summary table of existing results for the first foreign language in each country, adjusted on the basis of the comparability study.

11. Goals of the study
A new approach for measuring language competences at the European level
To develop an evidence base and understanding of language competences making use of existing national language tests

12. Main research questions
How comparable are existing language tests across EU Member States?
How could comparability be improved?
Focusing on:
Existing national language tests
In 28 EU Member States
At ISCED 2 and ISCED 3 levels

13. Scope of the study To complete five main tasks:
Assessment of comparability of the existing national language tests
Proposals for ex-post adjustment of existing results
Proposals for development work in existing exams
Proposals for the development of new national language exams
Comparative overview of existing national results
Based on the goals of the study, we were asked to complete 5 tasks

15. Comparative judgement judges
Organisation
Project Board
Project Team
Project Director
Project Coordinator
Data Management Officer
Project Assistant
Analysts
Content analysts
Comparative judgement judges
+ country representatives
+ SurveyLang language partners

16. Existing national tests
133 exams included in total
77 - ISCED 2
56 - ISCED 3
English
French
German
Spanish
Italian
Swedish
Dutch
ISCED 2 29 15 24 6 2 1
ISCED 3 23 13 3

17. Task 1 - Comparability of existing national tests
QUALITATIVE DATA
Data collected using an online content analysis tool
Focused on comparability of
Constructs
Interpretations
Populations
Measurement characteristics
Used CEFR as framework of reference
Employed expert content analysts

18. Task 1 - Comparability of existing national tests
QUANTITATIVE DATA
Data collected using Comparative Judgement
Innovative online approach using No More Marking website
Focused on first foreign language in each jurisdiction
i.e. English for most, French for some
Only reading + writing tasks
2 tasks per skill per exam
Involved 49 raters from across Europe

19. Task 2 - Proposals for ex-post adjustment
Comparative judgement is a suitable methodology for ex-post adjustment of existing results
This builds a common scale of language proficiency to which national language exams could be mapped
But a number of conditions would need to be met:
a common approach to reporting national results
a commitment to provide relevant evidence (more data)
an agreed schedule to carry it out and a responsible body to monitor the work

20. Task 3 – Proposals to increase comparability
Main recommendations:
Construct
Interpretations
Population
Measurement Characteristics

21. Task 4 - Proposals for new developments
Design the CEFR into the test
Develop procedures to continually improve the test
Develop a process to maintain standards
E.g. through item banking

22. Task 5 - Comparative overview of existing national results
Out of 62 exams (33 jurisdictions)…
4 jurisdictions mentioned CEFR levels in their reports.
National results for 16 tests were reported by what percentage of students passed the test (10 jurisdictions).
National results for 28 tests were reported by what percentage of students achieved each grade (20 jurisdictions).
National results for 20 tests reported results for each skill (12 jurisdictions)
National results for 19 tests reported results by different levels or candidate groups (12 jurisdictions).
The data for 20 jurisdictions (34 tests) contained information including average, maximum, minimum and standard deviation of test scores.

23. Conclusions However, such comparability will only become feasible if:
The study showed how an approach to comparability using national results of language examinations across EU Member States could be successfully implemented in future.
However, such comparability will only become feasible if:
The data collected have sufficient elements in common
The quality of the language examinations is addressed

24. European Commission’s follow-up
Better understanding and use of the CEFR within national education systems
Effective methods of formative assessment in national education systems
Focus on developing students’ language learning skills
Online Linguistic Support tool

25. Implementation in the Member States:
Thank you!
Full report:
Multilingualism policy at EU level:
Implementation in the Member States:

26. Flourishing diversity