1. Paris, France, 6-7 September 2006
Language proficiency evaluation (testing) Selection and development of language tests in the aviation context
Dr Jeremy Mell
Head of Language Studies, ENAC, Toulouse, France
Member PRICE SG, ICAO
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

2. Options for assessing language proficiency for aviation
informal observations
« line checks », « inspections »
classroom assessments
formal language tests
direct/semi-direct/indirect
simulated language use
most direct
integrative: performance samples are matched to rating scales
paper and pencil OR screen and mouse tasks
most indirect
discrete items/skills: favour numerical scores
but « juggling with only one ball »….
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

3. Paris, France, 6-7 September 2006
Language tests
Definition
structured events or procedures
to elicit performances as samples of test-taker ’s language skills
in a standardised way
enabling reliable inferences to be made concerning his/her level of competence
and possibility of reproducing those skills at that level of competence consistently over time
adapted from Carrol (1968) and Douglas (2000)
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

4. Paris, France, 6-7 September 2006
Language testing
A well-developed domain of intellectual activity:
academic
body of research, cross-disciplinary links, international associations,...
codes of ethics, codes of practice
institutions of excellence, « chapels », « gurus », feuds
commercial
international testing service providers
general, academic, business,…
Attempts to link different rating scales/test results
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

5. Types of language tests
entry
placement
diagnostic
progress/achievement
proficiency
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

6. Entry/placement tests
Purposes
ab initio training:
recruitment
form level groups
recurrent training:
benchmarking a population
Characteristics
items chosen to cover a broad range of levels of general language
formats reflect previous education
scores tend to form a « bell curve »
stakes
high (recruitment)
mid (benchmarking)
As slide
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

7. Paris, France, 6-7 September 2006
Diagnostic tests
Purposes
identify specific areas of skill/knowledge for improvement in subsequent training
individualisation of training programmes
Characteristics
each item chosen to represent a single significant area of knowledge/skill
overall score less important than analysis of right/wrong responses - focus on errors
stakes: low
As slide
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

8. Progress/achievement tests
Purposes
to measure effectiveness of a phase of learning
to allow access by learner to next phase of learning
Characteristics
items chosen to closely reflect content and methodology of preceding training phase
scores will often be interpreted with regard to average or pre-established norm
may refer to intermediate levels (3,5; 4,5;…)
stakes: low to medium
As slide
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

9. Paris, France, 6-7 September 2006
Proficiency tests
Purposes
to establish the competence of candidate to exercise language skills in operational conditions
Characteristics
items chosen to resemble real-world tasks
overall scores are holistic: YES/NO
stakes: very high
As slide
While diagnostic and progress/achievement tests for aviation personnel are devised and administered internally within institutions - and properly so - , the ICAO level 4 requirements, and their associated high-stakes, will lead to a need for some external reassurance in the matters of reliability and validity of entry/placement and proficiency testing. The remainder of this paper will focus exclusively on these.
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

10. Testing needs of aviation
Ab initio populations
entry
selection
local education vs aviation training policies
progress/achievement
intermediate training objectives
continuation/curtailment
initial proficiency
licensing
Qualified populations
placement/diagnostic
benchmarking populations
individual training needs
acceptance by takers
(progress/achievement)
degree and speed
proficiency renewal
recurrent
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

11. Paris, France, 6-7 September 2006
Language test quality
qualities of tests in general
validity
face validity
reliability
washback effect
practicality
conflicting qualities
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

12. Test design specifications (ICAO)
“As of 5 March 2008, aeroplane and helicopter pilots, air traffic controllers and aeronautical station operators shall demonstrate the ability to speak and understand the language used for radiotelephony communications to the level specified in the language proficiency requirements in the Appendix.” (Annex 1, )
formal evaluation
“An individual must demonstrate proficiency at level 4 in all categories in order to receive a level 4 score.” (Doc 9835, Manual on the Implementation of ICAO LPRs, 2.8.4)
profile evaluation
The international requirement for civil pilots and controllers to demonstrate, as of March 2008, a minimum level of proficiency in the languages used for aviation radiotelephony communications - and to do so repeatedly throughout their careers - has established an unprecedented need world-wide for language testing procedures and systems that can elicit job-specific (radiotelephony communications) language performances which can be related to all the categories defined in the ICAO language proficiency rating scale (Appendix to Annex 1, Personnel Licensing).
(Quote Annex 1 and Manual from slide)
The current situation with regard to fulfilling this need internationally is widely recognised as being unsatisfactory, resulting in local searches by those invested with this responsibility for solutions ranging from “off the peg” general tests to development of special purpose tests. Meanwhile, the signs are that the language testing industry itself is gearing up to be able provide solutions, and, in the absence of any independent international oversight, the aviation community must prepare itself to be able to assess the suitability of what is on offer.
In the words of Ms Elizabeth Mathews, linguistic consultant to the ICAO PRICE Study Group “ ‘any test at all’ is NOT better than ‘no test’ ”.
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

13. Paris, France, 6-7 September 2006
Test design teams
Tasks
develop test specifications
construct test
evaluate test
ensure ongoing test maintenance
Qualifications
operational expertise
language test development expertise
linguistic expertise
item writers
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

14. Test administration teams
Tasks
ensure advance access to test samples
schedule test
manage pre-test preparations
manage test
Qualifications
knowledge of administration guidelines
interlocutor expertise
operational
linguistic
rater expertise (min 2 raters)
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

15. Rater/interlocutor qualifications
3 basic qualifications
language level
aviation background + familiarity with ICAO LPRs
principles of language proficiency and language testing
suitable candidates
aviation personnel
language trainers
L1 background
native speakers(NS)/non-native speakers (NNS)
risk of familiarity with a given form of NNS spoken English
A final consideration, arising from the limitations of technology, is the profile required for test interlocutors and raters. A high language level is clearly a requirement, although this need not be level 6 in the case of interlocutors. Background knowledge of aviation is also essential, and this will need to be based on some practical experience in the case of interlocutors conducting interactive scenarios of radiotelephony communications. Thirdly, interlocutors and raters must undergo specific training in the principles of language proficiency and language testing. Testing personnel meeting these requirements can be drawn from both the language training and the ATC/piloting communities. They may be native or non-native speakers. However, since perceptions of intelligibility are more “tolerant” on the part of people who are of the same first language (L1) background - or simply extremely familiar with the speech of a given L1 background, exchanges of testing personnel between different states should be encouraged as an antidote.
Conclusion
Without claiming to be exhaustive, I have attempted in this presentation to outline the particular requirements for the development of appropriate testing procedures to meet ICAO language proficiency requirements at all stages of the ab initio and recurrent training of pilots and controllers. I would like to conclude with an invitation to all stake-holders to be critical and methodical in their actions with a view to acquiring or developing their own testing procedures.
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

16. Institutional context
need for independent international oversight to ensure:
validity of testing procedures used
comparability of testing procedures and outcomes
testing outcomes are linked with personnel licensing
testing may not be in the sole hands of the ELT community
partnerships with CAA, service provider, airline,...
system cannot tolerate a high failure rate
waste of expensive training investment
staffing levels must ensure continuity of service
prior knowledge of test procedures for positive “washback” on training
trainee motivation and attitude, reassurance of candidates
development of appropriate training systems
BUT unsuitability of “cramming”, test item banks
A number of institutional constraints conspire to reinforce this message.
As slide
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

17. A common test for ATC and pilots?
pilots and controllers are partners in R/T communication
but they are set apart by:
different interactive roles
complementary passive/active competencies
opportunity to use other job-related language uses to extend speech sample:
controller: telephone co-ordinations, report to supervisor, ...
pilot: pre-flight, intra-cockpit, cabin announcements, ground staff, ….
a possible solution?
common core test
job-specific components
It would appear, on the basis that pilots and controllers participate in the same communicative discourse, that great amounts of time and money could be saved if the same test could be used for both populations targeted by ICAO language proficiency requirements. For language testing purposes though, just as for evaluation of other respective professional competencies, this would pose a number of problems.
Pilots and controllers play different, albeit complementary, interactive roles in radiotelephony communications. If a test candidate is to demonstrate interactive ability in a job-related scenario, this role will govern the design of the testing format, which will have to be different for each population.
Additionally, the passive and productive language repertoires of pilots and controllers are complementary. Stated simply, this means that what controllers need to be able to understand, pilots need to be able to say, and that what pilots need to be able to understand, controllers need to be able to say. This distinction results in differing testing objectives for each population in terms of the lexical and functional content of listening and speaking tasks.
Finally, our experience has shown that, given the restrictions on varied language content in radiotelephony communications, testing formats can usefully call on other job-related language uses as a pretext for extending the speech sample obtained from the candidate. For example, in Paper 3 of the PELA test, candidates are required to report a radiotelephony interaction as if they were debriefing their supervisor.
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

18. Need for testing services
time/space constraints
dispersed locations (recurrent)
irregular schedules and limited availability of test-takers (recurrent)
availability of technologies
test/re-test
test components
single event, integrative?
different item designs to test separate skills?
standardisation of results
extensive trialling
comparable conditions of test administration
examiner training/auditing
interlocutors
raters
test security
secure storage and transfer of test materials
multiple parallel versions
Also features of test delivery must also be taken into account.
As slide
“What surely has to be demonstrated is the ability to perform at a satisfactory level on all the parameters at the same time within an aviation context.” ICAEA_WORLD discussion group, 19th April 2004.
“It seems to me that we may find ourselves having to design different parts of any test.. To address the different issues. That may mean one part for pronunciation, another for structure, yet another for vocabulary, fluency, and comprehension, and so on. I can’t see that any one form of test design will be able to accomplish all those tasks.” ICAEA_WORLD discussion group, 15th April 2004.
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

19. Appropriate technologies
voice-only
telephone, 2-way radio, language laboratories, training simulators
input delivery
analogue players, computer screen (multimedia)
performance storage and access
analogue recorders, computer sound files (rapid access)
speech recognition
standardise perception of intelligibility, save rating costs
BUT needs to filter all possible speech variations
no real interaction
« interactions » need to be human-human
initiate and maintain exchanges
deal with misunderstandings
Consideration of means of test delivery will certainly lead to the evaluation of appropriate technologies.
The voice-only character of radiotelephony communications means that telephones, 2-way radios, language laboratories or training simulators would all be preferred environments for mediating interactions.
Delivery of test input (text, sound, or graphics) is clearly simplified by computerisation. Similarly, candidate spoken performances are more rapidly accessed for rating if they are stored as digital sound files rather than recorded on analogue tape.
Finally, the possibility for speech recognition technologies to automate the rating of spoken performances is a tantalising prospect, but their acceptability would depend on prior standardisation of the parameters of “intelligibility” and they would need to demonstrate their capacity to filter all the possible speech variations for the region in which such testing technology is intended to be used.
In any case, the choice of delivery technologies will usually have to be aligned to the lowest level of equipment available to the target users. Additionally, current proven computer technology is unable to simulate the natural features of interaction. It is difficult to see how an ICAO-compliant test could exclude a human-human encounter allowing opportunities to evaluate the ability to initiate and maintain exchanges and to deal with misunderstandings. It is significant that all specific tests mentioned so far have maintained this component.
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

20. Paris, France, 6-7 September 2006
Standardisation
operationalisation of the rating scale
speech samples
« can do » statements
rater training
initial
refresher
rating protocols
paired rating
multiple rating
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

21. Paris, France, 6-7 September 2006
Assessment of level 6
NOT a safety issue, but raises concerns about:
resources ( no re-test)
NS/NNS distinctions
Speakers may be:
mother-tongue
non-mother tongue
intranational use
non-intranational use
Candidacy on basis of biographical criteria
citizenship, educational background, residency, ...
Protocols for assessment
admissible evidence
assessor qualification
appropriate documentation
Subsequent checking and oversight must address:
adherence to standard ICAO phraseologies
ICAO LPRs for
intelligibility
appropriateness
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

22. Paris, France, 6-7 September 2006
Informal evaluations
Within the initial or continuing training process, language trainers can identify 3 categories of trainee:
well-below level 4
approximately at level 4 (confirm by formal evaluation)
comfortably above above level 4
…thus enabling informed decisions to be made on
possible access to professional functions
further training required
benchmarking of populations
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

23. Formal tests for ICAO level 4
Placement (entry)
must relate to ALL 6 skills in ICAO scale
must measure distances above and below level 4
may have diagnostic capability
tasks/topics/input materials focus on general or job-related skills
development requires some statistical validation
Proficiency (exit)
must relate to ALL 6 skills in ICAO scale
must characterise test-takers as BELOW or AT levels 4, 5 or 6
tasks/topics/input material must reflect radiotelephony language competencies (including standardized phraseology)
development requires strong statistical validation (high stakes)
As slide
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

24. Detailed specifications (proficiency)
provide a representative range of intelligible international accents as input for comprehension;
provide a professionally relevant format for candidates to display comprehension;
elicit an adequate continuous speech sample to test fluency/pronunciation;
provide a voice-only setting for “diadic” (2-person) interactions;
provide examples of routine and unexpected events in a work-related context;
allow the candidate to use basic grammatical structures creatively;
allow the candidate to demonstrate ability to paraphrase;
allow the candidate to change between rehearsed/formulaic speech and spontaneous interaction;
simulate unexpected events to create opportunities for misunderstanding.
EUROCONTROL/ENAC preliminary feasibility study 2004
More specifically, a recent Eurocontrol/ENAC feasibility study has specified that proficiency testing will need to:
as slide
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

25. Some existing specific tests
Controllers
ab initio
entry: EPT/FEAST(Eurocontrol)
progress: APRO (ENAC, F)
proficiency: PELA (Eurocontrol)
recurrent
placement: TNP (DGAC, F)
proficiency: ELPAC (Eurocontrol)
under development for 2007
Pilots
proficiency: FCL (DGAC, F)
Other tests
TEA (Mayflower, UK)
TELLCAP (Russia)
TOEFA (Peru),
TELPA (IAES, Korea)
TELAP (CA Flight University,China)
RELTA (RMIT, Australia) ….
Tests in current use, or under consideration for use, in aviation for entry/placement or proficiency purposes include the Oral Proficiency Interview (OPI) in the USA, the Test in the Proficiency in English Language for Air Traffic Controllers (PELA) in Europe and the Test in English for Aviation (TOEFA) in Peru. All of these tests will be the subject of separate presentations during this symposium.
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006

26. Paris, France, 6-7 September 2006
TNP (DGAC, France)
developed by national language and subject-matter experts (controllers and ATC management) as a placement/diagnostic tool
3 papers cover professional language content:
a written multiple choice test of language knowledge
a tape-mediated written test of listening comprehension (RT messages)
an individual oral interview using paper-mediated graphic and text input
expresses results in terms of the ICAO level chart
used since 1999 with qualified air traffic controllers:
benchmarking to establish national training needs
formulating periodic individualised language development and maintenance programmes in order to meet national qualification renewal requirements
NOT a proficiency test
lacks robust validation
As slide
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006