1. REGULATORY ISSUES ON TESTING
WORKSHOP
LANGUAGE PROFICIENCY REQUIREMENTS IMPLEMENTATION
REGULATORY ISSUES ON TESTING
Eleonora Italia
Enac Personnel Licensing Director
3 - 5 March 2010
Rome - Italy

2. TO THE LEVEL SPECIFIED IN THE LANGUAGE PROFICIENCY REQUIREMENTS IN:
ICAO ANNEX 1
1.2.9 LANGUAGE PROFICIENCY
Refers to pilots and flight navigators, ATCOs and aeronautical station operators
“Shall demonstrate the ability to speak and understand the language used for radiotelephony communications”
TO THE LEVEL SPECIFIED IN THE LANGUAGE PROFICIENCY REQUIREMENTS IN:
- ICAO ANNEX 1 APPENDIX 1
- ICAO doc. 9835
- ICAO Circular 318-AN/180

3. in a manner acceptable to the licensing authority
ICAO ANNEX 1 APPENDIX 1
To meet the language proficiency requirements contained in section an applicant shall demonstrate
compliance with
the holistic descriptors at section 2 and
the ICAO Operational level of the ICAO rating scale
As a whole, the scale serves as benchmarks for training and testing, and in assisting candidates to attain the ICAO operational level
in a manner acceptable to the licensing authority

4. ACCEPTABLE TO THE AUTHORITY
WHICH ARE THE KEY CONCEPTS TO FOCUS ON FOR CONSIDERING THE DEMONSTRATION OF PROFICIENCY
ACCEPTABLE ?
THE STATE AVIATION AUTHORITIES MAINTAIN OVERSIGHT RESPONSIBILITIES

5. AVIATION LANGUAGE TESTING
DOC CHAPTER 6
PURPOSES OF TESTING:
To fulfil ICAO provisions requiring that pilots and controllers demonstrate language proficiency sufficient for SAFE and EFFICIENT radiotelephony communications
To provide BENCHMARKS by which the EFFECTIVENESS of language training and learning may be evaluated
LANGUAGE TESTING IN AVIATION HAS HIGH STAKES BECAUSE CAREERS AND SAFETY ARE AT STAKE
Proficiency test ⇒ evaluate OVERALL ability against a set of criteria
TESTS PURPOSE:

6. AVIATION LANGUAGE TESTING
1. Provides RELIABLE and VALID results
2. Are VALID – RELIABLE – EFFECTIVE – APPROPRIATE- PRACTICABLE
VALID: indicates the degree to which a test measures what is supposed to measure
RELIABLE: refers to the stability of the test, in terms of standard of evaluation and rating
PRACTICABLE: signifies the balance between resources required and resources available

7. HIGH QUALITY TESTING WILL ENCOURAGE HIGH QUALITY TRAINING !!!
TEST WASHBACK
ICAO CIRCULAR 318-AN/180
influence of testing on teaching and learning
POSITIVE WASHBACK ⇒ the test encourages learners and trainers to focus on PROFICIENT-BUILDING LANGUAGE learning activities, to improve proficiency level,
and NOT
to focus on the easier way to get a higher score
HIGH QUALITY TESTING WILL ENCOURAGE HIGH QUALITY TRAINING !!!

8. TESTING SERVICES RATHER THAN “A TEST” ARE REQUIRED
⇒ A TESTING SERVICE THAT CAN CONTINUALLY PROVIDE NEW TEST ITEMS
⇒ THAT CAN TAKE INTO CONSIDERATION DIFFERENT SCENARIOUS (AB INITIO PILOS AND ATCOs, PPL, PROFESSIONAL PILOTS AND ATCOs, HELICOPTER PILOTS, AEROPLANE PILOTS)
AND
⇒ A LARGE NUMBER OF RATERS AND ADMINISTRATIVE SERVICES
IS REQUIRED (6.6.1)
⇒ A TESTING SERVICE THAT CAN CONTINUALLY ASSURE SECURITY AND QUALITY CONTROL

9. BEST PRACTICE FOR LANGUAGE TESTING IN AVIATION (6.5.4)
TEST PREPARED WITH THE INPUT OF PROFESSIONAL TEST DEVELOPERS
WRITING TEST SPECIFICATIONS
DECIDING TEST METHOD AND CONTENT
DEVELOPING TEST ITEMS TRIALLING TEST ITEMS
ANALYSING THE RESULTS
REVISING TEST ITEMS
RE-TRIALLING THE TEST AND TEST ITEMS
VALIDATING THE TEST
ESTABLISHING A RATING PROCEDURE
ESTABLISHING A RATER TRAINING PROCESS AND A QUALITY CONTROL PROCESS
ESTABLISHING RECORD KEEPING ADMINISTRATIVE FUNCTIONS

10. BEST PRACTICE FOR LANGUAGE TESTING IN AVIATION
RATERS AT LEAST TWO
- TRAINED, QUALIFIED, EXPERIENCED
- ABLE TO IDENTIFY DEFICIENCIES IN PERFORMANCE
ABLE TO GUIDE CANDIDATES TOWARDS LANGUAGE LEARNING ACTIVITIES
INSPIRE TRUST AND GAIN THE CONFIDENCE OF BOTH THE COMMUNITY AND CANDIDATES
AGREE UPON STANDARDIZED PROCEDURES
COMMIT THE BEST PRACTICE (CODE OF ETHICS) ↓
THEN, THE RATING SCALE IS USED TO MAKE INFORMED JUDGEMENTS ABOUT CANDIDATES

11. STANDARDIZED AND TRAINED RATERS
Initial and recurrent training documented – rater training records maintained – audit conducted and documented periodically
CAN ONLY GUARANTEE THE VALIDITY AND THE RELIABILITY OF THE TEST

12. PROFICIENCY TESTS DIRECT SEMI-DIRECT
Candidate interacts with a tester directly or with other candidates while being observed by a tester.
Requires more time and human resources
Benefits: more natural, communicative
because of the interaction there is an infinite supply of tests prompts
SEMI-DIRECT
Speech samples are elicited based on pre-recorded prompts
Inflexibility
Benefits: test takers receive the same prompts
less time (more candidates tested at the same time)
Appropriate strategy: incorporate both direct and semi direct methods in a single testing system

13. AUDIO AND VIDEO RECORDING
HOWEVER
AUDIO AND VIDEO RECORDING
OF THE TEST PERFORMANCE IS STRONGLY RECOMMENDED FOR
VERIFICATION AND RECORD-KEEPING PURPOSES
IT ALSO HELPS IN APPEALS PROCEDURES, WHICH HAVE TO BE ESTABLISHED

14. TEST ADMINISTRATION CIRCULAR 318-AN/180
TO PUBLISH A COMPLETE SAMPLE OF THE TEST
DOCUMENTED TEST RATING PROCESS AVAILABLE
DOCUMENTED INSTRUCTIONS TO TEST TAKERS, TEST ADMINISTRATION TEAMS, TEST RATERS AVAILABLE
TEST ADMINISTRATION POLICIES AND PROCEDURE AVAILABLE
DOCUMENTED APPEALS PROCEDURE ESTABLISHED

15. TEST SECURITY
ENFORCE SECURITY MEASURES TO ENSURE THE INTEGRITY OF THE TESTING PROCESS
TEST MATERIAL HELD IN CONFIDENCE AND WELL STORED, NOT PUBLISHED OR PROVIDED TO TEST TAKERS BEFORE THE TEST

16. RECORD-KEEPING EVALUATION SHEETS AND SUPPORTING DOCUMENTATION
AUDIO AND/OR VIDEO RECORDING
EVALUATION SHEETS AND SUPPORTING DOCUMENTATION
SCORE-REPORTING PROCESS

17. HOW ENAC ENFORCED LPR KEY CONCEPTS AND REQUIREMENTS
ONE TEST FOR PILOTS AND ONE TEST FOR ATCOs
standardisation and uniformity of evaluation need
BOTH DIRECT AND SEMI-DIRECT METHODS IN THE TEST
EXAMINERS AND TEST ADMINISTRATORS APPLY TO ENAC
EXAMINERS CERTIFIED AFTER SUCCESSFULLY COMPLETING 20 HOURS ON SELF STUDYING, OVERSIGHT BY DISTANCE AND TWO DAYS OF STANDARDISATION AND CERTIFICATION SESSION HELD BY THE TEST DEVELOPER AND ENAC ⇒ 46 ACTIVE EXAMINERS
TEST ADMINISTRATORS BRIEFED AND TRAINED ON TEST ADMINISTRATION, SECURITY PROCEDURES AND RECORD-KEEPING
TEST CENTRES REQUIREMENTS EVALUATED BY ENAC, CERTIFIED BY THE TEST DEVELOPER ⇒ 39 TEST CENTRES CERTIFIED
THE LIST OF THE TEST CENTRES PUBLISHED ON ENAC WEB SITE

18. TEST CENTRES AUDITED BY ENAC
TWO RATERS
RESULTS CONFIRMED AFTER THE SECOND RATE
TEST RESULT ENDORSED BY ENAC ON THE PILOT’S LICENCE:
“language proficiency : ENGLISH Level ….. EXP. ……
date of the test + 3 years (Lv. 4)
date of the test + 6 years (Lv. 5)
ITALIAN Level 6
DATA BASE WITH TEST RESULTS AVAILABLE TO ALL ENAC AIRPORT OFFICERS
IN CASE OF DEMONSTRATION OF PROFICIENCY BELOW LEVEL 4:
“restricted to VFR domestic activity, other than public transportation”

19. Any questions?
La base della slide è presente da prima che inizi l’intervento
Il click iniziale attiva benvenuti e rullo di tamburi come bang di apertura e spazio alla spiegazione: la clip art del forcon e il piè di pagina hanno lo scopo di illustrare i numeri il conectto di di pungolo PER TUTTI, verso la base e verso il top dell’ENAC, verso la periferia e dalla periferia al centro, CTN=Direzione Centrale Regolazione Tecnica; HF OPS= Progetto Human Factor per la parte Operazioni che si affianca a quello di HF in Maintenance parimenti presente nel piano). Dire che in base ad esso l’obiettivo è farne 4/5 all’anno e che il piano è stato rispettato finora. STOP!!