1. R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation Workshop Russia, St-Petersburg, May 24-26, 2011

2. R ATERS T RAINING AS A S AFETY C ONCERN Associate Member - RATER AS A DECISION MAKER - SELECTION OF PERSONNEL - AB INITIO TRAINING - RATERS CERTIFICATION - INTER-RATER RELIABILITY - INTRA-RATER RELIABILITY - RASCH ANALYSIS - RECURRENT TRAINING - EVALUATION OF RATER’S ADEQUACY ICAO LPRI Workshop, St-Petersburg, Russia, 2011

3. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY

4. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member Raters Training as a Safety Concern Aviation English test results are felt around the globe

5. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern Associate Member Inadequacy in Level Evaluation Level 3 instead of Level 4: Loss of job and/or occupation Financial losses for AO or ANSP Market losses Redistribution of air transportation services Lack of confidence for test-takers

6. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member Inadequacy in Level Evaluation Level 4 instead of Level 3: Aviation safety is compromised Degradation to Level 2 Overconfidence for test-takers Raters Training as a Safety Concern

7. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY

8. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member Raters Training as a Safety Concern ICAO recommendations on interlocutors and raters qualifications

9. Raters Training as a Safety Concern ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member ICAO recommendations on interlocutors and raters qualifications (Doc 9835) Familiarity with ICAO documentation Annex 1; Doc 9835; ICAO Rated Speech Samples CD; ICAO Circ 318

10. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern ICAO recommendations on interlocutors and raters qualifications (Doc 9835) Familiarity with ICAO documentation Operational and/or language specialist expertise Level 5 or Level 6 proficiency Familiarity with Aviation English RATERS CODE OF ETHICS Associate Member

11. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY

12. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member Raters Training as a Safety Concern Theory of language testing ICAO documents: Annex 1, Annex 10, Doc 9432, Doс 9835, Circ 318 The ICAO Proficiency Rating Scale and holistic descriptors Correlating speech samples with the ICAO LPR Scale

13. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member Raters Training as a Safety Concern Raters training course should be aimed at experienced non-native and native English language teaching professionals and subject-matter experts who are currently teaching Aviation English and who want to be able to rate according to the ICAO Language Proficiency Scale

14. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY

15. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member Raters Training as a Safety Concern Certification by national Civil Aviation Authority or by international aviation organization is mandatory

16. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member Raters Training as a Safety Concern Is geographical principle necessary?

17. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern Associate Member Are quotas necessary?

18. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY

19. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member Raters Training as a Safety Concern Different raters but same results

20. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY

21. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern Associate Member Different times but same results

22. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY

23. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member Raters Training as a Safety Concern Evidence of the validity of the responses

24. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member Raters Training as a Safety Concern Evidence of the validity of the responses Correctional refresher course

25. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern Associate Member Evidence of the validity of the responses Correctional refresher course Administrative measures to marginal raters

26. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY

27. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member Raters Training as a Safety Concern Annual refresher course for raters

28. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member Raters Training as a Safety Concern Annual refresher course for raters New materials

29. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern Associate Member Annual refresher course for raters New materials Practical training

30. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY

31. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member Raters Training as a Safety Concern Rater is not a lifetime profession RATER TRAINING IS A SAFETY CONCERN

32. ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Associate Member Raters Training as a Safety Concern Thank you