1. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Research & Technology Europe Usability Study of Lufthansa’s Performance Evaluation System Hans-Juergen Hoermann, PhD Boeing Research & Technology Europe RAeS, April 30, 2004

2. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Overview Introduction Design of the usability study Quantitative results concerning the practicability and training benefits of the new Proficiency Reporting Form (PPR) Summary of written user comments Recommendations for a train-the-trainer course Conclusions

3. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Coinciding Inputs From Different Initiatives LH Flight safety survey Update of performance criteria for LH flight-crew members New concepts for proficiency reporting forms JAA requirements (NOTECHS/JARTEL)

4. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Special Features of the New PPR Based on the LH Competence System with the three areas: technical, procedural, and interpersonal Frequency scale for quantitative observations Four-point grading scale for quality of performance Competence codes for feedback and further training recommendations Training goals explicitly included

5. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Usability Study User Input & Feedback Form: –15 yes/no items with plenty of room for free comments, concerning... Structure, layout and contents of the new Evaluation System Practicability and use of different scales Clearness and adequacy of results Value of Comp. Codes for debriefing and training

6. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Usability Study Design and procedure: –N = 45 TCs from 3 LH fleets –One-day introductory course –Telephone hotline –4 months trial period with three forms –N = 20 returned with full data –Based on training events with 27 captain candidates –Follow-up crosscheck of the revised form within another LH subsidiary (N = 13 TCs)

7. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Usability Study User feedback in form of: –76 (98) specific comments –13 (18) general comments

8. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Structure, Layout, and Contents Sufficiently clear? 85 - 90% Criteria complete? 80% Comments: no gaps, suggestions for reducing items, clear structure significantly reduces efforts to fill out PPR clear structure significantly reduces efforts to fill out PPR

9. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Practicability and Use of Scales Famil- course OK? 95% Freq count practicable? Req. time justifiable? 94% 85%

10. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Practicability and Use of Scales Comments: The clear structure makes it less time-consuming than before, Difficult to handle the papers especially when also active as CM Quantity is not always equivalent with quality

11. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Clearness and Adequacy of Results Unambiguous? Definite and adequate? Different to previous? 15% 90% 65% Comments: Scale anchor „Always“ is too strict. Minor errors would lead to a downgrading of otherwise good performance 75%

12. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Use of Comp. Codes for Training & Debrief Useful for training? Facilitate debrief? 90% 50-60% Comments: Training goals more transparent for trainer and trainee. Weaknesses can be identified earlier. CCs helpful for finding adequate training recommendations.

13. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Distribution of Comp. Codes in the Study

14. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 General Comments Observation is determined by more specific criteria instead of the overall impression Debriefing could become a little longer but also more objective and supportive All in all, a little more effort for the benefit of a more adequate evaluation and feedback of the candidate‘s performance More objective because of more observation items (CLH) Because of the versatility of the observation items the form is good for a fair and more standardised evaluation (CLH)

15. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Conclusions User feedback has provided valuable insights for further revisions of the PPR and contents of the familiarization course Acceptance of deviation from absolute perfection (100%  99+%) High degree of acceptance for the integration of interpersonal skills, training should precede checking Full transparency of training goals offers better opportunity for self-evaluation and preparation of trainees A long-term evaluation study is scheduled to detect trends and tailor future training methods

16. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Training Course for NOTECHS Examiners In addition to the required training of instructors and examiners it is recommended to have at minimum A two-day training session with a trainee-trainer ratio not exceeding 12:1 Timed in vicinity to the first training mission of the new instructor/examiner A thorough briefing on the respective non-technical skills concept including categories, elements, and desired/undesired behaviour anchors A thorough briefing on the rating scale, the evaluation criteria (cut-offs), and the forms A practical training session with specific video examples for each of the evaluation categories. This training includes feedback and discussion. Guidance material and practical training sessions of how to conduct a constructive debriefing on non-technical skills. Both in examiner and examinee roles. Regular refresher for re-calibration Written material, including definitions and criteria should be available and agreed throughout the company

17. BR&TE H.-J. Hoermann, NOTECHS & LH-IPCs, RAeS, Apr. 2004 Lufthansa‘s Basic Competence System Finally, we arrived at the summary…

18. What is left? BR&TE Principles of Behaviour Assessment in NOTECHS/JARTEL Development of Lufthansa‘s New Behaviour Marker System Crosslinking NOTECHS to LH Interpersonal Competences The Implementation Case: Lufthansa‘s Evaluation Method Usability Study Summary