Qantas Flying Operations Audit September 2005

LOSA Overview LOSA was carried out: May - July 2002 Across 29 Qantas domestic and international ports 263 sectors B737, B767, B744 LOSA facilitated by the University Of Texas with James conducting the training of the Auditors.

LOSA Process LOSA Report: Undesired Aircraft State Australian law - disclosure Exec Report to QF Safety Board members Safety Board gave Chief Pilot three months to present plan for corrective actions, to be implemented within six months In our LOSA report we had 153 UAS’ - given that this represents increased risk then it cannot be hidden

Flight Crew Response All Flight Crew called in for meeting with Chief Pilot Putting final results directly to pilots meant they interpreted the poor result as being their fault. Lead to adverse reaction to LOSA. Flight Crew thought it was not a practical reflection of Qantas operations That meant somewhere between 50 and 60 meetings. The crews did not understand threat error methodology. They were asked to do what they normally do but then told that their error rate was high Could not see many of the UASs had any real risk associated with them eg: doing 265kts below 10,000ft may well be a handling error but could not possibly be conceived as “the closest thing to an incident or accident without actually having one”. The same applied to being 150ft off an altitude at 35,000ft - pre RVSM.

Flight Crew Response They didn’t just take it out on Flight Ops management… ……….. suddenly posters appeared

Flight Operations Actions re LOSA Chief Pilot strengthened Audit department Audit no longer a ‘monitoring’ department but a continual LOSA process Benchmarking other airlines QF shown to be insular Permission from University of Texas

Head of Flight Ops and Chief Pilot Audit Department Structure GM Flying Ops Audit (Captain) full time Two full time ground staff 25 Auditors (Captains) used as required Head of Flight Ops and Chief Pilot GM Flying Ops Audit Flying Ops Audit Mgr Flying Ops Audit Co-ordinator 25 Pilot Auditors GM - flies 1/3 of a line. Mgr - ground staff there 5 days per week. Does all the administration work. Receives reports, prints and organises them for the wash, researches issues, compiles statistics, formats final reports, follows up with Fleet Managers the agreed actions to the findings. Co-ordinator - does all rostering (security) separate to crew scheduling. Does other tasks within Flight Operations. Auditors work as required for Audit

Program Qantas mainline B744, B747, B767, B737, A330. Each fleet every two years Qantas Group Airlines - audits conducted as requested Australian Airlines B767 Jetstar B717 JetConnect B737 Standard fleet audit is 50 sectors. B737 65, B747 35 - decided by Safety Department New Approach Procedures - audited during simulator training and after line introduction. Checklist Audit. Queenstown audit will be narrative only recording threats, errors and UASs in relation to the actual Queenstown operation. New procedures New Approach Procedures, B737 RNP ZQN

Examples The following examples show how the University of Texas’ Threat/Error methodology translates into tangible outcomes for Flight Crew Enable Flight Crew to see the practical application of Threat/Error methodology Conversation with James he has commented that he is keen to see what are some of the end results of threat / error auditing Without question the tangible outcomes are the biggest “selling point” for auditing to Flight Crew. It immediately gives them ownership into the SOPs of the airline. In terms of the report, there are only two areas they are interested in: Issues arising, and Contribution to Crew Effectiveness. (later)

4. Initial departure climb altitude is 3000’ or as directed by ATC

Differences Perfect flight - optimum flight Modified to make it less theoretical and more practical eg. LAX - busy airport All audits must have a base line so there is no criticism intended of the base line used in LOSA. It is simply that we could not sell that to crews. It doesn’t exist. Modified behaviour must be from standard procedures. Recall great argument during LOSA training re cloud off the end of the runway. In the end James ruled that what mattered most was consistency with threat / error methodology across participating airlines, therefore it was a threat. We have done the same but changed the base line. Use Bob Dodd to referee.

We only use Captains as Auditors Acceptance by crew Decision errors Auditor opinions and comments One of the biggest complaints was also that we had used F/Os to carry out observations during LOSA. How could an F/O, whose decision making skills have not yet been fully tested in the cauldron of command training possibly decide on decision errors and, in some cases, intentional non compliance errors. Whilst they may well have been able to it was “politcally unacceptable”. See next slide Four qualities for Auditors are: Must have sound flying skills recognised by the pilot body. Must not have an “axe to grind” with the Company, Pilot’s Union or anyone else. Must be someone who other pilots might go to seeking advice regarding their own operation. Must be able to write it as they see it, knowing that sections of their narratives will go out to crews for perusal. Auditor opinions and comments have led to the uncovering of many issues eg: arrival into Perth

Decision Errors Changed rules for recording decision errors Must be a consequence to the decision Or; Decision is changed after external input We started out not having decision errors. Crews questioned how could the Auditor possibly know what was “unsaid” in the mind of the pilot being observed. Airmanship is a “grey” area. However there were clearly decision errors made when the above two requirements were met. Since introducing those two rules we have had acceptance by crews.

UAS / NSAPs Changed title of Undesired Aircraft State to Non Standard Aircraft Profile (NSAP) After LOSA, UAS was irrevocably defined as “The closest an airline gets to having an accident without actually having one” NSAP is understood by Flight Crew to mean ‘where the aircraft is placed in a state of increased risk’

Issues

Contribution to Crew Effectiveness In the LOSA report, crews did not understand the ratings given to them in terms of Contribution to Crew Effectiveness Completely different set of ratings Probably the first thing that crews look for in the report so we had to have a clear definition of what they meant. We have slowly worked on them in word terms and deleted any reference to the word “Regulatory” as that identifies it with a “check” Use them at EP days as we believe it is one of the best ways to allow a crewmember to see how they compare to others.

Well Above Company Standard Awarded when a crewmember’s “contribution to crew effectiveness” could be filmed and used as a training example to other pilots. It would have been a performance where strict adherence to Standard Operating Procedures, excellent CRM and sound manipulative (PF) or support (PNF) skills were observed. All facets of the performance must equally be to the high “training video” standard otherwise the “Well Above Company Standard” rating cannot be given. Company Standard Awarded when a crewmember’s “contribution to crew effectiveness” is highly professional and in line with what is expected by the Chief Pilot. It would not be without error, however, the Captain must demonstrate sound error management skills. Each crewmember must be able to recognise threats to the flight deck operation and the Captain must be able to manage such threats effectively. Point out that the rating must be achieved in all areas before it can be given. Also point out that we have some way to go with crewmember input to the words very important.

Below Company Standard Awarded when the Auditor is of the opinion that, given the opportunity to conduct the observed sector once again, the crewmember would have increased their contribution to crew effectiveness in order to achieve a more professional standard of flight deck operation. The performance may have included examples of unintentional deviation from SOPs, threats and/or errors that were not recognised or managed to Company expectations, or manipulative support skills that were below a professional aviator’s ability on the day. All crewmembers have had “days like these” and this rating does not mean that the crewmember was not trying to give of his/her best. A professional crewmember would have reviewed the sector with the intention of learning from the experience and improving their contribution to the crew effectiveness accordingly.

Well Below Company Standard Awarded when a crewmember’s contribution to crew effectiveness falls well short of what the Chief Pilot would reasonably expect of a professional crewmember. There is never any doubt in the mind of the Auditor, when awarding this rating, that the observed performance on the day was unacceptable. Either a complete lack of CRM, poor sector management, intentional violation of SOPs, or manipulative or support skills that did not meet the challenges of the day would have been observed during the crewmember’s performance.

Costs for Audit Program Audit and narrative writing - 50 sectors B767 $80 000 Wash process (4 weeks, average 3 sectors per day) $50 000 Report writing (2 weeks) $10 000 TOTAL $140 000 Cost of Audit per sector $2 800 Audit meeting (quarterly) $25 000

Where are we now in terms of Threat/Error methodology? Second B767 Audit - crew response was no different from the first audit De-identification Cabin Auditing Quite frankly the more we use threat / error methodology auditing the more we appreciate it’s value. It tells you not only what happened, but more importantly it tells you why it happened. Like ANZ we want our Regulator to see the benefits of this, and head down the path of getting rid of route checks, which are an expensive waste of time and money for zero tangible benefit. The money would be far better spent auditing, especially if we could get outside Auditors from other airlines to have a look at us every now and again eg: ANZ in our area. (Security drawbacks) As we get better at it in terms of Auditor skills the more we get out of it. For example, the error rate on the B767 audit that we just completed will be double the first one. You can put that down almost exclusively to the fact that Auditors are far better at our powers of observation and ability to take notes that can be transcribed into the report. Go to screen. We think it to be the most significant step forward that we have been able to take in terms of flight deck safety and congratulate UT on both its development and freedom of use.