1. Qantas Flying Operations Audit
September 2005

2. LOSA Overview LOSA was carried out: May - July 2002
Across 29 Qantas domestic and international ports
263 sectors
B737, B767, B744
LOSA facilitated by the University Of Texas with James conducting the training of the Auditors.

3. LOSA Process LOSA Report: Undesired Aircraft State
Australian law - disclosure
Exec Report to QF Safety Board members
Safety Board gave Chief Pilot three months to present plan for corrective actions, to be implemented within six months
In our LOSA report we had 153 UAS’ - given that this represents increased risk then it cannot be hidden

4. Flight Crew Response
All Flight Crew called in for meeting with Chief Pilot
Putting final results directly to pilots meant they interpreted the poor result as being their fault. Lead to adverse reaction to LOSA.
Flight Crew thought it was not a practical reflection of Qantas operations
That meant somewhere between 50 and 60 meetings.
The crews did not understand threat error methodology.
They were asked to do what they normally do but then told that their error rate was high
Could not see many of the UASs had any real risk associated with them eg: doing 265kts below 10,000ft may well be a handling error but could not possibly be conceived as “the closest thing to an incident or accident without actually having one”. The same applied to being 150ft off an altitude at 35,000ft - pre RVSM.

5. Flight Crew Response
They didn’t just take it out on Flight Ops management…
……….. suddenly posters appeared

7. Flight Operations Actions re LOSA
Chief Pilot strengthened Audit department
Audit no longer a ‘monitoring’ department but a continual LOSA process
Benchmarking other airlines
QF shown to be insular
Permission from University of Texas

8. Head of Flight Ops and Chief Pilot
Audit Department Structure
GM Flying Ops Audit (Captain) full time
Two full time ground staff
25 Auditors (Captains) used as required
Head of Flight Ops and Chief Pilot
GM Flying Ops Audit
Flying Ops Audit Mgr
Flying Ops Audit
Co-ordinator
25 Pilot Auditors
GM - flies 1/3 of a line.
Mgr - ground staff there 5 days per week. Does all the administration work. Receives reports, prints and organises them for the wash, researches issues, compiles statistics, formats final reports, follows up with Fleet Managers the agreed actions to the findings.
Co-ordinator - does all rostering (security) separate to crew scheduling. Does other tasks within Flight Operations.
Auditors work as required for Audit

9. Program Qantas mainline
B744, B747, B767, B737, A330. Each fleet every two years
Qantas Group Airlines - audits conducted as requested
Australian Airlines B767
Jetstar B717
JetConnect B737
Standard fleet audit is 50 sectors. B737 65, B decided by Safety Department
New Approach Procedures - audited during simulator training and after line introduction. Checklist Audit.
Queenstown audit will be narrative only recording threats, errors and UASs in relation to the actual Queenstown operation.
New procedures
New Approach Procedures, B737 RNP ZQN

10. Examples
The following examples show how the University of Texas’ Threat/Error methodology translates into tangible outcomes for Flight Crew
Enable Flight Crew to see the practical application of Threat/Error methodology
Conversation with James he has commented that he is keen to see what are some of the end results of threat / error auditing
Without question the tangible outcomes are the biggest “selling point” for auditing to Flight Crew. It immediately gives them ownership into the SOPs of the airline.
In terms of the report, there are only two areas they are interested in:
Issues arising, and
Contribution to Crew Effectiveness. (later)

11. 4. Initial departure climb altitude is 3000’ or as directed by ATC

17. Differences Perfect flight - optimum flight
Modified to make it less theoretical and more practical
eg. LAX - busy airport
All audits must have a base line so there is no criticism intended of the base line used in LOSA. It is simply that we could not sell that to crews. It doesn’t exist.
Modified behaviour must be from standard procedures. Recall great argument during LOSA training re cloud off the end of the runway. In the end James ruled that what mattered most was consistency with threat / error methodology across participating airlines, therefore it was a threat. We have done the same but changed the base line. Use Bob Dodd to referee.

18. We only use Captains as Auditors Acceptance by crew Decision errors
Auditor opinions and comments
One of the biggest complaints was also that we had used F/Os to carry out observations during LOSA.
How could an F/O, whose decision making skills have not yet been fully tested in the cauldron of command training possibly decide on decision errors and, in some cases, intentional non compliance errors. Whilst they may well have been able to it was “politcally unacceptable”. See next slide
Four qualities for Auditors are:
Must have sound flying skills recognised by the pilot body.
Must not have an “axe to grind” with the Company, Pilot’s Union or anyone else.
Must be someone who other pilots might go to seeking advice regarding their own operation.
Must be able to write it as they see it, knowing that sections of their narratives will go out to crews for perusal.
Auditor opinions and comments have led to the uncovering of many issues eg: arrival into Perth

19. Decision Errors Changed rules for recording decision errors
Must be a consequence to the decision
Or;
Decision is changed after external input
We started out not having decision errors. Crews questioned how could the Auditor possibly know what was “unsaid” in the mind of the pilot being observed. Airmanship is a “grey” area. However there were clearly decision errors made when the above two requirements were met.
Since introducing those two rules we have had acceptance by crews.

20. UAS / NSAPs Changed title of Undesired Aircraft State to
Non Standard Aircraft Profile (NSAP)
After LOSA, UAS was irrevocably defined as “The closest an airline gets to having an accident without actually having one”
NSAP is understood by Flight Crew to mean ‘where the aircraft is placed in a state of increased risk’

21. Issues

24. Contribution to Crew Effectiveness
In the LOSA report, crews did not understand the ratings given to them in terms of Contribution to Crew Effectiveness
Completely different set of ratings
Probably the first thing that crews look for in the report so we had to have a clear definition of what they meant. We have slowly worked on them in word terms and deleted any reference to the word “Regulatory” as that identifies it with a “check”
Use them at EP days as we believe it is one of the best ways to allow a crewmember to see how they compare to others.

25. Well Above Company Standard
Awarded when a crewmember’s “contribution to crew effectiveness” could be filmed and used as a training example to other pilots. It would have been a performance where strict adherence to Standard Operating Procedures, excellent CRM and sound manipulative (PF) or support (PNF) skills were observed. All facets of the performance must equally be to the high “training video” standard otherwise the “Well Above Company Standard” rating cannot be given.
Company Standard
Awarded when a crewmember’s “contribution to crew effectiveness” is highly professional and in line with what is expected by the Chief Pilot. It would not be without error, however, the Captain must demonstrate sound error management skills. Each crewmember must be able to recognise threats to the flight deck operation and the Captain must be able to manage such threats effectively.
Point out that the rating must be achieved in all areas before it can be given.
Also point out that we have some way to go with crewmember input to the words very important.

26. Below Company Standard
Awarded when the Auditor is of the opinion that, given the opportunity to conduct the observed sector once again, the crewmember would have increased their contribution to crew effectiveness in order to achieve a more professional standard of flight deck operation. The performance may have included examples of unintentional deviation from SOPs, threats and/or errors that were not recognised or managed to Company expectations, or manipulative support skills that were below a professional aviator’s ability on the day.
All crewmembers have had “days like these” and this rating does not mean that the crewmember was not trying to give of his/her best. A professional crewmember would have reviewed the sector with the intention of learning from the experience and improving their contribution to the crew effectiveness accordingly.

27. Well Below Company Standard
Awarded when a crewmember’s contribution to crew effectiveness falls well short of what the Chief Pilot would reasonably expect of a professional crewmember. There is never any doubt in the mind of the Auditor, when awarding this rating, that the observed performance on the day was unacceptable. Either a complete lack of CRM, poor sector management, intentional violation of SOPs, or manipulative or support skills that did not meet the challenges of the day would have been observed during the crewmember’s performance.

28. Costs for Audit Program
Audit and narrative writing - 50 sectors B $80 000
Wash process (4 weeks, average 3 sectors per day) $50 000
Report writing (2 weeks) $10 000
TOTAL $
Cost of Audit per sector $2 800
Audit meeting (quarterly) $25 000

29. Where are we now in terms of Threat/Error methodology?
Second B767 Audit - crew response was no different from the first audit
De-identification
Cabin Auditing
Quite frankly the more we use threat / error methodology auditing the more we appreciate it’s value. It tells you not only what happened, but more importantly it tells you why it happened.
Like ANZ we want our Regulator to see the benefits of this, and head down the path of getting rid of route checks, which are an expensive waste of time and money for zero tangible benefit. The money would be far better spent auditing, especially if we could get outside Auditors from other airlines to have a look at us every now and again eg: ANZ in our area. (Security drawbacks)
As we get better at it in terms of Auditor skills the more we get out of it. For example, the error rate on the B767 audit that we just completed will be double the first one. You can put that down almost exclusively to the fact that Auditors are far better at our powers of observation and ability to take notes that can be transcribed into the report.
Go to screen.
We think it to be the most significant step forward that we have been able to take in terms of flight deck safety and congratulate UT on both its development and freedom of use.