2LOSA Overview LOSA was carried out: May - July 2002 Across 29 Qantas domestic and international ports263 sectorsB737, B767, B744LOSA facilitated by the University Of Texas with James conducting the training of the Auditors.
3LOSA Process LOSA Report: Undesired Aircraft State Australian law - disclosureExec Report to QF Safety Board membersSafety Board gave Chief Pilot three months to present plan for corrective actions, to be implemented within six monthsIn our LOSA report we had 153 UAS’ - given that this represents increased risk then it cannot be hidden
4Flight Crew ResponseAll Flight Crew called in for meeting with Chief PilotPutting final results directly to pilots meant they interpreted the poor result as being their fault. Lead to adverse reaction to LOSA.Flight Crew thought it was not a practical reflection of Qantas operationsThat meant somewhere between 50 and 60 meetings.The crews did not understand threat error methodology.They were asked to do what they normally do but then told that their error rate was highCould not see many of the UASs had any real risk associated with them eg: doing 265kts below 10,000ft may well be a handling error but could not possibly be conceived as “the closest thing to an incident or accident without actually having one”. The same applied to being 150ft off an altitude at 35,000ft - pre RVSM.
5Flight Crew ResponseThey didn’t just take it out on Flight Ops management………….. suddenly posters appeared
7Flight Operations Actions re LOSA Chief Pilot strengthened Audit departmentAudit no longer a ‘monitoring’ department but a continual LOSA processBenchmarking other airlinesQF shown to be insularPermission from University of Texas
8Head of Flight Ops and Chief Pilot Audit Department StructureGM Flying Ops Audit (Captain) full timeTwo full time ground staff25 Auditors (Captains) used as requiredHead of Flight Ops and Chief PilotGM Flying Ops AuditFlying Ops Audit MgrFlying Ops AuditCo-ordinator25 Pilot AuditorsGM - flies 1/3 of a line.Mgr - ground staff there 5 days per week. Does all the administration work. Receives reports, prints and organises them for the wash, researches issues, compiles statistics, formats final reports, follows up with Fleet Managers the agreed actions to the findings.Co-ordinator - does all rostering (security) separate to crew scheduling. Does other tasks within Flight Operations.Auditors work as required for Audit
9Program Qantas mainline B744, B747, B767, B737, A330. Each fleet every two yearsQantas Group Airlines - audits conducted as requestedAustralian Airlines B767Jetstar B717JetConnect B737Standard fleet audit is 50 sectors. B737 65, B decided by Safety DepartmentNew Approach Procedures - audited during simulator training and after line introduction. Checklist Audit.Queenstown audit will be narrative only recording threats, errors and UASs in relation to the actual Queenstown operation.New proceduresNew Approach Procedures, B737 RNP ZQN
10ExamplesThe following examples show how the University of Texas’ Threat/Error methodology translates into tangible outcomes for Flight CrewEnable Flight Crew to see the practical application of Threat/Error methodologyConversation with James he has commented that he is keen to see what are some of the end results of threat / error auditingWithout question the tangible outcomes are the biggest “selling point” for auditing to Flight Crew. It immediately gives them ownership into the SOPs of the airline.In terms of the report, there are only two areas they are interested in:Issues arising, andContribution to Crew Effectiveness. (later)
114. Initial departure climb altitude is 3000’ or as directed by ATC
17Differences Perfect flight - optimum flight Modified to make it less theoretical and more practicaleg. LAX - busy airportAll audits must have a base line so there is no criticism intended of the base line used in LOSA. It is simply that we could not sell that to crews. It doesn’t exist.Modified behaviour must be from standard procedures. Recall great argument during LOSA training re cloud off the end of the runway. In the end James ruled that what mattered most was consistency with threat / error methodology across participating airlines, therefore it was a threat. We have done the same but changed the base line. Use Bob Dodd to referee.
18We only use Captains as Auditors Acceptance by crew Decision errors Auditor opinions and commentsOne of the biggest complaints was also that we had used F/Os to carry out observations during LOSA.How could an F/O, whose decision making skills have not yet been fully tested in the cauldron of command training possibly decide on decision errors and, in some cases, intentional non compliance errors. Whilst they may well have been able to it was “politcally unacceptable”. See next slideFour qualities for Auditors are:Must have sound flying skills recognised by the pilot body.Must not have an “axe to grind” with the Company, Pilot’s Union or anyone else.Must be someone who other pilots might go to seeking advice regarding their own operation.Must be able to write it as they see it, knowing that sections of their narratives will go out to crews for perusal.Auditor opinions and comments have led to the uncovering of many issues eg: arrival into Perth
19Decision Errors Changed rules for recording decision errors Must be a consequence to the decisionOr;Decision is changed after external inputWe started out not having decision errors. Crews questioned how could the Auditor possibly know what was “unsaid” in the mind of the pilot being observed. Airmanship is a “grey” area. However there were clearly decision errors made when the above two requirements were met.Since introducing those two rules we have had acceptance by crews.
20UAS / NSAPs Changed title of Undesired Aircraft State to Non Standard Aircraft Profile (NSAP)After LOSA, UAS was irrevocably defined as “The closest an airline gets to having an accident without actually having one”NSAP is understood by Flight Crew to mean ‘where the aircraft is placed in a state of increased risk’
24Contribution to Crew Effectiveness In the LOSA report, crews did not understand the ratings given to them in terms of Contribution to Crew EffectivenessCompletely different set of ratingsProbably the first thing that crews look for in the report so we had to have a clear definition of what they meant. We have slowly worked on them in word terms and deleted any reference to the word “Regulatory” as that identifies it with a “check”Use them at EP days as we believe it is one of the best ways to allow a crewmember to see how they compare to others.
25Well Above Company Standard Awarded when a crewmember’s “contribution to crew effectiveness” could be filmed and used as a training example to other pilots. It would have been a performance where strict adherence to Standard Operating Procedures, excellent CRM and sound manipulative (PF) or support (PNF) skills were observed. All facets of the performance must equally be to the high “training video” standard otherwise the “Well Above Company Standard” rating cannot be given.Company StandardAwarded when a crewmember’s “contribution to crew effectiveness” is highly professional and in line with what is expected by the Chief Pilot. It would not be without error, however, the Captain must demonstrate sound error management skills. Each crewmember must be able to recognise threats to the flight deck operation and the Captain must be able to manage such threats effectively.Point out that the rating must be achieved in all areas before it can be given.Also point out that we have some way to go with crewmember input to the words very important.
26Below Company Standard Awarded when the Auditor is of the opinion that, given the opportunity to conduct the observed sector once again, the crewmember would have increased their contribution to crew effectiveness in order to achieve a more professional standard of flight deck operation. The performance may have included examples of unintentional deviation from SOPs, threats and/or errors that were not recognised or managed to Company expectations, or manipulative support skills that were below a professional aviator’s ability on the day.All crewmembers have had “days like these” and this rating does not mean that the crewmember was not trying to give of his/her best. A professional crewmember would have reviewed the sector with the intention of learning from the experience and improving their contribution to the crew effectiveness accordingly.
27Well Below Company Standard Awarded when a crewmember’s contribution to crew effectiveness falls well short of what the Chief Pilot would reasonably expect of a professional crewmember. There is never any doubt in the mind of the Auditor, when awarding this rating, that the observed performance on the day was unacceptable. Either a complete lack of CRM, poor sector management, intentional violation of SOPs, or manipulative or support skills that did not meet the challenges of the day would have been observed during the crewmember’s performance.
28Costs for Audit Program Audit and narrative writing - 50 sectors B $80 000Wash process (4 weeks, average 3 sectors per day) $50 000Report writing (2 weeks) $10 000TOTAL $Cost of Audit per sector $2 800Audit meeting (quarterly) $25 000
29Where are we now in terms of Threat/Error methodology? Second B767 Audit - crew response was no different from the first auditDe-identificationCabin AuditingQuite frankly the more we use threat / error methodology auditing the more we appreciate it’s value. It tells you not only what happened, but more importantly it tells you why it happened.Like ANZ we want our Regulator to see the benefits of this, and head down the path of getting rid of route checks, which are an expensive waste of time and money for zero tangible benefit. The money would be far better spent auditing, especially if we could get outside Auditors from other airlines to have a look at us every now and again eg: ANZ in our area. (Security drawbacks)As we get better at it in terms of Auditor skills the more we get out of it. For example, the error rate on the B767 audit that we just completed will be double the first one. You can put that down almost exclusively to the fact that Auditors are far better at our powers of observation and ability to take notes that can be transcribed into the report.Go to screen.We think it to be the most significant step forward that we have been able to take in terms of flight deck safety and congratulate UT on both its development and freedom of use.