1. The LOSA Archive: The data and how it can be used
Ashleigh Merritt, Ph.D.
The University of Texas Human Factors Research Project (UT)
2nd ICAO Global Symposium on Threat and Error Management (TEM) and Normal Operations Safety Survey (NOSS) in Air Traffic Control
7-8 February 2007

2. Purpose of Presentation
Show how the LOSA Archive is used in the airline world
Show how a similar NOSS Archive could benefit the ATC world

3. What is LOSA? Line Operations Safety Audits
Forerunner/Prototype for NOSS
Normal operations monitoring based on cockpit observations of regularly scheduled commercial flights
Trained expert observers using TEM taxonomy to frame, describe, code the flight
ICAO: Industry best practice for normal operations monitoring (ICAO LOSA Manual, Doc 9803)
FAA: (FAA Advisory Circular )

4. TEM in the Airline World
Threats: Adverse weather, traffic congestion, terrain, airport conditions, aircraft malfunctions, problems with ground, maintenance, dispatch, cabin, and of course….. ATC
Errors: Handling, automation, systems, procedures, briefings, communication
Undesired Aircraft States: Speed, lateral and vertical deviations, unstable approaches, incorrect engine/system configurations, taxiway incursions

5. The LOSA Archive (Archie)
A database maintained and updated by UT
Data from airlines who do a LOSA with TLC
As of February 2007, data from:
30 airline LOSAs (including 4 repeat LOSAs)
~6000 regularly scheduled commercial flights
22,000+ threats
15,000+ errors
2,500+ undesired aircraft states

6. LOSA Airlines

7. Archive: Three Main Uses
Benchmarking
Industry Trends
Interface with other safety information sources

8. Archive Use #1: Benchmarking
Empirical TEM data
New form of data - “One hand clapping”
Example: Airline X
36% of flights had a mismanaged threat
30% of flights had a mismanaged Handling error
26% of flights had some form of intentional noncompliance.
Is that good, bad or average?

9. Airline X and the LOSA Archive
36% of flights had a mismanaged threat
Average across 30 LOSAs is 34%. Range 16-63%
30% of flights had a mismanaged Handling error
Average is 37%. Range 13-88%
26% of flights had some form of intentional noncompliance.
Average is 46%. Range 22-90%

10. Archive Use #2: Industry Trends
Normal Ops monitoring provides important base rate information
Example: Unstable Approaches
We know how many incidents/accidents involve an unstable approach, but do we know how many unstable approaches occur without incident?
Average is 5% of flights. Range 0-15% across 30 LOSAs
10% of UASs are linked to a mismanaged ATC threat

11. ATC Threats in the LOSA Archive
Data are based on ~2400 Flights (~9500 threats)
ATC the most common threat
¼ of all threats were ATC threats
10% of all ATC threats were mismanaged by the crews
In all, 30% of all mismanaged threats were ATC

12. Challenging clearances/ late change (46%) Language difficulty (7%)
ATC threats
2426 Flights
Challenging clearances/ late change (46%)
2349 ATC threats
Runway Changes (14%)
ATC error (13%)
Other ATC threats (20%)
Language difficulty (7%)

13. Mismanaged ATC threats
2426 Flights
Challenging clearances (53%)
2349 ATC threats
236 mismanaged
Runway Changes (14%)
Language difficulty (8%)
Similar call signs (5%)
ATC error (7%)

14. ATC threats -> Crew Errors
Communication errors (27%) - misinterpret ATC instructions
2426 Flights
280 errors
236 mismanaged ATC threats
Aircraft handling (24%) – unintentional speed, lateral, vertical deviations
Ground navigation errors (6%)
Cross-verification errors (7%)
Automation errors (20%) – wrong MCP/FCU altitude setting dialed

15. ATC threats -> Crew Errors -> UAS
76 Communication errors
2426 Flights 2
UAS
280 errors
236 mismanaged ATC threats
67 Aircraft handling errors 58
UAS
16 Ground navigation errors
19 Cross-verification errors
56 Automation errors 9
UAS 21
UAS 5
UAS

16. 236 mismanaged ATC threats ¾ occurred during descent/approach/land
ATC threats -> UAS
2426 Flights
Lateral deviation %
Speed too high %
Vertical deviation %
Unstable Approach %
Continued Landing %
Incorrect Automation configuration %
Taxiway/runway incursion %
Speed too low %
Incorrect Aircraft configuration % Other %
280 errors
236 mismanaged ATC threats
106 UAS
Bottom line:
4% of flights had a UAS arising from an ATC threat that was mismanaged.
¾ occurred during descent/approach/land

17. Archive Use #2: Industry Trends What does Noncompliance signify?
Has to meet one of three conditions to be coded as intentional noncompliance in LOSA:
Committed multiple times during one phase of flight (e.g., missing multiple altitude callouts during descent). (coded as one intentional noncompliance error)
Crew openly discusses they are intentionally committing an action that is against published SOP
Observer determines the crew is time-optimizing SOP when time is otherwise available (i.e., performing a checklist from memory)
Most errors are coded as unintentional/ ‘honest’ mistakes. If observer is in doubt, error is coded as unintentional.

18. Is Noncompliance a Problem?
The average (across 30 LOSAs): 46% of flights have one or more noncompliance errors
Range: 22% of flights at one airline to 90% at another
Most common noncompliance errors
checklist performed from memory / nonstandard checklist use
failure to cross-verify MCP/FCU altitude changes
PF makes own MCP/FCU changes when hand-flying
Conclusion 1: Noncompliance is just experienced pilots taking optimizing shortcuts… no big deal

19. Industry Trend: What does Noncompliance signify?
Across 30 LOSAs, airlines that have higher rates of intentional noncompliance also have higher rates of:
Mismanaged threats (r = .60)
Mismanaged errors - handling (r = .81)
Mismanaged errors - procedural (r = .80)
Undesired aircraft states UAS (r = .81)
Mismanaged UASs (r =.73)
Conclusion 2: Increased noncompliance decreases safety margins… Noncompliance reflects the safety culture…

20. Archive Use #3: Interface with other safety information sources
Queries from / data-sharing with:
Airlines
Incident Reporting systems
IATA/ICAO (ITA)
Transport Safety Boards
Boeing
ATC (NOSS Archive?)

21. And so - a NOSS Archive?
Have to be patient while the methodology matures and the Archive grows, but once in place:
You can benchmark within and across facilities to determine strengths and vulnerabilities
You can trend to determine industry best practices & problems
A TEM-based NOSS Archive would allow ATC to “converse” freely with the LOSA Archive on matters of mutual interest to pilots and controllers

22. The University of Texas
Human Factors Research Project