1. Human Performance Contributions to Safety in Commercial Aviation
Presented by Jon Holbrook, PhD
National Aeronautics and Space Administration
February 2019

2. What is NASA’s role in civil aviation?
NASA Aeronautics goal: Support safe, efficient, flexible, and environmentally sustainable air transportation
Guidance for technology/concept feasibility and maturation
Recommendations for design of systems, policies, & procedures
Tools to support performance assessment, VV&C
NASA maintains cooperative relationships with
Aeronautics industry (e.g., manufacturers, operators)
Government agencies concerned with operations and regulations (e.g., FAA)
Academic community engaged in aerospace research

3. Aviation is a data-driven industry
We (rightly) want to make data-driven decisions about safety management and system design.
The data that are available to us affect how we think about problems and solutions (and vice versa).
In current-day civil aviation, we collect large volumes of data on the failures and errors that result in incidents and accidents, BUT…

4. Aviation is a data-driven industry
We rarely collect or analyze data on resilient behaviors that result in successful outcomes.
We currently make safety management and system design decisions based on a small sample of non- representative data
As a consequence, many in the industry are making some dubious assumptions, such as:
The safest future state is the one that removes humans from the safety decision making loop.

5. (Some of the) Safety-II challenges in aviation
Overarching challenge: Safety-I thinking is so ingrained in current practices and vocabulary that it effectively filters or excludes data from consideration that, from a Safety-II perspective, would be relevant.

6. (Some of the) Safety-II challenges in aviation
Convincing policy makers that there is a problem with thinking about safety exclusively in terms of adverse outcomes
How can we demonstrate the magnitude of the problem?
How can we show that this problem is relevant to policy makers?
Identifying data-driven solutions to the problem
How can we characterize operators’ resilient performance?
Where can we get data on operators’ resilient performance?

7. 1. Demonstrate magnitude of the problem
Human error implicated in 80% of accidents in civil aviation (Weigmann & Shappell, 2001)
Pilots intervene to manage malfunctions on 20% of flights (PARC/CAST, 2013)
World-wide jet data from (Boeing, 2016)
244 million departures
388 accidents
Outcome
Not Accident
Accident No ? ? ?
Attributed to Human Intervention
Yes
20%
80% ? ?
388
244,000,000

8. 1. Demonstrate magnitude of the problem
When we characterize safety only in terms of errors and failures, we ignore the vast majority of human impacts on the system.
Outcome
Not Accident
Accident No
195,199,690 78
195,199,768
Attributed to Human Intervention
Yes
48,799,922
310
48,800,232
243,999,612
388
244,000,000

9. 2. Show problem is relevant to policy makers
Push for machines to take on tasks and responsibilities currently performed by humans
Without understanding how humans contribute to safety, any estimate of predicted safety of autonomous capabilities is incomplete and inherently suspect.
Provides the necessary basis for designing machine systems that could perform safety-producing behaviors
Push for development of in-time safety monitoring, prediction, and mitigation technologies
Requires a more complete understanding of safety – both Protective (Safety-I) and Productive (Safety-II)

10. 3. Characterizing resilient performance
Lots of failure taxonomies, few success taxonomies
“Positive” taxonomies largely focused on positive outcomes (e.g., flight canceled/delayed, rejected takeoff, proper following of radio procedures)
Can we use Safety-II/Resilience Engineering approaches to identify “universally desired” behaviors, regardless of subsequent outcomes?
Can we identify “language” of resilience?
How do operators in YOUR domain talk about resilient performance?
Behaviors are complex, and occur within a rich context
How can we systematically capture “situated” performance without losing that richness?

11. 3. Characterizing resilient performance
No single data source can provide all of this information
Strategy
Resilience Capability:
Anticipate, Monitor, Respond, Learn
Actors / Interactions:
Crew, ATC, Dispatch, Ground Ops, Airline…
Context:
External & Internal
Objectives: Intentions, Goals, Pressures
Resources:
Tools & Knowledge
(Adapted from Rankin, et al., 2014)
is a function of
is an action of type
is an action by
Observable Behavior:
Direct & Indirect
manifests as

12. 4. Sources of resilient performance data: Data that are collected but not analyzed
What data are currently available?
Operator-, observer-, and system-generated
Access challenge
How and why are those data collected?
Sunk cost challenge
Happenstance reporting challenge
How and why are those data analyzed?
Implications for post-hoc coding
Big-data challenge, and the need for tools to support analysis of narrative data
There is no silver bullet
Fusing data into a coherent picture
De-identification challenge

13. 4. Sources of resilient performance data: Data that are not collected but could be
For system-generated data, what parameters could be captured that aren’t (simply because nobody thought to capture them)?
For observer-generated data, what could observers be trained to record that they aren’t?
For operator-generated data, what questions could be included on the data forms that aren’t?
For operator-generated data, how do we overcome non- reporting bias (e.g., being resilient is part of my everyday job – why would I report that)?

14. Recommendations
Redefine safety in terms of the presence of desired behaviors and the absence of undesired behaviors.
Leverage existing data to identify strategies and behaviors that support resilient performance.
Develop tools to capture new data on strategies and behaviors that support resilient performance.
Observer-based, operator-based, system-based
Develop a system-level framework for integrating across data types to facilitate understanding of work-as-done
Develop organization-level strategies that promote recognition and reporting of behaviors that support resilient performance.

15. Questions or Comments?