System Safety and Risk Management

System Safety and Risk Management
September 19, 2018 System Safety and Risk Management Presented by AFS-900 Wednesday, September 19, 2018 Amer M. Younossi

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary Modules 1 Why Flight Standards Is Implementing System Safety and Risk Management 2 Risk Concepts 3 System and Organizational Principles 4 Human Factors 5 Data and Analysis 6 Risk Management

Module 1 Outline How did we get here? Why do system safety?
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary Module 1 Outline How did we get here? Why do system safety? AFS-900 objectives and functions Where is our authority? System safety challenges Summary

How Did The FAA Get Here? 90-Day Safety Review
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary How Did The FAA Get Here? 90-Day Safety Review White house and congressional mandates to reduce the accident rate by 80% in 10 years

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary 90-Day Safety Review The task force categorized issues into six general areas: certification policy and process resource targeting to address safety risks newly certificated air carrier operations and growth outsourcing and varied fleet mix inspector and air carrier guidance material inspector resources

Policy Statement of the FAA
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary Policy Statement of the FAA Aggressive attitude with respect to aviation safety Pro-active approach Air carrier to maintain the highest possible degree of safety Achieve this in the most economical and efficient manner to the government and the carrier Source: FAA Order A

Flight Standards Vision
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary FAA Mission Statement Setting regulations and standards for: air carriers air commerce air agencies airmen, etc. Accomplishing certification, surveillance, inspection, investigation and enforcement activities Managing the systems for registry of ... aircraft airmen Flight Standards Vision To be recognized and respected world- wide as the foremost authority on system safety approaches for regulation and certification

Why Do System Safety Air traffic is increasing
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary Why Do System Safety Air traffic is increasing Aviation operations are becoming more complex FAA oversight staff and resources are constrained We can no longer afford to function as a direct source of QC Systems approach is proactive - Stresses process improvements System safety is good business

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary System Safety Defined “The application of special technical and managerial skills in a systematic, forward-looking manner to identify and control hazards throughout the life cycle of a project, program, or activity.” Source: System Safety Course, FAA Academy

System Safety Philosophy
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary System Safety Philosophy System safety is: Validation of regulatory compliance Review and validation of systems and entire programs, not portions Application of safety attributes Application of risk management

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary AFS-900 Objectives “Air carrier certificates are our gold standard to measure how well we are accomplishing our responsibilities.” Dave Gilliom ~ AFS-900 Standardize air carrier certification Standardize systems and targeting risks Centralize information management functions Six Processes are ISO-9000 Certified

ATOS Model Module 3 Module 3 Module 2 Module 2 Policy
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary ATOS Model Module 3 Surveillance Resource Mgmt Module 3 Surveillance Resource Mgmt Module 2 Certificate Management Module 2 Certificate Management Policy Module 1 System Configuration Module 1 System Configuration Module 4 Surveillance Module 4 Surveillance Module 8 Implementation Module 8 Implementation Module 5 Reporting Module 5 Reporting Module 6 Evaluation Module 6 Evaluation Module 7 Analysis Module 7 Analysis

AFS-900 Functions Air Transportation Oversight System (ATOS)
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary AFS-900 Functions Certification, Standardization, and Evaluation Team (CSET) Certification Surveillance/Evaluation Planning Aircraft conformity team Technical support program Air Transportation Oversight System (ATOS) Improve the certification and surveillance processes for air carriers Ensure regulatory compliance and incorporate a systems approach Enhance the surveillance process Standardize the surveillance processes Flight Standards Safety Analysis Information Center (FSAIC) Provide analytical support to Flight Standards Identify industry-wide risk areas Partnership with aviation safety research institutions

AFS-800 Function System Safety Approach for General Aviation (SAGA)
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary AFS-800 Function System Safety Approach for General Aviation (SAGA) Improve the oversight process for general aviation Open communications with air operators. Identify focus areas to target, and provide feedback to operators.

Integrated System Safety Efforts
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary Integrated System Safety Efforts SASO ~ Systems Approach for Safety Oversight Business process procedures, guidelines, and automation tools Data quality enhancement and database standardization Risk management/analysis procedures and automation Interconnectivity of agency and industry databases

Where Is Our Authority? Title 49 44701, 44702 Current 121 Rules
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary Where Is Our Authority? Title , 44702 Current 121 Rules

USC Title 49 Sec. 44702. Issuance of certificates ...
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary USC Title 49 Sec Issuance of certificates ... (B) considerations. - When issuing a certificate under this chapter, the administrator shall - (1) consider - (a) the duty of an air carrier to provide service with the highest possible degree of safety in the public interest.

USC Title 49 Sec. 44701. General requirements
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary USC Title 49 Sec General requirements (A) promoting safety. - The administrator of the FAA shall promote safe flight of civil aircraft in air commerce by prescribing - ... (2) regulations and minimum standards in the interest of safety

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary 14 CFR Part 121 Rules 14 CFR part (a) (1) each manual required by § must - (1) include instructions and information necessary to allow the personnel concerned to perform their duties and responsibilities with a high degree of safety; ...

System Safety Challenges
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary System Safety Challenges Operator’s organizational structures Systemic orientation and completeness of regulations Defining and measuring safety FAA and industry cultures Human factors Availability of valid, reliable data that are representative of the system Risk management application

Summary 90 day safety review System Safety AFS-900 September 19, 2018
Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline How Why Objectives/Functions Authority Challenges Summary Summary 90 day safety review System Safety AFS-900

September 19, 2018 System Safety and Risk Management Module 2 Risk Concepts Wednesday, September 19, 2018 Presented by AFS-900

Module 2 Outline Fundamentals of Risk Definitions of Risk
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Module 2 Outline Fundamentals of Risk Definitions of Risk Common Concepts of Risks Risk Factors/Controls System Safety Summary

Risk Management Risk management is a core process to system safety.
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Risk Management Risk management is a core process to system safety. Risk management can be practiced at all levels and in all types of aviation. “Safety is not equivalent to risk free” – U.S. Supreme Court, 1972 “Risk management is a more realistic term than safety…” – Jerome Lederer, 1928

Basic Risk Concepts Event Hazard Risk Accident, Incident, Occurrence
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Basic Risk Concepts Event Accident, Incident, Occurrence Undesired event resulting in loss Hazard Condition that can cause event Risk Likelihood / severity of event associated with the hazard Exposure to hazard... Likelihood of occurrence Severity of consequences

Schools of Thought Product Development (“Traditional” SS)
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Schools of Thought Product Development (“Traditional” SS) Organizational Psych (e.g. James Reason) Project Management (Risk Management – e.g. SEI) AFS’ approach is a blend

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Definition(s) of Risk Several definitions exist, based upon the context for which they were developed More than one are used in AFS programs All have a common underlying set of concepts

Common Concepts What is it that I will run into, hit, trip on, etc.?
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Common Concepts What is it that I will run into, hit, trip on, etc.? Hazard, condition, failure, threat Something bad can happen to us Undesired event, mishap, consequence How likely is it to happen? Likelihood, probability (impact)1 How bad will it hurt if it does? Severity, impact (impact)1 1 Source: Product school

AFS Accepted Definition
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary AFS Accepted Definition Safety risk is an expression of the impact of an undesired event in terms of event severity and event likelihood. Source: FAA Order , Safety Risk Management - Product School

Risk Factors and Control
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Risk Factors and Control Most system risk is incurred through factors involving severity or likelihood Most deal with factors increasing likelihood of a hazardous condition Risk controls typically: Help detect likelihood-increasing factors Lower the likelihood Eliminate the hazard

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Risk Factors Things about the inherent hazard, system design, or organizational performance that affect risk likelihood or severity. Task complexity Environmental factors Difficult procedures Poor or missing supervisory controls Poor Communications

Risk Controls (Mitigation)
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Risk Controls (Mitigation) Serve to reduce likelihood or severity Be wary of substitution risk Controls must recognize human factors and culture

Risk Involving Organizations
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Risk Involving Organizations Inherent risk Cause of the problem Design deficiencies Failure provide controls on the problem Performance Deficiency Failure to comply with design failure to the design to control risk

Inherent Hazards and Risk
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Inherent Hazards and Risk ? Environment Operation Hazard

System Design and Risk System Procedures Process Controls Interfaces
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary System Design and Risk System Procedures Interfaces Resources Process Controls Process Measures

System Performance Conformance/compliance Effectiveness
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary System Performance Conformance/compliance Are we complying with the design of the system? Effectiveness Does the system accomplish its intended function (control the risk factors)?

Regulations as Risk Controls
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Regulations as Risk Controls Based upon public law (49 USC 447) Regulations and approved programs may be risk controls. Effectiveness is bounded by specificity of regulations, programs, manuals, etc. Proper application is essential!

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary System Safety Model The FAA will use the System Safety Process Model as a basis. Model was adapted from Nicholas J. Bahr by AFS-900 and ASY-300 for use by Flight Standards

System Safety Process Documentation Risk Management Define Objectives
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary System Safety Process Risk Management Define Objectives Hazard Identification: Identify Hazards & Consequences System Descriptions Risk Analysis: Analyze Hazards & Identify Risks Risk Assessment: Consolidate & Prioritize Risks Here are the steps that make up the System Safety Process: “Define Objectives” - what the system is designed to accomplish. “System Descriptions” - how the system will accomplish its objectives. “Hazard Identification and Consequences” - what can go wrong? "Risk analysis" looks at what can go wrong and determines what consequences might be, how likely it is to happen, how severe the consequences will be if it does happen. ”Risk assessment", we compare, combine, sort and rank risks according to how serious they are. This helps us when we are trying to decide what should be done about those risks. “Decision-making” what options are available to ensure that the air carrier is managing those risks? “Validation of Control” Did our plan work? Was our plan effective in ensuring that the air carrier is lowering risk levels? No? system/process need to be modified. Yes? Closed Loop SS safety approach includes continually searching for hazards.. Documentation is key throughout all these steps. Decision Making: Develop an Action Plan Modify System/Process System/Process Review Validation of Control: Results for Further Action Documentation

System Safety Process 1 - Identify the task(s)/functions –
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary System Safety Process 1 - Identify the task(s)/functions – 2 - Describe the environment – 3 - Identify the hazards – 4 - Evaluate the risk ...

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary System Safety Process 5 - Identify and evaluate mitigating strategies – 6 - Decide on a course of action 7 - Evaluate effectiveness of controls –

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Risk Perceptions “Carelessness and overconfidence are more dangerous than deliberately accepted risk” - Wilbur Wright, 1901 Wilbur Wright gliding, 1901 Photographs: Library of Congress

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Summary

September 19, 2018 System Safety and Risk Management Module 3 System and Organizational Principles Wednesday, September 19, 2018 Presented by AFS-900

Module 3 Outline What is a system? System attributes
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Attributes Operator Objectives Regulator’s Objectives Summary Module 3 Outline What is a system? Tasks, functions and environments System attributes Operator's Objectives Regulator’s Objectives Summary

System and Organizational Principles
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Attributes Operator Objectives Regulator’s Objectives Summary System and Organizational Principles “Accidents Are Not Due to Lack of Knowledge, but Failure to Use the Knowledge We Have.” - Trevor Kletz, “What Went Wrong?” July 17, :31:10 PM TWA Flight 800 Boeing , N rd aircraft built (1971)

People Software Procedures What’s a System? Systems Tools Equipment
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Attributes Operator Objectives Regulator’s Objectives Summary What’s a System? “A system is a composite of people, tools, procedures, materials, equipment, facilities, and software operating in a specific environment to perform a specific task or achieve a specific purpose, support, or mission requirement.” - Roland and Moriarty, System Safety Engineering and Management “A system is a composite of people, tools, procedures, materials, equipment, facilities, and software operating in a specific environment to perform a specific task or achieve a specific purpose, support, or mission requirement.” - Roland and Moriarty, System Safety Engineering and Management ...a set of components that act together as a whole to achieve a common goal Systems People Software Tools Equipment Procedures Facilities Materials

Situation - Environment
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Attributes Operator Objectives Regulator’s Objectives Summary Situation - Environment What Are the Conditions? Physical environment Facilities Airports Weather patterns Operational environment Type of operation Type of equipment Employee demographics Financial factors

Situation - Tasks and Functions
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Attributes Operator Objectives Regulator’s Objectives Summary Situation - Tasks and Functions What Is Being Done? Tasks - accomplished by humans Pilot tasks - Advanced Qualification Program (AQP) Maintenance tasks - Work cards Functions - organizational structures to accomplish tasks

System Safety Attributes
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Attributes Operator Objectives Regulator’s Objectives Summary System Safety Attributes Responsibility Interfaces Authority System Safety Attributes Process Measurement The safety attributes are: a. Responsibility. There is a clearly identifiable, qualified, and knowledgeable person who is accountable for the quality of a process. b. Authority. There is a clearly identifiable, qualified, and knowledgeable person with the authority to set up and change a process. c. Procedures. There are documented methods for doing a process. d. Controls. There are checks and restraints designed into a process in order to get a desired result. e. Process Measurement. The air carrier measures and assesses its processes to identify and correct problems or potential problems. f. Interfaces. The air carrier identifies and manages the interactions between processes. Procedures Controls

September 19, 2018 What is a Process? A Process is a set of tasks, work-flows and information flows that produce a desired result. Source: System Safety Course, FAA Academy If you can't describe what you are doing as a process, you don't know what you're doing. W. Edward Deming US business advisor & author ( )

System Based Process Inputs Outputs Process Controls Resources
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Attributes Operator Objectives Regulator’s Objectives Summary System Based Process Controls Inputs Process Outputs Resources

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Responsibility Outline System Attributes Operator Objectives Regulator’s Objectives Summary Authority Jobs and Tasks System Safety Attributes Procedures Responsibility / Authority Procedures Function Job Input Task Output This is a little more complicated process then the last, but same general components apply. Additionally, now we will ensure that all six attributes are being considered in the design of this process. We begin with a particular function which needs to be accomplished, as an example Control of calibrated tools and test equipment. This function contains tasks within it and input and an output. By using the six safety attributes, there has to be someone responsible for the quality of the process and who has the authority to set up and/or change the process.

Interfaces Between Tasks
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Responsibility Outline System Attributes Operator Objectives Regulator’s Objectives Summary Interfaces Authority Interfaces Between Tasks System Safety Attributes Procedures Controls Responsibility / Authority Procedures Interfaces Controls Function Job Job Input Task Task Output All the tasks that must be performed to achieve our desired output must have written procedures and controls to ensure those tasks are being performed properly. For example, the calibration lab personnel must,disassemble, inspect, clean, measure, test and then reassemble the tools and/or equipment in accordance to those procedures and controls. Additionally, interface must exist between each job and shown in this diagram.

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Responsibility Outline System Attributes Operator Objectives Regulator’s Objectives Summary Interfaces Authority And More Tasks System Safety Attributes Procedures Controls Responsibility / Authority Procedures Interfaces Controls Function Job Job Input Task Task Output Proper interfaces would include consistency across manual systems with other departments in the air carrier’s organization. For example, the calibration lab could interface with the Inspection area to assure all the measurements and/or tests are within mfg. Standards or traceable to the National Institute of Standards Technology. Additionally, now we have controls that to ensure that results are within the requirements. Job Input Task Resources Process Controls

Feedback and Information
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Responsibility Outline System Attributes Operator Objectives Regulator’s Objectives Summary Interfaces Authority Feedback and Information System Safety Attributes Process Measurement Procedures Controls Responsibility / Authority Procedures Interfaces Controls Function Job Job Input Task Task Status Process Measures A well designed system would include a process measurement to measure and assess its processes to identify and correct problems or potential problems. The companies Quality Assurance would provide feedback to the respective person who has the responsibility and authority to make any necessary changes within this function. Job Input Task Resources Process Controls

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Responsibility Outline System Attributes Operator Objectives Regulator’s Objectives Summary Interfaces Authority Completed Function System Safety Attributes Process Measurement Procedures Controls Responsibility / Authority Procedures Interfaces Controls Function Job Job Input Task Task Output Status Process Measures By incorporating all of the 6 safety attributes within this function, for control of calibrated tools and test equipment, an operator would be “designing-in” system safety for this function or process. They would still need to take into account the environments for which they operate in, the set of components that act together as a whole to achieve a common goal and the specific tasks entailed for achieving the mission requirement. One other item the operator must consider in the development of their system is to assure they look at the regulatory requirements and what the requirements are trying to achieve Job Input Task Resources Process Controls

Operator’s Objectives
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Attributes Operator Objectives Regulator’s Objectives Summary Operator’s Objectives What regulatory requirements are trying to achieve Designing systems that ensure regulatory compliance and safety

Organizational Decision Factors
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Attributes Operator Objectives Regulator’s Objectives Summary Organizational Decision Factors Delicate and Complex Balance Outcome relatively certain Measured in positive terms Outcome relatively uncertain Measured by absence of negative events Resources Available Money Equipment Personnel/Expertise Available Time Safety Goals Production Goals Decision Makers Protection Production Defensive Filters

Regulator’s Objectives
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Attributes Operator Objectives Regulator’s Objectives Summary Regulator’s Objectives Understand what the system is designed to do Assure the operator’s are complying with applicable regulations Assure that operator’s follow it’s approved or accepted programs

System Safety Philosophy
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Attributes Operator Objectives Regulator’s Objectives Summary System Safety Philosophy System safety is: Validation of regulatory compliance Review and validation of systems and entire programs, not portions Application safety attributes Application of risk management

Summary Importance of system design
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Attributes Operator Objectives Regulator’s Objectives Summary Summary Importance of system design Delicate balance within an organization Understanding the objectives The problems are usually related to human error

September 19, 2018 System Safety and Risk Management Module 4 Human Factors Wednesday, September 19, 2018 Presented by AFS-900

Module 4 Outline Human Performance Human Error Understand Causes
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Human Error Understand Causes Assessing Organizational Health Inspections Summary Module 4 Outline Human Performance Human Error Understand Causes Summary

Human Performance Basic motor reflexes Skill-based performance
September 19, 2018 Human Performance Basic motor reflexes Skill-based performance Rule-based performance Knowledge-based performance

Human Factors in the Decision Process
September 19, 2018 Human Factors in the Decision Process Information processing Situation awareness Cognitive (Mental) workload CRM (Cognitive Resource Management)

September 19, 2018 Situation Awareness Situation awareness is the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future. -Mica Endsley, 1993

Human Thought Process Mental Model Perception Comprehension Projection
September 19, 2018 Human Thought Process Mental Model Perception Comprehension Projection Stored Patterns Source: Dr. Mica Endsley

Situation Awareness Errors
September 19, 2018 Situation Awareness Errors Level 1 - Failure to correctly perceive the situation. Level 2 - Failure to comprehend the situation. Level 3 - Failure to comprehend the situation into the future.

Situation Awareness Analysis
September 19, 2018 Situation Awareness Analysis ASRS data. 169 errors in 113 incidents. 80.2% Level 1 Errors 16.9% Level 2 Errors 2.9% Level 3 Errors

Cognitive (Mental) Workload
September 19, 2018 Cognitive (Mental) Workload Human working memory is limited The magic 7 plus or minus 2 (Miller, 1956) Fatigue can reduce this Interruptions can “bump” material in working memory Information needed for a typical instrument approach can exceed the capacity available

Fatigue and Workload Activities late in flight have high workload
September 19, 2018 Fatigue and Workload Activities late in flight have high workload Capacity is diminished by fatigue High accident potential (Boeing studies)

Mechanical/Environmental
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Human Error Understand Causes Assessing Organizational Health Inspections Summary Human Error Most accidents are human related Accident Rates 1977 1992 What can we do? Human Error Mechanical/Environmental Factors

Human Error Error is ubiquitous - it’s a human thing.
September 19, 2018 Human Error Error is ubiquitous - it’s a human thing. Most accidents are human related. Rules and attention to detail are essential in a systematic approach to safety, but... Regulations and exhortations often have limited effectiveness in error prevention. “Be careful” only works so far.

September 19, 2018 Human Error Slips/lapses are most often associated with psychomotor or automatic skills. Skill or “action”-type problems most often responsible for frequent, low-severity events. Mistakes are often associated with cognitive tasks such as decision-making. Decision-type problems more often responsible for high-severity events (fatal accidents). Many inadvertent rule “violations” are really decision or knowledge problems.

New Zealand GA Study 200 GA accidents analyzed Decision errors:
September 19, 2018 New Zealand GA Study 200 GA accidents analyzed Decision errors: 31% of minor/non-injury accidents 63% of fatal accidents Action (skill) errors: 48% of non-injury accidents 16% of fatal accidents

Error Mitigation Skill-based errors Rule-based errors Knowledge-based
September 19, 2018 Error Mitigation Skill-based errors Rule-based errors Knowledge-based

Understand Causes Active Failures Latent Failures Air Ontario - Dryden
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Human Error Understand Causes Assessing Organizational Health Inspections Summary Understand Causes Active Failures Latent Failures Air Ontario - Dryden Pilot Error Organizational Failures

Latent Failures Operational Personnel Diminishing Returns
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Human Error Understand Causes Assessing Organizational Health Inspections Summary Latent Failures Operational Personnel Diminishing Returns Redirect Safety Efforts Judging Safety Health

James Reason Model Accidents Organization Workplace People/Team
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Human Error Understand Causes Assessing Organizational Health Inspections Summary James Reason Model Organization Workplace People/Team Controls Organizational Factors Communication Management Structure Goals Policy & Procedure Local Factors Physical Environment Operational Environment Task Design Active Failures Performance Errors - Mistakes - Slips - Lapses Violations Accidents Outcome Active Error Path Latent Error Path Source: The organizational connections to hurt bodies and bent metal, by: Dr. James Reason

Individual Differences - Attitudes
September 19, 2018 Individual Differences - Attitudes Antiauthority (don’t tell me!). Impulsivity (do something quickly!). Invulnerability (it won’t happen to me). Macho (I can do it). Resignation (what’s the use?).

Individual Differences – Personality (“Big 5”)
September 19, 2018 Individual Differences – Personality (“Big 5”) Openness to experience Conscientiousness Extraversion/Introversion Agreeableness Neuroticism (emotional stability)

Group Differences - Culture
September 19, 2018 Group Differences - Culture Uncertainty Avoidance Masculinity/Femininity Individualism/Collectivism Power Distance Short/Long-Term Orientation

Managing Human Factors
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Human Error Understand Causes Assessing Organizational Health Inspections Summary Managing Human Factors Safety Culture Incident Investigation Feedback Increase Tolerance Limit Error Potential Closed Loop

Organizational Response
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Human Error Understand Causes Assessing Organizational Health Inspections Summary Organizational Response Denial Repair Reform

Addressing Human Factors
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Human Error Understand Causes Assessing Organizational Health Inspections Summary Addressing Human Factors Humans Will Make Mistakes Controls Increase Tolerance Lessen Potential “Using the checklist is statistically safer than thinking.” - British Airways Check Airman

Summary Why Study Human Factors Active and Latent Failures
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Human Error Understand Causes Assessing Organizational Health Inspections Summary Summary Why Study Human Factors Active and Latent Failures Errors and Violations Controls Managing Human Factors

September 19, 2018 System Safety and Risk Management Module 5 Data and Analysis Wednesday, September 19, 2018 Presented by AFS-900

Module 5 Outline Decision-Making Hierarchy Question Analysis Data
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Module 5 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data to System Safety Summary

Decision-Making Hierarchy
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Decision-Making Hierarchy Action Decision Knowledge Information Data Observation

Biases Humans tend to use simple “rules of thumb” called heuristics.
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Biases Humans tend to use simple “rules of thumb” called heuristics. These heuristics simplify mental processes but they can also lead to over simplification and error. Typical biases include: Representativeness heuristic Salience - most attention-getting rather than most useful Availability heuristic - recency Confirmation bias - tendency to focus on initial hypothesis “As if” weighting (all info assumed of same quality) Conservatism

Decision-Making Hierarchy
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Decision-Making Hierarchy Question/Problem Action Decision Knowledge Analysis Information Data Data Observation

Bias can be reduced through the careful use of data
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Counteracting Biases Question Clear Empirical Question Bias can be reduced through the careful use of data Analysis Data Good Quality Data Appropriate Data Analysis

Question Problem Definition - What do we want to know?
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Question Problem Definition - What do we want to know? System definition and analysis Description of the situation Risk or Problem statement Operational definitions are important

Analysis Types of information
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Analysis Types of information Records Inferences Judgments Important to separate reports (facts) from inferences and judgments Recording only judgments or conclusions will usually decrease data reliability Data must fit: Analysis needs Nature of observation Observation situation

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Analysis Aggregation Cannot create system level analyses by “rolling up” unrelated data Result will depend more on mix of data than performance of any element Trends are especially sensitive Can’t mix apples and oranges and compare fruit baskets

Analysis Comparisons Many comparisons are legitimate:
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Analysis Comparisons Many comparisons are legitimate: Norms / thresholds Association with factors Comparisons between subgroups (e.g. location, aircraft fleet) Trends Comparisons should be avoided in uses to “rate” offices or people

Analysis Trend Analysis A special type of comparison
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Analysis Trend Analysis A special type of comparison Often confused with “generalization” Must be based upon a reliable, quantifiable metric at progressive time points Must have good internal validity Must have data available in adequate quantity, regular intervals

Data Data Torturing Inappropriate use of data
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Data Data Torturing Inappropriate use of data Inappropriate comparisons to averages, specifications, targets and percentages Smoothing Limited point trends Use of raw counts Source: Dr. Marcey Abate, Sandia National Labs, 1998

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Data What is data? Information organized for analysis or used as a basis for decision making Factual data is not a record of conclusions, concerns, or opinion Factual data is a record of facts upon which to base conclusions

Data Automation and Data
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Data Automation and Data Good quality automation is essential to manage the workload of providing data, but… Neither automation nor analysis can add information which is not in the data “Computers do not produce new sorts of errors. They merely provide new and easier opportunities for making old errors.” Source: Trevor Kletz, Wise After the Event

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Data Sampling Sample data represents a larger group with data from a small group Distribution of observations is important Must identify factors that can effect performance Problems can be created in “rollups” Normal trends (e.g. season) may be involved Statistical confidence is unlikely. Confidence in what?

Data Text and Narratives
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Data Text and Narratives Text is very flexible, however, very unreliable as as a data source Narratives are valuable adjuncts to explain observations Text may be coded into quantitative form using an appropriate taxonomy; however, this procedure is very time-consuming and adds another source of error Anecdotes may not be representative of any larger group

Data Data Quality Attributes Validity Reliability Sensitivity
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Data Data Quality Attributes Validity Reliability Sensitivity Representativeness

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Data Data Validity Validity means that the data measure what they are intended to measure Validity is weakened or reduced by: Vague Definitions Insufficient Evaluator Training Casual Data Collection Methods Operator Discretion Source: AQP Data Management Guide, 1998

Data Data Reliability Consistency between raters
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Data Data Reliability Consistency between raters Consistency among items Stability in measurements over time Factors effecting data reliability Definition of item(s) to be observed Standards of evaluation Training of inspectors Type of measurement used Source: AQP Data Management Guide, 1998

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Data Data Sensitivity Sensitivity means that small gradations or variations in the parameter being measured are reflected in some variation in the measurement Multi-point scales allow more sensitivity than pass/fail grading Using multiple items to measure individual components of performance and then combining them into a composite score or index could increase sensitivity Source: AQP Data Management Guide, 1998

Data Data Problems Confounding Contamination
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Data Data Problems Confounding Confounding occurs when two different factors can explain the same result When research is confounded, it is impossible to determine what variable is responsible for any obtained difference in performance When no confounding is present, a research study is said to have internal validity Contamination Behavior changes as a result of evaluation process Observer effects Announced inspections

Data Improving Data Collection
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Data Improving Data Collection Develop focused factual information requirements Document the requirements Seek answers that are: System configuration information System performance information Data must be factual and representative

Contribution of Data to System Safety
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Contribution of Data to System Safety System Validation Hazard Identification Risk Analysis/Assessment

September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Decision-Making Hierarchy Question Analysis Data Contribution of Data Summary Summary Collection of pertinent and quality data, and appropriate analysis will enhance current and future programs Data-driven surveillance programs will result in enhanced System Safety, whereby issues and hazards are identified early and addressed adequately

September 19, 2018 System Safety and Risk Management Module 6 Risk Management Wednesday, September 19, 2018 Presented by AFS-900

Module 6 Outline System Safety Process Practical Application
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Safety Process Practical Application Case Study Summary Module 6 Outline System Safety Process Practical Application Case Study Summary

What, How, Why System Safety’s primary objective
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Safety Process Practical Application Case Study Summary What, How, Why System Safety’s primary objective Consider the entire system Understanding the operator’s accident prevention program

Hazard Identification: Identify Hazards & Consequences
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Safety Process Practical Application Case Study Summary System Safety Process Risk Management Define Objectives Define Objectives Hazard Identification: Identify Hazards & Consequences Hazard Identification: Identify Hazards & Consequences System/Process Review Yes Modify System/Process No System Descriptions System Descriptions Risk Analysis: Analyze Hazards & Identify Risks Risk Analysis: Analyze Hazards & Identify Risks Risk Assessment: Consolidate & Prioritize Risks Risk Assessment: Consolidate & Prioritize Risks Decision Making: Develop an Action Plan Decision Making: Develop an Action Plan Modify System/Process System/Process Review Validation of Control: Results for Further Action Validation of Control: Results for Further Action Documentation

Risk Assessment Matrix
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline Fundamentals Definitions Concepts Risk Factors System Safety Summary Risk Assessment Matrix

September 19, 2018 Outline System Safety Process Practical Application Case Study Summary April 28, Near Maui, Hawaii Flight Boeing (N7371I) Puzzle

System Safety Process Documentation Risk Management Define Objectives
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Safety Process Practical Application Case Study Summary System Safety Process Risk Management Define Objectives Hazard Identification: Identify Hazards & Consequences System Descriptions Risk Analysis: Analyze Hazards & Identify Risks Risk Assessment: Consolidate & Prioritize Risks Objective - transport passengers and goods for profit System - fairly old, high time airplanes - Operational - high cycle (typical is 2 hours - actual was probably about 30 minutes because they were island hopping) Physical - oceanic, somewhat corrosive Hazard - Condition - The basic hazard is flying a pressurized aluminum tube into the lower stratosphere. Consequence can be airframe failure Severity - critical - loss of pressure and structural strength Likelihood - Environmental factors will determine our level of risk. Fairly old, high time airplanes - one of the ATOS Risk IndicatorsI Increase the risk of failure when we operate in a high cycle island hopping environment humid, salty air that acilitates corrosion in aluminum. Assessment - High Priority - Risk Level is High Decision Making- look at environmental factors -how can AC lower risk Buy newer airplanes. Move islands farther part. Dry out the air. Environment won’t change, high cycle and high time won’t change. Simplest way is for the AC to concentrate on corrosion inspection and structural inspection program. The air carrier should spend more time in surveillance to detect problems as early as possible. Validate Controls - Air carrier should have some sort of process and performance measures to assure the quality and performance of their programs. The FAA should increased emphasis on the design and auditing of corrosion prevention and structural inspection programs. Decision Making: Develop an Action Plan Modify System/Process System/Process Review Validation of Control: Results for Further Action Documentation

Summary Understand... the System Safety Process
September 19, 2018 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Outline System Safety Process Practical Application Case Study Summary Summary Understand... the System Safety Process how It Will Assist You in Your Job

Additional Sources Training Available at FAA Academy in Oklahoma City
September 19, 2018 Additional Sources Training Available at FAA Academy in Oklahoma City ATOS Overview Course Length: 24 hours Course Description: This course is designed to familiarize industry personnel, international students, and government agencies other than FAA with ATOS. It will provide instruction on the ATOS access, ATOS system, including the tools, the policy and guidance, that make the ATOS system work. System Safety Course Length: 24 hours Course Description: The purpose of this course is to train FAA aviation safety inspectors on system safety and risk management processes and the importance of both as they relate to systems management. This will be accomplished by demonstrating how system safety and risk management are used throughout the FAA environment.This course is available to industry personnel based on class availability

Additional Sources http://www.faa.gov/avr/afs/
September 19, 2018 Additional Sources System Safety and ATOS Overview Course Length: 32 hours Course Description: This seminar is designed to provide an introduction to system safety and risk management processes as they are used by the FAA for air carrier oversight. The seminar also familiarizes industry personnel with the FAA's Air Transportation Oversight System (ATOS). The four-day seminar is conducted on location by special arrangement with the sponsoring organization for up to 50 students. Contact: Mr. Joseph Souza Don Arendt - AFS-900 FSAIC Manager

System Safety and Risk Management

Similar presentations

Presentation on theme: "System Safety and Risk Management"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

System Safety and Risk Management

Similar presentations

Presentation on theme: "System Safety and Risk Management"— Presentation transcript:

Similar presentations

About project

Feedback