1. Normal Operations Data: Air Traffic Facility Evaluations and NOSS
Presented to: Second ICAO Global Symposium on
TEM/NOSS in Air Traffic Control
By: Karen Burcham
Date: February 8, 2007

2. Briefing Objectives FAA Safety Data Sources
Purpose, objectives, and philosophy of Safety Evaluations and Audits
Internal Evaluation and Audit processes
NOSS role and status

3. Heinrich’s Triangle and Safety
Leading Indicators
Lagging Indicators 1
Aircraft
Accident
~30
Incidents
~300
Hazardous
Conditions
~1,000
Unreported
“Unsafe Acts”
(OE, OD, NMAC, PD)
(Non-Compliance, Bad
Documentation, bad phraseology, etc.)
(Non-Reported OE’s/OD’s, Missed Readbacks, etc.)

4. Safety Data Reporting Requirements

5. Safety Data Sources

6. FAA Safety Data Sources (cont.)

7. So We Went Looking For The Big Picture.
Some data only shows the tip of the problem.
Multiple databases may provide a lot of information but what does it all mean?
What is needed to identify better precursors?
Photo courtesy of Claudia VIRLAN, ANSS Directorate, Romanian CAA

8. Sources of Leading Indicators
Data Collected During Normal Operations—not as the result of an Incident/Accident
Revamped Air Traffic Facility Evaluations/Audit Process
NOSS

9. Safety Evaluations Serve the Assurance Component of SMS
Supporting Requirements
ATO must continue to meet its performance targets for safety
Reduce operational errors
Reduce runway incursions
To date, ATO collects the number, rate, and severity of these incidents (“lagging” safety indicators)
ATO does not consistently collect data that may identify precursors to these incidents (“leading” safety indicators)
Continual tracking and analysis of lagging and leading indicators will drive toward the identification of mitigation strategies that have a greater impact on the achievement of safety performance targets
Policy
Architecture
FAA
Safety Management
System
Assurance
Safety
Promotion
Assurance Component of SMS 1
Safety assurance and evaluations 2
Safety data tracking and analysis

10. Safety Evaluations Purpose/Mission
Conduct audits, program assessments, operational assessments, and facilitate evaluations of ATC facilities to promote safety, quality, and efficiency of the ATC system.
Measure, through the evaluation and audit processes, ATC system compliance with established policies, procedures, and requirements.
Identify leading indicators of operational errors and deviations (OE/Ds), near mid-air collisions, controlled flight into terrain, and other safety hazards.
Monitor and report trends of system compliance with established policies, procedures, and requirements.

11. Goals of New Evaluations Process
Knowledge sharing through sharing of mitigation plans across facilities/service areas
Ability to compare facility performance on key safety indicators
Data collection/analysis capability that results in:
Identification of safety hazards
Identification of leading indicators
Communication of Evaluations/Audit findings throughout the ATO

12. Safety Evaluations Data Sharing
Field Customers
-- Service Areas
-- Hubs
-- Facilities
Headquarters Customers
-- OIG / GAO
-- Administrator
-- COO & Executive Council
-- Safety Oversight

13. Facility Internal Evaluations
Purpose is to identify compliance issues and safety hazards at a facility by evaluating and rating a set of checklist items.
Conducted by every Facility in the NAS except Flight Service and NFCTs (approximately 520 facilities)
Completed at least once per year
Due August 1st each year (beginning August 1, 2006)

14. Safety Audits
An audit is a method of assessing an air traffic facility’s performance and compliance with FAA directives and procedures.
Audit is Independent Oversight
Audit is NOT a check of the facility Internal Evaluation
All AT Facilities audited at least every 3 years. Federal Contract Towers (FCT’s) audited every other year.
Conducted by Safety Employees. Support from Service Areas/Hubs.
Audit sites determined through application of criteria
Audits are little or no notice to the Facility.

15. Facility Safety Assessment System (FSAS)
Web-based database tool that supports Safety Evaluations and Audits
Facilitates conduct of facility internal evaluations and audits by serving as a central source of information on requirements as contained in current directives.
Serves as a central collection point of the evaluation/audit data as it is completed.
Serves to facilitate coordination and approval throughout the process.
Plans to expand FSAS to support other safety processes

16. Where NOSS Fits NOSS provides data about normal situations
Identify situations that resulted in safe and efficient operations
Profile effective system performance
A new technique for the Evaluations office toolbox
More in-depth look at a facility or portion of a facility than is provided by current audit process
Provides structured look at operations
Assess precursors to operational errors (leading indicators)
Use results to help target safety improvements

17. FAA NOSS Status
The ATO-P Human Factors Research and Engineering Group focused first on the science of NOSS
Conducted a study of NOSS observer inter-rater reliability at WJHTC (8/06) to assess the consistency in how operational experts make the same interpretations of data
Partial training of 4 Evaluations specialists on the TEM model and NOSS observation approach
Plan to conduct a field trial at an En Route facility in the April/May timeframe

18. Summary
FAA ATO-Safety Evaluations focuses on Safety Assurance portion of SMS (leading indicators)
Most data currently collected on incidents (lagging indicators)
NOSS could provide a more in-depth tool for collecting leading indicators