7 So We Went Looking For The Big Picture .Some data only shows the tip of the problem.Multiple databases may provide a lot of information but what does it all mean?What is needed to identify better precursors?Photo courtesy of Claudia VIRLAN, ANSS Directorate, Romanian CAA
8 Sources of Leading Indicators Data Collected During Normal Operations—not as the result of an Incident/AccidentRevamped Air Traffic Facility Evaluations/Audit ProcessNOSS
9 Safety Evaluations Serve the Assurance Component of SMS Supporting RequirementsATO must continue to meet its performance targets for safetyReduce operational errorsReduce runway incursionsTo date, ATO collects the number, rate, and severity of these incidents (“lagging” safety indicators)ATO does not consistently collect data that may identify precursors to these incidents (“leading” safety indicators)Continual tracking and analysis of lagging and leading indicators will drive toward the identification of mitigation strategies that have a greater impact on the achievement of safety performance targetsPolicyArchitectureFAASafety ManagementSystemAssuranceSafetyPromotionAssurance Component of SMS1Safety assurance and evaluations2Safety data tracking and analysis
10 Safety Evaluations Purpose/Mission Conduct audits, program assessments, operational assessments, and facilitate evaluations of ATC facilities to promote safety, quality, and efficiency of the ATC system.Measure, through the evaluation and audit processes, ATC system compliance with established policies, procedures, and requirements.Identify leading indicators of operational errors and deviations (OE/Ds), near mid-air collisions, controlled flight into terrain, and other safety hazards.Monitor and report trends of system compliance with established policies, procedures, and requirements.
11 Goals of New Evaluations Process Knowledge sharing through sharing of mitigation plans across facilities/service areasAbility to compare facility performance on key safety indicatorsData collection/analysis capability that results in:Identification of safety hazardsIdentification of leading indicatorsCommunication of Evaluations/Audit findings throughout the ATO
12 Safety Evaluations Data Sharing Field Customers-- Service Areas-- Hubs-- FacilitiesHeadquarters Customers-- OIG / GAO-- Administrator-- COO & Executive Council-- Safety Oversight
13 Facility Internal Evaluations Purpose is to identify compliance issues and safety hazards at a facility by evaluating and rating a set of checklist items.Conducted by every Facility in the NAS except Flight Service and NFCTs (approximately 520 facilities)Completed at least once per yearDue August 1st each year (beginning August 1, 2006)
14 Safety AuditsAn audit is a method of assessing an air traffic facility’s performance and compliance with FAA directives and procedures.Audit is Independent OversightAudit is NOT a check of the facility Internal EvaluationAll AT Facilities audited at least every 3 years. Federal Contract Towers (FCT’s) audited every other year.Conducted by Safety Employees. Support from Service Areas/Hubs.Audit sites determined through application of criteriaAudits are little or no notice to the Facility.
15 Facility Safety Assessment System (FSAS) Web-based database tool that supports Safety Evaluations and AuditsFacilitates conduct of facility internal evaluations and audits by serving as a central source of information on requirements as contained in current directives.Serves as a central collection point of the evaluation/audit data as it is completed.Serves to facilitate coordination and approval throughout the process.Plans to expand FSAS to support other safety processes
16 Where NOSS Fits NOSS provides data about normal situations Identify situations that resulted in safe and efficient operationsProfile effective system performanceA new technique for the Evaluations office toolboxMore in-depth look at a facility or portion of a facility than is provided by current audit processProvides structured look at operationsAssess precursors to operational errors (leading indicators)Use results to help target safety improvements
17 FAA NOSS StatusThe ATO-P Human Factors Research and Engineering Group focused first on the science of NOSSConducted a study of NOSS observer inter-rater reliability at WJHTC (8/06) to assess the consistency in how operational experts make the same interpretations of dataPartial training of 4 Evaluations specialists on the TEM model and NOSS observation approachPlan to conduct a field trial at an En Route facility in the April/May timeframe
18 SummaryFAA ATO-Safety Evaluations focuses on Safety Assurance portion of SMS (leading indicators)Most data currently collected on incidents (lagging indicators)NOSS could provide a more in-depth tool for collecting leading indicators