Air Carrier Continuing Analysis and Surveillance System (CASS) Verifying Performance of the Maintenance Programs

Objectives Determine how to assess the performance of the maintenance program Identify “Performance” processes within the air carrier’s maintenance program Describe the carrier’s procedures in its CASS for monitoring performance of its maintenance program

Overview Review: Audit types and procedures Audit plan Audit process Air carrier’s responsibility for CASS Maintenance program goals Major CASS activities Audit types and procedures Audit plan Audit process Audit analysis

Maintenance Program Goals As stated in CFR 14, Sections 121.367 and 135.425: All maintenance performed by the air carrier or by other persons is performed according to the air carrier manual.  Competent personnel and adequate facilities and equipment are provided for the proper performance of maintenance. Each airplane released to service is airworthy and has been properly maintained for operations in air transportation.

Air Carrier’s Responsibility Maintain a system for the Continuing Analysis and Surveillance System (CASS) to measure the performance and effectiveness of its maintenance program Culture of safety helps establish a properly functioning CASS Properly functioning CASS should include an audit function to: Review the actual work performed within its maintenance program

The CASS Cycle CASS monitors maintenance program performance and effectiveness through a closed-loop system of four major activities.

Verifying Maintenance Program Performance

Surveillance: Audits Process What do Audits ask: Are they doing what they are supposed to be doing?

For purposes of CASS, and in line with general audit methodology, an audit is defined as a formal examination of the activities of a department or area of an air carrier’s maintenance program based on compliance with the standard, which is the air carrier’s manual. Remember that the one of the requirements of air carrier programs is that the program itself must ensure that all three of those regulatory objectives are being achieved. KEY POINT Ask the class what the three regulatory objectives are? What Is an Audit? An audit is a formal examination of the activities of an air carrier’s departments or areas as compared to a standard, which is the air carrier’s program as written in its manual.   Audits are intended to measure an air carrier’s compliance with maintenance program requirements. The maintenance program itself must ensure that all maintenance activities are accomplished in accordance with the processes and procedures of the maintenance program. §§121.367 and 135.425

Types of Audits These are the four types of audits usually utilized by air carriers. The four items are self-explanatory. Generally, the differences are who accomplishes the audits and the different audit targets. Internal Audits are performed by air carrier audit personnel on air carrier activities. External Audits are performed by air carrier personnel on activities of outside entities or contract maintenance providers. Self-Audits are performed “in-house” by a specific individual or position within a department or shop. Third Party Audits are performed by third parties (FAA, C.A.S.E., DOD, etc.) on the carrier.

Audit Methods These are the audit methods and their objectives that are used in one or more of the four audit types that we just discussed in the last slide. The primary type of audit method that we expect the air carrier to use is the work-in-progress method. The purpose of these audits is to determine if the worker is following the manual. This is the §121.367(a) and §135.425(a) requirement. A negative finding is a program deficiency under the CASS rules and must be addressed. This follows the plain language meaning of the CASS regulation, i.e., the performance of the maintenance program; are they doing what the program requires. The transaction method is used primarily for records review and again serves to see if the maintenance program standard for the record form, record procedures, record completion and record accuracy standards of the program are being achieved. The Systems method is a high level, comprehensive, and documented examination of all of the activities, records, processes, and other elements of an air carrier’s various systems to determine their conformity with the requirements of a standard such as the air carrier’s maintenance program. These audits also identify various latent faults in the air carrier’s overall systems. These audits are usually, but not always, accomplished by a professional audit agency and result in a written and detailed report that is used by senior management to make corrections to the systems. Work-in-Progress Audits examine adherence to the execution of the maintenance program (“As the task is being performed”). This is a primary proactive audit method. Transaction Audits check compliance against a standard or requirement, usually an after-the fact verification such as a records review. Systems Audits are independent, periodic and documented examinations of all of the activities, records, processes, and other elements of an air carrier’s various systems to determine their conformity with the requirements of a standard such as the air carrier’s maintenance program.

Planning Audits Risk-based resource allocation based on company experience Software-supported risk analysis Planning process should systematically make decisions compatible with the size and complexity of operations

Planning Audits (continued) Air carrier’s audit plan should include: What should be audited or collected Who performs audits or data collection How audits and data collection should be performed How often audits and data collection should occur

Auditors Thorough knowledge of auditing and the air carrier environment Should be able to objectively identify and document deficiencies

Planning Priority Regulatory compliance (promotes safety) Operational efficiency Consider company experience and outside factors, such as: Inspections – “OIG” Reports – “NTSB” Special Studies – “FAA” DoD – “DOT”

Audit Verification Government Publications Program Description Flight Operation Ground Operations Maintenance Alterations Conformity Repairs Government Publications Industry Documents Operator Documents Manufacturer Documents Program Description Records Reliability Scheduling/Tracking Aircraft Loading Deicing Taxiing and Towing Fueling e.g., regulations, airworthiness directives e.g., ATA 100 index e.g., General Maintenance Manual e.g., maintenance manual, service manual

Red Rock Airline: D091 Quarterly Audit v 3.0

Effective Communication within CASS Maintenance personnel should be able to discuss concerns they may have Auditors should: Ask questions not included in the checklists Adopt a “what if?” proactive approach during surveillance activities

CASS Audits Proactive – search for potential problems before undesired events Responsive – procedures established to address unscheduled audits driven by findings and events (IFSDs, accidents, incidents, negative trends) Productive – encourage open lines of communication throughout the organization Priority – regulatory compliance, efficiency

Verifying Maintenance Program Performance 22

Objectives of Audit Analysis Identify the maintenance program deficiencies. To accomplish that: Audit information should be factual Who? What? When? How? Make no prejudgments about root cause Analysis should be: Objective Systematic

What to Watch For Blame culture is an obstacle to improving safety.

What Is a Blame Culture? Blame Fixes the blame, move on… Focuses on the individual who made the error Stops short of identifying systemic problems Stops short of identifying root cause Never fixes the problem Tends to allow mishaps/mistakes to recur Blame

Inspector’s Oversight Determine desired outcome – Review process Does the process contain data points? Do data points provide useful data for analysis (are data repeatable)? Collect data – Procedures to perform the audit Review data How are data used? How are data analyses documented? Do the data support the air carrier’s finding? How are data analyses results communicated? (How do results get to the next step?)

Inspector’s Oversight (continued) How does the carrier analyze audit results? Identify deficiencies Can either be a real or a potential hazard? Are audit results used in part to allocate resources? Do audit procedures include calling upon additional technical expertise if required?

Practice Exercise PE-01-PER

Practice Exercise PE-02-PER

Summary Review: Audit types and procedures Audit plan Audit process Air carrier’s responsibility for CASS Maintenance program goals Major CASS activities Audit types and procedures Audit plan Audit process Audit analysis

Objectives Determine how to assess the performance of the maintenance program Identify “Performance” processes within the air carrier’s maintenance program Describe the carrier’s procedures in its CASS for monitoring performance of its maintenance program

QUESTIONS ?