USOAP Comprehensive Systems Approach By COSCAP-NA USOAP Comprehensive Systems Approach Module 3 Introduction to USOAP

Contents From an Annex-based to systems-based audits The Comprehensive Systems Approach Tools used in the implementation of the comprehensive systems approach Module 3 Introduction to USOAP

An Annex by Annex-based approach Former auditing practices: Assesses the implementation of specific Annexes at a time; A lengthy process which may take decades to address all ICAO Annexes; Does not give a clear picture of the overall safety level provided by the aviation system in the State (inter-relationship between Annex provisions may not be evident); Expensive to implement effectively. Module 3 Introduction to USOAP Introduction to USOAP

An Annex by Annex-based approach An Annex by Annex-based approach was practiced from 1996 – start of the voluntary assessment to 2004 – end of first cycle of mandatory audits Experience shows that it was successfully implemented and served the intended purpose: Raised State awareness of their safety oversight responsibilities; Provided reliable information on the status of implementation of ICAO SARPs; Enabled the identification of deficiencies in the global safety of aviation activities in the three areas Module 3 Introduction to USOAP Introduction to USOAP

Graphic presentation of the results of the first audit cycle Initial Audit - 181 Contracting States (32.62%) Audit Follow-up – 181 Contracting States (17.46%) 40 30 20 10 Primary Aviation Legislation Specific Operating Regulations Civil Aviation System And safety Oversight Functions Qualification and Training of Technical Staff Procedures and Technical Guidance Licensing and Certification Obligations Surveillance Obligations Resolution of Safety Issues Module 3 Introduction to USOAP

An Annex by Annex-based approach The challenge represented: Aviation activities are interrelated. A problem in one area affects the “overall” safety of the whole system; Safe aircraft operation is not limited to licensing, operations and airworthiness; State oversight capabilities with respect to other aviation activities have an equal impact on safety; Module 3 Introduction to USOAP Introduction to USOAP

An Annex by Annex-based approach The challenge represented: Continuing on an Annex-by-Annex approach would have been very lengthy and expensive; It would also represent an administrative problem, replacing experts on a three-year or six year cycle; Overall picture of the health of the aviation system would have been impossible to assess. Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach A comprehensive systems approach (CSA) refers to: “the implementation of a structured process and methodology for the planning, preparation, conduct, reporting, follow-up and evaluation of ICAO safety oversight audits.” Module 3 Introduction to USOAP

Comprehensive systems approach A structured approach enabling an audit process, methodically conducted and documented to ensure: Standardization, Traceability, and Accountability Compatible with USOAP Principles Module 3 Introduction to USOAP

Comprehensive systems approach A systems approach: Looks at States’ safety oversight systems from a process perspective; Provides a timely picture of the overall health of a State’s safety oversight system and its effectiveness; Focuses attention on critical safety areas; Allows for flexibility depending on the size and complexity of the aviation system established in a State. Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Benefits of a systems approach: Enables the assessment of a State’s capability for safety oversight; Enables the assessment of the effective implementation of the critical elements of a safety oversight system; Evaluates the overall aviation safety system established and avoids a piecemeal approach of evaluating aviation safety Standards; Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Benefits of a systems approach: Promotes uniform implementation of international safety Standards; Promotes the building confidence among States in that each State would be aware of the other States’ capability for safety oversight; Overall, it will significantly contribute to the safety and efficiency of international air transport system. Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Implementation required the development of a plan, processes, procedures and specific tools to be used both by ICAO and Contracting States It also required the establishment of a Programme which is: Fully transparent; and Works closely with all Contracting States. Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Endorsed by Assembly Resolution A35-6 (2004) USOAP to be further expanded to include the safety-related provisions in all safety-related Annexes; USOAP to adopt a comprehensive systems approach in conducting safety oversight audits; PEL, OPS, AIR, ATS, AGA and AIG to constitute the core elements (subjects) of the audit Programme. Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Endorsed by Assembly Resolution A35-6 (2004) Audit reports to reflect the critical elements of a safety oversight system; Final audit reports to be made available to all Contracting States; USOAP to continuously ensure the maintenance of the quality assurance system. Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach States (Assembly Resolution A35-6 (2004)) to: Second qualified and experienced technical staff to ICAO on long- or short-term basis; Submit, on a timely manner, and keep up-to-date, all information and documentation associated with the preparation and conduct of an audit; Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach States (Assembly Resolution A35-6 (2004)) to: Cooperate with ICAO and, as much as possible, accept audit missions as scheduled by ICAO; Accept the primacy of USOAP audit results as meeting the established SARPs; Secretary General to foster coordination and cooperation between USOAP and other audit programmes (IATA, EUROCONTROL, etc.) (A35-7 refers). Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach (core audit areas) Aircraft Operations Annexes 6, 18 PANS-OPS Personnel Licensing Annex 1 Airworthiness of Aircraft Annexes 7, 8, 16 Effective State Safety Oversight System Organization & Safety Oversight Functions SAAQ Legislation and Regulations Convention Annex 2 Aerodromes Annex 14 Accident & Incident Investigation Annex 13 Air Navigation System Annexes 3,4,5 10,11,12,15 & PANS-ATM Module 3 Introduction to USOAP

Comprehensive systems approach Audit processes and procedures: A three-phase audit process established (to be discussed phase-by-phase, in Modules 5, 6 and 7); Process and procedures highly transparent and available to all Contracting States; Initial planning/scheduling to the publication of a final safety oversight report would take approximately two years to complete. Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Selection criteria for States to be audited: Expressed urgency to audit a particular State Requests by a Contracting State Size and complexity of aviation activities Results of previously conducted audits Regional balance Aircraft accident and incident rates Completion of documentations required for the audit (auditors’ tools) Language requirements Proximity to a State selected for an audit Module 3 Introduction to USOAP

Comprehensive systems approach Audit tools: Audit tools are used throughout the three phases of the audit process starting with the initial planning stage to the publishing of the final audit report; Audit tools developed include: SOA Quality Manual; State Aviation Activities Questionnaire (SAAQ); Compliance Checklist (CC); and Audit Protocols Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach State Aviation Activities Questionnaire (SAAQ) Designed to enable ICAO to collect useful information on the organization and system established by a State to meet its safety-related obligations as a signatory to the Convention on International Civil Aviation. Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach State Aviation Activities Questionnaire (SAAQ) Completed SAAQs to be submitted to ICAO by 31 May 2005 (extended for some States on request); States to update information contained in their respective SAAQ whenever there is a change in their civil aviation system; An updated SAAQ to be submitted at least 90 days prior to the conduct of the on-site phase of the audit; SAAQ questionnaire available at: http://www.icao.int/icaonet/ Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Contents of the State Aviation Activity Questionnaire (SAAQ): Part I ─ General administrative information Part II ─ Legislation Part III ─ Organization Part IV ─ Operational activities Part V ─ Air navigation services Part VI ─ Aerodromes Part VII ─ Aircraft accident and incident investigation Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach SAAQ, Web-based version Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Compliance Checklist: A tool which will provide ICAO with information on a State’s level of implementation of international Standards and Recommended Practices (SARPs); A tool which will enable States to identify any difference which may exist between their own practices and those established by international standards (Article 38 of the Chicago Convention) Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Compliance Checklist: States submitting a completed compliance checklist will enable ICAO to maintain a database on a States level of compliance to the ICAO SARPs; States are required to maintain the compliance checklist up-to-date and/or update the compliance checklist at least 90 days prior to the scheduled audit. Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Compliance checklist contents Standards and Recommended Practices contained in all safety-related Annexes (all ICAO Annexes except Annexes 9 and 17); Each Annexes compliance checklist starts with a Foreword, providing States with instructions on how to complete the document; The Foreword also defines the categories to be considered in determining differences to be identified and submitted to SOA; Currently, CCs are used for auditing purposes. Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Compliance Checklist – an example Provision for comments Level of compliance SARP Identifier Annex Reference Annex SARP Legislation Reference Text of difference Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Compliance checklist, Web-based version Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Audit Protocols (PQ): Developed by SOA with the assistance of relevant ANB Sections; Used for the conduct of the on-site audit; Enables auditing against the critical elements of a safety oversight system; Provides guidelines to the auditor on what evidence should be reviewed; Could also be used by States to conduct an internal audit (Doc 9735). Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Audit Protocols (PQ): Use of standardized protocols as the primary tool for the conduct of the on-site audit ensures transparency, consistency and standardization; It enhances confidence and reliability in the conduct of audits; Designed to reflect the core areas of the audit Programme. Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Areas covered by the Audit Protocols: Primary Aviation legislation and civil aviation regulations State aviation system and safety oversight functions Personnel licensing system Operation of aircraft Airworthiness of aircraft Aircraft accident and incident investigations Air navigation services Aerodromes Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Audit Protocols – an example ICAO reference Aspects to be audited or questions to be answered Status Example of evidence to be reviewed Implementation as applicable Comments CE Auditor Assessment OPS 4.000 – Legislation and regulations for aircraft operations State Response GM Doc 8335 1.2.1 & 2.1.2 Doc 9734 Part A 3.3 OPS 4.001 Has the State developed and promulgated specific operating regulations applicable to aircraft operations to enable the State to implement the provisions of annex 6?   “ Yes “ No Title and date of applicability of all regulations related to operation of aircrafts. Review the compliance checklist.     2 “ Satisfactory “ Not satisfactory “ Not applicable Auditor Notes GM Doc 8335 1.2.1 & 2.1.2 Doc 9734 Part A 3.3 OPS 4.002 Has the State developed and promulgated specific operating regulations applicable to aircraft operations to enable the State to implement the provisions of annex 18?     “ Yes “ No Title and date of applicability of all regulations related to operation of aircrafts. Review the compliance checklist.     2 “ Satisfactory “ Not satisfactory “ Not applicable Audit Protocol Question Review Evidence OPS 4.060 – Availability of qualified technical personnel and training GM Doc 8335 9.3.1 & 9.4 Doc 9734 Part A 3.6   OPS 4.062 Has the CAA established minimum flight and work experience to become operations inspectors?   “ Yes “ No Review recruitment requirements. This can include: 1. Five or more years in air transport 2. 5.000 flight hours, 3. Training experience, etc.     4 “ Satisfactory “ Not satisfactory “ Not applicable Applicable Critical Element Source Reference Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Additional/Support Tools - Guidance material: Designed to assist States in the implementation of SARPs and associated Procedures: Doc 9734 Part A – provides guidance on the establishment and management of a State’s safety oversight system (2nd Edition, 2005) Doc 9734 Part B – provides guidance on the establishment and management of a regional safety oversight system (1st Edition, 2005) Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Additional/Support Tools - Guidance material: Designed to assist States in the implementation of SARPs and associated Procedures: Doc 9735 – provides detailed information on USOAP and on the conduct of safety oversight audits (2nd Edition, 2005) QMSP-007-Programme Management and Implementation Provisions (SOA Quality Manual, updated on continuous basis) Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Additional/Support Tools - Seminars and workshops One of the main tools used to assist States to establish an effective safety oversight system; A tool for providing information and “how to” to States in their preparation for an ICAO audit; A means of sharing knowledge and experience. Module 3 Introduction to USOAP Introduction to USOAP

Comprehensive systems approach Safety oversight audit process: Pre-audit Audit Post audit On-site Location Montreal Montreal Sec Gen letter to State Opening meeting Team meeting Final report Activities Update SAAQ & CC Closing meeting SPO handover to TL Time Frame 12 months days 8 months+ Module 3 Introduction to USOAP

Comprehensive systems approach The audit process feedback loop Process Analysis of Differences Determination of level of SARPs Implementation Analysis of Audit Findings Determination of State capability for Safety Oversight Output Recommendations Audit Report Resolution of identified deficiencies Annex supplement Input SAAQ Compliance checklist Protocols Other sources The Porter’s competitive force model. Feedback Loop Module 3 Introduction to USOAP Introduction to USOAP

Review The Annex-based system The Comprehensive Systems Approach Tools used in the implementation of the comprehensive systems approach The overall process Module 3 Introduction to USOAP

Questions ? INTRODUCTION TO USOAP By COSCAP-NA Module 3 Introduction of C-WP/12006 to Council Thank you Mr. President, and good morning everyone, C-WP/12006 presents the overall progress made in the implementation of the ICAO Universal Safety Oversight Audit Programme since its inception and in particular during the period from January to March 2003. Questions ? Module 3 Introduction to USOAP