1. Federal Aviation Administration The United States Government’s USOAP Experience By: Michele Cappelle, National USOAP Coordinator Date: December 15, 2007

2. 2 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 Safety oversight audit process Location Time Frame Activities Montreal Pre-audit Sec Gen letter to State 12 months SPO handover to TL Team meeting Update SAAQ & CC Post-audit Montreal Final report 9 months+ days On-site Audit Opening meeting Closing meeting

3. 3 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/20153 Preparation Recommendations Select a Coordinator with strong team building skills and knowledge of the State’s organizational and legal structure Ensure the Coordinator takes the ICAO Regional Audit Preparation Seminar/Workshop –Outlines USOAP process and ICAO expectations –Opportunity to meet others who have been audited

4. 4 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/20154 Preparation Recommendations Build a strong inter-agency working group to respond to each of the audit areas –Include all State entities involved with safety oversight as well as representatives from the appropriate offices in the CAA –Ensure the legal offices are involved in the coordination and development of State responses

5. 5 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 5 Preparation Recommendations Complete the compliance checklists and the State Activity Questionnaire in a timely manner –Offer clear, accurate and concise explanations –Answers should represent the “official” government position –There are a lot of overlapping and repetitious questions in the State Activity Questionnaire and in the Protocols. Ensure that the answers are compatible

6. 6 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/20156 Preparation Recommendations Develop answers to the audit protocols questions in advance of the audit –For internal use –Coordinate answers –Identify and organize evidence to support answer –Identify the subject-matter-expert for each question –Identify the appropriate person to respond to the auditor for each question

7. 7 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/20157 Preparation Recommendations Brief high-level government officials as well as the CAA staff –Ensure they understand the importance of the audit –Explain the USOAP comprehensive systems approach –Prepare them for the extensive resource demands for the pre-audit activities, on-site audit activities and post-audit activities

8. 8 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/20158 Preparation Recommendations Brief industry representatives for ICAO industry visits –Explain USOAP process and purpose of the visit

9. 9 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/20159 Preparation Recommendations Learn to use and navigate audit tools on the SOA website Learn how ICAO determines a State’s level of compliance with the critical elements –Bias toward less complex civil aviation oversight systems –Ensure high-level officials understand how this number is developed

10. 10 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/201510 Preparation Recommendations Review audit results of other States –Focus on audits which were conducted by the same auditors who will audit your State –Focus on audits of the most comparable oversight systems to your State –Become familiar with typical findings and begin to evaluate whether they would be applicable to your State as well

11. 11 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/201511 Preparation Recommendations Conduct practice audits for each audit area –Use people who have served as ICAO USOAP auditors of other countries –Use compliance inspectors and train them on the conduct of ICAO USOAP audits –Identify officials who will meet with the auditors and have them participate in the practice audits –Identify weaknesses –Ensure internal auditors are familiar with other State’s audit findings

12. 12 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/201512 Preparation Recommendations Manage expectations by developing a credible list of potential audit findings –Ensure that the management team is aware of the potential findings –Avoid findings by correcting weaknesses where possible

13. 13 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/201513 Lessons Learned The State’s USOAP team is critical to the success of the audit Strong internal coordination is essential Develop good communication w/ USOAP team

14. 14 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/201514 Lessons Learned Develop a detailed audit workplan –ensure each auditor is scheduled to meet with the appropriate officials for answers to all protocol questions. This will avoid last-minute surprises and unprepared or unavailable interviewees Insist on organized daily debriefs –Support a clear understanding by all participants –Prevent the development of inaccurate audit findings –Helps manage the IOUs

15. 15 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/201515 Lessons Learned Use all the resources available to understand and prepare for the audit –ICAO SOA website, USOAP team leader, State auditing expertise, audit results of other states Develop a strong internal communication strategy –Widely disseminate information concerning the USOAP –Brief important officials Ensure they are familiar with potential audit findings

16. 16 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/201516 Lessons Learned Train interviewees –On the USOAP process and expected attitude –To stick to the subject –To only answer questions which are in his/her purview –On the limits of their authority to respond to ICAO on behalf of the State

17. 17 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/201517 Lessons Learned Test all components of the audit for understanding and effectiveness –Are the compliance checklists complete and accurate –Do the State’s answers to the State Activity Questionnaire completely reflect the State’s official government response to each issue? –Are the protocol answers fully coordinated and supported by evidence? –Are the interviewees conversant with the protocol answers and evidence? –Do you know with whom the auditors need to meet and are those appointments scheduled?

18. 18 Federal Aviation Administration The United States Government’s USOAP experience December 12, 2007 8/21/201518 U.S. Government USOAP Point of Contact Michele Cappelle National USOAP Coordinator Tel: 202-385-8887 Email: michele.cappelle@faa.gov