1. Findings/Analysis/CAPs
Surveillance
Findings/Analysis/CAPs
ATAC 2010 1

2. Transport Canada Civil Aviation Prairie Northern Region
David White
SMS Coordinator PNR
Oversight Coordination Division
Based in Edmonton, Alberta
(780)

3. Contents Define surveillance activities
Discuss surveillance procedures (SI-SUR-001)
Refer to areas of review, SMS, QAP,Ops Control
Analysing Issues identified during surveillance
Causal analysis (root cause)
Generating acceptable Corrective Action Plans (CAP)
Effective communication during the process 3

4. What is an Assessment?
Assessment means the surveillance activity conducted to evaluate effectiveness and level of compliance with the CARs.
Staff Instruction (SI) SUR-001
Sometimes referred to as 17 PVIs.

5. What is a PVI?
A systematic review of one or more SMS components of an organization OR other regulated areas of a certificate holder.
Used to determine if regulatory requirements are documented, implemented, in use and effective within certified organizations.
It is a means to focus on the effectiveness of management systems & make judgments on performance.
Quality Assurance (QA) and the Ops control system are examples of management systems

6. Additional distinction between terms
Assessment – Evaluation of entire SMS
Limited to companies that meet SMS requirements
PVI – Evaluation of regulated area of company’s control system or specific SMS component
Applicable to all companies
Process Inspection – In-depth review of steps to obtain specific output
Our focus will be on Assessments/ PVIs here today.

7. Surveillance Activity Overview
The activity consists of:
Documentation review (preparation)
On-site review (including interviews & sampling)
Analysis
Reporting

8. Surveillance Procedures in SUR-001
Notification
Team selection
PVI plan
Preliminary doc review
Documentation review
Team meeting
Sampling procedures
On-site procedures
PVI team on-site briefing
On-site review
Interviewing
Analysis
PVI worksheets
Findings
Exit meeting
PVI report
PVI CAP procedures
PVI follow-up & closure

9. Program Validation Inspection (PVI)
PVI will review the following areas for certificate holder without Quality Assurance (QA) Program:
Part III - Airport operator’s obligations
Part IV - Company operational control & supervision
Part VII - Subpart 3 or 4 - Company operational control
Part VII - Subpart 5 - Safety Oversight component
Part VIII - Operating certificate requirements

10. Quality Assurance Program
Regulations for certificate holders to have QA
Will be implemented in phased manner
PVIs for Parts III, IV, VII & VIII without fully implemented QA Program use customized PVI worksheets
SUR-001 Appendix A contains expanded list of all PVI worksheets organized by operating rule

11. Quality Assurance Program
Certificate holders that have QAP:
Inspectors use worksheet SUR-001 Appendix A Table 5.1 Quality Assurance
This includes certificate pursuant to:
CAR 561
STD 566
CAR 573
Maintenance requirements for all Part IV & VII operators

12. Some Examples of Worksheets referenced in SI-SUR-001
(k) TABLE 3.1 – Reactive Processes
(l) TABLE 3.2 – Proactive Processes
(m) TABLE 3.3 – Investigation & Analysis
(n) TABLE 3.4 – Risk Management
(p) TABLE 5.1 – Quality Assurance
(r) Program Validation Inspection Worksheet – CAR 302
(s) Program Validation Inspection Worksheet – CAR 406
(t) Program Validation Inspection Worksheet – CAR 703/704
(u) Program Validation Inspection Worksheet – CAR 801
(ee) Program Validation Inspection Worksheet EASA
NOTE: These Worksheets are internal documents only

13. Surveillance Activity Responsibilities
Convening Authority
Identifies - Who
Defines - Scope & depth
PVI/Assessment Manager
Plans
Manages
Team Member(s)
Conduct

14. Documentation Review
Doc review 4-6 weeks before on-site review –Assessment
Doc review 1-2 weeks before on-site review –PVI
Includes review of all files & docs relevant to certificate holder and scope of the surveillance activity
Ensure all reference manuals/docs are available & include latest amendments
Confirm revision status of any manual subject to review
Review approved manuals for conformance to standard
Identify areas that require further review during on-site.
Develop specific questions for certificate holder and appropriate management based on intelligence gathered during review.

15. “A general policy statement is not usually enough to satisfy requirements. Document review team members must verify that there is a policy in place and that it is support by an appropriately documented procedure or process as required ”
This varies dependent on the area being reviewed and the type of certificate issued but is basically the “Write what you do and do what you write” principle.
TC may request additional documentation if the initial submission does not meet the team’s requirement.

16. Onsite Activities Entrance Meeting
Interviews ( Management and selected employees)
Sampling
Based on size, scope and complexity of the organization.
Determined through process in SI-SUR-001
Used in conjunction with interviews to determine regulatory compliance
Non-compliance issues Identified
Purpose of the activity is to determine regulatory compliance.
Company briefing(s)
Exit meeting

17. Surveillance Findings
We will concentrate on PVI examples today.
Ineffective Quality Assurance
Loss of Operational Control
Examples of Non-compliance with regulatory requirements.
Examples of Non-compliance with Company Policy/Procedures and processes.

18. Surveillance Reporting
Report generated with findings.
Required actions determined through review with CA.
Issues are typically handled through the Corrective Action Plan (CAP) process.
Required additional actions are based on the severity of non-compliance issues identified.
Additional actions may involve Certificate Action and/or Enhanced Monitoring.

19. Review of Findings Recognizing the System Failure
Addressing the immediate concerns
Causal Analysis
Planning the short-term corrections
Planning the long term corrections
Informing TC

20. Causal Analysis SI-SUR-001, Section 10.4 CAP Evaluation …
A Root Cause Analysis of the Non-conformance:
The certificate holder’s analysis shall include a summary of the root cause as well as any causal factors that may have contributed to the non-conformance. Even though the certificate holder may use a causal analysis method that is not familiar to the CAP reviewer (e.g.: MEDA process, “5 Why’s”, etc.), the certificate holder must be able to demonstrate how they arrived at the root cause and it should be clear what caused the non-conformance to occur.

21. Causal Analysis
Causal Analysis seeks to identify and understand the reasons why things are as they are and hence enabling focus of change activity.
Some Examples:
The 5 Whys
Meda
Fishbone
The key is just to keep asking until you get to a real 'aha' that you can treat.

22. CAP Submission TC Inspector Expectations Causal Analysis
Appropriate Timelines
Documentation
Communication
SI SUR-001 Section 10.4 (2) states:
*** “The role of the principal inspector responsible for the acceptance of the CAP shall be limited to the assessment of the process used by the certificate holder in reaching conclusions regarding the findings. It is not the inspector’s role to second-guess the CAP solutions, but rather to evaluate the process used in developing those solutions.”***

23. CAP Acceptance and Follow-up
Remember it is your Plan
Monitor the implementation to ensure it is implemented
Communicate Issues if implementation not as per plan.
Document the issues and resolution with the regulator.
Be prepared for the Activity follow-up…Confirming your systems are working not just that the examples identified during the inspection have been addressed

24. Surveillance Activity Closure
Effective systems are a continuing process.
Your responsibility for day to day compliance
Systems will mature and improve as we continue to use them.

25. Thank-you