Company Confidential Registration Management Committee (RMC) 1 AQMS Accreditation Programs ANAB Findings Atlanta, GA July 22, 2010 Steve Holladay ANAB, Accreditation Assessor Auditor Workshop Atlanta, GA July 22-23, 2010

Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 Objectives Provide an overview of the NCR’s identified from the witness audits. Present in process approach. Discuss steps that could have been taken to prevent the NCR’s. Rules of engagement. 2

Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 Goal Provide knowledge/information to the AS auditor pool with the goal to improve auditor/CB performance that will reduce repeat NCR’s which will, in turn, add value to the assessment process and the industry as a whole. 3

Registration Management Committee (RMC) Atlanta, GA July 22-23, NCR’s in 54 Witnessed Audits (WA) –2.22 Average per WA 50 NCR’s in approx 40 Office Assessments (OA) –1.25 Average per OA Overall

Registration Management Committee (RMC) Atlanta, GA July 22-23, NCR DETAIL: –Assigning Auditors who either did not have the proper NACE code or AS91XX qualifications. –Client not made aware of the OP assessor. LEARNINGS: –Verify auditors are qualified for the full dynamics of the audit. –Communicate clearly with the client. Pre-Audit Assignment

Registration Management Committee (RMC) Atlanta, GA July 22-23, NCR’s DETAIL: 19011, c –(4) The audit plan did not ID the organizations functional units and processes to be audited. LEARNINGS: –The audit plan ensure the client will have the proper resources available to the audit. –Clearly demonstrates what processes are intended to be audited. Pre-Audit Planning

Registration Management Committee (RMC) Atlanta, GA July 22-23, NCR’s DETAIL: –(2) Audit Planning not effective in assuring the AQMS is assessed to the min requirements for surveillance audits. LEARNINGS: –Ensure the auditors understand where the full auditing requirements are for surv audits if not clearly ID in the audit plan. Pre-Audit Planning

Registration Management Committee (RMC) Atlanta, GA July 22-23, NCR’s DETAIL: MD 5 and AS9104 –(5) Insufficient Auditor Days in the planning without clear justification. LEARNINGS: –If deviation…..JUSTIFY. –If the audit is being witnessed, provide to ANAB in the pre-audit information. Pre-Audit Planning

Registration Management Committee (RMC) Atlanta, GA July 22-23, NCR’s, several OFI’s. DETAIL: –Information missing or not correct in EQM. –Client Profile form not complete or accurate. (scope, head count, regulatory, ITAR) LEARNINGS: –Ensure the person entering information is knowledgeable about the information. –Verify with Client and lead auditor before submitting. Pre-Audit Preparation

Registration Management Committee (RMC) Atlanta, GA July 22-23, Total of 24 NCR’s DETAIL: –Sampling part of an element and making a conclusion on the whole requirement. LEARNINGS: –Audit plan and report accuracy is critical. On-site Audit Depth of auditing

Registration Management Committee (RMC) Atlanta, GA July 22-23, DETAIL: –Auditing to the scope of certification is not adequate. LEARNINGS: –Ensuring the scope of certification, scope of audit and audit plan is consistent. –Validate the exclusions. Is scope consistent with the QM. –Ensure the address and scope on the Certificate is correct. On-site Audit Depth of auditing

Registration Management Committee (RMC) Atlanta, GA July 22-23, DETAIL: –Not following audit trails when objective evidence suggest otherwise. LEARNINGS: –A plan is a plan. Follow the trail to its natural conclusion if potential findings are evident. –Keep head on a swivel. Don’t ignore clear findings. On-site Audit Depth of auditing

Registration Management Committee (RMC) Atlanta, GA July 22-23, DETAIL: –Not fully verifying the effectiveness of the actions taken on nonconformities identified during the previous audit. LEARNINGS: –Ensure the evidence is more than a record review. On-site Audit Depth of auditing

Registration Management Committee (RMC) Atlanta, GA July 22-23, NCR’s DETAIL: –Design Applicability LEARNINGS –More of an issue with AS9110. On-site Audit Interpretation

Registration Management Committee (RMC) Atlanta, GA July 22-23, DETAIL: –Not fully auditing Outsourcing when applicable. LEARNINGS –Standard clearly requires “control of such outsourced processes shall be identified” and therefore subject to audit. Not just a document verification. On-site Audit Interpretation

Registration Management Committee (RMC) Atlanta, GA July 22-23, DETAIL: –Changes to the capability listing of the client. LEARNINGS –Similar to the scoping discussion. Verify any changes to the capability listing as this could affect the scope of certification or introduce new processes and technology. On-site Audit Interpretation

Registration Management Committee (RMC) Atlanta, GA July 22-23, NCR’s DETAIL: –Calling the Clients CA/PA system effective with ineffective/inadequate correction, root cause or corrective action. LEARNINGS: –Deep dive the CA/PA system. Validate the information. If it doesn’t make since….make the call. On-site Audit Decisions

Registration Management Committee (RMC) Atlanta, GA July 22-23, DETAIL: –Soft Grading 8 NCR’s –Clearly stating findings during the audit and not raising the NCR or improper categorization. – Accepting informal correction during the audit. LEARNINGS –Learn the definitions and follow them. Keep track of the “verbal” findings identified during the audit. Accepting correction to verbal findings without reporting is consulting. –As an auditor it is not your job to justify an NCR down. On-site Audit Decisions

Registration Management Committee (RMC) Atlanta, GA July 22-23, DETAIL: –Continuing the audit when the audit objectives are clearly unattainable. LEARNINGS: –Only the client can make the decision to continue an audit. –There should be a clear conclusion that the objectives are unattainable and reported to the client with options. »At the time this is realized not at the end!! On-site Audit Decisions

Registration Management Committee (RMC) Atlanta, GA July 22-23, DETAIL: –Closing meeting does not address all of the requirements. LEARNINGS –Clearly address the CA follow up and impacts on the existing cert. –Ensure the CA are presented to the correct requirements. On-Site Audit Closing

Registration Management Committee (RMC) Atlanta, GA July 22-23, NCR’s DETAIL: –Accuracy of the Audit Report and Checklist to the observed audit. LEARNINGS –Anecdotal evidence should be documented as such. Same for deductive evidence. On-site/Off-site Audit Reporting

Registration Management Committee (RMC) Atlanta, GA July 22-23, DETAIL: –Report not in conformance with AS9014/AS91XX LEARNINGS –Justify/explain all NA. –Clear evidence to support the conclusion. –Clear evidence to support the SCOPE. –Include detailed notes On Site/Off-Site Audit Reporting

Registration Management Committee (RMC) Atlanta, GA July 22-23, NCR’s DETAIL: –Report did not contain mandatory items. LEARNINGS –Ensure the Checklist are complete or references the location of the information. –Explain and differences between the information left on site to the published report. Post Audit Reporting

Registration Management Committee (RMC) Atlanta, GA July 22-23, NCR’s DETAIL: Improper closing of NCR’s and Improper Certification Decision. LEARNINGS: –Clear evidence of Correction, RC and CA. –Ensure persons closing NCR are AEA. –Ensure the NCR’s are closed with appropriate evidence PRIOR to the cert decision. Post Audit NCR Closure

Registration Management Committee (RMC) Atlanta, GA July 22-23, 2010 Office Audit NCR’s OASIS data base admin/accuracy Verification of Client OASIS data admin Justification of auditor days. Improper qualification of AS cert decision maker. Improper decision made with certification information. 25

Registration Management Committee (RMC) Atlanta, GA July 22-23, Questions