1. Americas Aerospace Quality Group Registration Management Committee
Boeing Supplemental Oversight Assessment Results
January 22, 2008
Presented by: Brian Simons
Boeing Oversight Representative
The Boeing Company

2. Topics for discussion Boeing supplemental oversight process
Assessment Activities
Strengths and Issues
Corrective Action Expectations
Assessment results
Statistics
Significance
Corrective Action Responses
Questions and comments

3. Boeing Supplemental Oversight Process
Supplemental assessment activities:
Semi-annual CB performance review to identify 6 month planned activity
Risk-Based assessment for each CB using:
- Aggregate client (supplier) performance (product and QMS)
- Previous Boeing oversight activity
- Total quantity and change of client base
Ad Hoc or unplanned (Throughout the year)
- Primarily based on unsatisfactory supplier performance
- BOR (Boeing Oversight Representative) Support Requests
- Management requests
- Field Representative questions and inquiries
Approximately 80% of Boeing’s supplemental oversight is Ad Hoc

4. Boeing Supplemental Oversight Process Jan – June 2008 - America’s Sector - Oversight Opportunities
Y Axis = Influence
X Axis = Performance

5. Boeing Supplemental Oversight Process
Types of supplemental oversight activity:
OASIS database and certificate information validation
Audit report review (Audit summary, AS9101 checklist, non-conformances, corrective action, etc.)
Client validation (on-site, including audit report review)
- Required for risk CBs identified on RAT
CB auditor witness audit
CB/ANAB office audit

6. Boeing Supplemental Oversight Process
Strengths and Issues (S&I) document:
Issue (CA required) or Comment
Issues documented in a 3 part format:
Requirement – Reference clause number and verbatim statement of requirement (e.g. AS9104 clause 8.4 The AQMS audits shall be documented on the Aerospace Assessment Report….).
Issue (Nonconformance) – Restatement of requirement in the negative (e.g. The AQMS audits are not documented on the ….)
Evidence – Documentation of evidence collected, proceeded by “For example” (e.g. For example, no evidence of completed AS9101 checklist for surveillance assessment # 123, dated 1/22/08)
The BOR team will begin providing ANAB with a courtesy copy of all S&Is issued to CBs beginning 2008

7. Boeing Supplemental Oversight Process
Sample S&I - Issue #1
Requirements:
AS9104 clause 8.3 Nonconformity: No certificates or approvals to AQMS Standards or any combination of AQMS Standards with ISO 9001 shall be issued unless all major and minor nonconformities have been satisfactorily corrected with the root cause analysis and corrective action verified by the CRB.
IAF Guidance on the Application of ISO/IEC Guide 62: G The implementation of the corrective action should not be deemed to have been completed until its effectiveness has been demonstrated and the necessary changes made in the procedures, documentation and records.
Issue:
The CRB does not verify the satisfactory implementation, completion and effectiveness of corrective action for nonconformities, including demonstration that the necessary changes were made in the procedures, documentation and records, as necessary.
Evidence:
For example, The client corrective action for NCRs #2 and #3 from the recertification audit were not completed prior to CB X closing the NCRs. NCR #2 client corrective action states that inspector training will be completed by February 15, The training record provided by the client for this action is dated 4/4/2007. NCR #2 was closed on 2/5/07. NCR #3 client corrective action states that before February 10, 2007 the client will implement a revision to the procedures manual addressing AS9100 clause As of November 1, 2007, there is no evidence of a revision to the client's Quality Procedure Manual for clause (Procedure does not exist) or the Quality Assurance Manual (current revision December 2003). NCR #3 was closed on 2/5/07.

8. Boeing Supplemental Oversight Process
S&I Corrective Action expectations (5 part format):
Immediate Correction
- Address and correct the evidence
- Identify the Direct Cause
- Notify affected personnel
Root Cause Statement
- Identify fundamental process deficiency
- Personnel error or mistake is unacceptable
Root Cause Corrective Action (CA) Plan
- Address the Root Cause
- Identify what, who and when
- Include reference to revised QMS policy, process, etc. (title/number)

9. Boeing Supplemental Oversight Process
S&I Corrective Action expectations (5 part format):
(Continued)
Verification of Corrective Action Plan
- Verify RCCA actions have been completed per commitments
- QMS process updated, training completed, clients assessed, etc.
Follow-up audit
- Determine of CA plan is effective in preventing recurrence
- Conducted 6-9 months following verification

10. Boeing Oversight Assessment Results
BOR oversight activity from January 1, 2006 through December 31, 2007:
19 out of 34 domestic Certification Bodies Assessed (CBs associated with Boeing suppliers)
CBs assessed based on RAT and Ad Hoc requests
69 out of 79 oversight activities resulted in at least 1 issue
(87% rejection rate)
32 - OASIS database and Certificate validations
26 - Client Validations (includes audit report review)
9 - Audit Report Reviews (Desktop review)
1 - Office Audit
1 - Witness Audit
143 total issues identified (Nonconformance to industry requirement)

11. Boeing Oversight Assessment Results

12. Boeing Oversight Assessment Results
Supplemental oversight from January 1, 2006 through December 31, 2007 :
Inadequate QMS assessment (31)
OASIS update > 30 days (21)
Audit report deficiency (16)
Multi-site issue (13)
Assessment days below minimum requirement (11)
CB contract deficiency (7)
Certificate deficiency/error (7)
Certificate issued with incomplete client corrective action (7)
NC soft grading (6)
OASIS errors (5)

13. Boeing Oversight Assessment Results
Supplemental oversight from January 1, 2006 through December 31, 2007 :
Certificate validity
Inadequate QMS assessment (31)
Audit report deficiency (16)
Multi-site issue (13)
Assessment days below minimum requirement (11)
Certificate issued with incomplete client corrective action (7)
NC soft grading (6)
CB Administration
OASIS update > 30 days (21)
CB contract deficiency (7)
Certificate deficiency/error (7)
OASIS errors (5)

14. Boeing Oversight Assessment Results
Inadequate Quality Management System Assessment (31):
Ineffective QMS management processes (Client validation)
Management Review, Quality Objectives, Internal Audit, Corrective Action, Analysis of Data, Continual Improvement, etc.
Ineffective product control processes (Client validation)
Configuration Management, Production Documentation, Identification, Accountability, Control of Nonconforming Product, etc.
Completed audit reports lack sufficient evidence to justify “satisfactory” result
- Prompts Boeing oversight investigation and results comparison
Unsatisfactory Boeing supplier performance
Compare Boeing 2nd party QMS audit results vs. CB audit results
Prompts Boeing oversight investigation

15. Boeing Oversight Assessment Results
Certification Body Corrective Action Response:
Corrective action response due within 30 days (typical)
Initial submittal rejection rate approx %
- 5 part format not followed
- Evidence cited in S&I not addressed and corrected in Immediate Action
Root Cause identifies human error or mistake
Root Cause is an explanation or rationalization of the condition
Root Cause Corrective Action does not address Root Cause
Root Cause Corrective Action does not address long term process improvement
CA Verification and Follow-up are confused and mixed together
BOR policy is to elevate unsatisfactory S&I CA to ANAB for resolution
- 2 S&Is were elevated in 2007 for ANAB investigation and action

16. That’s all folks!
Questions or Comments?