1. Dan Conlon 951.273.5210 daniel.conlon@navy.mil 2008 Auditor Workshop “Output Matters” 14 July 2008

2. 2 Agenda AS9100 Importance to MDA MDA Assurance Provisions (MAP) MDA Audit Process AS9100 Findings Analysis Examples of AS9100 Audit Findings Common AS9100 Issues How You Can Help Conclusion

3. 3 AS9100 Important to MDA AS9100 is a key QMS document – MDA Core standard The MAP builds on AS9100 requirements/disciplines to create a very rigorous set of QMS implementation requirements MAP includes additional Quality, Safety, and Mission Assurance (QSMA) disciplines for developing mission and safety critical items such as: –Systems Engineering –Reliability, Maintainability, and Availability –Parts and Materials Management –Software –Risk Management –Configuration Management –Safety and Environmental Management –Interface Management

4. 4 MDA Assurance Provisions (MAP) A standardized set of 14 QSMA provisions, requirements, and processes for mission and safety critical items: –Authored by joint Gov/Industry Team –Goal is to implement on all MDA contracts –160 pages, 1800+ shall statements Provide methods to measure, verify, and validate mission success through: –Collection and analysis of metrics –Risk Assessment, technical evaluations, independent assessment, reviews –Conduct of qualification, ground, and flight tests Promote continual process improvement to ensure QSMA, reduce cost, and improve productivity Proven Mission Assurance Requirements on Contracts MDA Assurance Provisions (MAP) 3.1 Management 3.2 Design and Development 3.3 Software and Firmware 3.4 Technical and Mission Assurance Reviews 3.5 Reliability, Maintainability, and Availability 3.6 Parts and Materials Control Program 3.7 Integrated Test and Evaluation Program 3.8 Test, Meas., and Diagnostic Equip. & Stds. 3.9 Interface Management 3.10 Configuration Management 3.11 Control of Nonconforming Items & Materials 3.12 Fabrication and Quality 3.13 Supplier Management 3.14 Safety

5. 5 MDA MAP Program Usage As part of the MDA Acquisition Strategy the MAP was developed for MDA Program Office implementation and flow down to prime contractors and suppliers –Use existing processes, when adequate, to achieve and sustain mission success –Tailored by the MDA Program Offices for each program –Further tailored for subsystem components through supplier management Implementation of MAP (including AS9100) assessed via: –Mission Assurance audits –Supplier audits –On-Site Mission Assurance Representatives (MARs) –Focus Teams – (VAFB Flight Test Team, PMRF, NGST)

6. 6 Why Audit? QS Organization founded in Late 2002 Majority of Failures Were Lack of Process Discipline 199719981999200020012002200320042005200620072008 Aegis NMD/GMD THAAD FM-5 6/03FTR-1 7/00 IFT-01 1/97 IFT-04 1/00 IFT-05 7/00 IFT-10 12/02 IFT-13c 12/04 IFT-14 2/05 FT-07 3/97 FT-08 5/98 FT-09 3/99 No Interceptor Failures BV-2 8/01 BV-3 12/01BV-5 1/04 22 Flight Test / 10 Failures (45% failure rate) 32 Flight Test / 4 Failures (13% failure rate) 17 Flight Test / 0 Failures (0% failure rate) First MDA/QS Mission Assurance Audit performed

7. 7 GMD Flight Test-03a 28 Sept 07 Engages -Target launch from Kodiak -Interceptor launched VAFB -In midcourse phase of flight -EKV discrimination

8. 8 MDA QS Audit Program Purpose –Conduct Mission Assurance audits of MDA prime contractors and suppliers MDA Audit Objectives –Validate that the end item at a selected level conforms to engineering design requirements –Validate that the end item at a selected level is qualified –Compare fabrication methods and acceptance test levels against MDA and industry standards as appropriate for MDA use –Validate production readiness –Assess contractor/government Mission Assurance practices, procedures, and implementation MDA Audit Findings –Deficiency - a non-compliance to a contractual requirement or internal procedure/process Includes AS9100 deficiencies (if AS9100 is on contract or if audit site is certified or compliant) –Observation - when an MDA or Industry “best-practice” should be used

9. 9 All MDA Audits CY05-CY08

10. 10 AS9100 Findings

11. 11 Analysis Majority of MDA deficiencies can be tied to an AS9100 element AS9100 related deficiencies are primarily in the following areas: –Production and Service Provision: Quality Records, Procedural Discipline, Foreign Object Debris/Damage (FOD), and Electrostatic Discharge (ESD) –Documentation Requirements: Control of Documents (Change and Approval), Quality Management System Records –Purchasing: Supplier Requirements Flow Down, Supplier Rating System –Design and Development: Inputs/Outputs, Traceability, Design Reviews, Design Verification/Validation –Control of Monitoring and Measuring Devices: Design and Selection, Calibration, and Tool Control –Planning of Product Realization: Establishment and Documentation of Processes –Control of Nonconforming Product: Nonconformance Disposition and Nonconforming Material Identification and Segregation

12. 12 Finding Examples Production and Service Provision, AS9100, paragraph 7.5 –Drawing parameters and/or pass/fail criteria were not accurately represented in the build paper. –Foreign Object Debris/Damage (FOD) was found in the Building Powder Blend Area. –Limited life materials were expired and available for use in the production of deliverable hardware. –Electrostatic discharge equipment continuity and environmental limits are not adequately controlled in the Bonded Stores and Electrical Inspection Areas. –Desiccant used to minimize moisture in motor manufacturing is not used per Manufacturing and Inspection Records (M&IR) or drawing requirements. –Preventive Maintenance Work Orders are not verified and Repair Work Orders are not generated to correct deficiencies.

13. 13 Finding Examples Documentation Requirements, AS9100, paragraph 4.2 –The audited program did not implement enterprise company practices for requirement traceability, and did not have an approved process for the unique implementation of their particular practices. –X-Ray inspection procedures are not controlled. –Changes to a Mix and Cast Building checklist were not approved before use. –Receiving Inspection Records were missing required documentation (Inspection report, Government Source Inspection Stamp, Material Certification). –The contractor could not provide supporting information that correlated or related to waivers and deviations. –Uncontrolled and improperly marked documentation were posted in the Process Control Lab.

14. 14 Finding Examples Purchasing, AS9100, paragraph 7.4 –Parts that did not comply with all supplier quality assurance requirements were accepted at Receiving Inspection. –The contractor’s Approved Vendor List does not include some vendors and does not reflect the vendor’s quality rating. –The prime contractor Statement of Work imposes an incorrect quality requirement. –The contractor did not impose core quality requirements (corrective action system, inspection procedures, and quality records) on suppliers. –The contractor accepted a supplier certificate of analysis that certified material for a storage temperature in conflict with the material specification.

15. 15 Finding Examples Design and Development, AS9100, paragraph 7.3 –Contractor Automated Test Station configuration identification does not reflect the current approved configuration. –The contractor does not have a formal process for documenting and tracking design review action items and responses. Control of Monitoring and Measuring Devices, AS9100, paragraph 7.6 –Calibrated equipment at a work station in the Destructive Lot Acceptance Test Area had broken integrity seals. –Test and Measuring Equipment with expired calibration was in the receiving inspection area.

16. 16 Finding Examples Planning of Product Realization, AS9100, paragraph 7.1 –The contractor’s software development practices were not documented and were dependent on the cognizant engineer. –The mass test limit for the Inertial Measurement Unit was incorrectly noted on the test sheet in the Engineering Note Book. Control of Nonconforming Product, AS9100, paragraph 8.3 –The contractor processed “Use as Is” dispositions for hardware without proper review and approval, including the customer. –Undocumented items (various mechanical, electrical and optical parts) were noted in the bonded Material Review Board lockers and cages which appear to be scrap.

17. 17 Common AS9100 Issues Interpretation varies between contractors and government organizations –AS9100 Paragraph 7.5.1.1, Production Documentation: Production operations shall be carried out in accordance with approved data MDA Interpretation: Mission and Safety Critical processes shall be carried out in accordance with approved detailed plans and procedures –AS9100 Paragraph 8.3, Control of Nonconforming Product: The organization shall ensure that product that does not conform to product requirements is identified and controlled to prevent its unintended use or delivery MDA Interpretation: Nonconforming product is conspicuously identified (tagged), segregated, and retained in a hold status until dispositioned by formal review process –AS9100 Paragraph 4.3 Configuration Management: The organization shall establish, document and maintain a configuration management process appropriate to the product. NOTE: Guidance on configuration management is given in ISO 10007 MDA Interpretation: A formal Configuration Management Program includes: a CM Plan which addresses Configuration Planning, Configuration Identification, Configuration Change Management (Waivers/Deviations), Configuration Status Accounting, Configuration Verification / Audit, and CM of Digital Data

18. 18 Common AS9100 Issues Continued Compliance and interpretation varies from contractor to contractor –Some contractors manufacture mission and safety critical hardware per an approved drawing, while others use an approved drawing, traveler, and detailed procedures –Contractors advertise AS9100 Certification, however, implementation varies based on programs and funding Certification achieved at division level, but not implemented across all programs –MDA experience: Many contractors have stated that AS9100 requires them to have a documented system, but the documented system does not have to be effective or implemented for compliance.

19. 19 AS9100 How You Can Help Strengthen Implementation Requirements –MDA experience: There is a gap between system assessment and implementation. AS9100 certification should include assessment of critical processes and the associated detailed planning and procedures. Audit/Assess MDA Programs as part of Certification –Identify MDA Programs with MAP on Contract –Assess effectiveness of Corrective Actions from MDA Audit Findings –Provide feedback for high risk MDA Suppliers

20. 20 Conclusion Majority of MDA deficiencies can be tied to an AS9100 Element Most audit findings are violations of internal processes –Not following their own procedures or processes Our audits have indicated gaps in PDCA implementation –Most sites do the “Plan” portion –They do less well with the “Do” portion –Even do less well with the “Check” portion –Most “Act” actions are short term, focusing on immediate fixes rather than well thought out long term actions that address systemic issues and improve performance across the enterprise Output of Requirements Must be Defined

21. Backup

22. 22 MDA Audit Findings MAP to AS9100 Correlation Nonconformances